Understanding Caddy - Avoiding tls_insecure_skip_verify and 502 Errors

Help
1

1. The problem I’m having:

I’ve been transitioning my home lab setup from an old QNAP running Alpine Linux and Nginx Proxy Manager to a Minisforum UM690 with Proxmox VE 8.1.x, aiming to use Caddy as a reverse proxy for services like Portainer, Pi-Hole, and Bitwarden. Despite my efforts, I encounter issues like blank pages, 502 errors, and the need to use tls_insecure_skip_verify in order to get my site to show . Initially using Namecheap DNS, I switched to Cloudflare, hoping to resolve these issues but faced similar problems. Despite configuring Caddy with the dns-cloudflare module and setting up DNS challenges, the problems persist.

On Cloudflare, I am using their Proxied service (orange cloud) and their Full SSL/TLS option. I am not entering CNAMEs records for subdomains in Cloudflare. I use a wildcard to send everything home.

No question, I am doing something wrong or not understanding fundamentals. I am trying and have spent hours on the forum to get nowhere because those with similar issues get told “not a Caddy problem.” While I totally agree, it would be helpful to get assistance in understanding the issue so that corrective actions can take place. Additionally, it can be seen that modifying the Caddyfile can impact the results (i.e. using tls_insecure_skip_verify).

It is not a great feeling to think, I’m not smart enough to use Caddy so I should just go back to Nginx Proxy Manager. I’d rather learn and conquer.

2. Error messages and/or full log output:

There is way too much data and I got a message about exceeding the character limit.

Providing data from TODAY

Mar 18 00:49:36 caddy systemd[1]: Reloading caddy.service - Caddy...
Mar 18 00:49:36 caddy caddy[4839]: {"level":"info","ts":1710722976.4351082,"msg":"using provided configuration","config_file":"/etc/caddy/Caddyfile","config_adapter":""}
Mar 18 00:49:36 caddy systemd[1]: Reloaded caddy.service - Caddy.
Mar 18 00:50:14 caddy systemd[1]: Reloading caddy.service - Caddy...
Mar 18 00:50:14 caddy caddy[4846]: {"level":"info","ts":1710723014.7547262,"msg":"using provided configuration","config_file":"/etc/caddy/Caddyfile","config_adapter":""}
Mar 18 00:50:14 caddy caddy[4846]: {"level":"warn","ts":1710723014.7561529,"msg":"Caddyfile input is not formatted; run 'caddy fmt --overwrite' to fix inconsistencies","adapter":"caddyfile","file":"/etc/caddy/Caddyfile","line":47}
Mar 18 00:50:14 caddy systemd[1]: Reloaded caddy.service - Caddy.
Mar 18 00:56:34 caddy systemd[1]: Reloading caddy.service - Caddy...
Mar 18 00:56:34 caddy caddy[4854]: {"level":"info","ts":1710723394.6590383,"msg":"using provided configuration","config_file":"/etc/caddy/Caddyfile","config_adapter":""}
Mar 18 00:56:34 caddy caddy[4854]: {"level":"warn","ts":1710723394.6605506,"msg":"Caddyfile input is not formatted; run 'caddy fmt --overwrite' to fix inconsistencies","adapter":"caddyfile","file":"/etc/caddy/Caddyfile","line":47}
Mar 18 00:56:34 caddy systemd[1]: Reloaded caddy.service - Caddy.
Mar 18 01:03:59 caddy systemd[1]: Reloading caddy.service - Caddy...
Mar 18 01:03:59 caddy caddy[4868]: {"level":"info","ts":1710723839.2264497,"msg":"using provided configuration","config_file":"/etc/caddy/Caddyfile","config_adapter":""}
Mar 18 01:03:59 caddy caddy[4868]: {"level":"warn","ts":1710723839.227995,"msg":"Caddyfile input is not formatted; run 'caddy fmt --overwrite' to fix inconsistencies","adapter":"caddyfile","file":"/etc/caddy/Caddyfile","line":48}
Mar 18 01:03:59 caddy systemd[1]: Reloaded caddy.service - Caddy.
Mar 18 01:05:05 caddy systemd[1]: Reloading caddy.service - Caddy...
Mar 18 01:05:05 caddy caddy[4875]: {"level":"info","ts":1710723905.6266737,"msg":"using provided configuration","config_file":"/etc/caddy/Caddyfile","config_adapter":""}
Mar 18 01:05:05 caddy caddy[4875]: Error: adapting config using caddyfile: parsing caddyfile tokens for 'reverse_proxy': wrong argument count or unexpected line ending after 'tls_trusted_ca_certs', at /etc/caddy/Caddyfile:51
Mar 18 01:05:05 caddy systemd[1]: caddy.service: Control process exited, code=exited, status=1/FAILURE
Mar 18 01:05:05 caddy systemd[1]: Reload failed for caddy.service - Caddy.
Mar 18 01:05:57 caddy systemd[1]: Reloading caddy.service - Caddy...
Mar 18 01:05:57 caddy caddy[4884]: {"level":"info","ts":1710723957.0384095,"msg":"using provided configuration","config_file":"/etc/caddy/Caddyfile","config_adapter":""}
Mar 18 01:05:57 caddy caddy[4884]: Error: adapting config using caddyfile: parsing caddyfile tokens for 'reverse_proxy': wrong argument count or unexpected line ending after 'tls_trusted_ca_certs', at /etc/caddy/Caddyfile:51
Mar 18 01:05:57 caddy systemd[1]: caddy.service: Control process exited, code=exited, status=1/FAILURE
Mar 18 01:05:57 caddy systemd[1]: Reload failed for caddy.service - Caddy.
Mar 18 01:10:08 caddy systemd[1]: Reloading caddy.service - Caddy...
Mar 18 01:10:08 caddy caddy[4892]: {"level":"info","ts":1710724208.5224123,"msg":"using provided configuration","config_file":"/etc/caddy/Caddyfile","config_adapter":""}
Mar 18 01:10:08 caddy caddy[4892]: {"level":"warn","ts":1710724208.524019,"msg":"Caddyfile input is not formatted; run 'caddy fmt --overwrite' to fix inconsistencies","adapter":"caddyfile","file":"/etc/caddy/Caddyfile","line":50}
Mar 18 01:10:08 caddy systemd[1]: Reloaded caddy.service - Caddy.
Mar 18 01:24:51 caddy systemd[1]: Reloading caddy.service - Caddy...
Mar 18 01:24:51 caddy caddy[4900]: {"level":"info","ts":1710725091.3027983,"msg":"using provided configuration","config_file":"/etc/caddy/Caddyfile","config_adapter":""}
Mar 18 01:24:51 caddy caddy[4900]: {"level":"warn","ts":1710725091.304349,"msg":"Caddyfile input is not formatted; run 'caddy fmt --overwrite' to fix inconsistencies","adapter":"caddyfile","file":"/etc/caddy/Caddyfile","line":48}
Mar 18 01:24:51 caddy systemd[1]: Reloaded caddy.service - Caddy.
Mar 18 01:30:36 caddy systemd[1]: Reloading caddy.service - Caddy...
Mar 18 01:30:37 caddy caddy[4908]: {"level":"info","ts":1710725437.0149498,"msg":"using provided configuration","config_file":"/etc/caddy/Caddyfile","config_adapter":""}
Mar 18 01:30:37 caddy caddy[4908]: {"level":"warn","ts":1710725437.0165677,"msg":"Caddyfile input is not formatted; run 'caddy fmt --overwrite' to fix inconsistencies","adapter":"caddyfile","file":"/etc/caddy/Caddyfile","line":50}
Mar 18 01:30:37 caddy systemd[1]: Reloaded caddy.service - Caddy.
Mar 18 01:37:27 caddy systemd[1]: Reloading caddy.service - Caddy...
Mar 18 01:37:27 caddy caddy[4916]: {"level":"info","ts":1710725847.1187215,"msg":"using provided configuration","config_file":"/etc/caddy/Caddyfile","config_adapter":""}
Mar 18 01:37:27 caddy caddy[4916]: {"level":"warn","ts":1710725847.1202173,"msg":"Caddyfile input is not formatted; run 'caddy fmt --overwrite' to fix inconsistencies","adapter":"caddyfile","file":"/etc/caddy/Caddyfile","line":50}
Mar 18 01:37:27 caddy systemd[1]: Reloaded caddy.service - Caddy.
Mar 18 01:38:11 caddy systemd[1]: Reloading caddy.service - Caddy...
Mar 18 01:38:11 caddy caddy[4923]: {"level":"info","ts":1710725891.4261084,"msg":"using provided configuration","config_file":"/etc/caddy/Caddyfile","config_adapter":""}
Mar 18 01:38:11 caddy caddy[4923]: {"level":"warn","ts":1710725891.4275792,"msg":"Caddyfile input is not formatted; run 'caddy fmt --overwrite' to fix inconsistencies","adapter":"caddyfile","file":"/etc/caddy/Caddyfile","line":46}
Mar 18 01:38:11 caddy systemd[1]: Reloaded caddy.service - Caddy.
Mar 18 12:09:57 caddy systemd[1]: Reloading caddy.service - Caddy...
Mar 18 12:09:57 caddy caddy[5094]: {"level":"info","ts":1710763797.8990571,"msg":"using provided configuration","config_file":"/etc/caddy/Caddyfile","config_adapter":""}
Mar 18 12:09:57 caddy caddy[5094]: {"level":"warn","ts":1710763797.9005766,"msg":"Caddyfile input is not formatted; run 'caddy fmt --overwrite' to fix inconsistencies","adapter":"caddyfile","file":"/etc/caddy/Caddyfile","line":46}
Mar 18 12:09:57 caddy systemd[1]: Reloaded caddy.service - Caddy.
Mar 18 12:29:22 caddy systemd[1]: Reloading caddy.service - Caddy...
Mar 18 12:29:22 caddy caddy[5102]: {"level":"info","ts":1710764962.966532,"msg":"using provided configuration","config_file":"/etc/caddy/Caddyfile","config_adapter":""}
Mar 18 12:29:22 caddy caddy[5102]: {"level":"warn","ts":1710764962.9679294,"msg":"Caddyfile input is not formatted; run 'caddy fmt --overwrite' to fix inconsistencies","adapter":"caddyfile","file":"/etc/caddy/Caddyfile","line":46}
Mar 18 12:29:22 caddy systemd[1]: Reloaded caddy.service - Caddy.
Mar 18 12:30:01 caddy systemd[1]: Reloading caddy.service - Caddy...
Mar 18 12:30:01 caddy caddy[5109]: {"level":"info","ts":1710765001.8865488,"msg":"using provided configuration","config_file":"/etc/caddy/Caddyfile","config_adapter":""}
Mar 18 12:30:01 caddy caddy[5109]: {"level":"warn","ts":1710765001.888001,"msg":"Caddyfile input is not formatted; run 'caddy fmt --overwrite' to fix inconsistencies","adapter":"caddyfile","file":"/etc/caddy/Caddyfile","line":46}
Mar 18 12:30:01 caddy systemd[1]: Reloaded caddy.service - Caddy.
Mar 18 12:30:35 caddy systemd[1]: Reloading caddy.service - Caddy...
Mar 18 12:30:35 caddy caddy[5116]: {"level":"info","ts":1710765035.5546908,"msg":"using provided configuration","config_file":"/etc/caddy/Caddyfile","config_adapter":""}
Mar 18 12:30:35 caddy caddy[5116]: {"level":"warn","ts":1710765035.5561953,"msg":"Caddyfile input is not formatted; run 'caddy fmt --overwrite' to fix inconsistencies","adapter":"caddyfile","file":"/etc/caddy/Caddyfile","line":46}
Mar 18 12:30:35 caddy systemd[1]: Reloaded caddy.service - Caddy.
Mar 18 12:31:00 caddy systemd[1]: Reloading caddy.service - Caddy...
Mar 18 12:31:00 caddy caddy[5123]: {"level":"info","ts":1710765060.5901177,"msg":"using provided configuration","config_file":"/etc/caddy/Caddyfile","config_adapter":""}
Mar 18 12:31:00 caddy caddy[5123]: {"level":"warn","ts":1710765060.59165,"msg":"Caddyfile input is not formatted; run 'caddy fmt --overwrite' to fix inconsistencies","adapter":"caddyfile","file":"/etc/caddy/Caddyfile","line":46}
Mar 18 12:31:00 caddy systemd[1]: Reloaded caddy.service - Caddy.
Mar 18 12:32:04 caddy systemd[1]: Reloading caddy.service - Caddy...
Mar 18 12:32:46 caddy systemd[1]: Reloading caddy.service - Caddy...
Mar 18 12:32:46 caddy caddy[5137]: {"level":"info","ts":1710765166.9503937,"msg":"using provided configuration","config_file":"/etc/caddy/Caddyfile","config_adapter":""}
Mar 18 12:32:46 caddy caddy[5137]: {"level":"warn","ts":1710765166.951897,"msg":"Caddyfile input is not formatted; run 'caddy fmt --overwrite' to fix inconsistencies","adapter":"caddyfile","file"
:"/etc/caddy/Caddyfile","line":46}
Mar 18 12:32:46 caddy systemd[1]: Reloaded caddy.service - Caddy.
Mar 18 12:33:13 caddy systemd[1]: Reloading caddy.service - Caddy...
Mar 18 12:33:13 caddy caddy[5144]: {"level":"info","ts":1710765193.7303245,"msg":"using provided configuration","config_file":"/etc/caddy/Caddyfile","config_adapter":""}
Mar 18 12:33:13 caddy caddy[5144]: {"level":"warn","ts":1710765193.73188,"msg":"Caddyfile input is not formatted; run 'caddy fmt --overwrite' to fix inconsistencies","adapter":"caddyfile","file":
"/etc/caddy/Caddyfile","line":46}
Mar 18 12:33:13 caddy systemd[1]: Reloaded caddy.service - Caddy.
Mar 18 12:33:48 caddy systemd[1]: Reloading caddy.service - Caddy...
Mar 18 12:33:48 caddy caddy[5151]: {"level":"info","ts":1710765228.7822459,"msg":"using provided configuration","config_file":"/etc/caddy/Caddyfile","config_adapter":""}
Mar 18 12:33:48 caddy caddy[5151]: {"level":"warn","ts":1710765228.7837832,"msg":"Caddyfile input is not formatted; run 'caddy fmt --overwrite' to fix inconsistencies","adapter":"caddyfile","file
":"/etc/caddy/Caddyfile","line":46}
Mar 18 12:33:48 caddy systemd[1]: Reloaded caddy.service - Caddy.
Mar 18 12:34:17 caddy systemd[1]: Reloading caddy.service - Caddy...
Mar 18 12:34:17 caddy caddy[5158]: {"level":"info","ts":1710765257.330224,"msg":"using provided configuration","config_file":"/etc/caddy/Caddyfile","config_adapter":""}
Mar 18 12:34:17 caddy caddy[5158]: {"level":"warn","ts":1710765257.331722,"msg":"Caddyfile input is not formatted; run 'caddy fmt --overwrite' to fix inconsistencies","adapter":"caddyfile","file"
:"/etc/caddy/Caddyfile","line":46}
Mar 18 12:34:17 caddy systemd[1]: Reloaded caddy.service - Caddy.
Mar 18 12:35:21 caddy systemd[1]: Reloading caddy.service - Caddy...
Mar 18 12:35:21 caddy caddy[5166]: {"level":"info","ts":1710765321.3313212,"msg":"using provided configuration","config_file":"/etc/caddy/Caddyfile","config_adapter":""}
Mar 18 12:35:21 caddy caddy[5166]: {"level":"warn","ts":1710765321.3329725,"msg":"Caddyfile input is not formatted; run 'caddy fmt --overwrite' to fix inconsistencies","adapter":"caddyfile","file
":"/etc/caddy/Caddyfile","line":46}
Mar 18 12:35:21 caddy systemd[1]: Reloaded caddy.service - Caddy.
Mar 18 12:36:11 caddy systemd[1]: Reloading caddy.service - Caddy...
Mar 18 12:36:11 caddy caddy[5173]: {"level":"info","ts":1710765371.7023664,"msg":"using provided configuration","config_file":"/etc/caddy/Caddyfile","config_adapter":""}
Mar 18 12:36:11 caddy caddy[5173]: {"level":"warn","ts":1710765371.703873,"msg":"Caddyfile input is not formatted; run 'caddy fmt --overwrite' to fix inconsistencies","adapter":"caddyfile","file"
:"/etc/caddy/Caddyfile","line":46}
Mar 18 12:36:11 caddy systemd[1]: Reloaded caddy.service - Caddy.
Mar 18 12:41:50 caddy systemd[1]: Reloading caddy.service - Caddy...
Mar 18 12:41:50 caddy caddy[5181]: {"level":"info","ts":1710765710.5788908,"msg":"using provided configuration","config_file":"/etc/caddy/Caddyfile","config_adapter":""}
Mar 18 12:41:50 caddy caddy[5181]: {"level":"warn","ts":1710765710.580476,"msg":"Caddyfile input is not formatted; run 'caddy fmt --overwrite' to fix inconsistencies","adapter":"caddyfile","file"
:"/etc/caddy/Caddyfile","line":50}
Mar 18 12:41:50 caddy systemd[1]: Reloaded caddy.service - Caddy.
Mar 18 12:44:09 caddy systemd[1]: Reloading caddy.service - Caddy...
Mar 18 12:44:09 caddy caddy[5187]: {"level":"info","ts":1710765849.254575,"msg":"using provided configuration","config_file":"/etc/caddy/Caddyfile","config_adapter":""}
Mar 18 12:44:09 caddy caddy[5187]: {"level":"warn","ts":1710765849.2560687,"msg":"Caddyfile input is not formatted; run 'caddy fmt --overwrite' to fix inconsistencies","adapter":"caddyfile","file
":"/etc/caddy/Caddyfile","line":48}
Mar 18 12:44:09 caddy systemd[1]: Reloaded caddy.service - Caddy.
Mar 18 12:45:23 caddy systemd[1]: Reloading caddy.service - Caddy...
Mar 18 12:45:23 caddy caddy[5194]: {"level":"info","ts":1710765923.7787962,"msg":"using provided configuration","config_file":"/etc/caddy/Caddyfile","config_adapter":""}
Mar 18 12:45:23 caddy caddy[5194]: {"level":"warn","ts":1710765923.780372,"msg":"Caddyfile input is not formatted; run 'caddy fmt --overwrite' to fix inconsistencies","adapter":"caddyfile","file"
:"/etc/caddy/Caddyfile","line":48}
Mar 18 12:45:23 caddy systemd[1]: Reloaded caddy.service - Caddy.
Mar 18 12:45:57 caddy systemd[1]: Reloading caddy.service - Caddy...
Mar 18 12:45:57 caddy caddy[5201]: {"level":"info","ts":1710765957.422241,"msg":"using provided configuration","config_file":"/etc/caddy/Caddyfile","config_adapter":""}
Mar 18 12:45:57 caddy caddy[5201]: {"level":"warn","ts":1710765957.423767,"msg":"Caddyfile input is not formatted; run 'caddy fmt --overwrite' to fix inconsistencies","adapter":"caddyfile","file"
:"/etc/caddy/Caddyfile","line":48}
Mar 18 12:45:57 caddy systemd[1]: Reloaded caddy.service - Caddy.
Mar 18 12:46:22 caddy systemd[1]: Reloading caddy.service - Caddy...
Mar 18 12:46:22 caddy caddy[5208]: {"level":"info","ts":1710765982.4347491,"msg":"using provided configuration","config_file":"/etc/caddy/Caddyfile","config_adapter":""}
Mar 18 12:46:22 caddy caddy[5208]: {"level":"warn","ts":1710765982.4362335,"msg":"Caddyfile input is not formatted; run 'caddy fmt --overwrite' to fix inconsistencies","adapter":"caddyfile","file
":"/etc/caddy/Caddyfile","line":50}
Mar 18 12:47:24 caddy systemd[1]: Reloading caddy.service - Caddy...
Mar 18 12:47:24 caddy caddy[5216]: {"level":"info","ts":1710766044.1948133,"msg":"using provided configuration","config_file":"/etc/caddy/Caddyfile","config_adapter":""}
Mar 18 12:47:24 caddy caddy[5216]: {"level":"warn","ts":1710766044.1963325,"msg":"Caddyfile input is not formatted; run 'caddy fmt --overwrite' to fix inconsistencies","adapter":"caddyfile","file":"/etc/caddy/Caddyfile","line":50}
Mar 18 12:47:24 caddy systemd[1]: Reloaded caddy.service - Caddy.
Mar 18 12:48:13 caddy systemd[1]: Reloading caddy.service - Caddy...
Mar 18 12:48:13 caddy caddy[5223]: {"level":"info","ts":1710766093.4903228,"msg":"using provided configuration","config_file":"/etc/caddy/Caddyfile","config_adapter":""}
Mar 18 12:48:13 caddy caddy[5223]: Error: adapting config using caddyfile: ambiguous site definition: bpmedia.wdpronovost.com
Mar 18 12:48:13 caddy systemd[1]: caddy.service: Control process exited, code=exited, status=1/FAILURE
Mar 18 12:48:13 caddy systemd[1]: Reload failed for caddy.service - Caddy.
Mar 18 12:48:52 caddy systemd[1]: Reloading caddy.service - Caddy...
Mar 18 12:48:52 caddy caddy[5230]: {"level":"info","ts":1710766132.2344477,"msg":"using provided configuration","config_file":"/etc/caddy/Caddyfile","config_adapter":""}
Mar 18 12:48:52 caddy caddy[5230]: {"level":"warn","ts":1710766132.2359815,"msg":"Caddyfile input is not formatted; run 'caddy fmt --overwrite' to fix inconsistencies","adapter":"caddyfile","file":"/etc/caddy/Caddyfile","line":50}
Mar 18 12:48:52 caddy systemd[1]: Reloaded caddy.service - Caddy.
Mar 18 12:49:29 caddy systemd[1]: Reloading caddy.service - Caddy...
Mar 18 12:49:29 caddy caddy[5237]: {"level":"info","ts":1710766169.9065013,"msg":"using provided configuration","config_file":"/etc/caddy/Caddyfile","config_adapter":""}
Mar 18 12:49:29 caddy caddy[5237]: {"level":"warn","ts":1710766169.9080188,"msg":"Caddyfile input is not formatted; run 'caddy fmt --overwrite' to fix inconsistencies","adapter":"caddyfile","file":"/etc/caddy/Caddyfile","line":50}
Mar 18 12:49:29 caddy systemd[1]: Reloaded caddy.service - Caddy.
Mar 18 12:49:56 caddy systemd[1]: Reloading caddy.service - Caddy...
Mar 18 12:49:56 caddy caddy[5244]: {"level":"info","ts":1710766196.2184203,"msg":"using provided configuration","config_file":"/etc/caddy/Caddyfile","config_adapter":""}
Mar 18 12:49:56 caddy caddy[5244]: {"level":"warn","ts":1710766196.2199206,"msg":"Caddyfile input is not formatted; run 'caddy fmt --overwrite' to fix inconsistencies","adapter":"caddyfile","file":"/etc/caddy/Caddyfile","line":48}
Mar 18 12:49:56 caddy systemd[1]: Reloaded caddy.service - Caddy.
Mar 18 12:56:42 caddy systemd[1]: Reloading caddy.service - Caddy...
Mar 18 12:56:42 caddy caddy[5252]: {"level":"info","ts":1710766602.3590293,"msg":"using provided configuration","config_file":"/etc/caddy/Caddyfile","config_adapter":""}
Mar 18 12:56:42 caddy caddy[5252]: {"level":"warn","ts":1710766602.3606205,"msg":"Caddyfile input is not formatted; run 'caddy fmt --overwrite' to fix inconsistencies","adapter":"caddyfile","file":"/etc/caddy/Caddyfile","line":49}
Mar 18 12:56:42 caddy systemd[1]: Reloaded caddy.service - Caddy.
Mar 18 12:57:51 caddy systemd[1]: Reloading caddy.service - Caddy...
Mar 18 12:57:51 caddy caddy[5259]: {"level":"info","ts":1710766671.3707004,"msg":"using provided configuration","config_file":"/etc/caddy/Caddyfile","config_adapter":""}
Mar 18 12:57:51 caddy caddy[5259]: Error: adapting config using caddyfile: /etc/caddy/Caddyfile:67: unrecognized directive: fs.wdpronovost.com
Mar 18 12:57:51 caddy systemd[1]: caddy.service: Control process exited, code=exited, status=1/FAILURE
Mar 18 12:57:51 caddy systemd[1]: Reload failed for caddy.service - Caddy.
Mar 18 12:58:49 caddy systemd[1]: Reloading caddy.service - Caddy...
Mar 18 12:58:49 caddy caddy[5268]: {"level":"info","ts":1710766729.8544536,"msg":"using provided configuration","config_file":"/etc/caddy/Caddyfile","config_adapter":""}
Mar 18 12:58:49 caddy caddy[5268]: {"level":"warn","ts":1710766729.8559735,"msg":"Caddyfile input is not formatted; run 'caddy fmt --overwrite' to fix inconsistencies","adapter":"caddyfile","file":"/etc/caddy/Caddyfile","line":49}
Mar 18 12:58:49 caddy systemd[1]: Reloaded caddy.service - Caddy.
Mar 18 13:00:24 caddy systemd[1]: Reloading caddy.service - Caddy...
Mar 18 13:00:24 caddy caddy[5275]: {"level":"info","ts":1710766824.022294,"msg":"using provided configuration","config_file":"/etc/caddy/Caddyfile","config_adapter":""}
Mar 18 13:00:24 caddy caddy[5275]: {"level":"warn","ts":1710766824.0238352,"msg":"Caddyfile input is not formatted; run 'caddy fmt --overwrite' to fix inconsistencies","adapter":"caddyfile","file":"/etc/caddy/Caddyfile","line":49}
Mar 18 13:00:24 caddy systemd[1]: Reloaded caddy.service - Caddy.
Mar 18 13:01:04 caddy systemd[1]: Reloading caddy.service - Caddy...
Mar 18 13:01:04 caddy caddy[5282]: {"level":"info","ts":1710766864.846578,"msg":"using provided configuration","config_file":"/etc/caddy/Caddyfile","config_adapter":""}
Mar 18 13:01:04 caddy caddy[5282]: {"level":"warn","ts":1710766864.8481574,"msg":"Caddyfile input is not formatted; run 'caddy fmt --overwrite' to fix inconsistencies","adapter":"caddyfile","file":"/etc/caddy/Caddyfile","line":49}
Mar 18 13:01:04 caddy systemd[1]: Reloaded caddy.service - Caddy.
Mar 18 13:01:35 caddy systemd[1]: Reloading caddy.service - Caddy...
Mar 18 13:01:35 caddy caddy[5289]: {"level":"info","ts":1710766895.66268,"msg":"using provided configuration","config_file":"/etc/caddy/Caddyfile","config_adapter":""}
Mar 18 13:01:35 caddy caddy[5289]: {"level":"warn","ts":1710766895.66418,"msg":"Caddyfile input is not formatted; run 'caddy fmt --overwrite' to fix inconsistencies","adapter":"caddyfile","file":"/etc/caddy/Caddyfile","line":49}
Mar 18 13:01:35 caddy systemd[1]: Reloaded caddy.service - Caddy.



curl -vL portainer.wdpronovost.com
*   Trying 104.21.9.47:80...
* Connected to portainer.wdpronovost.com (104.21.9.47) port 80 (#0)
> GET / HTTP/1.1
> Host: portainer.wdpronovost.com
> User-Agent: curl/7.88.1
> Accept: */*
>
< HTTP/1.1 301 Moved Permanently
< Date: Mon, 18 Mar 2024 14:47:58 GMT
< Transfer-Encoding: chunked
< Connection: keep-alive
< Cache-Control: max-age=3600
< Expires: Mon, 18 Mar 2024 15:47:58 GMT
< Location: https://portainer.wdpronovost.com/
< Report-To: {"endpoints":[{"url":"https:\/\/a.nel.cloudflare.com\/report\/v4?s=fuL4T86T%2BhSPTp05fdSMOdZlCKsGO%2BnEQ7QjBHywH8Hl4K%2FRr%2FSYv5lZfKsFa9cM6oQsSD3HEliYwRloUIT69F5YJwRe%2FbKsC8Kz%2BewNgM6LnC4JgVzh2gawqUYkzw1wBa6rwYyeZ6rRpF%2Fq"}],"group":"cf-nel","max_age":604800}
< NEL: {"success_fraction":0,"report_to":"cf-nel","max_age":604800}
< Server: cloudflare
< CF-RAY: 8666055d09b6c5c5-SEA
< alt-svc: h3=":443"; ma=86400
<
* Ignoring the response-body
* Connection #0 to host portainer.wdpronovost.com left intact
* Clear auth, redirects to port from 80 to 443
* Issue another request to this URL: 'https://portainer.wdpronovost.com/'
*   Trying 104.21.9.47:443...
* Connected to portainer.wdpronovost.com (104.21.9.47) port 443 (#1)
* ALPN: offers h2,http/1.1
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
*  CAfile: /etc/ssl/certs/ca-certificates.crt
*  CApath: /etc/ssl/certs
* TLSv1.3 (IN), TLS handshake, Server hello (2):
* TLSv1.3 (IN), TLS handshake, Encrypted Extensions (8):
* TLSv1.3 (IN), TLS handshake, Certificate (11):
* TLSv1.3 (IN), TLS handshake, CERT verify (15):
* TLSv1.3 (IN), TLS handshake, Finished (20):
* TLSv1.3 (OUT), TLS change cipher, Change cipher spec (1):
* TLSv1.3 (OUT), TLS handshake, Finished (20):
* SSL connection using TLSv1.3 / TLS_AES_256_GCM_SHA384
* ALPN: server accepted h2
* Server certificate:
*  subject: CN=wdpronovost.com
*  start date: Mar 17 17:07:46 2024 GMT
*  expire date: Jun 15 17:07:45 2024 GMT
*  subjectAltName: host "portainer.wdpronovost.com" matched cert's "*.wdpronovost.com"
*  issuer: C=US; O=Google Trust Services LLC; CN=GTS CA 1P5
*  SSL certificate verify ok.
* using HTTP/2
* h2h3 [:method: GET]
* h2h3 [:path: /]
* h2h3 [:scheme: https]
* h2h3 [:authority: portainer.wdpronovost.com]
* h2h3 [user-agent: curl/7.88.1]
* h2h3 [accept: */*]
* Using Stream ID: 1 (easy handle 0x5832c1c78c80)
> GET / HTTP/2
> Host: portainer.wdpronovost.com
> user-agent: curl/7.88.1
> accept: */*
>
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
* old SSL session ID is stale, removing
< HTTP/2 200
< date: Mon, 18 Mar 2024 14:47:58 GMT
< content-type: text/html; charset=utf-8
< alt-svc: h3=":443"; ma=86400
< cache-control: max-age=31536000
< last-modified: Thu, 07 Dec 2023 08:11:10 GMT
< vary: Accept-Encoding
< x-content-type-options: nosniff
< x-xss-protection: 1; mode=block
< cf-cache-status: DYNAMIC
< report-to: {"endpoints":[{"url":"https:\/\/a.nel.cloudflare.com\/report\/v4?s=y4GIhobfA3DWkq4MXhNi04zzz3rduHmCml42iBE0UWINfYA%2FLuSKstpTm9hzHQ9UyMxeD3%2BIWyAh5D%2BiY7jwpJ0%2B3%2Bopdlyp0mThc%2FYk7jrY%2Ffa0kVsf1hhVRyWYTzji%2BS5%2FMSJ8J0Uf3qk8"}],"group":"cf-nel","max_age":604800}
< nel: {"success_fraction":0,"report_to":"cf-nel","max_age":604800}
< server: cloudflare
< cf-ray: 8666055d7df330a1-SEA
< {everything else is HTML}

3. Caddy version:

caddy version
v2.7.6 h1:w0NymbG2m9PcvKWsrXO6EEkY9Ru4FJK8uQbYcev1p3A=

4. How I installed and ran Caddy:

a. System environment:

  • OS and Architecture: Debian 12 on a Minisforum UM690, running Proxmox VE 8.1.x.
  • Environment Setup: Caddy is installed in a Container (CT) on Proxmox, with its own private IP.

Also used xcaddy to get the dns-cloudflare

b. Command:

# I don't recall using anything other than:

systemctl start caddy
systemctl reload caddy

c. Service/unit/compose file:

?

d. My complete Caddy config:

# The Caddyfile is an easy way to configure your Caddy web server.
#
# Unless the file starts with a global options block, the first
# uncommented line is always the address of your site.
#
# To use your own domain name (with automatic HTTPS), first make
# sure your domain's A/AAAA DNS records are properly pointed to
# this machine's public IP, then replace ":80" below with your
# domain name.

#:80 {
# Set this path to your site's directory.
#root * /usr/share/caddy

# Enable the static file server.
#file_server

# Another common task is to set up a reverse proxy:
# reverse_proxy localhost:8080

# Or serve a PHP site through php-fpm:
# php_fastcgi localhost:9000
#}

# Refer to the Caddy docs for more information:
# https://caddyserver.com/docs/caddyfile
{
	email wdp@wdpronovost.com
	log {
		output file /var/log/caddy/debug.log
		format json
		level DEBUG
	}

	acme_dns cloudflare {redacted}
}

*.wdpronovost.com {
		tls {
			dns cloudflare {redacted}
			resolvers 1.1.1.1
		}
}

bpmedia.wdpronovost.com {
	tls wdp@wdpronovost.com
	reverse_proxy 192.168.1.3:7080 {
		header_up Host {upstream_hostport}
#		transport http {
#			tls
#			tls_insecure_skip_verify
#			tls_trusted_ca_certs
#		}
	}
}

bw.wdpronovost.com {
	tls wdp@wdpronovost.com
	reverse_proxy https://192.168.1.150:8443 {
		header_up Host {upstream_hostport}
		transport http {
			tls
			tls_insecure_skip_verify
		}
	}
}

fs.wdpronovost.com {
	reverse_proxy 192.168.1.150:8191
}

homebridge.wdpronovost.com {
	reverse_proxy 192.168.1.150:8581
}

pihole.wdpronovost.com {
	reverse_proxy /admin* 192.168.1.233:80
}

portainer.wdpronovost.com {
	reverse_proxy https://192.168.1.150:9443 {
		transport http {
			tls
			tls_insecure_skip_verify
		}
	}
}

pve.wdpronovost.com {
	reverse_proxy https://192.168.1.230:8006 {
		header_up Host {upstream_hostport}
		transport http {
			tls
			tls_insecure_skip_verify
		}
	}
}

router.wdpronovost.com {
	reverse_proxy https://192.168.1.1:4443 {
		transport http {
			tls
			tls_insecure_skip_verify
		}
	}
}

plex.wdpronovost.com {
	reverse_proxy 192.168.1.3:32400
	#transport http {
	#tls
	#tls_insecure_skip_verify
	#}
	#}
}

5. Links to relevant resources:

6. Additional Details / Comments

The journal was really long and I was having trouble grabbing all of the content.

You will notice that I am not getting any ‘errors’ at the moment, but that is because I have used the tls_insecure_skip_verify to get things working. You can also see that I have been trying a number of different configurations in my Caddyfile.

Some things that I have learned, but uncertain if that is good or not,… With bpmedia and Plex, I found that I had required HTTPS set. That is actually my preference (mistaken belief that is more secure?) but from comments I saw that was part of what would cause 502 errors. Changing them seemed to work.

For others, where I couldn’t figure out how to change from requiring https, I can only get them to work using the tls_insecure_skip_verify.

I tried to put the tls in the route block but that didn’t work and I’d get an error trying to reload Caddy.

#EXAMPLE

pve.wdpronovost.com {
	reverse_proxy https://192.168.1.230:8006 {
	        tls {
		       dns cloudflare {redacted}
	        }
		transport http {
			tls
		}
	}
}

Hopefully I provided everything that was being asked for and presented in a way that will facilitate a better understanding of Caddy and an ultimate resolution.

Thank you.

2

That’s usually because Caddy wasn’t configured to serve anything for a specific page. But 502 errors (i.e. failure to connect to a proxy upstream) can cause a blank page as well, you’d need to use handle_errors to show an error page if you want one.

That’s usually because you’re proxying to the HTTPS port of your upstream app. Proxy to the HTTP port instead. There’s no reason to proxy over HTTPS because Caddy terminates the TLS connection between the client and Caddy, i.e. the part that crosses the public internet. It also adds some overhead.

DNS and ACME challenges are unrelated to proxying, they’re at different layers. Probably didn’t need to do that, but whatever, no big deal, if you prefer Cloudflare that’s fine anyway.

You should probably do this. Use the caddy fmt -w command to clean up your Caddyfile’s syntax to quiet this warning. It’ll just clean up indentation, making sure tabs are used etc.

You can delete all this, you won’t need it again. It’s just meant as a quick little mini-tour of what the Caddyfile can do, but once you set up your own site you don’t need it anymore.

Here, you’re only matching /admin*, so any request to other paths like /foo will just serve an empty page with status 200. You could remove the /admin* matcher if it makes sense to proxy all paths to that upstream.

Yeah that’s wrong, tls is its own directive, it can’t go within reverse_proxy. The tls directive must always be at the top-level of a site block.

1 Like
3

Good morning Francis and thank you for taking time to help me.

I’d like to cover your responses one by one.

Understood here. While it never crossed my mind to try and server something else, I appreciate the insight into its possibilities and how that might be helpful to me in the future.

This is where I am struggling the most to understand and why I came here to post. I got a basic understanding through my troubleshooting Plex, but what I am stuck on is why it doesn’t work with requiring HTTPS port? It worked under Nginx Proxy Manager. Granted, not the same application. But what I am trying to get to is, if I say:

plex.wdpronovost.com {
    reverse_proxy https://IP:PORT
}

Why does this just not work as is? Even if I don’t need to force HTTPS, why can’t I? What if it was a system where I couldn’t change it to just use HTTP and I had to use HTTPS, are we saying Caddy is unable to support that?

What is the solution for when HTTPS is required?

Only concern here is simply that my header metadata (ACME and TLS) directives are correct. (and they appear to be from what I can see)

I do every so often once I think I am done tinkering.

Understood, I left it in because I didn’t know if I would need it.

PiHole enters at the URL/admin and everything is below that. Without this, I can’t access pihole directly without modifying the url manually. (At least my experience)

I saw this on a Github issue: TLS example but I see that I missed the distinction you are making. I had it within the reverse_proxy and you are saying it needs to be above / outside of that.

Would this format change mean I could use reverse_proxy https://IP:PORT without issue then?

4

That’s from Caddy v1, it’s not relevant.

Correct, tls goes outside/above reverse_proxy.

The issue is trust. TLS requires trust to be secure. The client needs to be able to trust that the server’s TLS certificate was issued by a trusted authority (a CA, Certificate Authority). If there’s no trust, then it would be possible for some attacker to get in between the client and server and perform a man-in-the-middle attack, using its own TLS cert and “adding a link in the chain” reading and/or manipulating the traffic.

When you use tls_insecure_skip_verify you’re basically saying “don’t do any trust checks”, so you’re turning off all the security that HTTPS offers. The data’s still encrypted, but nothing stops an attacker from “acting” like they’re the server even though they put themselves in front of the actual server.

The reason ACME is a thing is to provide an automated way for a CA (like Let’s Encrypt) to verify that you have control of the domain and that nobody else does. You prove that by showing Let’s Encrypt “look at this one URL, yes it matches the thing you told me to show you” which is proof that “yes you were able to change the contents you served, so you must control the domain”.

Like I said earlier, HTTPS over private networks is not necessary and wasteful, because the risk of someone getting in between Caddy (the client in this case) and your upstream app (the server) is extremely unlikely (unless you run untrusted software or allow access to untrusted users).

But if you absolutely must use HTTPS, the “correct” thing is to configure trust, and you can do that by configuring tls_trusted_ca_certs to point to the root CA cert that signed the upstream’s certificate. If the upstream is using a self-signed cert, then it would just be that cert.

No app that you can proxy to requires HTTPS, it just might be configured to only serve HTTPS by default. There’s no technical reason that HTTPS must be used. Even HTTPS over the internet is optional, but very very strongly recommended because of the privacy and tamper-resistance that it provides. And obviously Caddy makes that easy via ACME automation.

5

Once again, I thank you for your time and helping me to better understand. I have not yet explored the tls_trusted_ca_certs but you helped me understand and that was what this was about … learning!

1 Like
6

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.