Cannot access apps (501 error) with Proxmox LxC homelab and Caddy as reverse proxy for DuckDNS subdomains

Help
1

Hello,
I’m new to Caddy and not so proficient with TLS, SSL and so on, but I spent several hours over the past days trying things and I can’t get an https connection to my local apps.

What I try to achieve:

  • https connection to local apps from various LxC (no external access expected)
  • DuckDNS providing a domain
  • Setup is Proxmox VE 8.3.3 on a simple workstation
  • Use Caddy to manage SSL certificates and subdomain local forwarding.

1. The problem I’m having:

Curl header reports a 501 code with the different apps.

curl -I https://nc-local.thekeyandthefeather2.duckdns.org/
HTTP/2 501 
alt-svc: h3=":443"; ma=2592000
cache-control: max-age=0
date: Fri, 14 Feb 2025 22:33:12 GMT
expires: Fri, 14 Feb 2025 22:33:12 GMT
pragma: no-cache
server: Caddy
server: pve-api-daemon/3.0

2. Error messages and/or full log output:

I would like to show the journal info here but it seems to contain sensitive data in urls.
Please let me know what I should remove…

Feb 14 23:26:19 caddy systemd[1]: Starting caddy.service - Caddy...
Feb 14 23:26:19 caddy caddy[134]: caddy.HomeDir=/var/lib/caddy
Feb 14 23:26:19 caddy caddy[134]: caddy.AppDataDir=/var/lib/caddy/.local/share/caddy
Feb 14 23:26:19 caddy caddy[134]: caddy.AppConfigDir=/var/lib/caddy/.config/caddy
Feb 14 23:26:19 caddy caddy[134]: caddy.ConfigAutosavePath=/var/lib/caddy/.config/caddy/autosave.json
Feb 14 23:26:19 caddy caddy[134]: caddy.Version=v2.9.1 h1:OEYiZ7DbCzAWVb6TNEkjRcSCRGHVoZsJinoDR/n9oaY=
Feb 14 23:26:19 caddy caddy[134]: runtime.GOOS=linux
Feb 14 23:26:19 caddy caddy[134]: runtime.GOARCH=amd64
Feb 14 23:26:19 caddy caddy[134]: runtime.Compiler=gc
Feb 14 23:26:19 caddy caddy[134]: runtime.NumCPU=1
Feb 14 23:26:19 caddy caddy[134]: runtime.GOMAXPROCS=1
Feb 14 23:26:19 caddy caddy[134]: runtime.Version=go1.24.0
Feb 14 23:26:19 caddy caddy[134]: os.Getwd=/
Feb 14 23:26:19 caddy caddy[134]: LANG=en_US.UTF-8
Feb 14 23:26:19 caddy caddy[134]: PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
Feb 14 23:26:19 caddy caddy[134]: NOTIFY_SOCKET=/run/systemd/notify
Feb 14 23:26:19 caddy caddy[134]: HOME=/var/lib/caddy
Feb 14 23:26:19 caddy caddy[134]: LOGNAME=caddy
Feb 14 23:26:19 caddy caddy[134]: USER=caddy
Feb 14 23:26:19 caddy caddy[134]: INVOCATION_ID=405a2300e2cf400ba5a0caf1fd2cb10f
Feb 14 23:26:19 caddy caddy[134]: JOURNAL_STREAM=8:8269797
Feb 14 23:26:19 caddy caddy[134]: SYSTEMD_EXEC_PID=134
Feb 14 23:26:19 caddy caddy[134]: {"level":"info","ts":1739571979.9064178,"msg":"using config from file","file":"/etc/caddy/Caddyfile"}
Feb 14 23:26:19 caddy caddy[134]: {"level":"info","ts":1739571979.912486,"msg":"adapted config to JSON","adapter":"caddyfile"}
Feb 14 23:26:19 caddy caddy[134]: {"level":"info","ts":1739571979.913685,"logger":"admin","msg":"admin endpoint started","address":"localhost:2019","enforce_origin":false,"origins":["//localhost:2019","//[::1]:2019","//127.0.0.1:2019"]}
Feb 14 23:26:19 caddy caddy[134]: {"level":"info","ts":1739571979.9152322,"logger":"http.auto_https","msg":"server is listening only on the HTTPS port but has no TLS connection policies; adding one to enable TLS","server_name":"srv0","https_port":443}
Feb 14 23:26:19 caddy caddy[134]: {"level":"info","ts":1739571979.9152968,"logger":"http.auto_https","msg":"enabling automatic HTTP->HTTPS redirects","server_name":"srv0"}
Feb 14 23:26:19 caddy caddy[134]: {"level":"debug","ts":1739571979.9153574,"logger":"http.auto_https","msg":"adjusted config","tls":{"automation":{"policies":[{"subjects":["*.thekeyandthefeather2.duckdns.org"]},{}]}},"http":{"servers":{"remaining_auto_https_redirects":{"listen":[":80"],"routes":[{},{}]},"srv0":{"listen":[":443"],"routes":[{"handle":[{"handler":"subroute","routes":[{"handle":[{"handler":"reverse_proxy","upstreams":[{"dial":"192.168.1.40:8006"}]},{"handler":"reverse_proxy","upstreams":[{"dial":"192.168.1.209:80"}]}]}]}],"terminal":true}],"tls_connection_policies":[{}],"automatic_https":{}}}}}
Feb 14 23:26:19 caddy caddy[134]: {"level":"info","ts":1739571979.9176888,"logger":"tls.cache.maintenance","msg":"started background certificate maintenance","cache":"0xc0000dd400"}
Feb 14 23:26:19 caddy caddy[134]: {"level":"debug","ts":1739571979.9220905,"logger":"http","msg":"starting server loop","address":"[::]:443","tls":true,"http3":false}
Feb 14 23:26:19 caddy caddy[134]: {"level":"info","ts":1739571979.922626,"logger":"http","msg":"enabling HTTP/3 listener","addr":":443"}
Feb 14 23:26:19 caddy caddy[134]: {"level":"info","ts":1739571979.9232483,"msg":"failed to sufficiently increase receive buffer size (was: 208 kiB, wanted: 7168 kiB, got: 416 kiB). See https://github.com/quic-go/quic-go/wiki/UDP-Buffer-Sizes for details."}
Feb 14 23:26:19 caddy caddy[134]: {"level":"info","ts":1739571979.9250197,"logger":"http.log","msg":"server running","name":"srv0","protocols":["h1","h2","h3"]}
Feb 14 23:26:19 caddy caddy[134]: {"level":"debug","ts":1739571979.9251618,"logger":"http","msg":"starting server loop","address":"[::]:80","tls":false,"http3":false}
Feb 14 23:26:19 caddy caddy[134]: {"level":"warn","ts":1739571979.925228,"logger":"http","msg":"HTTP/2 skipped because it requires TLS","network":"tcp","addr":":80"}
Feb 14 23:26:19 caddy caddy[134]: {"level":"warn","ts":1739571979.9263344,"logger":"http","msg":"HTTP/3 skipped because it requires TLS","network":"tcp","addr":":80"}
Feb 14 23:26:19 caddy caddy[134]: {"level":"info","ts":1739571979.926396,"logger":"http.log","msg":"server running","name":"remaining_auto_https_redirects","protocols":["h1","h2","h3"]}
Feb 14 23:26:19 caddy caddy[134]: {"level":"info","ts":1739571979.9264398,"logger":"http","msg":"enabling automatic TLS certificate management","domains":["*.thekeyandthefeather2.duckdns.org"]}
Feb 14 23:26:19 caddy caddy[134]: {"level":"info","ts":1739571979.9302733,"msg":"autosaved config (load with --resume flag)","file":"/var/lib/caddy/.config/caddy/autosave.json"}
Feb 14 23:26:19 caddy caddy[134]: {"level":"info","ts":1739571979.9303262,"msg":"serving initial configuration"}
Feb 14 23:26:19 caddy systemd[1]: Started caddy.service - Caddy.
Feb 14 23:26:19 caddy caddy[134]: {"level":"info","ts":1739571979.9345584,"logger":"tls","msg":"storage cleaning happened too recently; skipping for now","storage":"FileStorage:/var/lib/caddy/.local/share/caddy","instance":"5975d3b4-8dbb-483d-8db4-65c0fbc53bf9","try_again":1739658379.9345572,"try_again_in":86399.999999623}
Feb 14 23:26:19 caddy caddy[134]: {"level":"info","ts":1739571979.934614,"logger":"tls","msg":"finished cleaning storage units"}
Feb 14 23:26:19 caddy caddy[134]: {"level":"info","ts":1739571979.9381268,"logger":"tls.obtain","msg":"acquiring lock","identifier":"*.thekeyandthefeather2.duckdns.org"}
Feb 14 23:26:19 caddy caddy[134]: {"level":"info","ts":1739571979.939563,"logger":"tls.obtain","msg":"lock acquired","identifier":"*.thekeyandthefeather2.duckdns.org"}
Feb 14 23:26:19 caddy caddy[134]: {"level":"info","ts":1739571979.9396899,"logger":"tls.obtain","msg":"obtaining certificate","identifier":"*.thekeyandthefeather2.duckdns.org"}
Feb 14 23:26:19 caddy caddy[134]: {"level":"debug","ts":1739571979.9397573,"logger":"events","msg":"event","name":"cert_obtaining","id":"<sensitive-info?>","origin":"tls","data":{"identifier":"*.thekeyandthefeather2.duckdns.org"}}
Feb 14 23:26:19 caddy caddy[134]: {"level":"debug","ts":1739571979.9404378,"logger":"tls","msg":"created CSR","identifiers":["*.thekeyandthefeather2.duckdns.org"],"san_dns_names":["*.thekeyandthefeather2.duckdns.org"],"san_emails":[],"common_name":"","extra_extensions":0}
[...]

3. Caddy version:

v2.9.1 h1:OEYiZ7DbCzAWVb6TNEkjRcSCRGHVoZsJinoDR/n9oaY=

4. How I installed and ran Caddy:

a. System environment:

Proxmox VE 8.3.3, LxC Debian 12, helper script:
https://raw.githubusercontent.com/community-scripts/ProxmoxVE/main/install/caddy-install.sh

b. Command:

systemctl start caddy

c. Service/unit/compose file:

● caddy.service - Caddy
     Loaded: loaded (/lib/systemd/system/caddy.service; enabled; preset: enabled)
     Active: active (running) since Fri 2025-02-14 23:26:19 CET; 2min 22s ago
       Docs: https://caddyserver.com/docs/
   Main PID: 134 (caddy)
      Tasks: 7 (limit: 9346)
     Memory: 47.9M
        CPU: 118ms
     CGroup: /system.slice/caddy.service
             └─134 /usr/bin/caddy run --environ --config /etc/caddy/Caddyfile

d. My complete Caddy config:

{
        debug
}

*.thekeyandthefeather2.duckdns.org {
        tls {
                dns duckdns {
                        api_token <mytoken>
                }
        }

        @pve host pve.thekeyandthefeather2.duckdns.org
        reverse_proxy 192.168.1.40:8006

        @nc-local host nc-local.thekeyandthefeather2.duckdns.org
        reverse_proxy 192.168.1.209
}

5. Links to relevant resources:

I tried reading many forum posts, tutorials, but I feel I might be missing something here.
Example:
Here I am not getting how I would specify the domain token…

Thanks in advance for your help,

2

The 501 is coming from one of your apps

Here you define the matchers but you never use them. How should Caddy know which directive should it execute?

I assume this is what you meant to do:

        @pve host pve.thekeyandthefeather2.duckdns.org
-        reverse_proxy 192.168.1.40:8006
+        reverse_proxy @pve 192.168.1.40:8006

        @nc-local host nc-local.thekeyandthefeather2.duckdns.org
-        reverse_proxy 192.168.1.209
+        reverse_proxy @nc-local 192.168.1.209

You need to read this page: