1. The problem I’m having:
caddy is installed on my docker host serving as a reverse proxy to jellyfin running in a container using step-ca (also in a container) as my acme server.
problem is, caddy fails to solve the acme challenge for the public dns of media.jlove.lol (i do own this domain).
step-ca’s logs says “The server could not connect to validation target”, but i have confirmed the domain is resolvable from within the step-ca container (you can resolve it yourself, too). and my router is configured to forward traffic - both ports 80 and 443 - to my docker host via nat. the firewall on my docker host (ufw) allows all traffic over ports 80 and 443.
what’s weird to me is if i make media.jlove.lol resolve to 10.0.60.254 (the site’s bind address in caddyfile) via dnsmasq on the docker host, the acme challenge succeeds and i am able to browse to the site via public dns just fine. i guess i could consider this a solution/workaround, but it doesn’t feel like the Right Way to do it.
for comparison, caddy successfully solves the acme challenge for the internal dns of media.home (dns entry in my router) which caddy reverse proxies to jellyfin, as well.
2. Error messages and/or full log output:
journalctl --no-pager -u caddy
Sep 08 02:51:03 heart systemd[1]: Starting caddy.service - Caddy...
Sep 08 02:51:03 heart caddy[352531]: caddy.HomeDir=/var/lib/caddy
Sep 08 02:51:03 heart caddy[352531]: caddy.AppDataDir=/var/lib/caddy/.local/share/caddy
Sep 08 02:51:03 heart caddy[352531]: caddy.AppConfigDir=/var/lib/caddy/.config/caddy
Sep 08 02:51:03 heart caddy[352531]: caddy.ConfigAutosavePath=/var/lib/caddy/.config/caddy/autosave.json
Sep 08 02:51:03 heart caddy[352531]: caddy.Version=2.6.2
Sep 08 02:51:03 heart caddy[352531]: runtime.GOOS=linux
Sep 08 02:51:03 heart caddy[352531]: runtime.GOARCH=amd64
Sep 08 02:51:03 heart caddy[352531]: runtime.Compiler=gc
Sep 08 02:51:03 heart caddy[352531]: runtime.NumCPU=24
Sep 08 02:51:03 heart caddy[352531]: runtime.GOMAXPROCS=24
Sep 08 02:51:03 heart caddy[352531]: runtime.Version=go1.19.4
Sep 08 02:51:03 heart caddy[352531]: os.Getwd=/
Sep 08 02:51:03 heart caddy[352531]: LANG=en_US.UTF-8
Sep 08 02:51:03 heart caddy[352531]: PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/snap/bin
Sep 08 02:51:03 heart caddy[352531]: NOTIFY_SOCKET=/run/systemd/notify
Sep 08 02:51:03 heart caddy[352531]: HOME=/var/lib/caddy
Sep 08 02:51:03 heart caddy[352531]: LOGNAME=caddy
Sep 08 02:51:03 heart caddy[352531]: USER=caddy
Sep 08 02:51:03 heart caddy[352531]: INVOCATION_ID=7744aa08f57f4a0e9786b0ff40da3fc1
Sep 08 02:51:03 heart caddy[352531]: JOURNAL_STREAM=8:873864
Sep 08 02:51:03 heart caddy[352531]: SYSTEMD_EXEC_PID=352531
Sep 08 02:51:03 heart caddy[352531]: {"level":"info","ts":1694141463.6635387,"msg":"using provided configuration","config_file":"/etc/caddy/Caddyfile","config_adapter":""}
Sep 08 02:51:03 heart caddy[352531]: {"level":"warn","ts":1694141463.6672287,"msg":"Caddyfile input is not formatted; run the 'caddy fmt' command to fix inconsistencies","adapter":"caddyfile","file":"/etc/caddy/Caddyfile","line":2}
Sep 08 02:51:03 heart caddy[352531]: {"level":"info","ts":1694141463.6681986,"msg":"redirected default logger","from":"stderr","to":"/var/log/caddy/caddy.log"}
Sep 08 02:51:03 heart systemd[1]: Started caddy.service - Caddy.
Sep 08 02:53:38 heart caddy[352531]: {"level":"info","ts":1694141618.5556219,"msg":"panic: certificate worker: interface conversion: interface {} is nil, not acme.Authorization\ngoroutine 84 [running]:\ngithub.com/caddyserver/certmagic.(*jobManager).worker.func1()\n\tgithub.com/caddyserver/certmagic/async.go:58 +0x65\npanic({0x164f980, 0xc00080ed80})\n\truntime/panic.go:884 +0x212\ngithub.com/mholt/acmez.(*Client).ObtainCertificateUsingCSR(0xc0005a2c00, {0x1b93918, 0xc0002fa810}, {{0xc00068fad0, 0x5}, {0x0, 0x0, 0x0}, 0x1, {0x0, ...}, ...}, ...)\n\tgithub.com/mholt/acmez/client.go:137 +0x13b4\ngithub.com/caddyserver/certmagic.(*ACMEIssuer).doIssue(0xc0002fa810?, {0x1b93918, 0xc0002fa810}, 0x0?, 0x0)\n\tgithub.com/caddyserver/certmagic/acmeissuer.go:385 +0x1b0\ngithub.com/caddyserver/certmagic.(*ACMEIssuer).Issue(0xc000209200, {0x1b93918, 0xc0002fa810}, 0xc0001b0e01?)\n\tgithub.com/caddyserver/certmagic/acmeissuer.go:314 +0xa9\ngithub.com/caddyserver/caddy/v2/modules/caddytls.(*ACMEIssuer).Issue(0x187e9d5?, {0x1b93918?, 0xc0002fa810?}, 0xc00010ee70?)\n\tgithub.com/caddyserver/caddy/v2/modules/caddytls/acmeissuer.go:233 +0x2a\ngithub.com/caddyserver/certmagic.(*Config).obtainCert.func2({0x1b93918, 0xc0002fa810})\n\tgithub.com/caddyserver/certmagic/config.go:554 +0x1133\ngithub.com/caddyserver/certmagic.doWithRetry({0x1b93870, 0xc0000eddc0}, 0x1b9b100?, 0xc000651b78)\n\tgithub.com/caddyserver/certmagic/async.go:104 +0x1eb\ngithub.com/caddyserver/certmagic.(*Config).obtainCert(0xc000168840, {0x1b93870, 0xc0000eddc0}, {0xc000228b70, 0xf}, 0x0)\n\tgithub.com/caddyserver/certmagic/config.go:611 +0x66b\ngithub.com/caddyserver/certmagic.(*Config).ObtainCertAsync(...)\n\tgithub.com/caddyserver/certmagic/config.go:462\ngithub.com/caddyserver/certmagic.(*Config).manageOne.func1()\n\tgithub.com/caddyserver/certmagic/config.go:359 +0x7f\ngithub.com/caddyserver/certmagic.(*jobManager).worker(0x2674220)\n\tgithub.com/caddyserver/certmagic/async.go:73 +0x11b\ncreated by github.com/caddyserver/certmagic.(*jobManager).Submit\n\tgithub.com/caddyserver/certmagic/async.go:50 +0x29b"}
vi /var/log/caddy/caddy.log
{"level":"warn","ts":"2023-09-08T02:51:03.668Z","logger":"admin","msg":"admin endpoint disabled"}
{"level":"info","ts":"2023-09-08T02:51:03.668Z","logger":"tls.cache.maintenance","msg":"started background certificate maintenance","cache":"0xc0003923f0"}
{"level":"info","ts":"2023-09-08T02:51:03.668Z","logger":"http","msg":"server is listening only on the HTTPS port but has no TLS connection policies; adding one to enable TLS","server_name":"srv1","https_port":443}
{"level":"warn","ts":"2023-09-08T02:51:03.668Z","logger":"http","msg":"automatic HTTP->HTTPS redirects are disabled","server_name":"srv1"}
{"level":"info","ts":"2023-09-08T02:51:03.668Z","logger":"http","msg":"server is listening only on the HTTPS port but has no TLS connection policies; adding one to enable TLS","server_name":"srv2","https_port":443}
{"level":"warn","ts":"2023-09-08T02:51:03.668Z","logger":"http","msg":"automatic HTTP->HTTPS redirects are disabled","server_name":"srv2"}
{"level":"info","ts":"2023-09-08T02:51:03.669Z","logger":"tls","msg":"cleaning storage unit","description":"FileStorage:/var/lib/caddy/.local/share/caddy"}
{"level":"info","ts":"2023-09-08T02:51:03.669Z","logger":"http.log","msg":"server running","name":"srv0","protocols":["h1","h2","h3"]}
{"level":"info","ts":"2023-09-08T02:51:03.669Z","logger":"http","msg":"enabling HTTP/3 listener","addr":"10.0.50.254:443"}
{"level":"info","ts":"2023-09-08T02:51:03.669Z","logger":"http.log","msg":"server running","name":"srv1","protocols":["h1","h2","h3"]}
{"level":"info","ts":"2023-09-08T02:51:03.669Z","logger":"http","msg":"enabling HTTP/3 listener","addr":"10.0.60.254:443"}
{"level":"info","ts":"2023-09-08T02:51:03.670Z","logger":"http.log","msg":"server running","name":"srv2","protocols":["h1","h2","h3"]}
{"level":"info","ts":"2023-09-08T02:51:03.670Z","logger":"http","msg":"enabling automatic TLS certificate management","domains":["docker.home","media.home","media.jlove.lol"]}
{"level":"info","ts":"2023-09-08T02:51:03.670Z","logger":"tls","msg":"finished cleaning storage units"}
{"level":"warn","ts":"2023-09-08T02:51:03.670Z","logger":"tls","msg":"stapling OCSP","error":"no OCSP stapling for [docker.home]: no OCSP server specified in certificate","identifiers":["docker.home"]}
{"level":"warn","ts":"2023-09-08T02:51:03.671Z","logger":"tls","msg":"stapling OCSP","error":"no OCSP stapling for [media.home]: no OCSP server specified in certificate","identifiers":["media.home"]}
{"level":"info","ts":"2023-09-08T02:51:03.671Z","msg":"autosaved config (load with --resume flag)","file":"/var/lib/caddy/.config/caddy/autosave.json"}
{"level":"info","ts":"2023-09-08T02:51:03.671Z","msg":"serving initial configuration"}
{"level":"info","ts":"2023-09-08T02:51:03.671Z","logger":"tls.obtain","msg":"acquiring lock","identifier":"media.jlove.lol"}
{"level":"info","ts":"2023-09-08T02:51:03.674Z","logger":"tls.obtain","msg":"lock acquired","identifier":"media.jlove.lol"}
{"level":"info","ts":"2023-09-08T02:51:03.674Z","logger":"tls.obtain","msg":"obtaining certificate","identifier":"media.jlove.lol"}
{"level":"info","ts":"2023-09-08T02:51:03.675Z","logger":"http","msg":"waiting on internal rate limiter","identifiers":["media.jlove.lol"],"ca":"https://step-ca.heart.home:9000/acme/acme/directory","account":""}
{"level":"info","ts":"2023-09-08T02:51:03.675Z","logger":"http","msg":"done waiting on internal rate limiter","identifiers":["media.jlove.lol"],"ca":"https://step-ca.heart.home:9000/acme/acme/directory","account":""}
{"level":"info","ts":"2023-09-08T02:51:03.703Z","logger":"http.acme_client","msg":"trying to solve challenge","identifier":"media.jlove.lol","challenge_type":"http-01","ca":"https://step-ca.heart.home:9000/acme/acme/directory"}
{"level":"error","ts":"2023-09-08T02:53:38.555Z","logger":"http.acme_client","msg":"deactivating authorization","identifier":"media.jlove.lol","authz":"https://step-ca.heart.home:9000/acme/acme/authz/FJMBQoAmjxswWdGPbzALOL2fD5G2IjMr","error":"request to https://step-ca.heart.home:9000/acme/acme/authz/FJMBQoAmjxswWdGPbzALOL2fD5G2IjMr failed after 1 attempts: HTTP 0 urn:ietf:params:acme:error:malformed - The request message was malformed"}
{"level":"info","ts":"2023-09-08T02:53:38.555Z","logger":"tls.obtain","msg":"releasing lock","identifier":"media.jlove.lol"}
docker container logs step-ca
time="2023-09-08T02:53:38Z" level=warning duration=7.164808ms duration-ns=7164808 error="nonce ckpwelVSWlNoUzJCYTBMczBXenFXZTlZeTNWeU44WjI not found" fields.time="2023-09-08T02:53:38Z" method=POST name=ca nonce=V0k5TEVvY0FlVjJmd1BqbW9YeFZkVmdYTXBDbjB3bUM path=/acme/acme/challenge/FJMBQoAmjxswWdGPbzALOL2fD5G2IjMr/Hn1KmAJvhQgCEbMKor9Xmy5r9nuo0Xa4 protocol=HTTP/2.0 referer= remote-address=172.20.0.1 request-id=cjt8pcgjv6ss739rm70g response="{\"type\":\"urn:ietf:params:acme:error:badNonce\",\"detail\":\"Unacceptable anti-replay nonce\"}" size=89 status=400 user-agent="Caddy/2.6.2 CertMagic acmez (linux; amd64)" user-id=
time="2023-09-08T02:53:38Z" level=warning duration=5.367369ms duration-ns=5367369 error="expected POST-as-GET" fields.time="2023-09-08T02:53:38Z" method=POST name=ca nonce=VTVaRTJCSzJ6Wkc1NTI4dlQzV0tLSWlNTnpqSHJza2E path=/acme/acme/authz/FJMBQoAmjxswWdGPbzALOL2fD5G2IjMr protocol=HTTP/2.0 referer= remote-address=172.20.0.1 request-id=cjt8pcgjv6ss739rm710 response="{\"type\":\"urn:ietf:params:acme:error:malformed\",\"detail\":\"The request message was malformed\"}" size=93 status=400 user-agent="Caddy/2.6.2 CertMagic acmez (linux; amd64)" user-id=
time="2023-09-08T02:53:53Z" level=info duration=30.008256576s duration-ns=30008256576 fields.time="2023-09-08T02:53:23Z" method=POST name=ca nonce=RFpMNWJMNmpIRXhhODdyTzNKNERTM2w4cHpFRmM3eTU path=/acme/acme/challenge/FJMBQoAmjxswWdGPbzALOL2fD5G2IjMr/Hn1KmAJvhQgCEbMKor9Xmy5r9nuo0Xa4 protocol=HTTP/2.0 referer= remote-address=172.20.0.1 request-id=cjt8p8ojv6ss739rm6vg response="{\"type\":\"http-01\",\"status\":\"pending\",\"token\":\"l9F0nlMVU9eCNvFwPG0KTVnHwEuzY1Fe\",\"url\":\"https://step-ca.heart.home:9000/acme/acme/challenge/FJMBQoAmjxswWdGPbzALOL2fD5G2IjMr/Hn1KmAJvhQgCEbMKor9Xmy5r9nuo0Xa4\",\"error\":{\"type\":\"urn:ietf:params:acme:error:connection\",\"detail\":\"The server could not connect to validation target\"}}" size=325 status=200 user-agent="Caddy/2.6.2 CertMagic acmez (linux; amd64)" user-id=
3. Caddy version:
2.6.2
4. How I installed and ran Caddy:
a. System environment:
ubuntu 23.04
docker 24.0.5
caddy was installed via apt
b. Command:
sudo service caddy start
c. Service/unit/compose file:
cat /usr/lib/systemd/system/caddy.service
[Unit]
Description=Caddy
Documentation=https://caddyserver.com/docs/
After=network.target network-online.target
Requires=network-online.target
[Service]
Type=notify
User=caddy
Group=caddy
ExecStart=/usr/bin/caddy run --environ --config /etc/caddy/Caddyfile
ExecReload=/usr/bin/caddy reload --config /etc/caddy/Caddyfile --force
TimeoutStopSec=5s
LimitNOFILE=1048576
LimitNPROC=512
PrivateTmp=true
ProtectSystem=full
AmbientCapabilities=CAP_NET_BIND_SERVICE
[Install]
WantedBy=multi-user.target
d. My complete Caddy config:
cat /etc/caddy/Caddyfile
{
acme_ca https://step-ca.heart.home:9000/acme/acme/directory
acme_ca_root /usr/local/share/ca-certificates/home_Root_CA_227792293483531640165956953935378510402.crt
admin off
auto_https disable_redirects
default_bind 10.0.50.254
log default {
format json {
time_format iso8601
}
output file /var/log/caddy/caddy.log {
roll_size 5MiB
roll_uncompressed
}
}
}
docker.home {
log
reverse_proxy portainer.heart.home:9000
}
media.home {
log
reverse_proxy jellyfin.heart.home:8096
}
media.jlove.lol {
bind 10.0.60.254
log
header -server
reverse_proxy jellyfin.heart.home:8096
}
http://docker.home:2375 {
log
reverse_proxy localhost:2375
}
note on the domains:
- *.heart.home is resolved by dnsmasq on my docker host
- *.home is resolved by my router
5. Links to relevant resources:
?