1. The problem I’m having:
In a setup with two Caddy instances, one (from here on called Outer) forwarding to the other (from here on called Inner), Inner can’t get a TLS certificate via the HTTP-01 challenge because Outer seems to try to interpret the challenge request itself, even if .well-known/acme-challenge is explicitly proxied.
Inner is providing a service that should only be available internally, but should still use TLS. In order to obtain a certificate, since I didn’t want to bother with setting up DNS-based verification yet, the public DNS record points to Outer (which is reachable from the public internet and has access to the internal network), it is supposed to forward the challenge to Inner (which is only reachable on the internal network), and all devices that can access the actual service need to be configured via hosts file or custom DNS to use the IP of Inner.
2. Error messages and/or full log output:
Outer:
May 19 23:39:05 erinome systemd[1]: Starting caddy.service - Caddy...
May 19 23:39:05 erinome caddy[981966]: caddy.HomeDir=/var/lib/caddy
May 19 23:39:05 erinome caddy[981966]: caddy.AppDataDir=/var/lib/caddy/.local/share/caddy
May 19 23:39:05 erinome caddy[981966]: caddy.AppConfigDir=/var/lib/caddy/.config/caddy
May 19 23:39:05 erinome caddy[981966]: caddy.ConfigAutosavePath=/var/lib/caddy/.config/caddy/autosave.json
May 19 23:39:05 erinome caddy[981966]: caddy.Version=2.6.2
May 19 23:39:05 erinome caddy[981966]: runtime.GOOS=linux
May 19 23:39:05 erinome caddy[981966]: runtime.GOARCH=amd64
May 19 23:39:05 erinome caddy[981966]: runtime.Compiler=gc
May 19 23:39:05 erinome caddy[981966]: runtime.NumCPU=6
May 19 23:39:05 erinome caddy[981966]: runtime.GOMAXPROCS=6
May 19 23:39:05 erinome caddy[981966]: runtime.Version=go1.19.8
May 19 23:39:05 erinome caddy[981966]: os.Getwd=/
May 19 23:39:05 erinome caddy[981966]: LANG=en_GB.UTF-8
May 19 23:39:05 erinome caddy[981966]: LANGUAGE=en_GB:en
May 19 23:39:05 erinome caddy[981966]: PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
May 19 23:39:05 erinome caddy[981966]: NOTIFY_SOCKET=/run/systemd/notify
May 19 23:39:05 erinome caddy[981966]: HOME=/var/lib/caddy
May 19 23:39:05 erinome caddy[981966]: LOGNAME=caddy
May 19 23:39:05 erinome caddy[981966]: USER=caddy
May 19 23:39:05 erinome caddy[981966]: INVOCATION_ID=de96c0800a7c426d8ace17751f129ceb
May 19 23:39:05 erinome caddy[981966]: JOURNAL_STREAM=8:50943495
May 19 23:39:05 erinome caddy[981966]: RUNTIME_DIRECTORY=/run/caddy
May 19 23:39:05 erinome caddy[981966]: SYSTEMD_EXEC_PID=981966
May 19 23:39:05 erinome caddy[981966]: {"level":"info","ts":1779226745.2057364,"msg":"using provided configuration","config_file":"/etc/caddy/Caddyfile","config_adapter":""}
May 19 23:39:05 erinome caddy[981966]: {"level":"info","ts":1779226745.2083566,"logger":"admin","msg":"admin endpoint started","address":"localhost:2019","enforce_origin":false,"origins":["//localhost:2019","//[::1]:2019","//127.0.0.1:2019"]}
May 19 23:39:05 erinome caddy[981966]: {"level":"warn","ts":1779226745.208531,"logger":"http","msg":"server is listening only on the HTTP port, so no automatic HTTPS will be applied to this server","server_name":"srv0","http_port":80}
May 19 23:39:05 erinome caddy[981966]: {"level":"info","ts":1779226745.208593,"logger":"tls.cache.maintenance","msg":"started background certificate maintenance","cache":"0xc0000a1c00"}
May 19 23:39:05 erinome caddy[981966]: {"level":"debug","ts":1779226745.208793,"logger":"http","msg":"starting server loop","address":"[::]:80","tls":false,"http3":false}
May 19 23:39:05 erinome caddy[981966]: {"level":"info","ts":1779226745.208809,"logger":"http.log","msg":"server running","name":"srv0","protocols":["h1","h2","h3"]}
May 19 23:39:05 erinome caddy[981966]: {"level":"info","ts":1779226745.2088535,"logger":"tls","msg":"cleaning storage unit","description":"FileStorage:/var/lib/caddy/.local/share/caddy"}
May 19 23:39:05 erinome caddy[981966]: {"level":"info","ts":1779226745.2089422,"msg":"autosaved config (load with --resume flag)","file":"/var/lib/caddy/.config/caddy/autosave.json"}
May 19 23:39:05 erinome caddy[981966]: {"level":"info","ts":1779226745.2089791,"msg":"serving initial configuration"}
May 19 23:39:05 erinome systemd[1]: Started caddy.service - Caddy.
May 19 23:39:05 erinome caddy[981966]: {"level":"info","ts":1779226745.2101057,"logger":"tls","msg":"finished cleaning storage units"}
May 19 23:39:11 erinome caddy[981966]: {"level":"error","ts":1779226751.3478131,"logger":"http","msg":"looking up info for HTTP challenge","host":"paperless.k1ba.eu","error":"no information found to solve challenge for identifier: paperless.k1ba.eu"}
May 19 23:39:11 erinome caddy[981966]: {"level":"error","ts":1779226751.3478732,"logger":"http","msg":"looking up info for HTTP challenge","host":"paperless.k1ba.eu","error":"no information found to solve challenge for identifier: paperless.k1ba.eu"}
Inner:
May 19 23:39:07 thebe systemd[1]: Starting Caddy web server...
May 19 23:39:07 thebe caddy[25828]: {"level":"info","ts":1779226747.0979247,"msg":"using config from fi
le","file":"/etc/caddy/Caddyfile"}
May 19 23:39:07 thebe caddy[25828]: {"level":"info","ts":1779226747.0995846,"msg":"adapted config to JS
ON","adapter":"caddyfile"}
May 19 23:39:07 thebe caddy[25828]: {"level":"info","ts":1779226747.0999503,"logger":"http.auto_https",
"msg":"server is listening only on the HTTPS port but has no TLS connection policies; adding one to ena
ble TLS","server_name":"srv0","https_port":443}
May 19 23:39:07 thebe caddy[25828]: {"level":"info","ts":1779226747.0999665,"logger":"http.auto_https",
"msg":"enabling automatic HTTP->HTTPS redirects","server_name":"srv0"}
May 19 23:39:07 thebe caddy[25828]: {"level":"debug","ts":1779226747.0999892,"logger":"http.auto_https","msg":"adjusted config","tls":{"automation":{"policies":[{}]}},"http":{"servers":{"remaining_auto_https_redirects":{"listen":[":80"],"routes":[{},{}]},"srv0":{"listen":[":443"],"routes":[{"handle":[{"handler":"subroute","routes":[{"handle":[{"handler":"vars","root":"/var/www/testing"},{"handler":"file_server","hide":["/etc/caddy/Caddyfile"]}]}]}],"terminal":true}],"tls_connection_policies":[{}],"automatic_https":{}}}}}
May 19 23:39:07 thebe caddy[25828]: {"level":"info","ts":1779226747.1000118,"logger":"tls.cache.maintenance","msg":"started background certificate maintenance","cache":"0x16f48fdce780"}
May 19 23:39:07 thebe caddy[25828]: {"level":"info","ts":1779226747.1000888,"logger":"http","msg":"servers shutting down with eternal grace period"}
May 19 23:39:07 thebe caddy[25828]: {"level":"info","ts":1779226747.1001308,"logger":"tls.cache.maintenance","msg":"stopped background certificate maintenance","cache":"0x16f48fdce780"}
May 19 23:39:07 thebe caddy[25828]: Valid configuration
May 19 23:39:07 thebe caddy[25838]: {"level":"info","ts":1779226747.1593273,"msg":"maxprocs: Leaving GOMAXPROCS=4: CPU quota undefined"}
May 19 23:39:07 thebe caddy[25838]: {"level":"info","ts":1779226747.1593535,"msg":"GOMEMLIMIT is updated","GOMEMLIMIT":14842106265,"previous":9223372036854775807}
May 19 23:39:07 thebe caddy[25838]: {"level":"info","ts":1779226747.1593597,"msg":"using config from file","file":"/etc/caddy/Caddyfile"}
May 19 23:39:07 thebe caddy[25838]: {"level":"info","ts":1779226747.159365,"msg":"adapted config to JSON","adapter":"caddyfile"}
May 19 23:39:07 thebe caddy[25838]: {"level":"info","ts":1779226747.1606445,"logger":"admin","msg":"admin endpoint started","address":"localhost:2019","enforce_origin":false,"origins":["//localhost:2019","//[::1]:2019","//127.0.0.1:2019"]}
May 19 23:39:07 thebe caddy[25838]: {"level":"info","ts":1779226747.1608176,"logger":"http.auto_https","msg":"server is listening only on the HTTPS port but has no TLS connection policies; adding one to enable TLS","server_name":"srv0","https_port":443}
May 19 23:39:07 thebe caddy[25838]: {"level":"info","ts":1779226747.160833,"logger":"http.auto_https","msg":"enabling automatic HTTP->HTTPS redirects","server_name":"srv0"}
May 19 23:39:07 thebe caddy[25838]: {"level":"info","ts":1779226747.1608424,"logger":"tls.cache.maintenance","msg":"started background certificate maintenance","cache":"0x3c7ea1a30880"}
May 19 23:39:07 thebe caddy[25838]: {"level":"debug","ts":1779226747.1608615,"logger":"http.auto_https","msg":"adjusted config","tls":{"automation":{"policies":[{}]}},"http":{"servers":{"remaining_auto_https_redirects":{"listen":[":80"],"routes":[{},{}]},"srv0":{"listen":[":443"],"routes":[{"handle":[{"handler":"subroute","routes":[{"handle":[{"handler":"vars","root":"/var/www/testing"},{"handler":"file_server","hide":["/etc/caddy/Caddyfile"]}]}]}],"terminal":true}],"tls_connection_policies":[{}],"automatic_https":{}}}}}
May 19 23:39:07 thebe caddy[25838]: {"level":"debug","ts":1779226747.1610215,"logger":"http","msg":"starting server loop","address":"[::]:443","tls":true,"http3":false}
May 19 23:39:07 thebe caddy[25838]: {"level":"info","ts":1779226747.1610432,"logger":"http","msg":"enabling HTTP/3 listener","addr":":443"}
May 19 23:39:07 thebe caddy[25838]: {"level":"info","ts":1779226747.1613054,"logger":"http.log","msg":"server running","name":"srv0","protocols":["h1","h2","h3"]}
May 19 23:39:07 thebe caddy[25838]: {"level":"debug","ts":1779226747.1613524,"logger":"http","msg":"starting server loop","address":"[::]:80","tls":false,"http3":false}
May 19 23:39:07 thebe caddy[25838]: {"level":"warn","ts":1779226747.1613588,"logger":"http","msg":"HTTP/2 skipped because it requires TLS","network":"tcp","addr":":80"}
May 19 23:39:07 thebe caddy[25838]: {"level":"warn","ts":1779226747.1613622,"logger":"http","msg":"HTTP/3 skipped because it requires TLS","network":"tcp","addr":":80"}
May 19 23:39:07 thebe caddy[25838]: {"level":"info","ts":1779226747.1613648,"logger":"http.log","msg":"server running","name":"remaining_auto_https_redirects","protocols":["h1","h2","h3"]}
May 19 23:39:07 thebe caddy[25838]: {"level":"info","ts":1779226747.1613686,"logger":"http","msg":"enabling automatic TLS certificate management","domains":["paperless.k1ba.eu"]}
May 19 23:39:07 thebe caddy[25838]: {"level":"debug","ts":1779226747.1614516,"logger":"events","msg":"event","name":"started","id":"9a27076d-d6ee-46e0-b3f1-8916a56534e1","origin":"","data":null}
May 19 23:39:07 thebe caddy[25838]: {"level":"info","ts":1779226747.1616285,"msg":"autosaved config (load with --resume flag)","file":"/var/lib/caddy/autosave.json"}
May 19 23:39:07 thebe caddy[25838]: {"level":"info","ts":1779226747.1616952,"msg":"serving initial configuration"}
May 19 23:39:07 thebe systemd[1]: Started Caddy web server.
May 19 23:39:07 thebe caddy[25838]: {"level":"info","ts":1779226747.1630955,"logger":"tls.obtain","msg":"acquiring lock","identifier":"paperless.k1ba.eu"}
May 19 23:39:07 thebe caddy[25838]: {"level":"info","ts":1779226747.1639771,"logger":"tls","msg":"storage cleaning happened too recently; skipping for now","storage":"FileStorage:/var/lib/caddy","instance":"8facf6bf-0725-4bd3-93f0-356b5a36fb74","try_again":1779313147.1639743,"try_again_in":86399.999999403}
May 19 23:39:07 thebe caddy[25838]: {"level":"info","ts":1779226747.1641836,"logger":"tls","msg":"finished cleaning storage units"}
May 19 23:39:07 thebe caddy[25838]: {"level":"info","ts":1779226747.16449,"logger":"tls.obtain","msg":"lock acquired","identifier":"paperless.k1ba.eu"}
May 19 23:39:07 thebe caddy[25838]: {"level":"info","ts":1779226747.1645997,"logger":"tls.obtain","msg":"obtaining certificate","identifier":"paperless.k1ba.eu"}
May 19 23:39:07 thebe caddy[25838]: {"level":"debug","ts":1779226747.1646228,"logger":"events","msg":"event","name":"cert_obtaining","id":"181dfdd6-18f2-47b0-87bc-105efdffa1e3","origin":"tls","data":{"identifier":"paperless.k1ba.eu"}}
May 19 23:39:07 thebe caddy[25838]: {"level":"debug","ts":1779226747.164757,"logger":"tls","msg":"created CSR","identifiers":["paperless.k1ba.eu"],"san_dns_names":["paperless.k1ba.eu"],"san_emails":[],"common_name":"","extra_extensions":0}
May 19 23:39:07 thebe caddy[25838]: {"level":"debug","ts":1779226747.1652873,"logger":"tls.obtain","msg":"trying issuer 1/1","issuer":"acme-staging-v02.api.letsencrypt.org-directory"}
May 19 23:39:07 thebe caddy[25838]: {"level":"debug","ts":1779226747.165664,"logger":"http","msg":"using existing ACME account because key found in storage associated with email","email":"default","ca":"https://acme-staging-v02.api.letsencrypt.org/directory"}
May 19 23:39:07 thebe caddy[25838]: {"level":"debug","ts":1779226747.1659267,"logger":"http","msg":"using existing ACME account because key found in storage associated with email","email":"","ca":"https://acme-staging-v02.api.letsencrypt.org/directory"}
May 19 23:39:07 thebe caddy[25838]: {"level":"info","ts":1779226747.165977,"logger":"http","msg":"waiting on internal rate limiter","identifiers":["paperless.k1ba.eu"],"ca":"https://acme-staging-v02.api.letsencrypt.org/directory","account":""}
May 19 23:39:07 thebe caddy[25838]: {"level":"info","ts":1779226747.1659894,"logger":"http","msg":"done waiting on internal rate limiter","identifiers":["paperless.k1ba.eu"],"ca":"https://acme-staging-v02.api.letsencrypt.org/directory","account":""}
May 19 23:39:07 thebe caddy[25838]: {"level":"info","ts":1779226747.1660068,"logger":"http","msg":"using ACME account","account_id":"https://acme-staging-v02.api.letsencrypt.org/acme/acct/196539914","account_contact":[]}
May 19 23:39:07 thebe caddy[25838]: {"level":"debug","ts":1779226747.6275492,"msg":"http request","method":"GET","url":"https://acme-staging-v02.api.letsencrypt.org/directory","headers":{"User-Agent":["Caddy/2.11.2 CertMagic acmez (linux; amd64)"]},"response_headers":{"Cache-Control":["public, max-age=0, no-cache"],"Content-Length":["1107"],"Content-Type":["application/json"],"Date":["Tue, 19 May 2026 21:39:07 GMT"],"Server":["nginx"],"Strict-Transport-Security":["max-age=604800"],"X-Frame-Options":["DENY"]},"status_code":200}
May 19 23:39:07 thebe caddy[25838]: {"level":"debug","ts":1779226747.627835,"msg":"creating order","account":"https://acme-staging-v02.api.letsencrypt.org/acme/acct/196539914","identifiers":["paperless.k1ba.eu"]}
May 19 23:39:07 thebe caddy[25838]: {"level":"debug","ts":1779226747.7697108,"msg":"http request","method":"HEAD","url":"https://acme-staging-v02.api.letsencrypt.org/acme/new-nonce","headers":{"User-Agent":["Caddy/2.11.2 CertMagic acmez (linux; amd64)"]},"response_headers":{"Cache-Control":["public, max-age=0, no-cache"],"Date":["Tue, 19 May 2026 21:39:07 GMT"],"Link":["<https://acme-staging-v02.api.letsencrypt.org/directory>;rel=\"index\""],"Replay-Nonce":["bmaVB79h6fe43FM3EgPcGSBXrShWlpKNzqQgjVc550vZWvIxSQE"],"Server":["nginx"],"Strict-Transport-Security":["max-age=604800"],"X-Frame-Options":["DENY"]},"status_code":200}
May 19 23:39:07 thebe caddy[25838]: {"level":"debug","ts":1779226747.920185,"msg":"http request","method":"POST","url":"https://acme-staging-v02.api.letsencrypt.org/acme/new-order","headers":{"Content-Type":["application/jose+json"],"User-Agent":["Caddy/2.11.2 CertMagic acmez (linux; amd64)"]},"response_headers":{"Boulder-Requester":["196539914"],"Cache-Control":["public, max-age=0, no-cache"],"Content-Length":["362"],"Content-Type":["application/json"],"Date":["Tue, 19 May 2026 21:39:07 GMT"],"Link":["<https://acme-staging-v02.api.letsencrypt.org/directory>;rel=\"index\""],"Location":["https://acme-staging-v02.api.letsencrypt.org/acme/order/196539914/38523282103"],"Replay-Nonce":["bmaVB79hrJfQEL43XYuqoJCbpfX1M8IuXjb-zzEyDvbKct3ii9g"],"Server":["nginx"],"Strict-Transport-Security":["max-age=604800"],"X-Frame-Options":["DENY"]},"status_code":201}
May 19 23:39:08 thebe caddy[25838]: {"level":"debug","ts":1779226748.064047,"msg":"http request","method":"POST","url":"https://acme-staging-v02.api.letsencrypt.org/acme/authz/196539914/1399728183","headers":{"Content-Type":["application/jose+json"],"User-Agent":["Caddy/2.11.2 CertMagic acmez (linux; amd64)"]},"response_headers":{"Boulder-Requester":["196539914"],"Cache-Control":["public, max-age=0, no-cache"],"Content-Length":["1077"],"Content-Type":["application/json"],"Date":["Tue, 19 May 2026 21:39:07 GMT"],"Link":["<https://acme-staging-v02.api.letsencrypt.org/directory>;rel=\"index\""],"Replay-Nonce":["bmaVB79hFPq-RH-wj8ynDgMKpWS0iNLdI8CaJcx0A70VPmhR8wM"],"Server":["nginx"],"Strict-Transport-Security":["max-age=604800"],"X-Frame-Options":["DENY"]},"status_code":200}
May 19 23:39:08 thebe caddy[25838]: {"level":"debug","ts":1779226748.0646036,"msg":"no solver configured","challenge_type":"dns-01"}
May 19 23:39:08 thebe caddy[25838]: {"level":"info","ts":1779226748.0646384,"msg":"trying to solve challenge","identifier":"paperless.k1ba.eu","challenge_type":"tls-alpn-01","ca":"https://acme-staging-v02.api.letsencrypt.org/directory"}
May 19 23:39:08 thebe caddy[25838]: {"level":"debug","ts":1779226748.0680737,"msg":"waiting for solver before continuing","identifier":"paperless.k1ba.eu","challenge_type":"tls-alpn-01"}
May 19 23:39:08 thebe caddy[25838]: {"level":"debug","ts":1779226748.0681796,"msg":"done waiting for solver","identifier":"paperless.k1ba.eu","challenge_type":"tls-alpn-01"}
May 19 23:39:08 thebe caddy[25838]: {"level":"debug","ts":1779226748.0682058,"logger":"http.stdlib","msg":"http: TLS handshake error from 127.0.0.1:60686: EOF"}
May 19 23:39:08 thebe caddy[25838]: {"level":"debug","ts":1779226748.2127702,"msg":"http request","method":"POST","url":"https://acme-staging-v02.api.letsencrypt.org/acme/chall/196539914/1399728183/BLOf7w","headers":{"Content-Type":["application/jose+json"],"User-Agent":["Caddy/2.11.2 CertMagic acmez (linux; amd64)"]},"response_headers":{"Boulder-Requester":["196539914"],"Cache-Control":["public, max-age=0, no-cache"],"Content-Length":["204"],"Content-Type":["application/json"],"Date":["Tue, 19 May 2026 21:39:08 GMT"],"Link":["<https://acme-staging-v02.api.letsencrypt.org/directory>;rel=\"index\"","<https://acme-staging-v02.api.letsencrypt.org/acme/authz/196539914/1399728183>;rel=\"up\""],"Location":["https://acme-staging-v02.api.letsencrypt.org/acme/chall/196539914/1399728183/BLOf7w"],"Replay-Nonce":["3gIrNIz7nwG4YoKCvL9jhapNQ7lfRGpv1wQnIbN5d4Vk2vEcHEI"],"Server":["nginx"],"Strict-Transport-Security":["max-age=604800"],"X-Frame-Options":["DENY"]},"status_code":200}
May 19 23:39:08 thebe caddy[25838]: {"level":"debug","ts":1779226748.2129743,"msg":"challenge accepted","identifier":"paperless.k1ba.eu","challenge_type":"tls-alpn-01"}
May 19 23:39:08 thebe caddy[25838]: {"level":"debug","ts":1779226748.6079948,"msg":"http request","method":"POST","url":"https://acme-staging-v02.api.letsencrypt.org/acme/authz/196539914/1399728183","headers":{"Content-Type":["application/jose+json"],"User-Agent":["Caddy/2.11.2 CertMagic acmez (linux; amd64)"]},"response_headers":{"Boulder-Requester":["196539914"],"Cache-Control":["public, max-age=0, no-cache"],"Content-Length":["1077"],"Content-Type":["application/json"],"Date":["Tue, 19 May 2026 21:39:08 GMT"],"Link":["<https://acme-staging-v02.api.letsencrypt.org/directory>;rel=\"index\""],"Replay-Nonce":["3gIrNIz7LNpTNNmBswW_kDog6DrUB7xsGeQB_K6Y2dQy65OCkSw"],"Server":["nginx"],"Strict-Transport-Security":["max-age=604800"],"X-Frame-Options":["DENY"]},"status_code":200}
May 19 23:39:09 thebe caddy[25838]: {"level":"debug","ts":1779226749.0027065,"msg":"http request","method":"POST","url":"https://acme-staging-v02.api.letsencrypt.org/acme/authz/196539914/1399728183","headers":{"Content-Type":["application/jose+json"],"User-Agent":["Caddy/2.11.2 CertMagic acmez (linux; amd64)"]},"response_headers":{"Boulder-Requester":["196539914"],"Cache-Control":["public, max-age=0, no-cache"],"Content-Length":["1077"],"Content-Type":["application/json"],"Date":["Tue, 19 May 2026 21:39:08 GMT"],"Link":["<https://acme-staging-v02.api.letsencrypt.org/directory>;rel=\"index\""],"Replay-Nonce":["3gIrNIz75LtV0Oi_J24T_NhmEFOYZe7UASFsq5erIc4w7f4O_4I"],"Server":["nginx"],"Strict-Transport-Security":["max-age=604800"],"X-Frame-Options":["DENY"]},"status_code":200}
May 19 23:39:09 thebe caddy[25838]: {"level":"debug","ts":1779226749.3982942,"msg":"http request","method":"POST","url":"https://acme-staging-v02.api.letsencrypt.org/acme/authz/196539914/1399728183","headers":{"Content-Type":["application/jose+json"],"User-Agent":["Caddy/2.11.2 CertMagic acmez (linux; amd64)"]},"response_headers":{"Boulder-Requester":["196539914"],"Cache-Control":["public, max-age=0, no-cache"],"Content-Length":["840"],"Content-Type":["application/json"],"Date":["Tue, 19 May 2026 21:39:09 GMT"],"Link":["<https://acme-staging-v02.api.letsencrypt.org/directory>;rel=\"index\""],"Replay-Nonce":["3gIrNIz7gt5pZCIpxyFobG7ObFkXl8grsNYXvjMTl1AWIB9aWns"],"Server":["nginx"],"Strict-Transport-Security":["max-age=604800"],"X-Frame-Options":["DENY"]},"status_code":200}
May 19 23:39:09 thebe caddy[25838]: {"level":"error","ts":1779226749.3986938,"msg":"challenge failed","identifier":"paperless.k1ba.eu","challenge_type":"tls-alpn-01","problem":{"type":"urn:ietf:params:acme:error:connection","title":"","detail":"45.136.18.82: Error getting validation data","instance":"","subproblems":null},"stacktrace":"github.com/mholt/acmez/v3.(*Client).pollAuthorization\n\t/build/caddy/src/caddy/vendor/github.com/mholt/acmez/v3/client.go:570\ngithub.com/mholt/acmez/v3.(*Client).solveChallenges\n\t/build/caddy/src/caddy/vendor/github.com/mholt/acmez/v3/client.go:391\ngithub.com/mholt/acmez/v3.(*Client).ObtainCertificate\n\t/build/caddy/src/caddy/vendor/github.com/mholt/acmez/v3/client.go:149\ngithub.com/caddyserver/certmagic.(*ACMEIssuer).doIssue\n\t/build/caddy/src/caddy/vendor/github.com/caddyserver/certmagic/acmeissuer.go:498\ngithub.com/caddyserver/certmagic.(*ACMEIssuer).Issue\n\t/build/caddy/src/caddy/vendor/github.com/caddyserver/certmagic/acmeissuer.go:391\ngithub.com/caddyserver/caddy/v2/modules/caddytls.(*ACMEIssuer).Issue\n\t/build/caddy/src/caddy/modules/caddytls/acmeissuer.go:292\ngithub.com/caddyserver/certmagic.(*Config).obtainCert.func2\n\t/build/caddy/src/caddy/vendor/github.com/caddyserver/certmagic/config.go:662\ngithub.com/caddyserver/certmagic.doWithRetry\n\t/build/caddy/src/caddy/vendor/github.com/caddyserver/certmagic/async.go:104\ngithub.com/caddyserver/certmagic.(*Config).obtainCert\n\t/build/caddy/src/caddy/vendor/github.com/caddyserver/certmagic/config.go:736\ngithub.com/caddyserver/certmagic.(*Config).ObtainCertAsync\n\t/build/caddy/src/caddy/vendor/github.com/caddyserver/certmagic/config.go:532\ngithub.com/caddyserver/certmagic.(*Config).manageOne.func1\n\t/build/caddy/src/caddy/vendor/github.com/caddyserver/certmagic/config.go:415\ngithub.com/caddyserver/certmagic.(*jobManager).worker\n\t/build/caddy/src/caddy/vendor/github.com/caddyserver/certmagic/async.go:73"}
May 19 23:39:09 thebe caddy[25838]: {"level":"error","ts":1779226749.398884,"msg":"validating authorization","identifier":"paperless.k1ba.eu","problem":{"type":"urn:ietf:params:acme:error:connection","title":"","detail":"45.136.18.82: Error getting validation data","instance":"","subproblems":null},"order":"https://acme-staging-v02.api.letsencrypt.org/acme/order/196539914/38523282103","attempt":1,"max_attempts":3,"stacktrace":"github.com/mholt/acmez/v3.(*Client).ObtainCertificate\n\t/build/caddy/src/caddy/vendor/github.com/mholt/acmez/v3/client.go:165\ngithub.com/caddyserver/certmagic.(*ACMEIssuer).doIssue\n\t/build/caddy/src/caddy/vendor/github.com/caddyserver/certmagic/acmeissuer.go:498\ngithub.com/caddyserver/certmagic.(*ACMEIssuer).Issue\n\t/build/caddy/src/caddy/vendor/github.com/caddyserver/certmagic/acmeissuer.go:391\ngithub.com/caddyserver/caddy/v2/modules/caddytls.(*ACMEIssuer).Issue\n\t/build/caddy/src/caddy/modules/caddytls/acmeissuer.go:292\ngithub.com/caddyserver/certmagic.(*Config).obtainCert.func2\n\t/build/caddy/src/caddy/vendor/github.com/caddyserver/certmagic/config.go:662\ngithub.com/caddyserver/certmagic.doWithRetry\n\t/build/caddy/src/caddy/vendor/github.com/caddyserver/certmagic/async.go:104\ngithub.com/caddyserver/certmagic.(*Config).obtainCert\n\t/build/caddy/src/caddy/vendor/github.com/caddyserver/certmagic/config.go:736\ngithub.com/caddyserver/certmagic.(*Config).ObtainCertAsync\n\t/build/caddy/src/caddy/vendor/github.com/caddyserver/certmagic/config.go:532\ngithub.com/caddyserver/certmagic.(*Config).manageOne.func1\n\t/build/caddy/src/caddy/vendor/github.com/caddyserver/certmagic/config.go:415\ngithub.com/caddyserver/certmagic.(*jobManager).worker\n\t/build/caddy/src/caddy/vendor/github.com/caddyserver/certmagic/async.go:73"}
May 19 23:39:10 thebe caddy[25838]: {"level":"debug","ts":1779226750.3992782,"msg":"creating order","account":"https://acme-staging-v02.api.letsencrypt.org/acme/acct/196539914","identifiers":["paperless.k1ba.eu"]}
May 19 23:39:10 thebe caddy[25838]: {"level":"debug","ts":1779226750.550265,"msg":"http request","method":"POST","url":"https://acme-staging-v02.api.letsencrypt.org/acme/new-order","headers":{"Content-Type":["application/jose+json"],"User-Agent":["Caddy/2.11.2 CertMagic acmez (linux; amd64)"]},"response_headers":{"Boulder-Requester":["196539914"],"Cache-Control":["public, max-age=0, no-cache"],"Content-Length":["362"],"Content-Type":["application/json"],"Date":["Tue, 19 May 2026 21:39:10 GMT"],"Link":["<https://acme-staging-v02.api.letsencrypt.org/directory>;rel=\"index\""],"Location":["https://acme-staging-v02.api.letsencrypt.org/acme/order/196539914/38523285333"],"Replay-Nonce":["3gIrNIz7cZb_3rUL7n6xanjfdCYhzRQBdy2jeu0oRGGuUB3Udcw"],"Server":["nginx"],"Strict-Transport-Security":["max-age=604800"],"X-Frame-Options":["DENY"]},"status_code":201}
May 19 23:39:10 thebe caddy[25838]: {"level":"debug","ts":1779226750.694721,"msg":"http request","method":"POST","url":"https://acme-staging-v02.api.letsencrypt.org/acme/authz/196539914/1399728893","headers":{"Content-Type":["application/jose+json"],"User-Agent":["Caddy/2.11.2 CertMagic acmez (linux; amd64)"]},"response_headers":{"Boulder-Requester":["196539914"],"Cache-Control":["public, max-age=0, no-cache"],"Content-Length":["1077"],"Content-Type":["application/json"],"Date":["Tue, 19 May 2026 21:39:10 GMT"],"Link":["<https://acme-staging-v02.api.letsencrypt.org/directory>;rel=\"index\""],"Replay-Nonce":["bmaVB79hdJQbmj5nfrq_y6sS7sKHPGAZd_FzUMRQ7DV7h-XX2mw"],"Server":["nginx"],"Strict-Transport-Security":["max-age=604800"],"X-Frame-Options":["DENY"]},"status_code":200}
May 19 23:39:10 thebe caddy[25838]: {"level":"debug","ts":1779226750.6950119,"msg":"no solver configured","challenge_type":"dns-01"}
May 19 23:39:10 thebe caddy[25838]: {"level":"info","ts":1779226750.6950407,"msg":"trying to solve challenge","identifier":"paperless.k1ba.eu","challenge_type":"http-01","ca":"https://acme-staging-v02.api.letsencrypt.org/directory"}
May 19 23:39:10 thebe caddy[25838]: {"level":"debug","ts":1779226750.6976097,"msg":"waiting for solver before continuing","identifier":"paperless.k1ba.eu","challenge_type":"http-01"}
May 19 23:39:10 thebe caddy[25838]: {"level":"debug","ts":1779226750.6976788,"msg":"done waiting for solver","identifier":"paperless.k1ba.eu","challenge_type":"http-01"}
May 19 23:39:10 thebe caddy[25838]: {"level":"debug","ts":1779226750.8425908,"msg":"http request","method":"POST","url":"https://acme-staging-v02.api.letsencrypt.org/acme/chall/196539914/1399728893/q4PPdQ","headers":{"Content-Type":["application/jose+json"],"User-Agent":["Caddy/2.11.2 CertMagic acmez (linux; amd64)"]},"response_headers":{"Boulder-Requester":["196539914"],"Cache-Control":["public, max-age=0, no-cache"],"Content-Length":["200"],"Content-Type":["application/json"],"Date":["Tue, 19 May 2026 21:39:10 GMT"],"Link":["<https://acme-staging-v02.api.letsencrypt.org/directory>;rel=\"index\"","<https://acme-staging-v02.api.letsencrypt.org/acme/authz/196539914/1399728893>;rel=\"up\""],"Location":["https://acme-staging-v02.api.letsencrypt.org/acme/chall/196539914/1399728893/q4PPdQ"],"Replay-Nonce":["bmaVB79hqDPTEhKt7WG8cVKLP9wsvUCu_e9TjSrLhOyGBnYaNrY"],"Server":["nginx"],"Strict-Transport-Security":["max-age=604800"],"X-Frame-Options":["DENY"]},"status_code":200}
May 19 23:39:10 thebe caddy[25838]: {"level":"debug","ts":1779226750.8428268,"msg":"challenge accepted","identifier":"paperless.k1ba.eu","challenge_type":"http-01"}
May 19 23:39:11 thebe caddy[25838]: {"level":"debug","ts":1779226751.237462,"msg":"http request","method":"POST","url":"https://acme-staging-v02.api.letsencrypt.org/acme/authz/196539914/1399728893","headers":{"Content-Type":["application/jose+json"],"User-Agent":["Caddy/2.11.2 CertMagic acmez (linux; amd64)"]},"response_headers":{"Boulder-Requester":["196539914"],"Cache-Control":["public, max-age=0, no-cache"],"Content-Length":["1077"],"Content-Type":["application/json"],"Date":["Tue, 19 May 2026 21:39:11 GMT"],"Link":["<https://acme-staging-v02.api.letsencrypt.org/directory>;rel=\"index\""],"Replay-Nonce":["bmaVB79hmKG8J9EQSixFqd4qtNZVEciGmVnfKxBdQf4XhmPrPis"],"Server":["nginx"],"Strict-Transport-Security":["max-age=604800"],"X-Frame-Options":["DENY"]},"status_code":200}
May 19 23:39:11 thebe caddy[25838]: {"level":"debug","ts":1779226751.662501,"msg":"http request","method":"POST","url":"https://acme-staging-v02.api.letsencrypt.org/acme/authz/196539914/1399728893","headers":{"Content-Type":["application/jose+json"],"User-Agent":["Caddy/2.11.2 CertMagic acmez (linux; amd64)"]},"response_headers":{"Boulder-Requester":["196539914"],"Cache-Control":["public, max-age=0, no-cache"],"Content-Length":["1142"],"Content-Type":["application/json"],"Date":["Tue, 19 May 2026 21:39:11 GMT"],"Link":["<https://acme-staging-v02.api.letsencrypt.org/directory>;rel=\"index\""],"Replay-Nonce":["bmaVB79hhcP8oFeNmXFvvugACIAF3g4vqIo-5-Xqql5RDkGknC0"],"Server":["nginx"],"Strict-Transport-Security":["max-age=604800"],"X-Frame-Options":["DENY"]},"status_code":200}
May 19 23:39:11 thebe caddy[25838]: {"level":"error","ts":1779226751.6630776,"msg":"challenge failed","identifier":"paperless.k1ba.eu","challenge_type":"http-01","problem":{"type":"urn:ietf:params:acme:error:unauthorized","title":"","detail":"The key authorization file from the server did not match this challenge. Expected \"f5NwD5NsSrslLmXi-YLzkY8PpzJIt9GkGu2F2D39n4A.KhbSNX4uaUbhpmHw0iAug7AhWLWPqdLHjYTUWBRBc0o\" (got \"\")","instance":"","subproblems":null},"stacktrace":"github.com/mholt/acmez/v3.(*Client).pollAuthorization\n\t/build/caddy/src/caddy/vendor/github.com/mholt/acmez/v3/client.go:570\ngithub.com/mholt/acmez/v3.(*Client).solveChallenges\n\t/build/caddy/src/caddy/vendor/github.com/mholt/acmez/v3/client.go:391\ngithub.com/mholt/acmez/v3.(*Client).ObtainCertificate\n\t/build/caddy/src/caddy/vendor/github.com/mholt/acmez/v3/client.go:149\ngithub.com/caddyserver/certmagic.(*ACMEIssuer).doIssue\n\t/build/caddy/src/caddy/vendor/github.com/caddyserver/certmagic/acmeissuer.go:498\ngithub.com/caddyserver/certmagic.(*ACMEIssuer).Issue\n\t/build/caddy/src/caddy/vendor/github.com/caddyserver/certmagic/acmeissuer.go:391\ngithub.com/caddyserver/caddy/v2/modules/caddytls.(*ACMEIssuer).Issue\n\t/build/caddy/src/caddy/modules/caddytls/acmeissuer.go:292\ngithub.com/caddyserver/certmagic.(*Config).obtainCert.func2\n\t/build/caddy/src/caddy/vendor/github.com/caddyserver/certmagic/config.go:662\ngithub.com/caddyserver/certmagic.doWithRetry\n\t/build/caddy/src/caddy/vendor/github.com/caddyserver/certmagic/async.go:104\ngithub.com/caddyserver/certmagic.(*Config).obtainCert\n\t/build/caddy/src/caddy/vendor/github.com/caddyserver/certmagic/config.go:736\ngithub.com/caddyserver/certmagic.(*Config).ObtainCertAsync\n\t/build/caddy/src/caddy/vendor/github.com/caddyserver/certmagic/config.go:532\ngithub.com/caddyserver/certmagic.(*Config).manageOne.func1\n\t/build/caddy/src/caddy/vendor/github.com/caddyserver/certmagic/config.go:415\ngithub.com/caddyserver/certmagic.(*jobManager).worker\n\t/build/caddy/src/caddy/vendor/github.com/caddyserver/certmagic/async.go:73"}
May 19 23:39:11 thebe caddy[25838]: {"level":"error","ts":1779226751.6633859,"msg":"validating authorization","identifier":"paperless.k1ba.eu","problem":{"type":"urn:ietf:params:acme:error:unauthorized","title":"","detail":"The key authorization file from the server did not match this challenge. Expected \"f5NwD5NsSrslLmXi-YLzkY8PpzJIt9GkGu2F2D39n4A.KhbSNX4uaUbhpmHw0iAug7AhWLWPqdLHjYTUWBRBc0o\" (got \"\")","instance":"","subproblems":null},"order":"https://acme-staging-v02.api.letsencrypt.org/acme/order/196539914/38523285333","attempt":2,"max_attempts":3,"stacktrace":"github.com/mholt/acmez/v3.(*Client).ObtainCertificate\n\t/build/caddy/src/caddy/vendor/github.com/mholt/acmez/v3/client.go:165\ngithub.com/caddyserver/certmagic.(*ACMEIssuer).doIssue\n\t/build/caddy/src/caddy/vendor/github.com/caddyserver/certmagic/acmeissuer.go:498\ngithub.com/caddyserver/certmagic.(*ACMEIssuer).Issue\n\t/build/caddy/src/caddy/vendor/github.com/caddyserver/certmagic/acmeissuer.go:391\ngithub.com/caddyserver/caddy/v2/modules/caddytls.(*ACMEIssuer).Issue\n\t/build/caddy/src/caddy/modules/caddytls/acmeissuer.go:292\ngithub.com/caddyserver/certmagic.(*Config).obtainCert.func2\n\t/build/caddy/src/caddy/vendor/github.com/caddyserver/certmagic/config.go:662\ngithub.com/caddyserver/certmagic.doWithRetry\n\t/build/caddy/src/caddy/vendor/github.com/caddyserver/certmagic/async.go:104\ngithub.com/caddyserver/certmagic.(*Config).obtainCert\n\t/build/caddy/src/caddy/vendor/github.com/caddyserver/certmagic/config.go:736\ngithub.com/caddyserver/certmagic.(*Config).ObtainCertAsync\n\t/build/caddy/src/caddy/vendor/github.com/caddyserver/certmagic/config.go:532\ngithub.com/caddyserver/certmagic.(*Config).manageOne.func1\n\t/build/caddy/src/caddy/vendor/github.com/caddyserver/certmagic/config.go:415\ngithub.com/caddyserver/certmagic.(*jobManager).worker\n\t/build/caddy/src/caddy/vendor/github.com/caddyserver/certmagic/async.go:73"}
May 19 23:39:11 thebe caddy[25838]: {"level":"error","ts":1779226751.6635296,"logger":"tls.obtain","msg":"could not get certificate from issuer","identifier":"paperless.k1ba.eu","issuer":"acme-staging-v02.api.letsencrypt.org-directory","error":"HTTP 403 urn:ietf:params:acme:error:unauthorized - The key authorization file from the server did not match this challenge. Expected \"f5NwD5NsSrslLmXi-YLzkY8PpzJIt9GkGu2F2D39n4A.KhbSNX4uaUbhpmHw0iAug7AhWLWPqdLHjYTUWBRBc0o\" (got \"\")"}
May 19 23:39:11 thebe caddy[25838]: {"level":"debug","ts":1779226751.663583,"logger":"events","msg":"event","name":"cert_failed","id":"c7a87e25-767b-4899-b5c8-7589e24dcb72","origin":"tls","data":{"error":{},"identifier":"paperless.k1ba.eu","issuers":["acme-staging-v02.api.letsencrypt.org-directory"],"renewal":false}}
May 19 23:39:11 thebe caddy[25838]: {"level":"error","ts":1779226751.6636565,"logger":"tls.obtain","msg":"will retry","error":"[paperless.k1ba.eu] Obtain: [paperless.k1ba.eu] solving challenge: paperless.k1ba.eu: [paperless.k1ba.eu] authorization failed: HTTP 403 urn:ietf:params:acme:error:unauthorized - The key authorization file from the server did not match this challenge. Expected \"f5NwD5NsSrslLmXi-YLzkY8PpzJIt9GkGu2F2D39n4A.KhbSNX4uaUbhpmHw0iAug7AhWLWPqdLHjYTUWBRBc0o\" (got \"\") (ca=https://acme-staging-v02.api.letsencrypt.org/directory)","attempt":1,"retrying_in":60,"elapsed":4.499112333,"max_duration":2592000}
3. Caddy version:
Outer: 2.6.2
Inner 2.11.2
4. How I installed and ran Caddy:
a. System environment:
Outer: Debian 6.1.170-1, installed via sudo apt install caddy
Inner: Arch Linux with kernel 7.0.3-arch1-2, installed via sudo pacman -S caddy
b. Command:
sudo systemctl enable --now caddy.service
c. Service/unit/compose file:
Outer
# /lib/systemd/system/caddy.service
# caddy.service
#
# For using Caddy with a config file.
#
# Make sure the ExecStart and ExecReload commands are correct
# for your installation.
#
# See https://caddyserver.com/docs/install for instructions.
#
# WARNING: This service does not use the --resume flag, so if you
# use the API to make changes, they will be overwritten by the
# Caddyfile next time the service is restarted. If you intend to
# use Caddy's API to configure it, add the --resume flag to the
# `caddy run` command or use the caddy-api.service file instead.
[Unit]
Description=Caddy
Documentation=https://caddyserver.com/docs/
After=network.target network-online.target
Requires=network-online.target
[Service]
Type=notify
User=caddy
Group=caddy
ExecStart=/usr/bin/caddy run --environ --config /etc/caddy/Caddyfile
ExecReload=/usr/bin/caddy reload --config /etc/caddy/Caddyfile --force
TimeoutStopSec=5s
LimitNOFILE=1048576
LimitNPROC=512
PrivateTmp=true
ProtectSystem=full
AmbientCapabilities=CAP_NET_BIND_SERVICE
[Install]
WantedBy=multi-user.target
# /etc/systemd/system/caddy.service.d/override.conf
[Service]
RuntimeDirectory=caddy # After a recent update (in 2026-05 I believe) this became necessary because otherwise /run/caddy would be missing
Inner
# /etc/systemd/system/caddy.service
# caddy.service
#
# For using Caddy with a config file.
#
# Make sure the ExecStart and ExecReload commands are correct
# for your installation.
#
# See https://caddyserver.com/docs/install for instructions.
#
# WARNING: This service does not use the --resume flag, so if you
# use the API to make changes, they will be overwritten by the
# Caddyfile next time the service is restarted. If you intend to
# use Caddy's API to configure it, add the --resume flag to the
# `caddy run` command or use the caddy-api.service file instead.
[Unit]
Description=Caddy web server
Documentation=https://caddyserver.com/docs/
After=network-online.target
Wants=network-online.target
StartLimitIntervalSec=14400
StartLimitBurst=10
[Service]
Type=notify
User=caddy
Group=caddy
Environment=XDG_DATA_HOME=/var/lib
Environment=XDG_CONFIG_HOME=/etc
ExecStartPre=/usr/bin/caddy validate --config /etc/caddy/Caddyfile
ExecStart=/usr/bin/caddy run --config /etc/caddy/Caddyfile
ExecReload=/usr/bin/caddy reload --config /etc/caddy/Caddyfile --force
ExecStopPost=/usr/bin/rm -f /run/caddy/admin.socket
# Do not allow the process to be restarted in a tight loop. If the
# process fails to start, something critical needs to be fixed.
Restart=on-abnormal
# Use graceful shutdown with a reasonable timeout
TimeoutStopSec=5s
LimitNOFILE=1048576
LimitNPROC=512
# Hardening options
AmbientCapabilities=CAP_NET_BIND_SERVICE
CapabilityBoundingSet=CAP_NET_BIND_SERVICE
DevicePolicy=closed
LockPersonality=true
MemoryAccounting=true
MemoryDenyWriteExecute=true
NoNewPrivileges=true
PrivateDevices=true
PrivateTmp=true
ProcSubset=pid
ProtectClock=true
ProtectControlGroups=true
ProtectHome=true
ProtectHostname=true
ProtectKernelLogs=true
ProtectKernelModules=true
ProtectKernelTunables=true
ProtectProc=invisible
ProtectSystem=strict
RemoveIPC=true
ReadWritePaths=/var/lib/caddy /var/log/caddy /run/caddy
RestrictNamespaces=true
RestrictRealtime=true
RestrictSUIDSGID=true
[Install]
WantedBy=multi-user.target
# /etc/systemd/system/caddy.service.d/override.conf
[Service]
RuntimeDirectory=caddy
d. My complete Caddy config:
Outer:
{
debug
}
http://paperless.k1ba.eu {
reverse_proxy /.well-known/acme-challenge 192.168.122.3
}
Inner:
{
debug
acme_ca https://acme-staging-v02.api.letsencrypt.org/directory
}
https://paperless.k1ba.eu {
root /var/www/testing
file_server
}