Hello!
1. The problem I’m having:
I am trying to setup to Use Caddy for local HTTPS (TLS) between front-end reverse proxy and LAN hosts but the connection between frontend and backend does not work (502). The services I want to reach via DNS+SSL are local only but should be secured by a wild card cert for my local subdomain *.home.mydomain.tld which terminates on the frontend caddy.
This right here might be most relevant error log (from the backend caddy startup) so I will start with it
{"level":"error","ts":1785054957.7002916,"logger":"tls.obtain","msg":"will retry","error":"\[backend1.home.mydomain.tld\] Obtain: registering account \[\] with server: provisioning client: response Content-Type is but target container is not io.Writer: acme.Directory","attempt":1,"retrying_in":60,"elapsed":0.064567736,"max_duration":2592000}
Curl example:
curl -vL https://backend1.home.mydomain.tld/
* Host backend1.home.mydomain.tld:443 was resolved.
* IPv6: (none)
* IPv4: 192.168.2.2
* Trying 192.168.2.2:443...
* Connected to backend1.home.mydomain.tld (192.168.2.2) port 443
* ALPN: curl offers h2,http/1.1
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* CAfile: /etc/ssl/certs/ca-certificates.crt
* CApath: /etc/ssl/certs
* TLSv1.3 (IN), TLS handshake, Server hello (2):
* TLSv1.3 (IN), TLS handshake, Encrypted Extensions (8):
* TLSv1.3 (IN), TLS handshake, Certificate (11):
* TLSv1.3 (IN), TLS handshake, CERT verify (15):
* TLSv1.3 (IN), TLS handshake, Finished (20):
* TLSv1.3 (OUT), TLS change cipher, Change cipher spec (1):
* TLSv1.3 (OUT), TLS handshake, Finished (20):
* SSL connection using TLSv1.3 / TLS_AES_128_GCM_SHA256 / X25519 / id-ecPublicKey
* ALPN: server accepted h2
* Server certificate:
* subject: CN=*.home.mydomain.tld
* start date: Jul 25 16:44:54 2026 GMT
* expire date: Oct 23 16:44:53 2026 GMT
* subjectAltName: host "backend1.home.mydomain.tld" matched cert's "*.home.mydomain.tld"
* issuer: C=US; O=Let's Encrypt; CN=YE2
* SSL certificate verify ok.
* Certificate level 0: Public key type EC/prime256v1 (256/128 Bits/secBits), signed using ecdsa-with-SHA384
* Certificate level 1: Public key type EC/secp384r1 (384/192 Bits/secBits), signed using ecdsa-with-SHA384
* Certificate level 2: Public key type EC/secp384r1 (384/192 Bits/secBits), signed using ecdsa-with-SHA384
* Certificate level 3: Public key type EC/secp384r1 (384/192 Bits/secBits), signed using ecdsa-with-SHA384
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
* using HTTP/2
* [HTTP/2] [1] OPENED stream for https://backend1.home.mydomain.tld/
* [HTTP/2] [1] [:method: GET]
* [HTTP/2] [1] [:scheme: https]
* [HTTP/2] [1] [:authority: backend1.home.mydomain.tld]
* [HTTP/2] [1] [:path: /]
* [HTTP/2] [1] [user-agent: curl/8.5.0]
* [HTTP/2] [1] [accept: */*]
> GET / HTTP/2
> Host: backend1.home.mydomain.tld
> User-Agent: curl/8.5.0
> Accept: */*
>
< HTTP/2 502
< alt-svc: h3=":443"; ma=2592000
< server: Caddy
< content-length: 0
< date: Sun, 26 Jul 2026 09:35:26 GMT
<
* Connection #0 to host backend1.home.mydomain.tld left intact
2. Error messages and/or full log output:
This right here might be most relevant error log from the backend log below
{"level":"error","ts":1785054957.7002916,"logger":"tls.obtain","msg":"will retry","error":"\[backend1.home.mydomain.tld\] Obtain: registering account \[\] with server: provisioning client: response Content-Type is but target container is not io.Writer: acme.Directory","attempt":1,"retrying_in":60,"elapsed":0.064567736,"max_duration":2592000}
Backend Log startup:
{"level":"info","ts":1785054957.5999954,"msg":"maxprocs: Leaving GOMAXPROCS=4: CPU quota undefined"}
{"level":"info","ts":1785054957.6000435,"msg":"GOMEMLIMIT is updated","GOMEMLIMIT":1740758630,"previous":9223372036854775807}
{"level":"info","ts":1785054957.600055,"msg":"using config from file","file":"/etc/caddy/Caddyfile"}
{"level":"info","ts":1785054957.6000626,"msg":"adapted config to JSON","adapter":"caddyfile"}
{"level":"warn","ts":1785054957.6000688,"msg":"Caddyfile input is not formatted; run 'caddy fmt --overwrite' to fix inconsistencies","adapter":"caddyfile","file":"/etc/caddy/Caddyfile","line":3}
{"level":"info","ts":1785054957.6025715,"logger":"admin","msg":"admin endpoint started","address":"localhost:2019","enforce_origin":false,"origins":\["//localhost:2019","//\[::1\]:2019","//127.0.0.1:2019"\]}
{"level":"info","ts":1785054957.6036751,"logger":"http.auto_https","msg":"server is listening only on the HTTPS port but has no TLS connection policies; adding one to enable TLS","server_name":"srv0","https_port":443}
{"level":"info","ts":1785054957.6037593,"logger":"http.auto_https","msg":"enabling automatic HTTP->HTTPS redirects","server_name":"srv0"}
{"level":"info","ts":1785054957.6042733,"logger":"tls.cache.maintenance","msg":"started background certificate maintenance","cache":"0x7f4bc37480"}
{"level":"debug","ts":1785054957.6042278,"logger":"http.auto_https","msg":"adjusted config","tls":{"automation":{"policies":\[{}\]}},"http":{"servers":{"remaining_auto_https_redirects":{"listen":\[":80"\],"routes":\[{},{}\]},"srv0":{"listen":\[":443"\],"routes":\[{"handle":\[{"handler":"subroute","routes":\[{"handle":\[{"body":"Hello from Backend 1 (pi4)","handler":"static_response"}\]}\]}\],"terminal":true}\],"tls_connection_policies":\[{}\],"automatic_https":{}}}}}
{"level":"debug","ts":1785054957.6162853,"logger":"http","msg":"starting server loop","address":"\[::\]:443","tls":true,"http3":false}
{"level":"info","ts":1785054957.6163795,"logger":"http","msg":"enabling HTTP/3 listener","addr":":443"}
{"level":"info","ts":1785054957.617311,"logger":"http.log","msg":"server running","name":"srv0","protocols":\["h1","h2","h3"\]}
{"level":"debug","ts":1785054957.6284058,"logger":"http","msg":"starting server loop","address":"\[::\]:80","tls":false,"http3":false}
{"level":"warn","ts":1785054957.6284728,"logger":"http","msg":"HTTP/2 skipped because it requires TLS","network":"tcp","addr":":80"}
{"level":"warn","ts":1785054957.6284833,"logger":"http","msg":"HTTP/3 skipped because it requires TLS","network":"tcp","addr":":80"}
{"level":"info","ts":1785054957.6284919,"logger":"http.log","msg":"server running","name":"remaining_auto_https_redirects","protocols":\["h1","h2","h3"\]}
{"level":"info","ts":1785054957.6285086,"logger":"http","msg":"enabling automatic TLS certificate management","domains":\["backend1.home.mydomain.tld"\]}
{"level":"debug","ts":1785054957.6286988,"logger":"events","msg":"event","name":"started","id":"64ac0fb3-18c2-474a-ba03-63bac631e997","origin":"","data":null}
{"level":"info","ts":1785054957.629227,"msg":"autosaved config (load with --resume flag)","file":"/config/caddy/autosave.json"}
{"level":"info","ts":1785054957.6292582,"msg":"serving initial configuration"}
{"level":"info","ts":1785054957.6331282,"logger":"tls.obtain","msg":"acquiring lock","identifier":"backend1.home.mydomain.tld"}
{"level":"info","ts":1785054957.6335924,"logger":"tls","msg":"storage cleaning happened too recently; skipping for now","storage":"FileStorage:/data/caddy","instance":"b15f9cfc-9a1b-4454-b018-a2b8a030fd5a","try_again":1785141357.6335874,"try_again_in":86399.999998111}
{"level":"info","ts":1785054957.6354802,"logger":"tls","msg":"finished cleaning storage units"}
{"level":"info","ts":1785054957.6357014,"logger":"tls.obtain","msg":"lock acquired","identifier":"backend1.home.mydomain.tld"}
{"level":"info","ts":1785054957.6360633,"logger":"tls.obtain","msg":"obtaining certificate","identifier":"backend1.home.mydomain.tld"}
{"level":"debug","ts":1785054957.6361423,"logger":"events","msg":"event","name":"cert_obtaining","id":"75c57c98-1408-41c6-8bf6-c40387d563c9","origin":"tls","data":{"identifier":"backend1.home.mydomain.tld"}}
{"level":"debug","ts":1785054957.636501,"logger":"tls","msg":"created CSR","identifiers":\["backend1.home.mydomain.tld"\],"san_dns_names":\["backend1.home.mydomain.tld"\],"san_emails":\[\],"common_name":"","extra_extensions":0}
{"level":"debug","ts":1785054957.6374838,"logger":"tls.obtain","msg":"trying issuer 1/1","issuer":"caddy-acme.home-local-directory"}
{"level":"info","ts":1785054957.6377134,"logger":"http","msg":"creating new account because no account for configured email is known to us","email":"","ca":"https://caddy-acme.home/local/directory","error":"open /data/caddy/acme/caddy-acme.home-local-directory/users/default/default.json: no such file or directory"}
{"level":"info","ts":1785054957.637842,"logger":"http","msg":"ACME account has empty status; registering account with ACME server","contact":\[\],"location":""}
{"level":"info","ts":1785054957.6404047,"logger":"http","msg":"creating new account because no account for configured email is known to us","email":"","ca":"https://caddy-acme.home/local/directory","error":"open /data/caddy/acme/caddy-acme.home-local-directory/users/default/default.json: no such file or directory"}
{"level":"debug","ts":1785054957.6995811,"logger":"http.acme_client","msg":"http request","method":"GET","url":"https://caddy-acme.home/local/directory","headers":{"User-Agent":\["Caddy/2.11.4 CertMagic acmez (linux; arm64)"\]},"response_headers":{"Alt-Svc":\["h3=\\":443\\"; ma=2592000"\],"Content-Length":\["0"\],"Date":\["Sun, 26 Jul 2026 08:35:57 GMT"\],"Server":\["Caddy"\]},"status_code":200}
{"level":"error","ts":1785054957.7001147,"logger":"tls.obtain","msg":"could not get certificate from issuer","identifier":"backend1.home.mydomain.tld","issuer":"caddy-acme.home-local-directory","error":"registering account \[\] with server: provisioning client: response Content-Type is but target container is not io.Writer: \*acme.Directory"}
{"level":"debug","ts":1785054957.7001913,"logger":"events","msg":"event","name":"cert_failed","id":"8357bf4c-34b3-4aeb-8455-c1a50fd665ee","origin":"tls","data":{"error":{},"identifier":"backend1.home.mydomain.tld","issuers":\["caddy-acme.home-local-directory"\],"renewal":false}}
{"level":"error","ts":1785054957.7002916,"logger":"tls.obtain","msg":"will retry","error":"\[backend1.home.mydomain.tld\] Obtain: registering account \[\] with server: provisioning client: response Content-Type is but target container is not io.Writer: \*acme.Directory","attempt":1,"retrying_in":60,"elapsed":0.064567736,"max_duration":2592000}
Backend Log when curl’ed:
{"level":"debug","ts":1785054860.1510007,"logger":"events","msg":"event","name":"tls_get_certificate","id":"2ef0a881-eea3-4b35-a9bf-60313b1cff42","origin":"tls","data":{"client_hello":{"CipherSuites":\[49195,49199,49196,49200,52393,52392,49161,49171,49162,49172,4865,4866,4867\],"ServerName":"pi4.home","SupportedCurves":\[4588,4587,4589,29,23,24,25\],"SupportedPoints":"AA==","SignatureSchemes":\[2052,1027,2055,2053,2054,1025,1281,1537,1283,1539\],"SupportedProtos":\["h2","http/1.1"\],"SupportedVersions":\[772,771\],"RemoteAddr":{"IP":"192.168.2.2","Port":46718,"Zone":""},"LocalAddr":{"IP":"172.18.0.2","Port":443,"Zone":""}}}}
{"level":"debug","ts":1785054860.151191,"logger":"tls.handshake","msg":"no matching certificates and no custom selection logic","identifier":"pi4.home"}
{"level":"debug","ts":1785054860.151224,"logger":"tls.handshake","msg":"no matching certificates and no custom selection logic","identifier":"\*.home"}
{"level":"debug","ts":1785054860.1512444,"logger":"tls.handshake","msg":"no matching certificates and no custom selection logic","identifier":"\*.\*"}
{"level":"debug","ts":1785054860.1512978,"logger":"tls.handshake","msg":"no certificate matching TLS ClientHello","remote_ip":"192.168.2.2","remote_port":"46718","server_name":"pi4.home","remote":"192.168.2.2:46718","identifier":"pi4.home","cipher_suites":\[49195,49199,49196,49200,52393,52392,49161,49171,49162,49172,4865,4866,4867\],"cert_cache_fill":0,"load_or_obtain_if_necessary":true,"on_demand":false}
{"level":"debug","ts":1785054860.1516151,"logger":"http.stdlib","msg":"http: TLS handshake error from 192.168.2.2:46718: no certificate available for 'pi4.home'"}
Frontend Log when curl’ed
{"level":"debug","ts":1785054860.0558243,"logger":"events","msg":"event","name":"tls_get_certificate","id":"2af4e3f1-d3ed-40c6-bd82-12fbbd5eae08","origin":"tls","data":{"client_hello":{"CipherSuites":\[4866,4867,4865,49196,49200,159,52393,52392,52394,49195,49199,158,49188,49192,107,49187,49191,103,49162,49172,57,49161,49171,51,157,156,61,60,53,47,255\],"ServerName":"backend1.home.mydomain.tld","SupportedCurves":\[29,23,30,25,24,256,257,258,259,260\],"SupportedPoints":"AAEC","SignatureSchemes":\[1027,1283,1539,2055,2056,2057,2058,2059,2052,2053,2054,1025,1281,1537,771,769,770,1026,1282,1538\],"SupportedProtos":\["h2","http/1.1"\],"SupportedVersions":\[772,771\],"RemoteAddr":{"IP":"192.168.2.50","Port":42020,"Zone":""},"LocalAddr":{"IP":"10.42.42.100","Port":443,"Zone":""}}}}
{"level":"debug","ts":1785054860.0559046,"logger":"tls.handshake","msg":"no matching certificates and no custom selection logic","identifier":"backend1.home.mydomain.tld"}
{"level":"debug","ts":1785054860.055911,"logger":"tls.handshake","msg":"choosing certificate","identifier":"\*.home.mydomain.tld","num_choices":1}
{"level":"debug","ts":1785054860.0559163,"logger":"tls.handshake","msg":"default certificate selection results","identifier":"\*.home.mydomain.tld","subjects":\["\*.home.mydomain.tld"\],"managed":true,"issuer_key":"acme-v02.api.letsencrypt.org-directory","hash":"582826872ae2a997a9ed2c8a18aa2d4be19b4da856fb47ae1559e45cded55b5e"}
{"level":"debug","ts":1785054860.0559244,"logger":"tls.handshake","msg":"matched certificate in cache","remote_ip":"192.168.2.50","remote_port":"42020","subjects":\["\*.home.mydomain.tld"\],"managed":true,"expiration":1792773894,"hash":"582826872ae2a997a9ed2c8a18aa2d4be19b4da856fb47ae1559e45cded55b5e"}
{"level":"debug","ts":1785054860.0730286,"logger":"http.handlers.reverse_proxy","msg":"selected upstream","dial":"pi4.home:443","total_upstreams":1}
{"level":"debug","ts":1785054860.1367788,"logger":"http.handlers.reverse_proxy","msg":"upstream roundtrip","upstream":"pi4.home:443","duration":0.063677812,"request":{"remote_ip":"192.168.2.50","remote_port":"42020","client_ip":"192.168.2.50","proto":"HTTP/2.0","method":"GET","host":"pi4.home:443","uri":"/","headers":{"X-Forwarded-Proto":\["https"\],"X-Forwarded-Host":\["backend1.home.mydomain.tld"\],"Via":\["2.0 Caddy"\],"User-Agent":\["curl/8.5.0"\],"Accept":\["\*/\*"\],"X-Forwarded-For":\["192.168.2.50"\]},"tls":{"resumed":false,"version":772,"cipher_suite":4865,"proto":"h2","server_name":"backend1.home.mydomain.tld","ech":false}},"error":"remote error: tls: internal error"}
{"level":"error","ts":1785054860.1368668,"logger":"http.log.error","msg":"remote error: tls: internal error","request":{"remote_ip":"192.168.2.50","remote_port":"42020","client_ip":"192.168.2.50","proto":"HTTP/2.0","method":"GET","host":"backend1.home.mydomain.tld","uri":"/","headers":{"User-Agent":\["curl/8.5.0"\],"Accept":\["\*/\*"\]},"tls":{"resumed":false,"version":772,"cipher_suite":4865,"proto":"h2","server_name":"backend1.home.mydomain.tld","ech":false}},"duration":0.063926501,"status":502,"err_id":"b9wij7p1i","err_trace":"reverseproxy.statusError (reverseproxy.go:1626)"}
3. Caddy version:
v2.11.4 h1:XKxkMTgNSizEvKG6QHue6cAsFOteU2qA61w2tKkCWi0=
(both instances)
4. How I installed and ran Caddy:
a. System environment:
My Setup is like this:
- Server 1: Pi5
- Frontend caddy (handles WAN + local requests)
- 192.168.2.2
- Server 2: Pi4
- Backend caddy
- 192.168.2.3
- Local DNS Server
- on Server1, being used network wide.
Local DNS setup:
- pi4.home → 192.168.2.3
- backend server
- caddy-acme.home → 192.168.2.2
- acme running on frontend caddy
- backend1.home.mydomain.tld → 192.168.2.2
- fqdn for backend service but pointing to frontend caddy for reverse proxying
- more entries for servies running on the frontend which are not relevant
My domain
- mydomain.tld (sorry this is redacted, but properly)
- *.home.mydomain.tld wildcard cert on frontend caddy
- This is being used for local only services
- subdomains not known outside LAN
b. Command:
Caddy via docker compose
c. Service/unit/compose file:
services:
caddy:
container_name: caddy
build: ./caddy-desec-dockerfile
restart: unless-stopped
ports:
- "80:80"
- "443:443"
- "443:443/udp"
networks:
home:
ipv4_address: 10.42.42.100
dns: 192.168.2.2
cap_add:
- NET_ADMIN
volumes:
- ${DOCKER_DATA}/caddy/conf:/etc/caddy:ro
- ${DOCKER_DATA}/caddy/site:/srv:ro
- ${DOCKER_DATA}/caddy/data:/data
- ${DOCKER_DATA}/caddy/caddy_config:/config
Backend:
services:
caddy:
container_name: caddy
image: caddy:latest
restart: unless-stopped
cap_add:
- NET_ADMIN
ports:
- "80:80"
- "443:443"
- "443:443/udp"
volumes:
- ~/docker/caddy/caddy_data/conf:/etc/caddy
- ~/docker/caddy/caddy_data/site:/srv
- ~/docker/caddy/caddy_data/frontend-root-CA.crt:/etc/ssl/certs/frontend-root-CA.crt:ro
- caddy_data:/data
- caddy_config:/config
volumes:
caddy_data:
caddy_config:
d. My complete Caddy config:
Frontend caddy config:
{
debug
}
# Wildcard cert for internal domains
*.home.mydomain.tld {
redacted
}
# acme server for frontend/backend communication
caddy-acme.home {
acme_server
tls internal
}
# exemplary frontend service (nginx docker container)
https://web.home.mydomain.tld:443 {
reverse_proxy nginx:80
}
# backend stuff that is not working
https://backend1.home.mydomain.tld {
reverse_proxy https://pi4.home {
header_up Host {upstream_hostport}
}
}
Backend caddy config:
{
# General option
debug
# TLS Options
acme_ca https://caddy-acme.home/local/directory
acme_ca_root /etc/ssl/certs/frontend-root-CA.crt
# this cert is mounted to the container and contains the cert created by the frontend acme
}
https://backend1.home.mydomain.tld:443 {
respond "Hello from Backend 1 (pi4)"
}