Caddy reverse proxy is using upstream proxy IP for TLS validation instead of Host header

1. The problem I’m having:

I have an existing machine running nginx, serving ~10 sub-domains of johnmaguire.me. I’m trying to setup Caddy to act as a frontend reverse proxy for this server, using DNS for automatic TLS.

When connecting to the upstream proxy, it seems to be using the upstream proxy’s IP as the hostname for TLS validation, instead of the hostname in the Host header sent by the client. This causes validation to fail, because the nginx server is configured for hostnames only.

How can I get Caddy to validate the upstream certificate based on the Host header?

❯ curl -vL https://johnmaguire.me
*   Trying 192.168.123.52:443...
* Connected to johnmaguire.me (192.168.123.52) port 443
* ALPN: curl offers h2,http/1.1
* (304) (OUT), TLS handshake, Client hello (1):
*  CAfile: /etc/ssl/cert.pem
*  CApath: none
* (304) (IN), TLS handshake, Server hello (2):
* (304) (IN), TLS handshake, Unknown (8):
* (304) (IN), TLS handshake, Certificate (11):
* (304) (IN), TLS handshake, CERT verify (15):
* (304) (IN), TLS handshake, Finished (20):
* (304) (OUT), TLS handshake, Finished (20):
* SSL connection using TLSv1.3 / AEAD-CHACHA20-POLY1305-SHA256
* ALPN: server accepted h2
* Server certificate:
*  subject: CN=johnmaguire.me
*  start date: Feb  1 02:48:03 2024 GMT
*  expire date: May  1 02:48:02 2024 GMT
*  subjectAltName: host "johnmaguire.me" matched cert's "johnmaguire.me"
*  issuer: C=US; O=Let's Encrypt; CN=R3
*  SSL certificate verify ok.
* using HTTP/2
* [HTTP/2] [1] OPENED stream for https://johnmaguire.me/
* [HTTP/2] [1] [:method: GET]
* [HTTP/2] [1] [:scheme: https]
* [HTTP/2] [1] [:authority: johnmaguire.me]
* [HTTP/2] [1] [:path: /]
* [HTTP/2] [1] [user-agent: curl/8.4.0]
* [HTTP/2] [1] [accept: */*]
> GET / HTTP/2
> Host: johnmaguire.me
> User-Agent: curl/8.4.0
> Accept: */*
>
< HTTP/2 502
< alt-svc: h3=":443"; ma=2592000
< server: Caddy
< content-length: 0
< date: Thu, 01 Feb 2024 04:09:43 GMT
<
* Connection #0 to host johnmaguire.me left intact

2. Error messages and/or full log output:

Feb 01 04:10:49 ingress systemd[1]: Started Caddy webserver.
Feb 01 04:10:49 ingress caddy[25064]: {"level":"info","ts":1706760649.659486,"msg":"using provided configuration","config_file":"/etc/caddy/Caddyfile","config_adapter":""}
Feb 01 04:10:49 ingress caddy[25064]: {"level":"warn","ts":1706760649.6607897,"msg":"Caddyfile input is not formatted; run 'caddy fmt --overwrite' to fix inconsistencies","adapter":"caddyfile","file":"/etc/caddy/Caddyfile","line":2}
Feb 01 04:10:49 ingress caddy[25064]: {"level":"info","ts":1706760649.6676884,"logger":"admin","msg":"admin endpoint started","address":"localhost:2019","enforce_origin":false,"origins":["//localhost:2019","//[::1]:2019","//127.0.0.1:2019"]}
Feb 01 04:10:49 ingress caddy[25064]: {"level":"info","ts":1706760649.667778,"logger":"http.auto_https","msg":"server is listening only on the HTTPS port but has no TLS connection policies; adding one to enable TLS","server_name":"srv0","https_port":443}
Feb 01 04:10:49 ingress caddy[25064]: {"level":"info","ts":1706760649.6677856,"logger":"http.auto_https","msg":"enabling automatic HTTP->HTTPS redirects","server_name":"srv0"}
Feb 01 04:10:49 ingress caddy[25064]: {"level":"info","ts":1706760649.6677854,"logger":"tls.cache.maintenance","msg":"started background certificate maintenance","cache":"0xc0003fa780"}
Feb 01 04:10:49 ingress caddy[25064]: {"level":"debug","ts":1706760649.6677961,"logger":"http.auto_https","msg":"adjusted config","tls":{"automation":{"policies":[{"subjects":["*.johnmaguire.me","johnmaguire.me"]},{}]}},"http":{"servers":{"remaining_auto_https_redirects":{"listen":[":80"],"routes":[{},{}]},"srv0":{"listen":[":443"],"routes":[{"handle":[{"handler":"subroute","routes":[{"handle":[{"handler":"reverse_proxy","transport":{"protocol":"http","tls":{}},"upstreams":[{"dial":"192.168.123.14:443"}]}]}]}],"terminal":true}],"tls_connection_policies":[{}],"automatic_https":{}}}}}
Feb 01 04:10:49 ingress caddy[25064]: {"level":"debug","ts":1706760649.6678927,"logger":"http","msg":"starting server loop","address":"[::]:80","tls":false,"http3":false}
Feb 01 04:10:49 ingress caddy[25064]: {"level":"info","ts":1706760649.6678998,"logger":"http.log","msg":"server running","name":"remaining_auto_https_redirects","protocols":["h1","h2","h3"]}
Feb 01 04:10:49 ingress caddy[25064]: {"level":"info","ts":1706760649.667909,"logger":"http","msg":"enabling HTTP/3 listener","addr":":443"}
Feb 01 04:10:49 ingress caddy[25064]: {"level":"info","ts":1706760649.6679323,"msg":"failed to sufficiently increase receive buffer size (was: 208 kiB, wanted: 2048 kiB, got: 416 kiB). See https://github.com/quic-go/quic-go/wiki/UDP-Buffer-Sizes for details."}
Feb 01 04:10:49 ingress caddy[25064]: {"level":"debug","ts":1706760649.667973,"logger":"http","msg":"starting server loop","address":"[::]:443","tls":true,"http3":true}
Feb 01 04:10:49 ingress caddy[25064]: {"level":"info","ts":1706760649.6679769,"logger":"http.log","msg":"server running","name":"srv0","protocols":["h1","h2","h3"]}
Feb 01 04:10:49 ingress caddy[25064]: {"level":"info","ts":1706760649.6679785,"logger":"http","msg":"enabling automatic TLS certificate management","domains":["*.johnmaguire.me","johnmaguire.me"]}
Feb 01 04:10:49 ingress caddy[25064]: {"level":"debug","ts":1706760649.6680858,"logger":"tls","msg":"loading managed certificate","domain":"*.johnmaguire.me","expiration":1714531684,"issuer_key":"acme-v02.api.letsencrypt.org-directory","storage":"FileStorage:/var/lib/caddy"}
Feb 01 04:10:49 ingress caddy[25064]: {"level":"debug","ts":1706760649.6682148,"logger":"tls.cache","msg":"added certificate to cache","subjects":["*.johnmaguire.me"],"expiration":1714531684,"managed":true,"issuer_key":"acme-v02.api.letsencrypt.org-directory","hash":"f33bc06d55d0f2f118a579a4ba19b4bdcd8f58fc57618f5fd7c9c8fd2d650186","cache_size":1,"cache_capacity":10000}
Feb 01 04:10:49 ingress caddy[25064]: {"level":"debug","ts":1706760649.6682231,"logger":"events","msg":"event","name":"cached_managed_cert","id":"5cf947ae-9353-4dd1-bf97-71ccd83ca69a","origin":"tls","data":{"sans":["*.johnmaguire.me"]}}
Feb 01 04:10:49 ingress caddy[25064]: {"level":"debug","ts":1706760649.6683135,"logger":"tls","msg":"loading managed certificate","domain":"johnmaguire.me","expiration":1714531683,"issuer_key":"acme-v02.api.letsencrypt.org-directory","storage":"FileStorage:/var/lib/caddy"}
Feb 01 04:10:49 ingress caddy[25064]: {"level":"debug","ts":1706760649.6683953,"logger":"tls.cache","msg":"added certificate to cache","subjects":["johnmaguire.me"],"expiration":1714531683,"managed":true,"issuer_key":"acme-v02.api.letsencrypt.org-directory","hash":"94d1884b8cc47393888b93065189455e11cc9f23a1db25c0276c82de66b6ac14","cache_size":2,"cache_capacity":10000}
Feb 01 04:10:49 ingress caddy[25064]: {"level":"debug","ts":1706760649.6684022,"logger":"events","msg":"event","name":"cached_managed_cert","id":"2bb5148d-5e18-4163-9c7b-10949a486b9c","origin":"tls","data":{"sans":["johnmaguire.me"]}}
Feb 01 04:10:49 ingress caddy[25064]: {"level":"error","ts":1706760649.6684153,"msg":"unable to autosave config","file":"/etc/caddy/autosave.json","error":"open /etc/caddy/autosave.json: read-only file system"}
Feb 01 04:10:49 ingress caddy[25064]: {"level":"info","ts":1706760649.66842,"msg":"serving initial configuration"}
Feb 01 04:10:49 ingress caddy[25064]: {"level":"warn","ts":1706760649.6709912,"logger":"tls","msg":"storage cleaning happened too recently; skipping for now","storage":"FileStorage:/var/lib/caddy","instance":"3d7f8c91-469d-4639-a66a-51967cee9563","try_again":1706847049.6709907,"try_again_in":86399.999999863}
Feb 01 04:10:49 ingress caddy[25064]: {"level":"info","ts":1706760649.6710343,"logger":"tls","msg":"finished cleaning storage units"}
Feb 01 04:11:12 ingress caddy[25064]: {"level":"debug","ts":1706760672.5990365,"logger":"events","msg":"event","name":"tls_get_certificate","id":"78261adf-e78d-41d7-b461-d4a635907043","origin":"tls","data":{"client_hello":{"CipherSuites":[4867,4866,4865,52393,52392,52394,49200,49196,49192,49188,49172,49162,159,107,57,65413,196,136,129,157,61,53,192,132,49199,49195,49191,49187,49171,49161,158,103,51,190,69,156,60,47,186,65,49169,49159,5,4,49170,49160,22,10,255],"ServerName":"johnmaguire.me","SupportedCurves":[29,23,24,25],"SupportedPoints":"AA==","SignatureSchemes":[2054,1537,1539,2053,1281,1283,2052,1025,1027,513,515],"SupportedProtos":["h2","http/1.1"],"SupportedVersions":[772,771,770,769],"RemoteAddr":{"IP":"192.168.123.7","Port":65404,"Zone":""},"LocalAddr":{"IP":"192.168.123.52","Port":443,"Zone":""}}}}
Feb 01 04:11:12 ingress caddy[25064]: {"level":"debug","ts":1706760672.5993311,"logger":"tls.handshake","msg":"choosing certificate","identifier":"johnmaguire.me","num_choices":1}
Feb 01 04:11:12 ingress caddy[25064]: {"level":"debug","ts":1706760672.5993595,"logger":"tls.handshake","msg":"default certificate selection results","identifier":"johnmaguire.me","subjects":["johnmaguire.me"],"managed":true,"issuer_key":"acme-v02.api.letsencrypt.org-directory","hash":"94d1884b8cc47393888b93065189455e11cc9f23a1db25c0276c82de66b6ac14"}
Feb 01 04:11:12 ingress caddy[25064]: {"level":"debug","ts":1706760672.5993743,"logger":"tls.handshake","msg":"matched certificate in cache","remote_ip":"192.168.123.7","remote_port":"65404","subjects":["johnmaguire.me"],"managed":true,"expiration":1714531683,"hash":"94d1884b8cc47393888b93065189455e11cc9f23a1db25c0276c82de66b6ac14"}
Feb 01 04:11:12 ingress caddy[25064]: {"level":"debug","ts":1706760672.6194866,"logger":"http.handlers.reverse_proxy","msg":"selected upstream","dial":"192.168.123.14:443","total_upstreams":1}
Feb 01 04:11:12 ingress caddy[25064]: {"level":"debug","ts":1706760672.6712964,"logger":"http.handlers.reverse_proxy","msg":"upstream roundtrip","upstream":"192.168.123.14:443","duration":0.051750528,"request":{"remote_ip":"192.168.123.7","remote_port":"65404","client_ip":"192.168.123.7","proto":"HTTP/2.0","method":"GET","host":"johnmaguire.me","uri":"/","headers":{"User-Agent":["curl/8.4.0"],"Accept":["*/*"],"X-Forwarded-For":["192.168.123.7"],"X-Forwarded-Proto":["https"],"X-Forwarded-Host":["johnmaguire.me"]},"tls":{"resumed":false,"version":772,"cipher_suite":4867,"proto":"h2","server_name":"johnmaguire.me"}},"error":"tls: failed to verify certificate: x509: cannot validate certificate for 192.168.123.14 because it doesn't contain any IP SANs"}
Feb 01 04:11:12 ingress caddy[25064]: {"level":"error","ts":1706760672.6714041,"logger":"http.log.error","msg":"tls: failed to verify certificate: x509: cannot validate certificate for 192.168.123.14 because it doesn't contain any IP SANs","request":{"remote_ip":"192.168.123.7","remote_port":"65404","client_ip":"192.168.123.7","proto":"HTTP/2.0","method":"GET","host":"johnmaguire.me","uri":"/","headers":{"User-Agent":["curl/8.4.0"],"Accept":["*/*"]},"tls":{"resumed":false,"version":772,"cipher_suite":4867,"proto":"h2","server_name":"johnmaguire.me"}},"duration":0.052061441,"status":502,"err_id":"4uz51jgd1","err_trace":"reverseproxy.statusError (reverseproxy.go:1267)"}
Feb 01 04:11:29 ingress caddy[25064]: {"level":"debug","ts":1706760689.0341105,"logger":"events","msg":"event","name":"tls_get_certificate","id":"77f66fa6-b296-48c9-97f3-214de0506106","origin":"tls","data":{"client_hello":{"CipherSuites":[10794,4865,4866,4867,49196,49195,52393,49200,49199,52392,49162,49161,49172,49171,157,156,53,47,49160,49170,10],"ServerName":"johnmaguire.me","SupportedCurves":[6682,29,23,24,25],"SupportedPoints":"AA==","SignatureSchemes":[1027,2052,1025,1283,515,2053,2053,1281,2054,1537,513],"SupportedProtos":["h2","http/1.1"],"SupportedVersions":[60138,772,771,770,769],"RemoteAddr":{"IP":"192.168.123.7","Port":65406,"Zone":""},"LocalAddr":{"IP":"192.168.123.52","Port":443,"Zone":""}}}}
Feb 01 04:11:29 ingress caddy[25064]: {"level":"debug","ts":1706760689.0342417,"logger":"tls.handshake","msg":"choosing certificate","identifier":"johnmaguire.me","num_choices":1}
Feb 01 04:11:29 ingress caddy[25064]: {"level":"debug","ts":1706760689.0342674,"logger":"tls.handshake","msg":"default certificate selection results","identifier":"johnmaguire.me","subjects":["johnmaguire.me"],"managed":true,"issuer_key":"acme-v02.api.letsencrypt.org-directory","hash":"94d1884b8cc47393888b93065189455e11cc9f23a1db25c0276c82de66b6ac14"}
Feb 01 04:11:29 ingress caddy[25064]: {"level":"debug","ts":1706760689.0342817,"logger":"tls.handshake","msg":"matched certificate in cache","remote_ip":"192.168.123.7","remote_port":"65406","subjects":["johnmaguire.me"],"managed":true,"expiration":1714531683,"hash":"94d1884b8cc47393888b93065189455e11cc9f23a1db25c0276c82de66b6ac14"}
Feb 01 04:11:29 ingress caddy[25064]: {"level":"debug","ts":1706760689.0806134,"logger":"http.handlers.reverse_proxy","msg":"selected upstream","dial":"192.168.123.14:443","total_upstreams":1}
Feb 01 04:11:29 ingress caddy[25064]: {"level":"debug","ts":1706760689.1128397,"logger":"http.handlers.reverse_proxy","msg":"upstream roundtrip","upstream":"192.168.123.14:443","duration":0.032069393,"request":{"remote_ip":"192.168.123.7","remote_port":"65406","client_ip":"192.168.123.7","proto":"HTTP/2.0","method":"GET","host":"johnmaguire.me","uri":"/","headers":{"Sec-Fetch-Mode":["navigate"],"User-Agent":["Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/17.2.1 Safari/605.1.15"],"Accept-Language":["en-US,en;q=0.9"],"Sec-Fetch-Dest":["document"],"Accept":["text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8"],"Upgrade-Insecure-Requests":["1"],"X-Forwarded-Host":["johnmaguire.me"],"Sec-Fetch-Site":["none"],"Accept-Encoding":["gzip, deflate"],"X-Forwarded-For":["192.168.123.7"],"X-Forwarded-Proto":["https"]},"tls":{"resumed":false,"version":772,"cipher_suite":4865,"proto":"h2","server_name":"johnmaguire.me"}},"error":"tls: failed to verify certificate: x509: cannot validate certificate for 192.168.123.14 because it doesn't contain any IP SANs"}
Feb 01 04:11:29 ingress caddy[25064]: {"level":"error","ts":1706760689.1129675,"logger":"http.log.error","msg":"tls: failed to verify certificate: x509: cannot validate certificate for 192.168.123.14 because it doesn't contain any IP SANs","request":{"remote_ip":"192.168.123.7","remote_port":"65406","client_ip":"192.168.123.7","proto":"HTTP/2.0","method":"GET","host":"johnmaguire.me","uri":"/","headers":{"Sec-Fetch-Mode":["navigate"],"User-Agent":["Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/17.2.1 Safari/605.1.15"],"Accept-Language":["en-US,en;q=0.9"],"Sec-Fetch-Dest":["document"],"Accept":["text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8"],"Sec-Fetch-Site":["none"],"Accept-Encoding":["gzip, deflate"],"Upgrade-Insecure-Requests":["1"]},"tls":{"resumed":false,"version":772,"cipher_suite":4865,"proto":"h2","server_name":"johnmaguire.me"}},"duration":0.032383901,"status":502,"err_id":"pncf5q9ym","err_trace":"reverseproxy.statusError (reverseproxy.go:1267)"}

3. Caddy version:

v2.7.6 h1:w0NymbG2m9PcvKWsrXO6EEkY9Ru4FJK8uQbYcev1p3A=

4. How I installed and ran Caddy:

yay -S caddy-cloudflare

AUR (en) - caddy-cloudflare contains the Cloudflare DNS module.

a. System environment:

Arch Linux, systemd, caddy-cloudflare 2.7.6-1

b. Command:

systemctl start caddy

c. Service/unit/compose file:

[Unit]
Description=Caddy webserver
Documentation=https://caddyserver.com/docs/
After=network-online.target
Wants=network-online.target systemd-networkd-wait-online.service
StartLimitIntervalSec=14400
StartLimitBurst=10

[Service]
User=caddy
Group=caddy

# environment: store secrets here such as API tokens
EnvironmentFile=-/var/lib/caddy/envfile
# data directory: uses $XDG_DATA_HOME/caddy
# TLS certificates and other assets are stored here
Environment=XDG_DATA_HOME=/var/lib
# config directory: uses $XDG_CONFIG_HOME/caddy
Environment=XDG_CONFIG_HOME=/etc

# do not print --environ here, as it may contain API tokens!!
ExecStart=/usr/bin/caddy run --config /etc/caddy/Caddyfile
ExecReload=/usr/bin/caddy reload --config /etc/caddy/Caddyfile

# Do not allow the process to be restarted in a tight loop.
Restart=on-abnormal

# Use graceful shutdown with a reasonable timeout
KillMode=mixed
KillSignal=SIGQUIT
TimeoutStopSec=5s

# Sufficient resource limits
LimitNOFILE=1048576
LimitNPROC=512

# Grants binding to port 443...
AmbientCapabilities=CAP_NET_BIND_SERVICE
# ...and limits potentially inherited capabilities to this
CapabilityBoundingSet=CAP_NET_BIND_SERVICE

# Hardening options
LockPersonality=true
NoNewPrivileges=true

PrivateTmp=true
PrivateDevices=true

ProtectControlGroups=true
ProtectHome=true
ProtectKernelTunables=true
ProtectKernelModules=true
ProtectSystem=strict

ReadWritePaths=/var/lib/caddy
ReadOnlyPaths=/etc/caddy
ReadOnlyPaths=-/var/lib/caddy/envfile

[Install]
WantedBy=multi-user.target

d. My complete Caddy config:

{
    debug
}

*.johnmaguire.me, johnmaguire.me {
    tls {

            dns cloudflare (redacted)

    }

    reverse_proxy https://192.168.123.14
}

5. Links to relevant resources:

Yeah, Caddy will use the upstream address for TLS-SNI unless otherwise configured. Use the tls_server_name transport option to override it.

It’s not the Host header that’s used, it’s TLS-SNI: Server Name Indication - Wikipedia

That’s because the Host header is part of the HTTP payload which is encrypted, so it’s not known until after the TLS handshake is completed. So TLS-SNI is used to announce the target.

1 Like