1. The problem I’m having:
I would like to have the following setup with two servers running Caddy:
client --HTTP--> Caddy1 --HTTPS--> Caddy2 --HTTP--> upstream
The Caddy1 and Caddy2 instances runs on two different servers, but Caddy2 and upstream runs on the same server.
I would like the connection from Caddy1 to Caddy2 to be encrypted.
However, when testing this with both Caddy1, Caddy2 and upstream running on the same computer, I can’t get my request to Caddy1 to reach upstream.
Here is the config for Caddy1 and Caddy2:
# Caddyfile for Caddy1
{
debug
}
http://s1.localhost:8002 {
reverse_proxy :8001 {
# Adding this doesn't solve the problem either:
# header_up Host {upstream_hostport}
transport http {
tls_server_name s2.localhost
}
}
}
# Caddyfile for Caddy2
{
debug
ocsp_stapling off
http_port 32132 # Prevent binding to port 80
}
https://s2.localhost:8001 {
reverse_proxy :8000
}
The upstream is just running python -mhttp.server in this test.
Sending requests directly to the upstream works:
$ curl -vI http://localhost:8000/
* Host localhost:8000 was resolved.
* IPv6: ::1
* IPv4: 127.0.0.1
* Trying 127.0.0.1:8000...
* Connected to localhost (127.0.0.1) port 8000
* using HTTP/1.x
> HEAD / HTTP/1.1
> Host: localhost:8000
> User-Agent: curl/8.11.1
> Accept: */*
>
* Request completely sent off
* HTTP 1.0, assume close after body
< HTTP/1.0 200 OK
HTTP/1.0 200 OK
< Server: SimpleHTTP/0.6 Python/3.13.1
Server: SimpleHTTP/0.6 Python/3.13.1
< Date: Thu, 16 Jan 2025 15:32:38 GMT
Date: Thu, 16 Jan 2025 15:32:38 GMT
< Content-type: text/html; charset=utf-8
Content-type: text/html; charset=utf-8
< Content-Length: 10733
Content-Length: 10733
<
* shutting down connection #0
Sending request to Caddy2 also works:
$ curl -vI https://s2.localhost:8001/
* Host s2.localhost:8001 was resolved.
* IPv6: ::1
* IPv4: 127.0.0.1
* Trying 127.0.0.1:8001...
* ALPN: curl offers h2,http/1.1
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* CAfile: /etc/ssl/certs/ca-certificates.crt
* CApath: none
* TLSv1.3 (IN), TLS handshake, Server hello (2):
* TLSv1.3 (IN), TLS change cipher, Change cipher spec (1):
* TLSv1.3 (IN), TLS handshake, Encrypted Extensions (8):
* TLSv1.3 (IN), TLS handshake, Certificate (11):
* TLSv1.3 (IN), TLS handshake, CERT verify (15):
* TLSv1.3 (IN), TLS handshake, Finished (20):
* TLSv1.3 (OUT), TLS change cipher, Change cipher spec (1):
* TLSv1.3 (OUT), TLS handshake, Finished (20):
* SSL connection using TLSv1.3 / TLS_AES_128_GCM_SHA256 / x25519 / id-ecPublicKey
* ALPN: server accepted h2
* Server certificate:
* subject:
* start date: Jan 16 15:20:16 2025 GMT
* expire date: Jan 17 03:20:16 2025 GMT
* subjectAltName: host "s2.localhost" matched cert's "s2.localhost"
* issuer: CN=Caddy Local Authority - ECC Intermediate
* SSL certificate verify ok.
* Certificate level 0: Public key type EC/prime256v1 (256/128 Bits/secBits), signed using ecdsa-with-SHA256
* Certificate level 1: Public key type EC/prime256v1 (256/128 Bits/secBits), signed using ecdsa-with-SHA256
* Certificate level 2: Public key type EC/prime256v1 (256/128 Bits/secBits), signed using ecdsa-with-SHA256
* Connected to s2.localhost (127.0.0.1) port 8001
* using HTTP/2
* [HTTP/2] [1] OPENED stream for https://s2.localhost:8001/
* [HTTP/2] [1] [:method: HEAD]
* [HTTP/2] [1] [:scheme: https]
* [HTTP/2] [1] [:authority: s2.localhost:8001]
* [HTTP/2] [1] [:path: /]
* [HTTP/2] [1] [user-agent: curl/8.11.1]
* [HTTP/2] [1] [accept: */*]
> HEAD / HTTP/2
> Host: s2.localhost:8001
> User-Agent: curl/8.11.1
> Accept: */*
>
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
* Request completely sent off
< HTTP/2 200
HTTP/2 200
< alt-svc: h3=":8001"; ma=2592000
alt-svc: h3=":8001"; ma=2592000
< content-type: text/html; charset=utf-8
content-type: text/html; charset=utf-8
< date: Thu, 16 Jan 2025 15:33:37 GMT
date: Thu, 16 Jan 2025 15:33:37 GMT
< server: Caddy
server: Caddy
< server: SimpleHTTP/0.6 Python/3.13.1
server: SimpleHTTP/0.6 Python/3.13.1
< content-length: 10733
content-length: 10733
<
* Connection #0 to host s2.localhost left intact
However sending a request to Caddy2 just produces a 200 OK response from Caddy1 without proxying it to the upstream:
$ curl -vI http://s1.localhost:8002/
* Host s1.localhost:8002 was resolved.
* IPv6: ::1
* IPv4: 127.0.0.1
* Trying 127.0.0.1:8002...
* Connected to s1.localhost (127.0.0.1) port 8002
* using HTTP/1.x
> HEAD / HTTP/1.1
> Host: s1.localhost:8002
> User-Agent: curl/8.11.1
> Accept: */*
>
* Request completely sent off
< HTTP/1.1 200 OK
HTTP/1.1 200 OK
< Date: Thu, 16 Jan 2025 15:34:53 GMT
Date: Thu, 16 Jan 2025 15:34:53 GMT
< Server: Caddy
Server: Caddy
< Server: Caddy
Server: Caddy
<
* Connection #0 to host s1.localhost left intact
2. Error messages and/or full log output:
Logs when sending one request to Caddy1 with curl -vI http://s1.localhost:8002/:
2025/01/16 15:36:27.992 INFO using adjacent Caddyfile
2025/01/16 15:36:27.994 INFO adapted config to JSON {"adapter": "caddyfile"}
2025/01/16 15:36:27.996 INFO admin admin endpoint started {"address": "localhost:2019", "enforce_origin": false, "origins": ["//[::1]:2019", "//127.0.0.1:2019", "//localhost:2019"]}
2025/01/16 15:36:27.996 DEBUG http.auto_https adjusted config {"tls": {"automation":{"policies":[{"disable_ocsp_stapling":true}]},"disable_ocsp_stapling":true}, "http": {"http_port":32133,"servers":{"srv0":{"listen":[":8002"],"routes":[{"handle":[{"handler":"subroute","routes":[{"handle":[{"handler":"reverse_proxy","transport":{"protocol":"http","tls":{"server_name":"s2.localhost"}},"upstreams":[{"dial":":8001"}]}]}]}],"terminal":true}],"automatic_https":{"skip":["s1.localhost"]}}}}}
2025/01/16 15:36:27.997 INFO tls.cache.maintenance started background certificate maintenance {"cache": "0xc000646700"}
2025/01/16 15:36:27.998 INFO tls storage cleaning happened too recently; skipping for now {"storage": "FileStorage:/home/tyilo/.local/share/caddy", "instance": "12e300f0-f82d-4a93-9f5d-fcc4bbec0be1", "try_again": "2025/01/17 15:36:27.998", "try_again_in": 86399.999999774}
2025/01/16 15:36:27.998 INFO tls finished cleaning storage units
2025/01/16 15:36:28.001 DEBUG http starting server loop {"address": "0.0.0.0:8002", "tls": false, "http3": false}
2025/01/16 15:36:28.001 INFO http.log server running {"name": "srv0", "protocols": ["h1", "h2", "h3"]}
2025/01/16 15:36:28.001 INFO autosaved config (load with --resume flag) {"file": "/home/tyilo/.local/share/caddy/autosave.json"}
2025/01/16 15:36:28.001 INFO serving initial configuration
2025/01/16 15:36:33.902 DEBUG http.handlers.reverse_proxy selected upstream {"dial": ":8001", "total_upstreams": 1}
2025/01/16 15:36:33.907 DEBUG http.handlers.reverse_proxy upstream roundtrip {"upstream": ":8001", "duration": 0.004832662, "request": {"remote_ip": "127.0.0.1", "remote_port": "58656", "client_ip": "127.0.0.1", "proto": "HTTP/1.1", "method": "HEAD", "host": "s1.localhost:8002", "uri": "/", "headers": {"X-Forwarded-Host": ["s1.localhost:8002"], "User-Agent": ["curl/8.11.1"], "Accept": ["*/*"], "X-Forwarded-For": ["127.0.0.1"], "X-Forwarded-Proto": ["http"]}}, "headers": {"Alt-Svc": ["h3=\":8001\"; ma=2592000"], "Server": ["Caddy"], "Date": ["Thu, 16 Jan 2025 15:36:33 GMT"]}, "status": 200}
Logs from Caddy2:
2025/01/16 15:36:26.395 INFO using adjacent Caddyfile
2025/01/16 15:36:26.396 INFO adapted config to JSON {"adapter": "caddyfile"}
2025/01/16 15:36:26.399 INFO admin admin endpoint started {"address": "localhost:2019", "enforce_origin": false, "origins": ["//localhost:2019", "//[::1]:2019", "//127.0.0.1:2019"]}
2025/01/16 15:36:26.399 INFO http.auto_https enabling automatic HTTP->HTTPS redirects {"server_name": "srv0"}
2025/01/16 15:36:26.399 INFO tls.cache.maintenance started background certificate maintenance {"cache": "0xc000119e00"}
2025/01/16 15:36:26.400 DEBUG http.auto_https adjusted config {"tls": {"automation":{"policies":[{"subjects":["s2.localhost"],"disable_ocsp_stapling":true},{"disable_ocsp_stapling":true}]},"disable_ocsp_stapling":true}, "http": {"http_port":32132,"servers":{"remaining_auto_https_redirects":{"listen":[":32132"],"routes":[{},{}]},"srv0":{"listen":[":8001"],"routes":[{"handle":[{"handler":"subroute","routes":[{"handle":[{"handler":"reverse_proxy","upstreams":[{"dial":":8000"}]}]}]}],"terminal":true}],"tls_connection_policies":[{}],"automatic_https":{}}}}}
2025/01/16 15:36:26.400 INFO pki.ca.local root certificate is already trusted by system {"path": "storage:pki/authorities/local/root.crt"}
2025/01/16 15:36:26.401 INFO tls storage cleaning happened too recently; skipping for now {"storage": "FileStorage:/home/tyilo/.local/share/caddy", "instance": "12e300f0-f82d-4a93-9f5d-fcc4bbec0be1", "try_again": "2025/01/17 15:36:26.401", "try_again_in": 86399.999999611}
2025/01/16 15:36:26.401 INFO tls finished cleaning storage units
2025/01/16 15:36:26.404 INFO http enabling HTTP/3 listener {"addr": ":8001"}
2025/01/16 15:36:26.404 INFO failed to sufficiently increase receive buffer size (was: 208 kiB, wanted: 7168 kiB, got: 416 kiB). See https://github.com/quic-go/quic-go/wiki/UDP-Buffer-Sizes for details.
2025/01/16 15:36:26.404 DEBUG http starting server loop {"address": "0.0.0.0:8001", "tls": true, "http3": true}
2025/01/16 15:36:26.404 INFO http.log server running {"name": "srv0", "protocols": ["h1", "h2", "h3"]}
2025/01/16 15:36:26.404 DEBUG http starting server loop {"address": "0.0.0.0:32132", "tls": false, "http3": false}
2025/01/16 15:36:26.404 INFO http.log server running {"name": "remaining_auto_https_redirects", "protocols": ["h1", "h2", "h3"]}
2025/01/16 15:36:26.404 INFO http enabling automatic TLS certificate management {"domains": ["s2.localhost"]}
2025/01/16 15:36:26.404 DEBUG tls.cache added certificate to cache {"subjects": ["s2.localhost"], "expiration": "2025/01/17 03:20:17.000", "managed": true, "issuer_key": "local", "hash": "55a6f235b9eabcc77fbca1043d3c30fe8ac54c55c4fab1374ab669e021226515", "cache_size": 1, "cache_capacity": 10000}
2025/01/16 15:36:26.404 DEBUG events event {"name": "cached_managed_cert", "id": "3d5a91cb-ac0a-45d0-813b-f7340827ba36", "origin": "tls", "data": {"sans":["s2.localhost"]}}
2025/01/16 15:36:26.404 INFO autosaved config (load with --resume flag) {"file": "/home/tyilo/.local/share/caddy/autosave.json"}
2025/01/16 15:36:26.404 INFO serving initial configuration
2025/01/16 15:36:33.904 DEBUG events event {"name": "tls_get_certificate", "id": "1c6731e1-5686-469d-b5c4-8babb4cce7ee", "origin": "tls", "data": {"client_hello":{"CipherSuites":[49195,49199,49196,49200,52393,52392,49161,49171,49162,49172,156,157,47,53,49170,10,4865,4866,4867],"ServerName":"s2.localhost","SupportedCurves":[29,23,24,25],"SupportedPoints":"AA==","SignatureSchemes":[2052,1027,2055,2053,2054,1025,1281,1537,1283,1539,513,515],"SupportedProtos":["h2","http/1.1"],"SupportedVersions":[772,771],"RemoteAddr":{"IP":"127.0.0.1","Port":58858,"Zone":""},"LocalAddr":{"IP":"127.0.0.1","Port":8001,"Zone":""}}}}
2025/01/16 15:36:33.904 DEBUG tls.handshake choosing certificate {"identifier": "s2.localhost", "num_choices": 1}
2025/01/16 15:36:33.904 DEBUG tls.handshake default certificate selection results {"identifier": "s2.localhost", "subjects": ["s2.localhost"], "managed": true, "issuer_key": "local", "hash": "55a6f235b9eabcc77fbca1043d3c30fe8ac54c55c4fab1374ab669e021226515"}
2025/01/16 15:36:33.904 DEBUG tls.handshake matched certificate in cache {"remote_ip": "127.0.0.1", "remote_port": "58858", "subjects": ["s2.localhost"], "managed": true, "expiration": "2025/01/17 03:20:17.000", "hash": "55a6f235b9eabcc77fbca1043d3c30fe8ac54c55c4fab1374ab669e021226515"}
3. Caddy version:
I used v2.8.4 so the logs would reflect that.
Using v2.9.1 downloaded from the GitHub releases page shows the same behavior.
4. How I installed and ran Caddy:
a. System environment:
Arch Linux, installed with pacman
b. Command:
For both Caddy1 and Caddy2 (in two different directories):
caddy run
d. My complete Caddy config:
See above