1. The problem I’m having:
Hi, I want to set up a reverse proxy between two Caddy instances.
Server 1: listens on public IP
Server 2: listens on a local IP
What I would like to have is HTTP/3 support for the transport between the Caddy servers, no matter if the client supports it or not.
If I omit the transport http {} Caddy defaults to HTTP/2 to the upstream.
An alternative solution would be if it was possible to proxy http/https traffic from server 1 to server 2. But as far as I can see, Caddy connects to upstream using the hostname from reverse_proxy hostname instead of using ALPN. Would proxy_protocol be a viable option?
2. Error messages and/or full log output:
Server 1: logfile
{"level":"debug","ts":1713619010.2977695,"logger":"events","msg":"event","name":"tls_get_certificate","id":"523ec077-9675-4b0d-8003-355692757a03","origin":"tls","data":{"client_hello":{"CipherSuites":[4866,4867,4865,49196,49200,159,52393,52392,52394,49195,49199,158,49188,49192,107,49187,49191,103,49162,49172,57,49161,49171,51,157,156,61,60,53,47,255],"ServerName":"gist.tnonline.net","SupportedCurves":[29,23,30,25,24,256,257,258,259,260],"SupportedPoints":"AAEC","SignatureSchemes":[1027,1283,1539,2055,2056,2057,2058,2059,2052,2053,2054,1025,1281,1537,771,769,770,1026,1282,1538],"SupportedProtos":["h2","http/1.1"],"SupportedVersions":[772,771],"RemoteAddr":{"IP":"158.174.155.174","Port":34338,"Zone":""},"LocalAddr":{"IP":"37.27.86.91","Port":443,"Zone":""}}}}
{"level":"debug","ts":1713619010.2979167,"logger":"tls.handshake","msg":"choosing certificate","identifier":"gist.tnonline.net","num_choices":1}
{"level":"debug","ts":1713619010.297944,"logger":"tls.handshake","msg":"custom certificate selection results","identifier":"gist.tnonline.net","subjects":["gist.tnonline.net"],"managed":false,"issuer_key":"","hash":"632dc9ecc2baee7266c4e7b8d4d3383a6ff2ea56f5d95e48c0fd7f8c8d70935d"}
{"level":"debug","ts":1713619010.2979589,"logger":"tls.handshake","msg":"matched certificate in cache","remote_ip":"158.174.155.174","remote_port":"34338","subjects":["gist.tnonline.net"],"managed":false,"expiration":1718849531,"hash":"632dc9ecc2baee7266c4e7b8d4d3383a6ff2ea56f5d95e48c0fd7f8c8d70935d"}
{"level":"debug","ts":1713619010.315176,"logger":"http.handlers.reverse_proxy","msg":"selected upstream","dial":"gist.tnonline.net:443","total_upstreams":1}
{"level":"debug","ts":1713619010.3310494,"logger":"http.handlers.reverse_proxy","msg":"upstream roundtrip","upstream":"gist.tnonline.net:443","duration":0.015765159,"request":{"remote_ip":"158.174.155.174","remote_port":"34338","client_ip":"158.174.155.174","proto":"HTTP/2.0","method":"GET","host":"gist.tnonline.net:443","uri":"/","headers":{"X-Forwarded-Host":["gist.tnonline.net"],"User-Agent":["curl/7.81.0"],"Accept":["*/*"],"X-Forwarded-For":["158.174.155.174"],"X-Forwarded-Proto":["https"]},"tls":{"resumed":false,"version":772,"cipher_suite":4865,"proto":"h2","server_name":"gist.tnonline.net"}},"headers":{"Date":["Sat, 20 Apr 2024 13:16:50 GMT"],"Server":["Caddy","TornadoServer/6.3.2"],"Set-Cookie":[],"Etag":["\"185a56e15dbf656f932996bc30e94ea14e7cb2ba\""],"Vary":["Accept-Encoding"],"Alt-Svc":["h3=\":443\"; ma=2592000"],"Content-Type":["text/html; charset=UTF-8"]},"status":200}
Server 2: logfile
{"level":"info","ts":1713619010.3248546,"logger":"http.log.access.log8","msg":"handled request","request":{"remote_ip":"10.5.1.1","remote_port":"48066","client_ip":"10.5.1.1","proto":"HTTP/1.1","method":"GET","host":"gist.tnonline.net:443","uri":"/","headers":{"X-Forwarded-Proto":["https"],"Accept-Encoding":["gzip"],"User-Agent":["curl/7.81.0"],"Accept":["*/*"],"X-Forwarded-For":["158.174.155.174"],"X-Forwarded-Host":["gist.tnonline.net"]},"tls":{"resumed":false,"version":772,"cipher_suite":4865,"proto":"","server_name":"gist.tnonline.net"}},"bytes_read":0,"user_id":"","duration":0.004571557,"size":6968,"status":200,"resp_headers":{"Content-Type":["text/html; charset=UTF-8"],"Date":["Sat, 20 Apr 2024 13:16:50 GMT"],"Etag":["\"185a56e15dbf656f932996bc30e94ea14e7cb2ba\""],"Content-Encoding":["gzip"],"Set-Cookie":[],"Vary":["Accept-Encoding"],"Server":["Caddy","TornadoServer/6.3.2"],"Alt-Svc":["h3=\":443\"; ma=2592000"]}}
3. Caddy version:
Server 1: v2.7.6-r2 (Alpine Linux package)
Server 2: v2.7.6 => /usr/src/caddy/git/caddy@(devel) (compiled from git sources)
4. How I installed and ran Caddy:
a. System environment:
Server 1: Alpine Linux aarch64
Server 2: Gentoo Linux x86_64
b. Command:
caddy run --config Caddyfile
d. My complete Caddy config:
Server 1:
{
debug
auto_https off
log {
output file /var/log/caddy/caddy_main.log {
roll_disabled
}
format json
}
servers {
metrics
}
}
gist.tnonline.net:443 {
tls /etc/caddy/certs/gist.tnonline.net_cert.pem /etc/caddy/certs/gist.tnonline.net_privkey.pem
log {
output file /var/log/caddy/gist.tnonline.net.log {
roll_disabled
}
format json
}
reverse_proxy https://gist.tnonline.net:443 {
header_up Host {upstream_hostport}
transport http {
tls
versions h3
}
}
}
Server 2:
{
# debug
auto_https off
log {
output file /var/log/caddy/caddy_main.log {
roll_disabled
}
format json
}
servers {
#enable_full_duplex
metrics
## Cloudflare IPs
trusted_proxies cloudflare
}
}
## Snippets
(main) {
tls /etc/letsencrypt/live/{args[0]}/fullchain.pem /etc/letsencrypt/live/{args[0]}/privkey.pem {
curves x25519 secp521r1 secp384r1 secp256r1
}
log {
output file /var/log/caddy/{args[0]}_443.log {
roll_disabled
}
format json
}
encode br zstd gzip
}
gist.tnonline.net:443 {
import main gist.tnonline.net
@php {
not path /public/files/* /rain/* /install/* /includes/*
}
root * /var/www/domains/gist.tnonline.net/
reverse_proxy 192.168.0.1:9000 {
trusted_proxies private_ranges
}
uri replace /files/ /public/files/
php_fastcgi @php unix//var/run/php-fpm/fpm-www.socket {
#max_requests 10
}
file_server
}
## <Other domain blocks were removed>
5. Additional information
I have attempted to force udp+tls for the upstream, but without success:
reverse_proxy udp/gist.tnonline.net:443 {
header_up Host {upstream_hostport}
transport http {
tls
}
}
Caddy still uses h2 for the upstream:
{"level":"debug","ts":1713618527.673246,"logger":"http.handlers.reverse_proxy","msg":"upstream roundtrip","upstream":"udp/gist.tnonline.net:443","duration":0.01389355,"request":{"remote_ip":"158.174.155.174","remote_port":"45024","client_ip":"158.174.155.174","proto":"HTTP/2.0","method":"GET","host":"gist.tnonline.net:443","uri":"/","headers":{"X-Forwarded-Host":["gist.tnonline.net"],"User-Agent":["curl/7.81.0"],"Accept":["*/*"],"X-Forwarded-For":["158.174.155.174"],"X-Forwarded-Proto":["https"]},"tls":{"resumed":false,"version":772,"cipher_suite":4865,"proto":"h2","server_name":"gist.tnonline.net"}},"headers":{"Content-Type":["text/html; charset=UTF-8"],"Date":["Sat, 20 Apr 2024 13:08:47 GMT"],"Vary":["Accept-Encoding"],"Alt-Svc":["h3=\":443\"; ma=2592000"],"Etag":["\"63d9eeb3ee1ef16d386b5b9da5d4ed1934a76081\""],"Server":["Caddy","TornadoServer/6.3.2"],"Set-Cookie":[]},"status":200}
The Caddy server upstream does support HTTP/3. This can be tested using
# curl -vL --http3 -4 --resolve 'gist.tnonline.net:443:10.5.2.1' https://gist.tnonline.net
* Added gist.tnonline.net:443:10.5.2.1 to DNS cache
* Hostname gist.tnonline.net was found in DNS cache
* Trying 10.5.2.1:443...
* found 146 certificates in /etc/ssl/certs/ca-certificates.crt
* GnuTLS ciphers: NORMAL:-ARCFOUR-128:-CTYPE-ALL:+CTYPE-X509:-VERS-SSL3.0
* SSL connection using TLS1.3 / ECDHE_RSA_AES_128_GCM_SHA256
* server certificate verification OK
* server certificate status verification SKIPPED
* common name: gist.tnonline.net (matched)
* server certificate expiration date OK
* server certificate activation date OK
* certificate public key: EC/ECDSA
* certificate version: #3
* subject: CN=gist.tnonline.net
* start date: Fri, 22 Mar 2024 02:12:11 GMT
* expire date: Thu, 20 Jun 2024 02:12:10 GMT
* issuer: C=US,O=Let's Encrypt,CN=E1
* Verified certificate just fine
* Connected to gist.tnonline.net (10.5.2.1) port 443
* using HTTP/3
* [HTTP/3] [0] OPENED stream for https://gist.tnonline.net/
* [HTTP/3] [0] [:method: GET]
* [HTTP/3] [0] [:scheme: https]
* [HTTP/3] [0] [:authority: gist.tnonline.net]
* [HTTP/3] [0] [:path: /]
* [HTTP/3] [0] [user-agent: curl/8.7.1]
* [HTTP/3] [0] [accept: */*]
> GET / HTTP/3
> Host: gist.tnonline.net
> User-Agent: curl/8.7.1
> Accept: */*
...