Http/3 upstream with reverse_proxy

1. The problem I’m having:

Hi, I want to set up a reverse proxy between two Caddy instances.

Server 1: listens on public IP
Server 2: listens on a local IP

What I would like to have is HTTP/3 support for the transport between the Caddy servers, no matter if the client supports it or not.

If I omit the transport http {} Caddy defaults to HTTP/2 to the upstream.

An alternative solution would be if it was possible to proxy http/https traffic from server 1 to server 2. But as far as I can see, Caddy connects to upstream using the hostname from reverse_proxy hostname instead of using ALPN. Would proxy_protocol be a viable option?

2. Error messages and/or full log output:

Server 1: logfile

{"level":"debug","ts":1713619010.2977695,"logger":"events","msg":"event","name":"tls_get_certificate","id":"523ec077-9675-4b0d-8003-355692757a03","origin":"tls","data":{"client_hello":{"CipherSuites":[4866,4867,4865,49196,49200,159,52393,52392,52394,49195,49199,158,49188,49192,107,49187,49191,103,49162,49172,57,49161,49171,51,157,156,61,60,53,47,255],"ServerName":"gist.tnonline.net","SupportedCurves":[29,23,30,25,24,256,257,258,259,260],"SupportedPoints":"AAEC","SignatureSchemes":[1027,1283,1539,2055,2056,2057,2058,2059,2052,2053,2054,1025,1281,1537,771,769,770,1026,1282,1538],"SupportedProtos":["h2","http/1.1"],"SupportedVersions":[772,771],"RemoteAddr":{"IP":"158.174.155.174","Port":34338,"Zone":""},"LocalAddr":{"IP":"37.27.86.91","Port":443,"Zone":""}}}}
{"level":"debug","ts":1713619010.2979167,"logger":"tls.handshake","msg":"choosing certificate","identifier":"gist.tnonline.net","num_choices":1}
{"level":"debug","ts":1713619010.297944,"logger":"tls.handshake","msg":"custom certificate selection results","identifier":"gist.tnonline.net","subjects":["gist.tnonline.net"],"managed":false,"issuer_key":"","hash":"632dc9ecc2baee7266c4e7b8d4d3383a6ff2ea56f5d95e48c0fd7f8c8d70935d"}
{"level":"debug","ts":1713619010.2979589,"logger":"tls.handshake","msg":"matched certificate in cache","remote_ip":"158.174.155.174","remote_port":"34338","subjects":["gist.tnonline.net"],"managed":false,"expiration":1718849531,"hash":"632dc9ecc2baee7266c4e7b8d4d3383a6ff2ea56f5d95e48c0fd7f8c8d70935d"}
{"level":"debug","ts":1713619010.315176,"logger":"http.handlers.reverse_proxy","msg":"selected upstream","dial":"gist.tnonline.net:443","total_upstreams":1}
{"level":"debug","ts":1713619010.3310494,"logger":"http.handlers.reverse_proxy","msg":"upstream roundtrip","upstream":"gist.tnonline.net:443","duration":0.015765159,"request":{"remote_ip":"158.174.155.174","remote_port":"34338","client_ip":"158.174.155.174","proto":"HTTP/2.0","method":"GET","host":"gist.tnonline.net:443","uri":"/","headers":{"X-Forwarded-Host":["gist.tnonline.net"],"User-Agent":["curl/7.81.0"],"Accept":["*/*"],"X-Forwarded-For":["158.174.155.174"],"X-Forwarded-Proto":["https"]},"tls":{"resumed":false,"version":772,"cipher_suite":4865,"proto":"h2","server_name":"gist.tnonline.net"}},"headers":{"Date":["Sat, 20 Apr 2024 13:16:50 GMT"],"Server":["Caddy","TornadoServer/6.3.2"],"Set-Cookie":[],"Etag":["\"185a56e15dbf656f932996bc30e94ea14e7cb2ba\""],"Vary":["Accept-Encoding"],"Alt-Svc":["h3=\":443\"; ma=2592000"],"Content-Type":["text/html; charset=UTF-8"]},"status":200}

Server 2: logfile

{"level":"info","ts":1713619010.3248546,"logger":"http.log.access.log8","msg":"handled request","request":{"remote_ip":"10.5.1.1","remote_port":"48066","client_ip":"10.5.1.1","proto":"HTTP/1.1","method":"GET","host":"gist.tnonline.net:443","uri":"/","headers":{"X-Forwarded-Proto":["https"],"Accept-Encoding":["gzip"],"User-Agent":["curl/7.81.0"],"Accept":["*/*"],"X-Forwarded-For":["158.174.155.174"],"X-Forwarded-Host":["gist.tnonline.net"]},"tls":{"resumed":false,"version":772,"cipher_suite":4865,"proto":"","server_name":"gist.tnonline.net"}},"bytes_read":0,"user_id":"","duration":0.004571557,"size":6968,"status":200,"resp_headers":{"Content-Type":["text/html; charset=UTF-8"],"Date":["Sat, 20 Apr 2024 13:16:50 GMT"],"Etag":["\"185a56e15dbf656f932996bc30e94ea14e7cb2ba\""],"Content-Encoding":["gzip"],"Set-Cookie":[],"Vary":["Accept-Encoding"],"Server":["Caddy","TornadoServer/6.3.2"],"Alt-Svc":["h3=\":443\"; ma=2592000"]}}

3. Caddy version:

Server 1: v2.7.6-r2 (Alpine Linux package)
Server 2: v2.7.6 => /usr/src/caddy/git/caddy@(devel) (compiled from git sources)

4. How I installed and ran Caddy:

a. System environment:

Server 1: Alpine Linux aarch64
Server 2: Gentoo Linux x86_64

b. Command:

caddy run --config Caddyfile

d. My complete Caddy config:

Server 1:

{
	debug
	auto_https off
	log {
		output file /var/log/caddy/caddy_main.log {
			roll_disabled
		}
		format json
	}
	servers {
		metrics
	}
}

gist.tnonline.net:443 {
	tls /etc/caddy/certs/gist.tnonline.net_cert.pem /etc/caddy/certs/gist.tnonline.net_privkey.pem
	log {
		output file /var/log/caddy/gist.tnonline.net.log {
			roll_disabled
		}
		format json
	}
	reverse_proxy https://gist.tnonline.net:443 {
		header_up Host {upstream_hostport}
		transport http {
			tls
			versions h3
		}
	}
}

Server 2:


{
	# debug
	auto_https off
	log {
		output file /var/log/caddy/caddy_main.log {
			roll_disabled
		}
		format json
	}
	servers {
		#enable_full_duplex
		metrics

		## Cloudflare IPs
		trusted_proxies cloudflare
	}
}

## Snippets

(main) {
	tls /etc/letsencrypt/live/{args[0]}/fullchain.pem /etc/letsencrypt/live/{args[0]}/privkey.pem {
		curves x25519 secp521r1 secp384r1 secp256r1
	}
	log {
		output file /var/log/caddy/{args[0]}_443.log {
			roll_disabled
		}
		format json
	}
	encode br zstd gzip
}

gist.tnonline.net:443 {
	import main gist.tnonline.net
	@php {
		not path /public/files/* /rain/* /install/* /includes/*
	}
	root * /var/www/domains/gist.tnonline.net/

	reverse_proxy 192.168.0.1:9000 {
		trusted_proxies private_ranges
	}

	uri replace /files/ /public/files/
	php_fastcgi @php unix//var/run/php-fpm/fpm-www.socket {
		#max_requests 10
	}
	file_server
}

## <Other domain blocks were removed>

5. Additional information

I have attempted to force udp+tls for the upstream, but without success:

	reverse_proxy udp/gist.tnonline.net:443 {
		header_up Host {upstream_hostport}
		transport http {
			tls
		}
	}

Caddy still uses h2 for the upstream:

{"level":"debug","ts":1713618527.673246,"logger":"http.handlers.reverse_proxy","msg":"upstream roundtrip","upstream":"udp/gist.tnonline.net:443","duration":0.01389355,"request":{"remote_ip":"158.174.155.174","remote_port":"45024","client_ip":"158.174.155.174","proto":"HTTP/2.0","method":"GET","host":"gist.tnonline.net:443","uri":"/","headers":{"X-Forwarded-Host":["gist.tnonline.net"],"User-Agent":["curl/7.81.0"],"Accept":["*/*"],"X-Forwarded-For":["158.174.155.174"],"X-Forwarded-Proto":["https"]},"tls":{"resumed":false,"version":772,"cipher_suite":4865,"proto":"h2","server_name":"gist.tnonline.net"}},"headers":{"Content-Type":["text/html; charset=UTF-8"],"Date":["Sat, 20 Apr 2024 13:08:47 GMT"],"Vary":["Accept-Encoding"],"Alt-Svc":["h3=\":443\"; ma=2592000"],"Etag":["\"63d9eeb3ee1ef16d386b5b9da5d4ed1934a76081\""],"Server":["Caddy","TornadoServer/6.3.2"],"Set-Cookie":[]},"status":200}

The Caddy server upstream does support HTTP/3. This can be tested using

# curl -vL --http3 -4 --resolve 'gist.tnonline.net:443:10.5.2.1' https://gist.tnonline.net

* Added gist.tnonline.net:443:10.5.2.1 to DNS cache
* Hostname gist.tnonline.net was found in DNS cache
*   Trying 10.5.2.1:443...
* found 146 certificates in /etc/ssl/certs/ca-certificates.crt
* GnuTLS ciphers: NORMAL:-ARCFOUR-128:-CTYPE-ALL:+CTYPE-X509:-VERS-SSL3.0
* SSL connection using TLS1.3 / ECDHE_RSA_AES_128_GCM_SHA256
*   server certificate verification OK
*   server certificate status verification SKIPPED
*   common name: gist.tnonline.net (matched)
*   server certificate expiration date OK
*   server certificate activation date OK
*   certificate public key: EC/ECDSA
*   certificate version: #3
*   subject: CN=gist.tnonline.net
*   start date: Fri, 22 Mar 2024 02:12:11 GMT
*   expire date: Thu, 20 Jun 2024 02:12:10 GMT
*   issuer: C=US,O=Let's Encrypt,CN=E1
* Verified certificate just fine
* Connected to gist.tnonline.net (10.5.2.1) port 443
* using HTTP/3
* [HTTP/3] [0] OPENED stream for https://gist.tnonline.net/
* [HTTP/3] [0] [:method: GET]
* [HTTP/3] [0] [:scheme: https]
* [HTTP/3] [0] [:authority: gist.tnonline.net]
* [HTTP/3] [0] [:path: /]
* [HTTP/3] [0] [user-agent: curl/8.7.1]
* [HTTP/3] [0] [accept: */*]
> GET / HTTP/3
> Host: gist.tnonline.net
> User-Agent: curl/8.7.1
> Accept: */*

...

Unfortunately our proxy doesn’t have HTTP/3 support yet. See this issue:

1 Like

Thanks. I had missed that ticket. I understand it isn’t a high priority. My use case is as someone else said in that ticket; inet->caddy->wiregurard->caddy. UDP is preferred for latency, but it is not big issue at all. h2 works, and that is good.

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.