HTTP/3 not working (Docker)

1. Caddy version (caddy version):

2.6.1

2. How I run Caddy:

a. System environment:

Debian 11, Docker (rootless mode)

b. Command:

docker compose up -d

c. Service/unit/compose file:

Dockerfile
FROM caddy:2.6.1-builder-alpine AS builder

RUN xcaddy build \
    --with github.com/caddy-dns/digitalocean

FROM caddy:2.6.1-alpine

COPY --from=builder /usr/bin/caddy /usr/bin/caddy
docker-compose.yml
version: "3.7"

services:
  caddy:
    container_name: caddy
    build: .
    restart: always
    networks:
      - etebase_default
      - miniflux_default
      - rssbridge_default
      - seafile_default
    ports:
      - 80:80
      - 443:443
      - 443:443/udp
    volumes:
      - ./Caddyfile:/etc/caddy/Caddyfile
      - ./data:/data
      - ./logs:/var/log/caddy
      - ./site:/srv
      - ../etebase/srv:/srv/etebase
      - config:/config

networks:
  etebase_default:
    external: true
  miniflux_default:
    external: true
  rssbridge_default:
    external: true
  seafile_default:
    external: true

volumes:
  config:

d. My complete Caddyfile or JSON config:

*.example.com {
	redir https://example.com{$uri}

	tls {
		dns digitalocean {env.DIGITALOCEAN_API_TOKEN}
	}

	log {
		output file /var/log/caddy/access.log
	}
}

example.com {
	root * /srv/index
	file_server

	header Strict-Transport-Security "max-age=31536000"

	log {
		output file /var/log/caddy/access.log
	}
}

bridge.example.com {
	reverse_proxy rssbridge:80

	log {
		output file /var/log/caddy/bridge.log
	}
}

cloud.example.com {
	reverse_proxy /seafdav* seafile:8080
	reverse_proxy seafile:80

	log {
		output file /var/log/caddy/seafile.log
	}
}

etebase.example.com {
	handle_path /static/* {
		root * /srv/etebase/static
		file_server
	}

	handle {
		reverse_proxy etebase:3735
	}

	log {
		output file /var/log/caddy/etebase.log
	}
}

nc.example.com {
	reverse_proxy 10.120.0.2:9090

	header Strict-Transport-Security "max-age=15552000"

	redir /.well-known/carddav /remote.php/carddav 301
	redir /.well-known/caldav /remote.php/caldav 301

	log {
		output file /var/log/caddy/nextcloud.log
	}
}

rss.example.com {
	reverse_proxy miniflux:8080

	log {
		output file /var/log/caddy/miniflux.log
	}
}

3. The problem I’m having:

This is one of two servers on which I’m running Caddy 2.6.1–even though the alt-svc header is showing h3=":443"; ma=2592000, no content is ever served over HTTP/3. Geekflare’s HTTP/3 testing tool gives the following error message:

Couldn’t connect over HTTP/3. Take advantage of the latest protocol HTTP/3 for better performance.

4. Error messages and/or full log output:

Logs on server launch:

{"level":"info","ts":1663989262.319634,"msg":"using provided configuration","config_file":"/etc/caddy/Caddyfile","config_adapter":"caddyfile"}
{"level":"warn","ts":1663989262.3222895,"msg":"Caddyfile input is not formatted; run the 'caddy fmt' command to fix inconsistencies","adapter":"caddyfile","file":"/etc/caddy/Caddyfile","line":6}
{"level":"info","ts":1663989262.3235857,"logger":"admin","msg":"admin endpoint started","address":"localhost:2019","enforce_origin":false,"origins":["//localhost:2019","//[::1]:2019","//127.0.0.1:2019"]}
{"level":"info","ts":1663989262.3238082,"logger":"tls.cache.maintenance","msg":"started background certificate maintenance","cache":"0xc000246fc0"}
{"level":"info","ts":1663989262.3238745,"logger":"http","msg":"server is listening only on the HTTPS port but has no TLS connection policies; adding one to enable TLS","server_name":"srv0","https_port":443}
{"level":"info","ts":1663989262.3238966,"logger":"http","msg":"enabling automatic HTTP->HTTPS redirects","server_name":"srv0"}
{"level":"info","ts":1663989262.3245602,"logger":"http","msg":"enabling HTTP/3 listener","addr":":443"}
{"level":"info","ts":1663989262.3253105,"logger":"tls","msg":"cleaning storage unit","description":"FileStorage:/data/caddy"}
{"level":"debug","ts":1663989262.3255713,"logger":"http","msg":"starting server loop","address":"[::]:443","tls":true,"http3":true}
{"level":"info","ts":1663989262.3255901,"logger":"http.log","msg":"server running","name":"srv0","protocols":["h1","h2","h3"]}
{"level":"debug","ts":1663989262.3256235,"logger":"http","msg":"starting server loop","address":"[::]:80","tls":false,"http3":false}
{"level":"info","ts":1663989262.3256304,"logger":"http.log","msg":"server running","name":"remaining_auto_https_redirects","protocols":["h1","h2","h3"]}

Logs when attempting to access one of the configured domains:

{"level":"debug","ts":1663989684.2364225,"logger":"http.handlers.reverse_proxy","msg":"upstream roundtrip","upstream":"miniflux:8080","duration":0.002274311,"request":{"remote_ip":"X.X.X.X","remote_port":"53913","proto":"HTTP/2.0","method":"GET","host":"rss.example.com","uri":"/icon/icon-192.png","headers":{"Sec-Fetch-Dest":["image"],"X-Forwarded-For":["X.X.X.X"],"X-Forwarded-Host":["rss.example.com"],"Accept":["image/avif,image/webp,*/*"],"Dnt":["1"],"Cookie":[],"Cache-Control":["no-cache"],"Sec-Fetch-Site":["same-origin"],"Te":["trailers"],"Accept-Language":["en-US,en;q=0.5"],"Sec-Fetch-Mode":["no-cors"],"Sec-Gpc":["1"],"X-Forwarded-Proto":["https"],"User-Agent":["Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:105.0) Gecko/20100101 Firefox/105.0"],"Pragma":["no-cache"],"Accept-Encoding":["gzip, deflate, br"]},"tls":{"resumed":false,"version":772,"cipher_suite":4865,"proto":"h2","server_name":"rss.example.com"}},"headers":{"Cache-Control":["public"],"Referrer-Policy":["no-referrer"],"Content-Length":["881"],"Etag":["b0b36eceb45f494fa4151a7ffce7a31d603f50e1a58c98e335c12776d24e755f"],"X-Xss-Protection":["1; mode=block"],"X-Content-Type-Options":["nosniff"],"X-Frame-Options":["DENY"],"Date":["Sat, 24 Sep 2022 03:21:24 GMT"],"Content-Type":["image/png"],"Expires":["Tue, 27 Sep 2022 03:21:24 UTC"],"Strict-Transport-Security":["max-age=31536000"]},"status":200}

5. What I already tried:

Attempting to run a similar configuration with rootful Docker and host networking produces the same result (non-working HTTP/3, errors on HTTP/3 testers).

Thanks in advance for any guidance you can provide! :blush:

Are you sure your firewall and router has port 443 over UDP open/forwarded? Your docker-compose looks fine, you have UDP published. So I figure it must be some other networking config between your clients and your server.

1 Like

As far as Docker is concerned, it seems like all of the port mappings are what they should be…

0.0.0.0:80->80/tcp, :::80->80/tcp, 0.0.0.0:443->443/tcp, :::443->443/tcp, 0.0.0.0:443->443/udp, :::443->443/udp

netstat shows the following for port 443:

> netstat -an | grep 0.0.0.0:443
tcp        0      0 0.0.0.0:443             0.0.0.0:*               LISTEN
udp        0      0 0.0.0.0:443             0.0.0.0:*

After a few more days, I haven’t had any success getting HTTP/3 working for either of my servers–just reporting back on what I’ve tried in case anyone might have ideas.

Attempting to connect to port 443/UDP on both servers via nc works without issue.

> nc -z -v -u [server-IP] 443
Connection to [server-IP] port 443 [udp/https] succeeded!

Reviewing the logs for the server detailed in my first post, I can see that the last successful connection over HTTP/3 was logged several days ago, when rootless Docker was still set to use the built-in port driver. There are just a handful of entries like this one, and they’re all 304 status codes:

{"level":"info","ts":1663913031.2618148,"logger":"http.log.access.log1","msg":"handled request","request":{"remote_ip":"[IP]","remote_port":"45549","proto":"HTTP/3.0","method":"GET","host":"example.com","uri":"/","headers":{"Accept-Language":["en-US,en;q=0.9"],"Cache-Control":["max-age=0"],"Sec-Ch-Ua-Mobile":["?0"],"Sec-Ch-Ua-Platform":["\"macOS\""],"Upgrade-Insecure-Requests":["1"],"Accept":["text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.9"],"Sec-Fetch-User":["?1"],"Accept-Encoding":["gzip, deflate, br"],"If-None-Match":["\"r95l3e2p7\""],"Sec-Ch-Ua":["\"Chromium\";v=\"105\", \"Not)A;Brand\";v=\"8\""],"Sec-Fetch-Site":["none"],"Sec-Fetch-Mode":["navigate"],"If-Modified-Since":["Tue, 22 Mar 2022 15:45:14 GMT"],"Dnt":["1"],"User-Agent":["Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/105.0.0.0 Safari/537.36"],"Sec-Fetch-Dest":["document"]},"tls":{"resumed":false,"version":0,"cipher_suite":0,"proto":"","server_name":""}},"user_id":"","duration":0.000126688,"size":0,"status":304,"resp_headers":{"Server":["Caddy"],"Alt-Svc":["h3=\":443\"; ma=2592000"],"Strict-Transport-Security":["max-age=31536000"],"Etag":["\"r95l3e2p7\""]}}

After switching to the slirp4netns port driver, the only successful connections happening over HTTP/3 are API calls, like this one:

{"level":"info","ts":1664320114.5114968,"logger":"http.log.access.log6","msg":"handled request","request":{"remote_ip":"[IP]","remote_port":"55588","proto":"HTTP/3.0","method":"POST","host":"rss.example.com","uri":"/fever/?api","headers":{"Priority":["u=3"],"User-Agent":["Reeder/4020.89.01 CFNetwork/1390 Darwin/22.0.0"],"Accept-Language":["en-us"],"Accept-Encoding":["gzip, deflate, br"],"Accept":["*/*"],"Content-Type":["application/x-www-form-urlencoded"]},"tls":{"resumed":false,"version":0,"cipher_suite":0,"proto":"","server_name":""}},"user_id":"","duration":0.002515496,"size":62,"status":200,"resp_headers":{"Date":["Tue, 27 Sep 2022 23:08:34 GMT"],"Content-Type":["application/json"],"Alt-Svc":["h3=\":443\"; ma=2592000"],"Content-Length":["62"],"Referrer-Policy":["no-referrer"],"Strict-Transport-Security":["max-age=31536000"],"X-Frame-Options":["DENY"],"X-Xss-Protection":["1; mode=block"],"Server":["Caddy"],"X-Content-Type-Options":["nosniff"]}}

The fact that some connections are getting through just has me more confused, to be honest.

Any suggestions would be hugely appreciated–I can’t understand why my otherwise-working setup seems to be broken in this one specific way! (Am I really the only person in the world running Caddy with rootless Docker?)

Are the clients for the non-HTTP/3 requests all browsers? I’ve found non-browser clients to be much more consistent with regards to their HTTP/3 support. Even Firefox is pretty flaky. curl --http3 is ever faithful.

I’ve never gotten Geekflare’s site to work, btw. I think they use an old version of HTTP/3 or something.

Oh, I’m so glad that you mentioned this–here I was thinking that Geekflare’s site was some sort of standard for HTTP/3 testing! I grabbed an HTTP/3-enabled build of curl from Docker, and it looks like everything is working just fine:

curl -IL https://example.com --http3
HTTP/3 200
last-modified: Fri, 23 Sep 2022 20:22:37 GMT
accept-ranges: bytes
content-length: 3530
server: Caddy
alt-svc: h3=":443"; ma=2592000
strict-transport-security: max-age=31536000
etag: "rioj9p2q2"
content-type: text/html; charset=utf-8

Hopefully this silly topic of mine can reassure someone else who’s in a similar situation. Thanks so much to both of you for your help! :smiley_cat:

1 Like

This topic was automatically closed after 30 days. New replies are no longer allowed.