MTU Problem with HTTP3

1. The problem I’m having:

I want to deploy Caddy on an IP6 network which only supports MTU 1310. When I switch to MTU 1310, Caddy doesn’t reply to the handshake request. I understand QUIC should support down to 1280.

When I switch back to 1500, everything works.

2. Error messages and/or full log output:

An example timeout:

$ sudo ip netns exec http3ns2 bash -c " all_proxy="" https_proxy="" curl --http3-only -v --cert /etc/caddy/certs/client.crt --key /etc/caddy/certs/client.key --resolve api.airlink.local:443:[fd00:dead::1]  --cacert /etc/caddy/certs/rootCA.pem https://api.airlink.local:443/v0/client/query?obcu_id=1"
* Added api.airlink.local:443:[fd00:dead::1] to DNS cache
* Uses proxy env variable no_proxy == '127.0.0.1,localhost,internal.domain'
* Hostname api.airlink.local was found in DNS cache
*   Trying [fd00:dead::1]:443...
* QUIC cipher selection: TLS_AES_128_GCM_SHA256:TLS_AES_256_GCM_SHA384:TLS_CHACHA20_POLY1305_SHA256:TLS_AES_128_CCM_SHA256
*  CAfile: /etc/caddy/certs/rootCA.pem
*  CApath: none
* ngtcp2_conn_handle_expiry returned error: ERR_HANDSHAKE_TIMEOUT
* Failed to connect to api.airlink.local port 443 after 10002 ms: Failed sending data to the peer
* Closing connection
curl: (55) ngtcp2_conn_handle_expiry returned error: ERR_HANDSHAKE_TIMEOUT

Caddy doesn’t actually error out, it just doesn’t forward to the reverse proxy etc.

2025/04/03 20:04:03.394 DEBUG   tls.handshake   choosing certificate    {"identifier": "api.airlink.local", "num_choices": 1}
2025/04/03 20:04:03.394 DEBUG   tls.handshake   custom certificate selection results    {"identifier": "api.airlink.local", "subjects": ["49677b3bcbb760a39ea5cf20db06c490.non-safety.unit.airlink.local", "airlink.local", "api.airlink.local", "console.airlink.local", "*.mgmt.airlink.local", "non-safety.role.airlink.local"], "managed": false, "issuer_key": "", "hash": "0778f5c453138465c7629af68378fec46884d5423c7b840a2f6f4d3d327f8612"}
2025/04/03 20:04:03.394 DEBUG   tls.handshake   matched certificate in cache    {"remote_ip": "fd00:dead::2", "remote_port": "37156", "subjects": ["49677b3bcbb760a39ea5cf20db06c490.non-safety.unit.airlink.local", "airlink.local", "api.airlink.local", "console.airlink.local", "*.mgmt.airlink.local", "non-safety.role.airlink.local"], "managed": false, "expiration": "2152/02/07 12:00:01.000", "hash": "0778f5c453138465c7629af68378fec46884d5423c7b840a2f6f4d3d327f8612"}

Caddy does successfully reply with an ACK_ECN packet instead of completing the handshake in the normal way.

3. Caddy version:

2.9.1

4. How I installed and ran Caddy:

I installed via NixOS and also tried the binary from the Caddy Github repo

a. System environment:

NixOS & Yocto

b. Command:

sudo ip netns exec http3ns1 bash -c "CADDY_LOG_LEVEL=DEBUG /nix/store/vdjnfdjnnqspc67fjys692b4hishmljd-caddy-2.9.1/bin/caddy run --config /home/john/caddy_config --adapter caddyfile"

c. Service/unit/compose file:

NA

d. My complete Caddy config:

{
    auto_https disable_redirects
    debug
    servers {
        protocols h3
    }
}

api.airlink.local:443 {

        # TODO: update to new IP when ready
        bind udp6/[fd00:3d2::300:64]                                  

	tls /run/http/ssl/server.crt /run/http/ssl/server.key {
		client_auth {
			mode require_and_verify
			trusted_ca_cert_file /run/http/ssl/roots.crt
		}
	}

        # Match only paths that begin with "client/"
        @client_routes {
                       path_regexp client ^/v0/(client/.+)$
                       }


        handle @client_routes {
               #TODO: make port configurable
               reverse_proxy localhost:8000
               }

        respond "Access denied" 403
}

5. Links to relevant resources:

NA

I found Qlog from Caddy, here is an example. It looks like what I see from the PCAP, But what’s interesting is both client and Caddy set a larger MTU…

{"qlog_format":"JSON-SEQ","qlog_version":"0.3","title":"quic-go qlog","configuration":{"code_version":"v0.48.2"},"trace":{"vantage_point":{"type":"server"},"common_fields":{"ODCID":"e9fc5fcc83f6c8a6e875ad0c7958f0b5a2e3f6de","group_id":"e9fc5fcc83f6c8a6e875ad0c7958f0b5a2e3f6de","reference_time":1743747845583.3948,"time_format":"relative"}}}
{"time":0.081378,"name":"recovery:congestion_state_updated","data":{"new":"slow_start"}}
{"time":0.088439,"name":"transport:parameters_set","data":{"owner":"local","original_destination_connection_id":"e9fc5fcc83f6c8a6e875ad0c7958f0b5a2e3f6de","stateless_reset_token":"625253d5007068eda5fd317f7ee912ea","initial_source_connection_id":"c0a97002","disable_active_migration":true,"max_idle_timeout":30000,"max_udp_payload_size":1452,"ack_delay_exponent":3,"max_ack_delay":26,"active_connection_id_limit":4,"initial_max_data":786432,"initial_max_stream_data_bidi_local":524288,"initial_max_stream_data_bidi_remote":524288,"initial_max_stream_data_uni":524288,"initial_max_streams_bidi":100,"initial_max_streams_uni":100}}
{"time":0.118152,"name":"security:key_updated","data":{"trigger":"tls","key_type":"client_initial_secret"}}
{"time":0.119653,"name":"security:key_updated","data":{"trigger":"tls","key_type":"server_initial_secret"}}
{"time":0.302155,"name":"transport:version_information","data":{"server_versions":["1","6b3343cf"],"chosen_version":"1"}}
{"time":0.303191,"name":"transport:connection_started","data":{"ip_version":"ipv6","src_ip":"fd00:dead::1","src_port":443,"dst_ip":"fd00:dead::2","dst_port":41703,"src_cid":"8c554415715f81b329eaff7ad41351b339122fb9","dst_cid":"e9fc5fcc83f6c8a6e875ad0c7958f0b5a2e3f6de"}}
{"time":0.782701,"name":"security:key_updated","data":{"trigger":"tls","key_type":"server_handshake_secret"}}
{"time":0.788163,"name":"security:key_updated","data":{"trigger":"tls","key_type":"client_handshake_secret"}}
{"time":5.727209,"name":"security:key_updated","data":{"trigger":"tls","key_type":"server_1rtt_secret","key_phase":0}}
{"time":5.729184,"name":"transport:parameters_set","data":{"owner":"remote","initial_source_connection_id":"8c554415715f81b329eaff7ad41351b339122fb9","disable_active_migration":false,"max_idle_timeout":120000,"max_udp_payload_size":4611686018427387903,"ack_delay_exponent":3,"max_ack_delay":25,"active_connection_id_limit":2,"initial_max_data":1310720,"initial_max_stream_data_bidi_local":131072,"initial_max_stream_data_bidi_remote":131072,"initial_max_stream_data_uni":131072,"initial_max_streams_bidi":262144,"initial_max_streams_uni":262144}}
{"time":5.745068,"name":"transport:packet_received","data":{"header":{"packet_type":"initial","packet_number":0,"version":"1","scil":20,"scid":"8c554415715f81b329eaff7ad41351b339122fb9","dcil":20,"dcid":"e9fc5fcc83f6c8a6e875ad0c7958f0b5a2e3f6de"},"raw":{"length":1200,"payload_length":1148},"frames":[{"frame_type":"crypto","offset":0,"length":330}],"ecn":"Not-ECT"}}
{"time":5.76709,"name":"transport:packet_sent","data":{"header":{"packet_type":"initial","packet_number":0,"version":"1","scil":4,"scid":"c0a97002","dcil":20,"dcid":"8c554415715f81b329eaff7ad41351b339122fb9"},"raw":{"length":184,"payload_length":150},"frames":[{"frame_type":"ack","acked_ranges":[[0]]},{"frame_type":"crypto","offset":0,"length":123}],"ecn":"Not-ECT"}}
{"time":5.767891,"name":"transport:packet_sent","data":{"header":{"packet_type":"handshake","packet_number":0,"version":"1","scil":4,"scid":"c0a97002","dcil":20,"dcid":"8c554415715f81b329eaff7ad41351b339122fb9"},"raw":{"length":1096,"payload_length":1063},"frames":[{"frame_type":"crypto","offset":0,"length":1041}],"ecn":"Not-ECT"}}
{"time":5.7709,"name":"recovery:metrics_updated","data":{"min_rtt":0,"smoothed_rtt":0,"latest_rtt":0,"rtt_variance":0,"congestion_window":40960,"bytes_in_flight":184,"packets_in_flight":1}}
{"time":5.771996,"name":"recovery:loss_timer_updated","data":{"event_type":"set","timer_type":"pto","packet_number_space":"initial","delta":199.976227}}
{"time":5.772863,"name":"recovery:metrics_updated","data":{"bytes_in_flight":1280,"packets_in_flight":2}}

hi!

i was able to replicate this problem.

it seems caddy http3 does not function under ipv6 with an MTU of 1327 or lower
and ipv4 with an MTU of 1308 or lower.

could you see if you run into the same limit on ipv6? just try MTU 1328, see if that works, and see if 1327 doesn’t

I think this is a problem with the underlying library that is used to handle http3/quic in caddy, so we will try to see what is going on here and maybe report upstream

2 Likes

Hi E Lee,

thanks for double checking.

I also see that 1327 doesn’t work for IP6 and 1328 does work.

Unfortunately 1310 is probably a fixed constraint for me. But thanks nevertheless.

John

yeah its unfortunate.

ive opened an issue that you are free to follow around or add information about your own environment to http3 handler not working when interface MTU is <1328 for IPV6 · Issue #5021 · quic-go/quic-go · GitHub

2 Likes

OK, Marten Seeman replied to the issue:

Actually, this makes sense. The IPv6 header is 40 bytes, and the UDP header is 8 bytes. So 1328 is the minimum MTU on IPv6.

Makes sense.

2 Likes

On second thoughts, I just read the below. Basically the packet size should include the headers… So therefore the MTU minimum should indeed be 1280.

14. Datagram Size

A UDP datagram can include one or more QUIC packets. The datagram size refers to the total UDP payload size of a single UDP datagram carrying QUIC packets. The datagram size includes one or more QUIC packet headers and protected payloads, but not the UDP or IP headers.

The maximum datagram size is defined as the largest size of UDP payload that can be sent across a network path using a single UDP datagram. QUIC MUST NOT be used if the network path cannot support a maximum datagram size of at least 1200 bytes.QUIC assumes a minimum IP packet size of at least 1280 bytes. This is the IPv6 minimum size [IPv6] and is also supported by most modern IPv4 networks. Assuming the minimum IP header size of 40 bytes for IPv6 and 20 bytes for IPv4 and a UDP header size of 8 bytes, this results in a maximum datagram size of 1232 bytes for IPv6 and 1252 bytes for IPv4. Thus, modern IPv4 and all IPv6 network paths are expected to be able to support QUIC.

OK, Marten Seemann suggested changing the InitialPacketSize to support lower MTUs (i.e. 1280-1328 which just matches the RFC).

Unsure whether best to change in Quic-Go or Caddy, but on the face of it, it seems straightforward in either.

The InitialPacketSize should be 1232.

Caddy fix, if useful to someone in future: