Firefox HTTP/3 0-RTT issue with named IP matcher

Help
1

1. The problem I’m having:

The recent issue with HTTP/3 and iOS18 caused me to upgrade Caddy to the latest master branch. Which solved that issue but has caused a new one.

I have Caddyfile config as seen below which uses a named matcher to allow a request if it matches a list of IP address ranges, or asks for Basic Auth if it doesn’t. This worked fine until I upgraded from 2.8.4 to master.

I now see Caddy send Firefox the www-authenticate header and a 401 response on the first request after being idle for 30 seconds or so causing a request for username/password. If I cancel the dialog and press refresh the page loads fine with a 200.

I have bisected the exact commit that causes this which is Reject 0-RTT early data in IP matchers and set Early-Data header when proxying

If I go into about:config and toggle network.http.http3.enable_0rtt then this behaviour goes away. So this is obviously the problem.

Note that this is not a problem in Edge, Chrome, or Safari. So I assume it’s only Firefox that enables 0-RTT.

2. Error messages and/or full log output:

192.168.0.1 - - "GET /unread HTTP/3.0" 401 16 "-" "Mozilla/5.0 (X11; Linux x86_64; rv:130.0) Gecko/20100101 Firefox/130.0"

I’m using the log transformer plugin here so this isn’t the JSON format. But you can see 401.

3. Caddy version:

v2.8.5-0.20240925120048-9dda8fbf846d h1:At3iU625S0FqzmJYSOHih0vPqvko8CqJY5igbDUW0lE=

4. How I installed and ran Caddy:

a. System environment:

Fedora Linux 40 amd64. Running docker-hub builder image using podman.

b. Command:

N/A

c. Service/unit/compose file:

N/A

d. My complete Caddy config:

I have only provided the relevant parts here, but I believe I’ve provided more than enough information already that this shouldn’t be a problem. If you do require the entire Caddyfile I’ll provide afterwards.

(auth) {                                                                                                                                                                                
        @auth{args[0]} {                                                                                                                                                                
                not remote_ip 127.0.0.1 ::1 fe80::/10 192.168.0.0/16                                                                                                                    
                path {args[0]}                                                                                                                                                          
        }                                                                                                                                                                               
                                                                                                                                                                                        
        basic_auth @auth{args[0]} {                                                                                                                                                     
                user <redacted hash>                                                                                                                                                    
        }                                                                                                                                                                               
} 

www.example.com {
        reverse_proxy 127.0.0.1:8080                                                                                                                                          
                                                                                                                                                                                        
        import auth /*
}

5. Links to relevant resources:

2

After reading the source code of that commit it occurs to me that this may be a deliberate feature and not a bug as such. It looks like Caddy may not trust the source IP if it’s received in a 0-RTT handshake because it can be spoofed?

If that does turn out to be the case, could there be something to mitigate it. Either forcing Caddy to not do 0-RTT in the case of using an IP matcher, or a global server option to be able to disable 0-RTT?

Because whilst I have temporarily disabled it in Firefox to work around this problem, if Apple decide to enable this on their iPhone I bet there won’t be any way to disable it on the browser, so it would have to be disabled server side.

3

FYI @marten-seemann in case you have any suggestions. Do you agree it makes sense to have an user-configurable option to turn off 0-RTT?

1 Like
4

0-RTT rejection happens at the QUIC / TLS layer. This is before Caddy even sees any request.

It’s the browsers decision what it wants to do with the request it sent in 0-RTT data. In most cases, resending it in 1-RTT is the right thing to do. It sounds like Firefox is not doing this correctly, although one would probably want to consult the Firefox logs to see what’s actually going on.

That said, other than working around browser bugs, I don’t think there’s any reason to disable 0-RTT on the server side.

5

@marten-seemann I’m confused. This is about Caddy’s HTTP routes seeing the request but without the TLS handshake being complete (e.g. caddyhttp: Reject 0-RTT early data in IP matchers and set Early-Data … · caddyserver/caddy@c3fb5f4 · GitHub) so effectively those requests are bypassing authentication because auth is set to only be applied for certain IPs.

Wouldn’t it be a valid reason to turn off 0-RTT to ensure authentication can’t be bypassed due to the IP address being unknown?

1 Like
6

I think the best thing to do here would be to disallow 0-RTT connections when the IP matcher is negated and authentication relies on it.

There is a new tls matcher in the mainline (build from source for now) that can match early data connections:

tls early_data

So you’d probably want to put that in a not matcher.

7

This seems to work, but not in the way you said and my brain can’t quite get my head around the logic. I just want to be sure I’m not introducing any open security holes with this.

Just adding “tls early_data” doesn’t work, I still get the 401. However negating it with “not tls early_data” does appear to work. The issue goes away.

These matchers are being logically AND’ed with each other aren’t they. So are we saying if it is /path* AND is not 192.168.0.1 etc. AND is not 0-RTT then 401 it.

1 Like
8

Yep, matchers in a set are AND’ed:

 @auth{args[0]} {
    not remote_ip 127.0.0.1 ::1 fe80::/10 192.168.0.0/16
    not tls early_data
    path {args[0]}
}
2 Likes
9

@francislavoie I might update the docs to mention nuances with using IP matchers in HTTP/3 contexts (like this one). Sound good to you?

1 Like
10

That’s what we’re already doing! If the IP matcher is set, we reject 0-RTT. The IP matcher is per path though, and we should accept 0-RTT for other paths that don’t match on the IP.

11

That’s not my reading of that code. What it does is say “no it doesn’t match the IP” which isn’t the same as “rejecting 0-RTT”. Like, it doesn’t fail the request, it just means “this one HTTP route will not happen but another route may happen” which causes it to fail open here (i.e. authentication could be bypassed by using 0-RTT, gaining access to the system without a password).

I was talking to Matt on Slack, what I think should happen is the IP matcher should not return false when the IP isn’t available, but instead return an error. The Match() bool interface doesn’t allow that, but we did add an escape hatch for the file matcher by adding an error in the request context which then gets looked at immediately after the matcher runs, and the middleware execution would bail out if it noticed the matcher put an error in context (instead invoking the error routes).

Sure we can augment the docs, but shouldn’t we also plug that hole in the matcher? It really feels wrong to me to return false like that without even considering the IPs. The matcher is lying, basically. If it’s impossible for the matcher to do its job, it doesn’t mean it doesn’t match, it means there’s an error in the preconditions to run that code.

I’m glad there’s a workaround with the tls matcher, but that’s still just a workaround. There’s still a problem here.

1 Like
12

I hate having to stuff an error into the context, but it’s my own fault. So yeah, maybe we should do that too. Sigh

1 Like
13

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.