Experimental_http3 behind firewall port-forwarding

1. Caddy version (caddy version):

v2.4.6 h1:HGkGICFGvyrodcqOOclHKfvJC0qTU7vny/7FhYp9hNw=

2. How I run Caddy:

a. System environment:

  • CentOS 7
  • x86_64

Important:

  • Firewall Port Forwarding of Port 443 to 8443 and likewise 80 to 8080
  • Caddy runs as non root user

b. Command:

./caddy start

c. Service/unit/compose file:

not used

d. My complete Caddyfile or JSON config:

{
        https_port 8443
        http_port 8080
        skip_install_trust true

        servers :8443 {
                protocol {
                        experimental_http3
                }
        }
}

https://playground.stbu.net:8443 {
        respond "It works."
}

3. The problem I’m having:

curl -v -s  https://playground.stbu.net |& grep 'alt-svc'
< alt-svc: h3=":8443"; ma=2592000,h3-29=":8443"; ma=2592000

Server does not advertise supported HTTP/3 or QUIC version on the same port.
The HTTPS Connection is established to Port 443, the firewall forwards to 8443 and the HTTP Response Header from Caddy contains that H3 support is available port 8443.

I want that Caddy returns Port 443 in the alt-svc response header.

4. Error messages and/or full log output:

5. What I already tried:

I have removed the https_port and http_port global options, but then caddy wants to bind to port 80 which is not possible.

2022/01/16 11:51:02.721 ERROR   admin.api       request error   {"error": "loading config: loading new config: http app module: start: tcp: listening on :80: listen tcp :80: bind: permission denied", "status_code": 400}
reload: sending configuration to instance: caddy responded with error: HTTP 400: {"error":"loading config: loading new config: http app module: start: tcp: listening on :80: listen tcp :80: bind: permission denied"}

6. Links to relevant resources:

@stbu , you need to allow the service to binding to privileged ports. Here is an example of the end-to-end reconfiguration. The important command is setcap.

sudo systemctl stop gatekeeper
sudo rm -rf /usr/local/bin/gatekeeper
sudo cp bin/caddy /usr/local/bin/gatekeeper
sudo setcap cap_net_bind_service=+ep /usr/local/bin/gatekeeper
sudo systemctl start gatekeeper
1 Like

Thanks for your input @greenpau
If setcap cap_net_bind_service=+ep isn’t allowed because of restrictive environments, shouldn’t there be any other possibility to enable h3 support?
Anything else from certificate handling to http->https redirects is working perfectly fine when I’m running caddy on 8080 and 8443 behind the firwall-port forwarding of these two ports. The only thing bothering me is that h3 is advertised on 8443.
I’m coming from the Java Servlet World and even when a Servlet Container is listening on Port 8443, the getServerPort() would still return 443 when the Request was sent on the default https port 443 and forwarded by let’s say firewall-cmd --zone=public --permanent --add-forward-port=port=443:proto=tcp:toport=8443 to the Servlet Container.
And that’s what I would expect from Caddy as well in the Alt-Svc header for h3.

The logic for this isn’t done in Caddy, but rather in the upstream package quic-go which Caddy uses for HTTP/3 support.

The relevant bit of code is here:

So basically, it depends on the server address it received the request on to build the Alt-Svc header.

Probably best if you open an issue on that project’s repo asking for a change to support an alternate mapping for the port.

2 Likes

I’ve been interested in a solution for this too as we are running into the same issue.

We currently have:

End User Request → Amazon ELB :80/:443 → Caddy :8080/:8443 → Some docker service.

We can’t get HTTP3 working until we can set the header to “advertise” the correct AWS ELB upstream port (eg. 443).

I’m wondering if there is a way to hard code/override this in the short term using the header_up/header_down functionality?

Can i just send a header like this? Given some arbitrary matcher?
Alt-Svc: quic=":443"; ma=2592000; v="33,32,31,30"

I haven’t read/researched the spec, but imagine this header goes away once the connection is established…

1 Like

Actually, doing some further research, apparently this has been resolved upstream already @francislavoie , see:

Issue Discussion:

PR:

The question I believe is, how could we best set this via a Caddyfile option, and pass it through to the lib. :pray:

Perhaps something like?

        servers :8443 {
                protocol {
                        experimental_http3 {
                                header_port 443
                        }
                }
        }
2 Likes

Oh, okay! Cool, thanks for looking into that.

Yeah, something like that maybe. Really what’s most important to think about is where it goes in the JSON config first, then how it’s configured in the Caddyfile comes second.

Take a look at the JSON config for the HTTP app, you’ll see that experimental_http3 is just a boolean flag for now:

I don’t think we really want to break this option for everyone by turning the boolean into an object, like "experimental_http3": {"enabled": true, "advertised_port": 443}, but we could do that since it’s still marked experimental. The Caddyfile adapter could still generate the right thing in a backwards compatible way.

We could also add a new http3_port option alongside http_port and https_port on the HTTP app which would do this. Downside is it’s further away from the option where http3 is enabled, but I think it’s probably a better place to put the option, since it’s related with the fact that https_port is configured to some non-default. We’d have to mark that option as experimental in the docs as well until HTTP3 is made stable.

PRs welcome :+1:

2 Likes

Thanks for the quick response!

That makes sense too, re: bool types and backwards compatibility.

My golang proficiency is quite basic (coming from a python background) - but i can read the docs and see how far i get. (I may be more suited to testing!) :sweat_smile:

If i’m not too far off though, i think it’s having to set the port prior to calling the setQuickHeaders method (line 147)

Like maybe(?):
s.h3server.Port = <some caddy config global option. eg. "http3_port">

Which may then set the value here:

Again, not sure how it all get’s wired up/passed through. (More reading for me required! :smile: )

1 Like

Yep, exactly.

Look for how "https_port" is wired up, it would be the same if we go the route of "http3_port", which is the simplest way.

1 Like

This topic was automatically closed after 30 days. New replies are no longer allowed.