Experimental_http3 behind firewall port-forwarding

stbu · January 16, 2022, 11:57am

1. Caddy version (`caddy version`):

v2.4.6 h1:HGkGICFGvyrodcqOOclHKfvJC0qTU7vny/7FhYp9hNw=

2. How I run Caddy:

a. System environment:

CentOS 7
x86_64

Important:

Firewall Port Forwarding of Port 443 to 8443 and likewise 80 to 8080
Caddy runs as non root user

b. Command:

./caddy start

c. Service/unit/compose file:

not used

d. My complete Caddyfile or JSON config:

{
        https_port 8443
        http_port 8080
        skip_install_trust true

        servers :8443 {
                protocol {
                        experimental_http3
                }
        }
}

https://playground.stbu.net:8443 {
        respond "It works."
}

3. The problem I’m having:

curl -v -s  https://playground.stbu.net |& grep 'alt-svc'
< alt-svc: h3=":8443"; ma=2592000,h3-29=":8443"; ma=2592000

Server does not advertise supported HTTP/3 or QUIC version on the same port.
The HTTPS Connection is established to Port 443, the firewall forwards to 8443 and the HTTP Response Header from Caddy contains that H3 support is available port 8443.

I want that Caddy returns Port 443 in the alt-svc response header.

4. Error messages and/or full log output:

5. What I already tried:

I have removed the https_port and http_port global options, but then caddy wants to bind to port 80 which is not possible.

2022/01/16 11:51:02.721 ERROR   admin.api       request error   {"error": "loading config: loading new config: http app module: start: tcp: listening on :80: listen tcp :80: bind: permission denied", "status_code": 400}
reload: sending configuration to instance: caddy responded with error: HTTP 400: {"error":"loading config: loading new config: http app module: start: tcp: listening on :80: listen tcp :80: bind: permission denied"}

6. Links to relevant resources:

greenpau · January 17, 2022, 1:13pm

@stbu , you need to allow the service to binding to privileged ports. Here is an example of the end-to-end reconfiguration. The important command is setcap.

sudo systemctl stop gatekeeper
sudo rm -rf /usr/local/bin/gatekeeper
sudo cp bin/caddy /usr/local/bin/gatekeeper
sudo setcap cap_net_bind_service=+ep /usr/local/bin/gatekeeper
sudo systemctl start gatekeeper

stbu · January 17, 2022, 6:50pm

Thanks for your input @greenpau
If setcap cap_net_bind_service=+ep isn’t allowed because of restrictive environments, shouldn’t there be any other possibility to enable h3 support?
Anything else from certificate handling to http->https redirects is working perfectly fine when I’m running caddy on 8080 and 8443 behind the firwall-port forwarding of these two ports. The only thing bothering me is that h3 is advertised on 8443.
I’m coming from the Java Servlet World and even when a Servlet Container is listening on Port 8443, the getServerPort() would still return 443 when the Request was sent on the default https port 443 and forwarded by let’s say firewall-cmd --zone=public --permanent --add-forward-port=port=443:proto=tcp:toport=8443 to the Servlet Container.
And that’s what I would expect from Caddy as well in the Alt-Svc header for h3.

francislavoie · January 17, 2022, 8:02pm

The logic for this isn’t done in Caddy, but rather in the upstream package quic-go which Caddy uses for HTTP/3 support.

The relevant bit of code is here:

github.com

lucas-clemente/quic-go/blob/976cc29c1648949a671d4d88358f316378ee556b/http3/server.go#L439-L473

    
      
          // SetQuicHeaders can be used to set the proper headers that announce that this server supports QUIC.
          // The values that are set depend on the port information from s.Server.Addr, and currently look like this (if Addr has port 443):
          //  Alt-Svc: quic=":443"; ma=2592000; v="33,32,31,30"
          func (s *Server) SetQuicHeaders(hdr http.Header) error {
          	port := atomic.LoadUint32(&s.Port)
          
          
	if port == 0 {
          		// Extract port from s.Server.Addr
          		_, portStr, err := net.SplitHostPort(s.Server.Addr)
          		if err != nil {
          			return err
          		}
          		portInt, err := net.LookupPort("tcp", portStr)
          		if err != nil {
          			return err
          		}
          		port = uint32(portInt)
          		atomic.StoreUint32(&s.Port, port)
          	}

This file has been truncated. show original

So basically, it depends on the server address it received the request on to build the Alt-Svc header.

Probably best if you open an issue on that project’s repo asking for a change to support an alternate mapping for the port.

jotterbot · January 19, 2022, 11:31pm

I’ve been interested in a solution for this too as we are running into the same issue.

We currently have:

End User Request → Amazon ELB :80/:443 → Caddy :8080/:8443 → Some docker service.

We can’t get HTTP3 working until we can set the header to “advertise” the correct AWS ELB upstream port (eg. 443).

I’m wondering if there is a way to hard code/override this in the short term using the header_up/header_down functionality?

Can i just send a header like this? Given some arbitrary matcher?
Alt-Svc: quic=":443"; ma=2592000; v="33,32,31,30"

I haven’t read/researched the spec, but imagine this header goes away once the connection is established…

jotterbot · January 19, 2022, 11:38pm

Actually, doing some further research, apparently this has been resolved upstream already @francislavoie , see:

Issue Discussion:

github.com/lucas-clemente/quic-go

Allow setting custom port in Alt-Svc header value

opened 01:25PM - 01 Sep 21 UTC

closed 09:10AM - 12 Oct 21 UTC

aaronriekenberg

I have an app using quic-go running as a non-privileged user process on port 844…3. A firewall redirects UDP and TCP traffic on port 443 to my app on port 8443. I am finding I need to rewrite the Alt-Svc header value to replace ":8443" with ":443" to make this work: https://github.com/aaronriekenberg/pi-web/blob/master/pi-web.go#L24-L56 Is there any better way to do this? Could it be possible to add some configuration to http3.ListenAndServe or quicServer.SetQuicHeaders to allow for a custom Alt-Svc port? I suspect this is a fairly common use-case. This is the form of ListenAndServe my app is calling: https://github.com/lucas-clemente/quic-go/blob/master/http3/server.go#L488 Thanks!

PR:

The question I believe is, how could we best set this via a Caddyfile option, and pass it through to the lib.

Perhaps something like?

        servers :8443 {
                protocol {
                        experimental_http3 {
                                header_port 443
                        }
                }
        }

francislavoie · January 20, 2022, 2:27am

Oh, okay! Cool, thanks for looking into that.

jotterbot:

The question I believe is, how could we best set this via a Caddyfile option, and pass it through to the lib.

Perhaps something like?
        servers :8443 {
                protocol {
                        experimental_http3 {
                                header_port 443
                        }
                }
        }

Yeah, something like that maybe. Really what’s most important to think about is where it goes in the JSON config first, then how it’s configured in the Caddyfile comes second.

Take a look at the JSON config for the HTTP app, you’ll see that experimental_http3 is just a boolean flag for now:

I don’t think we really want to break this option for everyone by turning the boolean into an object, like "experimental_http3": {"enabled": true, "advertised_port": 443}, but we could do that since it’s still marked experimental. The Caddyfile adapter could still generate the right thing in a backwards compatible way.

We could also add a new http3_port option alongside http_port and https_port on the HTTP app which would do this. Downside is it’s further away from the option where http3 is enabled, but I think it’s probably a better place to put the option, since it’s related with the fact that https_port is configured to some non-default. We’d have to mark that option as experimental in the docs as well until HTTP3 is made stable.

PRs welcome

jotterbot · January 20, 2022, 3:16am

Thanks for the quick response!

That makes sense too, re: bool types and backwards compatibility.

My golang proficiency is quite basic (coming from a python background) - but i can read the docs and see how far i get. (I may be more suited to testing!)

If i’m not too far off though, i think it’s having to set the port prior to calling the setQuickHeaders method (line 147)

github.com

caddyserver/caddy/blob/bf380d00ab62ccedcf5d7f32125bfbd4dd636d01/modules/caddyhttp/server.go#L146-L147

    
      
          	if s.h3server != nil {
          		err := s.h3server.SetQuicHeaders(w.Header())

Like maybe(?):
s.h3server.Port = <some caddy config global option. eg. "http3_port">

Which may then set the value here:

github.com

lucas-clemente/quic-go/blob/976cc29c1648949a671d4d88358f316378ee556b/http3/server.go#L443

    
      
          // CloseGracefully in combination with ListenAndServe() (instead of Serve()) may race if it is called before a UDP socket is established.
          func (s *Server) CloseGracefully(timeout time.Duration) error {
          	// TODO: implement
          	return nil
          }
          
          
// SetQuicHeaders can be used to set the proper headers that announce that this server supports QUIC.
          // The values that are set depend on the port information from s.Server.Addr, and currently look like this (if Addr has port 443):
          //  Alt-Svc: quic=":443"; ma=2592000; v="33,32,31,30"
          func (s *Server) SetQuicHeaders(hdr http.Header) error {
          	port := atomic.LoadUint32(&s.Port)
          
          
	if port == 0 {
          		// Extract port from s.Server.Addr
          		_, portStr, err := net.SplitHostPort(s.Server.Addr)
          		if err != nil {
          			return err
          		}
          		portInt, err := net.LookupPort("tcp", portStr)
          		if err != nil {
          			return err

Again, not sure how it all get’s wired up/passed through. (More reading for me required! )

francislavoie · January 20, 2022, 4:35am

Yep, exactly.

Look for how "https_port" is wired up, it would be the same if we go the route of "http3_port", which is the simplest way.

system · February 15, 2022, 11:58am

This topic was automatically closed after 30 days. New replies are no longer allowed.

1. Caddy version (caddy version):

2. How I run Caddy:

a. System environment:

b. Command:

c. Service/unit/compose file:

d. My complete Caddyfile or JSON config:

3. The problem I’m having:

4. Error messages and/or full log output:

5. What I already tried:

6. Links to relevant resources:

1. Caddy version (`caddy version`):