Combining the layer4 and http apps (SSL pass through + http file_server/reverse_proxy)

Help
5

Thanks to an old issue covering this topic in GitHub I think I fairly well understand what’s going on and the best way forward for my setup (ssl passthrough proxy + normal caddy http servers/reverse_proxies)!

A lot of interesting details are in the issue here

First, why was I getting errors with my posted config
My inintial config was trying to solve the problem by terminating TLS in layer4 and proxying to normal http setup in the http app. This wasn’t a complete solution but also strangely didn’t work!

The root cause was that curl and caddy use HTTP/2 by default but that doesn’t work when terminating in layer4 and then proxying http upstream. You get an error like:

http2 error: Remote peer returned unexpected data while we expected SETTINGS frame.  Perhaps, peer does not support HTTP/2 properly.

I don’t know the exact mechanics behind why this breaks (since the problem in the GitHub issue is not quite the same) but it can be fixed in two ways:

  1. Configure TLS handler to only advertise HTTP/1.1:
{
	"handler": "tls",
	"connection_policies": [
		{"alpn": ["http/1.1"]}
	}
}
  1. Or configure curl to only try HTTP/1.1:
    curl -Lv --http1.1 https://localhost

This is useful if your upstream doesn’t support HTTP/2 but if it does it’s not great since performance will be degraded IIUC.

It also breaks most of the automatic HTTPS functionality of caddy. You’ll be able to automate certificates via the tls app but you won’t get automatic HTTP → HTTPS redirects for example.

Luckily, the GitHub issue had the answer to that as well though it seemed it wasn’t exactly what that poster wanted. I’ll summarize the salient points:

TLS Passthrough proxy using layer4 app while serving HTTPS file_server and reverse_proxy
The main mental blocker I ran into when trying to solve this was thinking "Because layer4 must listen to :443 for TLS passthrough, I must terminate all regular HTTPS traffic there as well.

Reading the work-around in GitHub made me realize that there was no reason that the http app needed to listen to :443 to work. In fact, there are many posts/docs around how caddy should work when it can’t bind :443 directly! Just like any other service sitting in front of caddy, we just need to make sure layer4 forwards :443 to caddy for all required traffic (Our HTTPS services and ACME challenges mostly).

The key piece that the posted config in GitHub was missing was that it didn’t set https_port in the http app config which is required if we want caddy to figure out that we expect automatic HTTPS.

I’ll link my current working config below but first I’ll highlight the key steps in case my config isn’t exactly the same as others.

  1. Add a layer4 app with specific routes for handling your layer 4 traffic in special ways. In my case, I use an tls.sni matcher to forward specific HTTPS domains to a Java backend with a baked in cert.

  2. Your last route is a catch all with no matcher that should simply proxy to your chosen caddy https_port. I chose 1337. :slight_smile:

  3. Create an http app config and set https_port as before. (1337 in my case)

  4. Set up your http servers like normal using your https_port where you would have used :443

In retrospect, it’s not so unreasonable but it did take me a bit to understand what was happening end to end. Also note, that my config still has acme config set to the staging URL for testing so don’t copy that part if you already using prod CA. (My next step is to actually get this working with real URLs!)

My final config:

apps:
  layer4:
    servers:
      mdath:
        listen:
          - ':443'
        routes:
          - match:
              - tls:
                  sni:
                    - '*.e1cmyhhndp0ep.cdn.network'
            handle:
              - handler: 'proxy'
                proxy_protocol: 'v2'
                upstreams:
                  - dial:
                      - 'localhost:4433'
          - handle:
              - handler: proxy
                upstreams:
                  - dial:
                      - 127.0.0.1:1337

  http:
    https_port: 1337
    servers:
      blog.me:
        listen:
          - ':1337'
        routes:
          - match:
              - host:
                  - 'localhost'
          - handle:
              - handler: file_server
                root: /srv/popple/blog.me
      my_blog:
        listen: ['localhost:5733']
        routes:
          - handle:
              - handler: 'reverse_proxy'
                upstreams:
                  - dial: 'localhost:2368'

  tls:
    automation:
      policies:
        - issuers:
            - module: acme
              email: <email>
              ca: https://acme-staging-v02.api.letsencrypt.org/directory

logging:
  logs:
    default:
      level: 'DEBUG'

edit: Using localhost:1337 as the listen target caused connection refused errors on port 80 when I changed to using real domain names so I changed it to :1337. I had originally wanted to bind to localhost as to not expose the internal secure port but it’s fine since my firewall already blocks other ports. I’ll debug it later but I assume I just don’t understand listen.

4 Likes