How to combine layer4 module with tls and http modules?

Help
1

1. Caddy version (caddy version):

I’m using a docker image livekit/caddyl4:v2.5, which is basically Caddyl4 and caddy-yaml bundled together.

2. How I run Caddy:

In a docker container.

a. System environment:

Running docker with docker-compose v2.2.3 on Ubuntu 20.04.1 LTS.

b. Command:

docker-compose up

c. Service/unit/compose file:

version: "3.9"
services:
  caddy:
    image: livekit/caddyl4
    command: run --config /etc/caddy.yaml --adapter yaml
    restart: unless-stopped
    network_mode: "host"
    volumes:
      - ./caddy.yaml:/etc/caddy.yaml
      - ./caddy_data:/data

d. My complete Caddyfile or JSON config:

logging:
  logs:
    default:
      level: INFO
storage:
  "module": "file_system"
  "root": "/data"
apps:
  tls:
    certificates:
      automate:
        - livekit.spvw.de
        - livekit-turns.spvw.de
  layer4:
    servers:
      main:
        listen: [":443"]
        routes:
          - match:
            - tls:
                sni:
                  - "livekit-turns.spvw.de"
            handle:
              - handler: tls
              - handler: proxy
                upstreams:
                  - dial: ["localhost:5349"]
          - match:
              - tls:
                  sni:
                    - "livekit01.spvw.de"
            handle:
              - handler: tls
                connection_policies:
                  - alpn: ["http/1.1"]
              - handler: proxy
                upstreams:
                  - dial: ["localhost:7880"]

3. The problem I’m having:

Basically, I’m trying to achieve the following logic in pseudocode:

listen_on(443)
if (request.sni == "livekit-turns.spvw.de") {
    pass_to("localhost:5349")
} else if (request.sni == "livekit.spvw.de") {
    if (http.path == "/api") {
        add_header("Access-Control-Allow-Origin", "*")
        pass_to("localhost:8080")
    } else {
        pass_to("localhost:7880")
    }
}

Currently, with my config, it works as if I had the following logic instead:

listen_on(443)
if (request.sni == "livekit-turns.spvw.de") {
    pass_to("localhost:5349")
} else if (request.sni == "livekit.spvw.de") {
    pass_to("localhost:7880")
}

4. Error messages and/or full log output:

No errors, I’m just trying to figure out how to achieve the logic that I have in mind.

5. What I already tried:

I’ve read:

  • Getting started
  • Quick-starts
  • Caddy API
  • JSON Config Structure

sections from the official documentation.

However, I have not found a way to code the logic in question. I’ve gone to JSON Config Structure → apps → layer4 → handle. The list of modules available for handle doesn’t include any http options, so I’m not sure how can I add an additional match section inside the handler that would use the http module, filter things by the path and route them to different local services (optionally adding certain CORS headers to it). I’m unsure if the documentation has something missing or if I’m just searching in the wrong place or doing something wrong. Any help would be appreciated :slight_smile:

Basically, I want the internal handle section of the last match of the livekit01.spvw.de to correspond to the following nginx code if it makes any sense:

http {
    server {
        listen 443 ssl http2;
        server_name livekit01.spvw.de;
        location / {
            proxy_pass localhost:7880;
        }
        location /api {
            add_header 'Access-Control-Allow-Origin' '*';
            proxy_pass localhost:8080
        }
    }
}

(I approximately know how to write the given nginx logic for a single host without an SNI, i.e. without the layer4 module that I’m currently using, but I’m not sure how to write this logic in combination with the logic that I’ve had so far that respects the SNI etc)

Sorry, a caddy newbie here :slight_smile:

6. Links to relevant resources:

2

Are you sure you need Caddy-l4? If all the traffic is HTTP, then you can just use the HTTP app instead.

What you can do though is make an HTTP app which does what you need for your second domain, listening on some unused port, then proxy from l4 to the HTTP app.

3

If it’s all HTTP, you could just write a Caddyfile like this:

livekit-turns.spvw.de {
    reverse_proxy localhost:5349
}

livekit.spvw.de {
    handle /api* {
        header Access-Control-Allow-Origin *
        reverse_proxy localhost:8080
    }

    handle {
        reverse_proxy localhost:7880
    }
}
1 Like
4

Thanks for the quick reply :slight_smile:

It’s unfortunately not HTTP, the incoming traffic is always TLS traffic, so the only way for me to differentiate 2 services is to rely upon the SNI to proxy the traffic. I.e. one of the domains corresponds to the so-called TURN-over-TLS traffic, so I need to perform a TLS termination and then forward it to one application, whereas if the SNI is different, I should treat it as HTTP(s) traffic.

What you can do though is make an HTTP app that does what you need for your second domain, listening on some unused port, then proxy from l4 to the HTTP app.

Ah, that’s a nice idea! So, do I get it right: the idea is to leave the config as is, but add another app (or rather server) that listens on that port that receives the forwarding and performs this whole HTTP matching logic there?

1 Like
5

Yeah, you would set up an http app (sibling to your layer4 app) which has a server which listens on port (let’s say) :8443. You’d add routes to that server which match on the host (optional I guess but doesn’t hurt) and uses a header handler to set the field you want, and a reverse_proxy handler to send the request to the upstreams you want.

Then you’d simply proxy in the layer4 app to localhost:8443. You could either terminate TLS in the layer4 app or in the http app, it’s up to you, it probably won’t make a big difference. I.e. you don’t necessarily need handler: tls inside the layer4 app for that hostname, because the http app can terminate itself.

If you want a starting point for the HTTP app, take the Caddyfile I wrote for you above and run caddy adapt --pretty --config Caddyfile to see the JSON it adapts to. You can turn that into the YAML you need.

Another thing to think about, with this pattern, if you need to preserve the connecting client’s IP, you’d have to use the PROXY protocol so that the original client IP is preserved when hopping from layer4 to http; otherwise, the IP will always look like it’s coming from 127.0.0.1 or ::1. The layer4 proxy has built-in support to send PROXY upstream, but for the HTTP app you’ll need this plugin GitHub - mastercactapus/caddy2-proxyprotocol which you configure as a listener_wrapper for the server.

2 Likes
6

Thanks!

I’ve been finally able to implement the desired logic. In case readers of this issue are looking for a final result, my final result looks like this:

logging:
  logs:
    default:
      level: INFO
storage:
  "module": "file_system"
  "root": "/data"
apps:
  tls:
    certificates:
      automate:
        - livekit01.spvw.de
        - livekit-turns.spvw.de
  layer4:
    servers:
      main:
        listen: [":443"]
        routes:
          - match:
            - tls:
                sni:
                  - "livekit-turns.spvw.de"
            handle:
              - handler: tls
              - handler: proxy
                upstreams:
                  - dial: ["localhost:5349"]
          - match:
              - tls:
                  sni:
                    - "livekit01.spvw.de"
            handle:
              - handler: tls
                connection_policies:
                  - alpn: ["http/1.1"]
              - handler: proxy
                upstreams:
                  - dial: ["localhost:8000"]
  http:
    servers:
      api:
        listen:
        - ":8000"
        routes:
        - handle:
          - handler: subroute
            routes:
              - group: main-group
                match:
                  - path:
                    - "/api*"
                handle:
                  - handler: headers
                    response:
                      set:
                        Access-Control-Allow-Origin:
                          - "*"
                  - handler: headers
                    response:
                      set:
                        Access-Control-Allow-Methods:
                          - "GET, POST, PUT, DELETE, OPTIONS"
                  - handler: reverse_proxy
                    rewrite:
                      strip_path_prefix: "/api"
                    upstreams:
                      - dial: "localhost:8080"
              - group: main-group
                handle:
                  - handler: reverse_proxy
                    upstreams:
                      - dial: "localhost:7880"

I must admit, that the Caddyfile version looks way more readable and akin to Nginx :slight_smile: I wonder if my YAML version could be written simpler with less boilerplate. I would actually prefer to use Caddyfile, but as far as I got it (after reading all pages from the official docs apart from references for some modules that I don’t use), the Caddyfile is not as flexible as JSON (or any other adapter) and so writing the desired logic (like in my case) with Caddyfile is simply not possible due to the layer4 module not being available.

1 Like
7

Yeah. We want to improve the situation with layer4… We might later integrate it into Caddy’s standard distribution, which would make it possible to have the Caddyfile to directly support TCP/UDP servers.

But for now, you could check out caddy-ext/layer4 at master · RussellLuo/caddy-ext · GitHub which provides Caddyfile support for layer4 via global options. It’s not perfect, but it might be all you need for now.

4 Likes
8

This topic was automatically closed after 30 days. New replies are no longer allowed.