How do I match tls version in layer4?

1. The problem I’m having:

TL/DR;

  • What’s the matcher syntax to specify a layer4 matcher for TLS version in my otherwise-working json config? I have sni matching working OK.

I need to proxy (without TLS termination) traffic from clients that only support TLS v1.0 (ie, ancient, insecure client apps that I cannot change). On the same port I also wish to serve regular web traffic for modern clients, reverse_proxy’d to other servers.

I have been able to configure the layer4 module to do the proxying etc that I need, but I have not been able to work out how to get it to apply a match on the TLS version, which is provided in the clienthello.

If I use a matcher for “sni” the rest of my config works, I just don’t know the syntax for matching by tls version (or if it’s definitely possible - but mholt indicated it should be

My setup generally is:

  • I build a docker image that includes the layer4 module (and installs curl)
  • I start the container with a Caddyfile that holds my “normal” config (as it’s in a file format that supports comments)
  • caddy then is able to serve my normal sites on port 443
  • I then inject my layer4 config via the API
    • this adds the layer4 app which listens on port 4433.
    • it identifies the legacy clients (currently by SNI but I need tls version) and proxies their connections to the legacy server, without performing TLS termination.
    • un-matched connections get proxied to port 443, so all my normal services are served via the http and tls apps that are defined in the main Caddyfile.

I’ve done it this way so that I can keep the simple Caddyfile config, and at my firewall I redirect incoming port 443 to caddy’s port 4433.

2. Error messages and/or full log output:

When I try to inject my layer4 config block with the command

docker-compose exec caddy curl \
  -X POST \
  -H "Content-Type: application/json" \
  -d @caddy-inject.json \
  "http://127.0.0.1:2019/config/apps/layer4"

I get:
{"error":"loading new config: loading layer4 app module: provision layer4: server 'ajgtlswrap': route 0: loading matcher modules: module name 'tls': provision layer4.matchers.tls: loading TLS matchers: module name 'version': unknown module: tls.handshake_match.version"}

This is when the caddy-inject.json snippet contains (as opposed to the “working” config using SNI specified further below):

{
  "servers": {
    "ajgtlswrap": {
      "listen": [
        ":4433"
      ],
      "routes": [
        {
          "match": [
            {
              "tls": {
                "version": [
                  "TLS10"
                ]
              }
            }
          ],
          "handle": [
            {
              "handler": "proxy",
              "upstreams": [
                {
                  "dial": [
                    "legacyserver.internal.example.com:443"
                  ]
                }
              ]
            }
          ]
        },
        {
          "handle": [
            {
              "handler": "proxy",
              "upstreams": [
                {
                  "dial": [
                    "127.0.0.1:443"
                  ]
                }
              ]
            }
          ]
        }
      ]
    }
  }
}

3. Caddy version:

I build from caddy 2.7.5 locally using this Dockerfile:

ARG VERSION

FROM caddy:$VERSION-builder-alpine AS builder

RUN xcaddy build \
    --with github.com/abiosoft/caddy-json-schema \
    --with github.com/mholt/caddy-l4

FROM caddy:$VERSION-alpine

COPY --from=builder /usr/bin/caddy /usr/bin/caddy
RUN apk add curl

4. How I installed and ran Caddy:

a. System environment:

Debian 11.8 with its docker.io 20.10.5+dfsg1-1+deb11u2
docker-compose 1.25.0-1

b. Command:

docker-compose up -d

c. Service/unit/compose file:

My docker-compose.yaml includes:

services:
  caddy:
    build:
      context: ./caddy-build
      args:
        - VERSION=2.7.5
    restart: unless-stopped
    cap_add:
      - NET_ADMIN  # Required for tweaking of UDP buffers for http/3
    ports:
      - "80:80"
      - "443:443"
      - "443:443/udp"  # HTTP/3 support(!)
      - "4433:4433"  # Gateway directs traffic here, which handles proxying.
    volumes:
      - $PWD/Caddyfile:/etc/caddy/Caddyfile
      - $PWD/caddy-srv:/srv
      - $PWD/caddy-data:/data
      - $PWD/caddy-config:/config

d. My complete Caddy config:

Note: I am redacting info here because I don’t believe its inclusion is relevant to my problem (ie all certs, redirections etc are working OK) and because my situation requires obfuscating details due to the presence of legacy systems. I can provide details by DM if they are relevant.

Initial Caddyfile loaded at startup:

{
        email hostmaster@example.com
}

home.example.com {
        reverse_proxy 192.168.35.42:8088
}
media.example.com {
        reverse_proxy jellyfin:8096
}

“Injected” l4 config which DOES work, but I need to match on TLS version, not on SNI:

{
    "servers": {
      "ajgtlswrap": {
        "listen": [
          ":4433"
        ],
        "routes": [
          {
            "match": [
              {
                "tls": {
                  "sni": [
                    "legacy.example.com"
                  ]
                }
              }
            ],
            "handle": [
              {
                "handler": "proxy",
                "upstreams": [
                  {
                    "dial": [
                      "legacyserver.internal.example.com:443"
                    ]
                  }
                ]
              }
            ]
          },
          {
            "handle": [
              {
                "handler": "proxy",
                "upstreams": [
                  {
                    "dial": [
                      "127.0.0.1:443"
                    ]
                  }
                ]
              }
            ]
          }
        ]
      }
    }
}

5. Links to relevant resources:

This is essentially a feature request, I recommend opening an issue on the caddy-l4 repo. It should be simple to add tls > version matching.

Maybe you’d like to contribute a patch? Should be pretty easy, even if you don’t have any Go experience.

Thanks for taking a look!

I raised an issue back in May (Tunnelling legacy TLS1.0 clients to another host, terminating modern clients · Issue #119 · mholt/caddy-l4 · GitHub) and the impression I got was that it was already possible, but I may have misread the response. At any rate I’m not in a position to sponsor for this, but I am willing to have a crack at implementing it.

I’ve no Go experience, and I was unable to work out from the codebase how matchers were implemented - any time I look at Go code I feel like I’m not seeing several layers of abstraction, with key parts appearing to be done magically elsewhere :smile:

To be honest I couldn’t tell that it wasn’t implemented, and found the comment around https://github.com/mholt/caddy-l4/blob/a362a1fbf6524336ce3bd01879cb45493d9ed33f/modules/l4tls/matcher.go#L105 a bit confusing re where things happen. A few lines before there the version gets added to a “replacer” but I’ve no idea what that is or where/how it’s used…

Happy to have a go at it but I might need a little guidance, I’m afraid!

I think Matt was just saying “yes you can proxy raw TLS bytes regardless of the version”, not commenting on matching by version.

Look at Modules - Caddy Documentation for tls.handshake_match, those are the currently supported matchers. There isn’t one for version currently, only ALPN, SNI and Remote IP.

I think the matcher should probably be implemented in Caddy proper, because SNI & Remote IP are from there.

I’m not actually sure why the ALPN matcher is in caddy-l4, it should maybe be in Caddy itself :thinking: /cc @matt

Anyway, read Extending Caddy — Caddy Documentation to get an idea of how modules work. Basically we just need a new struct which has a CaddyModule() with ID tls.handshake_match.version and implements a Match(hello *tls.ClientHelloInfo) bool function (comparing against hello.SupportedVersions).

You can find the existing matchers here caddy/modules/caddytls/matchers.go at master · caddyserver/caddy · GitHub

1 Like

(edit: I posted before I saw your edit, will take a look at the extra pointers you gave, thanks…)

I think Matt was just saying “yes you can proxy raw TLS bytes regardless of the version”, not commenting on matching by version.

That’s fair, I might have just read more into “and you should be able to match TLS 1.0 connections” than was intended.

Yes I am a bit confused about the difference/relationship(s) between say layer4.matchers.tls and tls.handshake_match, and how matching “sni” inside a “layer4” application block gets handled by… the tls module? And the fact that there is no documentation further inside layer4.matchers.tls had me scratching my head trying to work out what params you can pass to a tls matcher.

That is:

           "match": [
              {
                "tls": {
                  "sni": [
                    "legacy.example.com"
                  ]
                }
              }

Isn’t a documented option of layer4.matchers.tls in the Docs site you linked, but it looks like it gets remapped/imported (I don’t know the terminology to use) to the core tls.handshake_match matcher. I think I might be misunderstanding some core concepts of how the overall app glues together and how modules interact.

I’m normally able to fumble my way around a codebase, but being quite unfamiliar with Go and the highly modular nature of Caddy has me a bit lost.

1 Like

layer4.matchers.tls is a module that provides a “layer4 matcher” (i.e. only usable in l4, and not usable in the http app) which is scoped to “tls”. It sources the actual matchers from tls.handshake_match.* matchers dynamically.

In the config you configure match > tls > (module_name) essentially, where sni in your quoted config is a module name.

So it loads the tls.handshake_match.sni module to perform that match, and the value of the sni key is passed to that module as its input.

Basically, everything(ish) is a module, in Caddy. For example, layer4 is an “app module”, and it calls down to other modules to do actual work, like matcher modules and handler modules.

It might also help to read Architecture — Caddy Documentation

Modules in a particular namespace are expected to implement a particular interface (i.e. have certain methods available). For example vanilla Caddy has all these module namespaces & associated interfaces Module Namespaces — Caddy Documentation (l4 ones aren’t covered here because it is itself a plugin and the README mostly covers it anyway).

1 Like

Thanks so much for taking the time to walk me through this, much appreciated.

Things make a lot more sense to me now (often a warning sign, I find - but fingers crossed!).

So I am thinking that I should be able to…

  • in caddy-layer4/modules/l4tls/
  • create a new version_matcher.go based heavily on alpn_matcher.go in the same directory, and your notes above re ID and Match etc.

Since I need to match versions of tls that caddy doesn’t support, I suspect I need to match against ClientHelloInfo.Version, which gets applied in caddy-l4’s clienthello.go rather than in caddy “core”, does that sound right?

I need to head off for a bit, but should be able to take a first crack at this in a couple of hours. Thanks again for your guidance.

2 Likes

Yeah, sounds about right.

FWIW, I recommend using VSCode, the Go plugin is excellent and does a lot of auto-completion for you.

2 Likes

Thanks, I was planning on using vscode and indeed it makes things a lot easier.

I’ve got a new matcher and it shows up in xcaddy list-modules --versions:

...
layer4.proxy.selection_policies.random v0.0.0-20231016112149-a362a1fbf652
layer4.proxy.selection_policies.random_choose v0.0.0-20231016112149-a362a1fbf652
layer4.proxy.selection_policies.round_robin v0.0.0-20231016112149-a362a1fbf652
tls.handshake_match.alpn v0.0.0-20231016112149-a362a1fbf652
tls.handshake_match.version v0.0.0-20231016112149-a362a1fbf652

  Non-standard modules: 27
...

I’ve pushed it up to my fork on github to test, but when I run xcaddy build from my test env I get errors about the module self-identifying as github.com/mholt/caddy-l4:

....
go: added github.com/abiosoft/caddy-json-schema v0.0.0-20220621031927-c4d6e132f3af
2023/11/04 13:47:02 [INFO] exec (timeout=0s): /usr/local/go/bin/go get -d -v github.com/agittins/caddy-l4 github.com/caddyserver/caddy/v2@v2.7.5
go: downloading github.com/agittins/caddy-l4 v0.0.0-20231104133937-2c17618df591
go: github.com/agittins/caddy-l4@v0.0.0-20231104133937-2c17618df591 found: parsing go.mod:
        module declares its path as: github.com/mholt/caddy-l4
                but was required as: github.com/agittins/caddy-l4
go: github.com/agittins/caddy-l4@upgrade (v0.0.0-20231104133937-2c17618df591) requires github.com/agittins/caddy-l4@v0.0.0-20231104133937-2c17618df591: parsing go.mod:
        module declares its path as: github.com/mholt/caddy-l4
                but was required as: github.com/agittins/caddy-l4
2023/11/04 13:47:04 [FATAL] exit status 1
ERROR: Service 'caddy' failed to build: The command '/bin/sh -c xcaddy build     --with github.com/abiosoft/caddy-json-schema     --with github.com/agittins/caddy-l4' returned a non-zero code: 1

I couldn’t see any flags for xcaddy that might be a simple fix, and I see that the repo path is scattered all through the module in the imports. I tried setting the import in my go.mod but that threw up more errors, so I suspect I am on the wrong track there.

Is there a simple thing I need to do to allow it to build from my fork, or do I need to search and replace every import path?

2 Likes

Aha! You already answered here: Build from source and include custom modification - #2 by francislavoie

:smiley:

2 Likes

Wow… it all works!

I’ve tested on my server using my fork (by the way, for anyone playing along at home, to use a fork of a module with xcaddy, you need to include a version tag for the replacement repo, unless you are using a local path. I used --with github.com/mholt/caddy-l4=github.com/agittins/caddy-l4@latest which made it happy).

My caddy is now matching and proxying connections based on their TLS version, without terminating tls itself.

I’ve raised a PR for the change: Add tls.handshake_match.version matcher by agittins · Pull Request #155 · mholt/caddy-l4 · GitHub

2 Likes

:grin:

Awesome! Now you get to say you have Go in your back pocket :sweat_smile:

2 Likes

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.