Custom certificates for TLS termination in layer4 app

Help
1

1. The problem I’m having:

I’m am trying to set up a TCP reverse proxy for MQTT on port 8883 that terminates TLS. Therefore I am using the layer4 app. I would like to use custom generated certificates for that.
The problem I am having is that I have not found a way to configure my own certificates for the TLS server. I would have expected to use a directive tls <cert_file> <key_file>, but that does not seem to work inside the layer4 block.

How can I configure custom certificates for TLS termination in the layer4 app?

2. Error messages and/or full log output:

3. Caddy version:

I run caddy built with the caddy-l4 and the caddy-docker-proxy inside docker compose.

caddy version
v2.8.4 h1:q3pe0wpBj1OcHFZ3n/1nl4V4bxBrYoSoab7rL9BMYNk=

4. How I installed and ran Caddy:

a. System environment:

Inside a docker container, Caddy was built with

xcaddy build \
    --with github.com/mholt/caddy-l4 \ 
    --with github.com/lucaslorentz/caddy-docker-proxy/v2

c. Service/unit/compose file:

services:
  caddy:
    container_name: reverse-proxy
    image: my_caddy:1.0
    networks:
      - app-network
    environment:
      - CADDY_DOCKER_CADDYFILE_PATH=/etc/caddy/Caddyfile
      - CADDY_INGRESS_NETWORKS=app-network
    volumes:
      - ./tls.crt:/etc/ssl/tls.crt:ro
      - ./tls.key:/etc/ssl/tls.key:ro
      - ./Caddyfile:/etc/caddy/Caddyfile:ro
      - /var/run/docker.sock:/var/run/docker.sock:ro
      - ./data:/data
    ports:
      - 1883:1883
      - 8883:8883

  mosquitto:
    container_name: mosquitto
    image: eclipse-mosquitto:2
    networks:
      - app-network
    volumes:
      - ./mosquitto.conf:/mosquitto/config/mosquitto.conf

networks:
  app-network:
    name: app-network

d. My complete Caddy config:

localhost 

{
    local_certs
    skip_install_trust

    layer4 {
        :1883 {
            route {
                proxy mosquitto:1883
            }
        }
        :8883 {
            route {
                tls
                proxy mosquitto:1883
            }
        }
    }
}

5. Links to relevant resources:

2

Howdy @rcm25, welcome to the Caddy community.

Doesn’t look like you can do this with a Caddyfile, or if you can, it’s not documented. Might be worth opening a feature request at Issues · mholt/caddy-l4 · GitHub.

I felt like seeing if I could do this in JSON, though, and I can see that layer4 provides a connection_policies field identical to the HTTP server’s tls_connection_policies.

That means we can use much the same logic as the HTTP Caddyfile but port it over to JSON.

So, I took your Caddyfile config (by the way, it didn’t validate for me with localhost at the top because Caddy treats that as a site address for the obviously global option block below it) and adapted it to JSON:

Caddyfile adapted to JSON 
{
  "apps": {
    "layer4": {
      "servers": {
        "srv0": {
          "listen": [
            ":1883"
          ],
          "routes": [
            {
              "handle": [
                {
                  "handler": "proxy",
                  "upstreams": [
                    {
                      "dial": [
                        "mosquitto:1883"
                      ]
                    }
                  ]
                }
              ]
            }
          ]
        },
        "srv1": {
          "listen": [
            ":8883"
          ],
          "routes": [
            {
              "handle": [
                {
                  "handler": "tls"
                },
                {
                  "handler": "proxy",
                  "upstreams": [
                    {
                      "dial": [
                        "mosquitto:1883"
                      ]
                    }
                  ]
                }
              ]
            }
          ]
        }
      }
    },
    "pki": {
      "certificate_authorities": {
        "local": {
          "install_trust": false
        }
      }
    },
    "tls": {
      "automation": {
        "policies": [
          {
            "issuers": [
              {
                "module": "internal"
              }
            ]
          }
        ]
      }
    }
  }
}

Then I took a simple Caddyfile:

example.com
tls cert key

And adapted that too:

tls cert key adapted to JSON 
{
  "apps": {
    "http": {
      "servers": {
        "srv0": {
          "listen": [
            ":443"
          ],
          "routes": [
            {
              "match": [
                {
                  "host": [
                    "example.com"
                  ]
                }
              ],
              "terminal": true
            }
          ],
          "tls_connection_policies": [
            {
              "match": {
                "sni": [
                  "example.com"
                ]
              },
              "certificate_selection": {
                "any_tag": [
                  "cert0"
                ]
              }
            },
            {}
          ]
        }
      }
    },
    "tls": {
      "certificates": {
        "load_files": [
          {
            "certificate": "cert",
            "key": "key",
            "tags": [
              "cert0"
            ]
          }
        ]
      }
    }
  }
}

So we can see it does a few specific things.

Under the tls field we see it loads the file and gives it a tag:

      "certificates": {
        "load_files": [
          {
            "certificate": "cert",
            "key": "key",
            "tags": [
              "cert0"
            ]
          }
        ]
      }

Then, we can see under the http app that we’re specifying this tagged cert for certificate selection:

          "tls_connection_policies": [
            {
              "match": {
                "sni": [
                  "example.com"
                ]
              },
              "certificate_selection": {
                "any_tag": [
                  "cert0"
                ]
              }
            },
            {}
          ]

I don’t think you need to match on SNI here since it seems like you just want this port for the one upstream, so you can just skip match and use certificate_selection.

So if we take the certificates > load files as well as the connection_policies adpated from the tls cert key Caddyfile and put them in the JSON we adapted from your Caddyfile, we get something that looks like:

{
  "apps": {
    "layer4": {
      "servers": {
        "srv0": {
          "listen": [
            ":1883"
          ],
          "routes": [
            {
              "handle": [
                {
                  "handler": "proxy",
                  "upstreams": [
                    {
                      "dial": [
                        "mosquitto:1883"
                      ]
                    }
                  ]
                }
              ]
            }
          ]
        },
        "srv1": {
          "listen": [
            ":8883"
          ],
          "routes": [
            {
              "handle": [
                {
                  "handler": "tls",
                  "connection_policies": [
                    {
                      "certificate_selection": {
                        "any_tag": [
                          "cert0"
                        ]
                      }
                    }
                  ]
                },
                {
                  "handler": "proxy",
                  "upstreams": [
                    {
                      "dial": [
                        "mosquitto:1883"
                      ]
                    }
                  ]
                }
              ]
            }
          ]
        }
      }
    },
    "pki": {
      "certificate_authorities": {
        "local": {
          "install_trust": false
        }
      }
    },
    "tls": {
      "certificates": {
        "load_files": [
          {
            "certificate": "/etc/ssl/tls.crt",
            "key": "/etc/ssl/tls.key",
            "tags": [
              "cert0"
            ]
          }
        ]
      },
      "automation": {
        "policies": [
          {
            "issuers": [
              {
                "module": "internal"
              }
            ]
          }
        ]
      }
    }
  }
}
Or a YAML variant for use with github.com/abiosoft/caddy-yaml 
apps:
  layer4:
    servers:
      srv0:
        listen:
        - ":1883"
        routes:
        - handle:
          - handler: proxy
            upstreams:
            - dial:
              - mosquitto:1883
      srv1:
        listen:
        - ":8883"
        routes:
        - handle:
          - handler: tls
            connection_policies:
            - certificate_selection:
                any_tag:
                - cert0
          - handler: proxy
            upstreams:
            - dial:
              - mosquitto:1883
  pki:
    certificate_authorities:
      local:
        install_trust: false
  tls:
    certificates:
      load_files:
      - certificate: "/etc/ssl/tls.crt"
        key: "/etc/ssl/tls.key"
        tags:
        - cert0
    automation:
      policies:
      - issuers:
        - module: internal

Which I imagine might just make it work for you.

4 Likes
3

Howdy @Whitestrake ,

thank you very much for your answer and the time you invested! I just tried it out and it works as expected.

However, my docker-compose configuration is a bit more complex and I would like to use the caddy-docker-proxy module to dynamically add/remove containers with labels that change the Caddy configuration.
As far as I’ve seen, this only works with the Caddyfile and not the json configuration (and using both at the same time is also not possible I think).

As a workaround, I could probably just run two docker containers with Caddy, one configured via the Caddyfile using caddy-docker-proxy and forwarding all MQTT traffic to a second container configured with the json file you suggested and doing the TLS termination.

Maybe I’ll open a feature request at caddy-l4 as well.

1 Like
4

Yep, that’s a big downside in this case. I did note that you had CDP compiled in, but were still loading a Caddyfile directly, so I wasn’t sure and figured I’d give the JSON config a shot anyway.

I know that CDP allows you to supply a base configuration with which to extend from, but the base config needs to be Caddyfile as well.

Your workaround sounds like a good one - but I don’t think you really even need to have one in front of the other at all. You can have your Caddy HTTP server on ports 80/443 and your Caddy L4 server on ports 1883 and 8883. If you DO need one in front of the other, I think it’d have to be the L4 server, because while L4 can handle HTTP and proxy on to the HTTP server, the HTTP server can’t handle passing TCP through to the L4 server, so it’s gotta be one way around.

I’m sure you won’t be the only one who’d benefit from this little feature addition, and I can’t imagine it would be a huge effort to add to the L4 Caddyfile, so definitely put that feature request in, I think it’d be a win.

2 Likes
5

You’re right, I could just set up two independent reverse proxies, one for ports 80/443 and one for ports 1883/8883.

I opened a feature request: Support for custom certificates in TLS handler · Issue #244 · mholt/caddy-l4 (github.com)

1 Like
6

Epic answer. Thanks for helping out so much!

2 Likes