Can't connect to database behind layer4 server

1. Caddy version (caddy version):

2.5.1

2. How I run Caddy:

I run Caddy as a docker container. I build the image for this container like this:

FROM caddy:2.5.1-builder AS builder

RUN xcaddy build \
    --with github.com/mholt/caddy-l4/layer4 \
    --with github.com/mholt/caddy-l4/modules/l4tls \
    --with github.com/mholt/caddy-l4/modules/l4proxy \
    --with github.com/mholt/caddy-l4/modules/l4proxyprotocol \
    --with github.com/mholt/caddy-l4/modules/l4echo

FROM caddy:2.5.1

COPY --from=builder /usr/bin/caddy /usr/bin/caddy

a. System environment:

Docker Desktop running on Windows 10

b. Command:

I run Caddy through the docker compose using

docker-compose up

c. Service/unit/compose file:

Here is a subset of the docker compose I use to run the Caddy container

services:
  hxd-proxy:
    container_name: hxd-proxy
    restart: ${RESTART_CONTAINERS}
    image: "hyperxdashboard-caddy:latest"
    ports:
      - 80:80
      - 443:443
      - 3317:3317
    volumes:
      - ./temp/caddy-config.json:/etc/caddy/caddy-config.json
      - ./certs/hx-dashboard.crt:/etc/caddy/hx-dashboard.crt
      - ./certs/hx-dashboard.key:/etc/caddy/hx-dashboard.key
      - hxd-dashboard-proxy:/data
    command: ["caddy", "run", "--config", "/etc/caddy/caddy-config.json"]
    networks:
      - hxd-network 
    extra_hosts:
      - "${DASH_HOSTNAME}:${DASH_IP}" 

d. My complete Caddyfile or JSON config:

{
{
  "apps": {
    "http": {
      "servers": {
        "srv0": {
          "listen": [":443"],
          "routes": [
            {
              "match": [
                {
                  "path": ["/auth*"]
                }
              ],
              "handle": [
                {
                  "handler": "reverse_proxy",
                  "upstreams": [
                    {
                      "dial": "hxd-keycloak:14040"
                    }
                  ]
                }
              ]
            },
            {
              "match": [
                {
                  "path": ["/api*"]
                }
              ],
              "handle": [
                {
                  "handler": "reverse_proxy",
                  "upstreams": [
                    {
                      "dial": "hxd-backend:35500"
                    }
                  ]
                }
              ]
            },
            {
              "handle": [
                {
                  "handler": "reverse_proxy",
                  "upstreams": [
                    {
                      "dial": "hxd-frontend:8080"
                    }
                  ]
                }
              ]
            }
          ],
          "tls_connection_policies": [
            {
              "certificate_selection": {
                "any_tag": ["cert0"]
              }
            }
          ]
        },
        "srv1": {
          "listen": [":80"],
          "routes": [
            {
              "handle": [
                {
                  "handler": "static_response",
                  "headers": {
                    "Location": [
                      "https://{http.request.host}{http.request.uri}"
                    ]
                  },
                  "status_code": 302
                }
              ]
            }
          ]
        }
      }
    },
    "layer4": {
        "servers": {
            "secure-imap": {
                "listen": [":3317"],
                "routes": [
                    {
                        "match": [
                            {
                                "tls": {
                                    "sni": ["r92"]
                                }
                            }
                        ],
                        "handle": [
                            {"handler": "tls"},
                            {"handler": "echo"},
                            {
                              "handler": "proxy",
                              "proxy_protocol": "v2",
                              "upstreams": [
                                {
                                  "dial": ["hxd-mariadb:3306"]
                                }
                              ]
                            }
                          ]
                    }
                ]
            }
        }
    },
    "tls": {
      "certificates": {
        "load_files": [
          {
            "certificate": "/etc/caddy/hx-dashboard.crt",
            "key": "/etc/caddy/hx-dashboard.key",
            "tags": ["cert0"]
          }
        ]
      }
    }
  }
}

3. The problem I’m having:

When I try to connect to my mariadb database using DBeaver (a database manager), DBeaver seems to get stuck trying to connect (it just shows a loading bar at 0% for 5+ minutes). No logs show up on the Caddy container during this time.

I’ve also tried connecting manually via command line using the command below (r92 is the hostname associated with my TLS cert, and points to my laptop).

mysql -u root --password=password -h r92 -P 3317

This command acts in a similar way to DBeaver, where it just gets stuck and doesn’t yield any output.

If I forcibly kill this command using CTRL+C, I get this output from Caddy:

hxd-proxy     | {"level":"error","ts":1654096574.4075031,"logger":"layer4","msg":"matching connection","error":"EOF"}

I can connect to the database if I expose the mariadb container to my laptop network and connect to the database directly. However, I’d like to hide the mariadb container behind Caddy so that I can encrypt traffic to and from the database using TLS.

It’s very possible the problem lies with how I’ve configured my mariadb container. However, if someone could take a look at my layer4 Caddy config and see if I’ve made any obvious configuration errors there, I would greatly appreciate it!

1 Like

Layer 4 stuff can be a little difficult to debug, because we don’t usually know what the clients are doing. For example, I am not sure that by specifying -h to the mysql command, it is necessarily using that as the SNI for the TLS name.

I would start by reducing your config to the minimal working example and removing all the other config parameters for now – and eliminating the container as complexity – and once you get even the basic, bare-bones config working (no TLS, etc.) then slowly add in functionality one thing at a time until it breaks.

2 Likes

Thanks Matt! I’ll work on reproducing my issue within a simpler setup. Just as a sanity check, does my layer4 configuration look right? I want Caddy to listen for traffic on port 3317, terminate TLS, and then forward the traffic to host hxd-mariadb on port 3306.

    "layer4": {
        "servers": {
            "secure-imap": {
                "listen": [":3317"],
                "routes": [
                    {
                        "match": [
                            {
                                "tls": {
                                    "sni": ["r92"]
                                }
                            }
                        ],
                        "handle": [
                            {"handler": "tls"},
                            {"handler": "echo"},
                            {
                              "handler": "proxy",
                              "proxy_protocol": "v2",
                              "upstreams": [
                                {
                                  "dial": ["hxd-mariadb:3306"]
                                }
                              ]
                            }
                          ]
                    }
                ]
            }
        }
    },
1 Like

As far as I can tell from a quick Google search, MySQL and MariaDB don’t support TLS-SNI at all. So you cannot use the sni matcher here.

3 Likes

Yeah, so like Francis said, it’s hard to know whether the config looks right, because I don’t know MariaDB’s network implementation. Sounds like it won’t work if they don’t support SNI, but you’d have to ask MariaDB’s community, documentation, or developers to know for sure.

I do notice that you have an echo handler before the proxy handler, so the bytes will be echoed instead of getting proxied.

(Besides, I imagine that echoing the bytes to the DB client will confuse it.)

1 Like

I tried reducing the complexity of my setup by removing all but the mariadb and proxy containers, and removing everything from the Caddy config except the layer4-related config. I also removed the sni matcher, the echo handler, and the proxy_protocol line from the config.

This works! I can connect to the database just fine.

Working config:

{
  "apps": {
    "layer4": {
        "servers": {
            "secure-imap": {
                "listen": [":3317"],
                "routes": [
                    {
                        "handle": [
                            {
                              "handler": "proxy",
                              "upstreams": [
                                {
                                  "dial": ["hxd-mariadb:3306"]
                                }
                              ]
                            }
                        ]
                    }
                ]
            }
        }
    }
  }
}

If I keep this same setup but try to add TLS, my problem returns and I cannot connect to the database (same symptoms as before, where the connection just loads indefinitely).

Problematic config:

{
  "apps": {
    "layer4": {
        "servers": {
            "secure-imap": {
                "listen": [":3317"],
                "routes": [
                    {
                        "handle": [
                            {"handler": "tls"},
                            {
                              "handler": "proxy",
                              "upstreams": [
                                {
                                  "dial": ["hxd-mariadb:3306"]
                                }
                              ]
                            }
                        ]
                    }
                ]
            }
        }
    },
    "tls": {
      "certificates": {
        "load_files": [
          {
            "certificate": "/etc/caddy/hx-dashboard.crt",
            "key": "/etc/caddy/hx-dashboard.key",
            "tags": ["cert0"]
          }
        ]
      }
    }
  }
}

I found a mysql proxy that terminates encryption and then proxies to a mysql database. In the description of how it works, it says the proxy removed the “client supports ssl” flag. Maybe the mariadb client connects to Caddy using this flag, but then runs into trouble when it tries to pass that flag along to mariadb. If that’s the case, I doubt there’s much I can do to get the database to work behind the proxy. Perhaps I can just configure mariadb to use TLS with the same certs that Caddy is using instead of routing traffic through Caddy to the database.

1 Like

As you can see in this line parses the proxy the MySQL package and search for the SSL flag

As you can see in this ticket MySQL Bugs: #104649: Contribution: Add the TLS SNI extension when connecting over TLS to a serve ... isn’t there too much progress to add SNI into MySQL.

The situation looks similar to the MariaDB [MDEV-10658] Support TLS SNI in MariaDB - Jira

At the end maybe you can make a pull request to add the mysql-proxy feature to GitHub - mholt/caddy-l4: Layer 4 (TCP/UDP) app for Caddy :slight_smile:

2 Likes

Probably best if someone who actually needs or will use it contributes the change, but yeah I’m sure Matt would be open to it. Seems simple enough.

1 Like

full ack. the message was for the origin poster @PWGCA :smiley:

1 Like

Yep, could definitely accept a contribution for l4 to have special support for detecting mysql.

1 Like

This topic was automatically closed after 30 days. New replies are no longer allowed.