`caddy` w/ `caddy-l4` and proxying udp

1. The problem I’m having:

I am trying to setup headscale[1] behind caddy with caddy-l4 module/plugin. I have been successful in proxying the tcp ports, but I am struggling with the udp port.

What I am trying to achieve:

  • udp/3478 (caddy)udp/13478 (headscale)

2. Error messages and/or full log output:

N/A

From what I observe in journalctl -u caddy --no-pager | less +G there are no log statements indicating traffic on udp/3478 is proxied to 127.0.0.1:13478

3. Caddy version:

2.11.4

4. How I installed and ran Caddy:

Installed using NixOS module – nixpkgs/nixos/modules/services/web-servers/caddy/default.nix at nixos-26.05 · NixOS/nixpkgs · GitHub, which generates configuration and systemd service unit.

a. System environment:

NixOS 26.11 (x86_64), caddy managed with systemd

b. Command:

# from systemd service unit
ExecStart=/nix/store/9nf7n51c8y8h9pyjw4b8jclh5m85zqyq-caddy-2.11.4/bin/caddy run --environ --config /etc/caddy/Caddyfile

c. Service/unit/compose file:

# caddy.service
#
# For using Caddy with a config file.
#
# Make sure the ExecStart and ExecReload commands are correct
# for your installation.
#
# See https://caddyserver.com/docs/install for instructions.
#
# WARNING: This service does not use the --resume flag, so if you
# use the API to make changes, they will be overwritten by the
# Caddyfile next time the service is restarted. If you intend to
# use Caddy's API to configure it, add the --resume flag to the
# `caddy run` command or use the caddy-api.service file instead.

[Unit]
Description=Caddy
Documentation=https://caddyserver.com/docs/
After=network.target network-online.target
Requires=network-online.target

[Service]
Type=notify
User=caddy
Group=caddy
ExecStart=/nix/store/9nf7n51c8y8h9pyjw4b8jclh5m85zqyq-caddy-2.11.4/bin/caddy run --environ --config /etc/caddy/Caddyfile
ExecReload=/nix/store/9nf7n51c8y8h9pyjw4b8jclh5m85zqyq-caddy-2.11.4/bin/caddy reload --config /etc/caddy/Caddyfile --force
TimeoutStopSec=5s
LimitNOFILE=1048576
PrivateTmp=true
ProtectSystem=full
AmbientCapabilities=CAP_NET_ADMIN CAP_NET_BIND_SERVICE

[Install]
WantedBy=multi-user.target

d. My complete Caddy config:

{
        layer4 {
                udp/:3478 {
                        route {
                                tls
                                proxy udp/127.0.0.1:13478
                        }
                }
        }
        servers :443 {
                protocols h1
        }
        servers {
                listener_wrappers {
                        layer4 {
                                @tls tls
                                route @tls {
                                        subroute {
                                                @clients tls sni mesh.ryz.dev
                                                route @clients {
                                                        tls
                                                        proxy {
                                                                upstream tcp/127.0.0.1:5443 {
                                                                        tls_server_name mesh.ryz.dev
                                                                        tls
                                                                }
                                                        }
                                                }
                                        }
                                }
                        }
                        tls
                }
        }

        log {
                level DEBUG
        }
}

mesh.ryz.dev {
        tls /var/lib/acme/mesh.ryz.dev/cert.pem /var/lib/acme/mesh.ryz.dev/key.pem
        log {
                output file /var/log/caddy/access-mesh.ryz.dev.log
        }

        reverse_proxy https://127.0.0.1:5443 {
                transport http {
                        tls_server_name mesh.ryz.dev
                }
        }
}

5. Links to relevant resources:

fyi: configuration is correct. I sent test messages to udp/3478 via nc –u mesh.ryz.dev 3478 and observe log messages in caddy logs on udp/3478.

However, observed headscale will directly advertise the listening address and port change – udp/0.0.0.0:3478udp/127.0.0.1:13478 – to tailnet clients (but they will fail to connect due to firewall). Which is why I did not observe traffic in caddy logs. Should have heeded the “Limitations” [1] section of documentation.

On another note:

A reverse proxy adds another layer of complexity that needs to be able to handle the Tailscale Control Protocol properly. Be sure to test your setup without a reverse proxy before raising an issue.

In the caddy logs with aforementioned configuration, I observe these messages:

# journalctl -u caddy --no-pager | less +G
Jul 06 19:23:42 nabob caddy[28335]: {"level":"debug","ts":1783365822.65814,"logger":"http.handlers.reverse_proxy","msg":"upgrading connection","upstream":"127.0.0.1:5443","duration":0.020126754,"request":{"remote_ip":"...","remote_port":"64798","client_ip":"...","proto":"HTTP/1.1","method":"POST","host":"127.0.0.1:5443","uri":"/ts2021","headers":{"Connection":["Upgrade"],"X-Forwarded-Host":["mesh.ryz.dev:443"],"User-Agent":["Go-http-client/1.1"],"Content-Length":["0"],"X-Forwarded-For":["..."],"Upgrade":["tailscale-control-protocol"],"X-Tailscale-Handshake":["..."],"X-Forwarded-Proto":["https"],"Via":["1.1 Caddy"]},"tls":{"resumed":false,"version":772,"cipher_suite":4865,"proto":"","server_name":"mesh.ryz.dev","ech":false}},"http_version":1}
..."logger":"http.handlers.reverse_proxy","msg":"upgrading connection" ...

Is it safe to assume this is upgrading connection to websocket as per requirement [2]?

The reverse proxy must be configured to support WebSockets in order to communicate with Tailscale clients and it needs to handle two peculiarities of the Tailscale Control Protocol:

Or do I need to configure this explicitly in Caddyfile?

[1] Reverse proxy - Headscale

[2] Reverse proxy - Headscale