`caddy` w/ `caddy-l4` and proxying udp

1. The problem I’m having:

I am trying to setup headscale[1] behind caddy with caddy-l4 module/plugin. I have been successful in proxying the tcp ports, but I am struggling with the udp port.

What I am trying to achieve:

  • udp/3478 (caddy)udp/13478 (headscale)

2. Error messages and/or full log output:

N/A

From what I observe in journalctl -u caddy --no-pager | less +G there are no log statements indicating traffic on udp/3478 is proxied to 127.0.0.1:13478

3. Caddy version:

2.11.4

4. How I installed and ran Caddy:

Installed using NixOS module – nixpkgs/nixos/modules/services/web-servers/caddy/default.nix at nixos-26.05 · NixOS/nixpkgs · GitHub, which generates configuration and systemd service unit.

a. System environment:

NixOS 26.11 (x86_64), caddy managed with systemd

b. Command:

# from systemd service unit
ExecStart=/nix/store/9nf7n51c8y8h9pyjw4b8jclh5m85zqyq-caddy-2.11.4/bin/caddy run --environ --config /etc/caddy/Caddyfile

c. Service/unit/compose file:

# caddy.service
#
# For using Caddy with a config file.
#
# Make sure the ExecStart and ExecReload commands are correct
# for your installation.
#
# See https://caddyserver.com/docs/install for instructions.
#
# WARNING: This service does not use the --resume flag, so if you
# use the API to make changes, they will be overwritten by the
# Caddyfile next time the service is restarted. If you intend to
# use Caddy's API to configure it, add the --resume flag to the
# `caddy run` command or use the caddy-api.service file instead.

[Unit]
Description=Caddy
Documentation=https://caddyserver.com/docs/
After=network.target network-online.target
Requires=network-online.target

[Service]
Type=notify
User=caddy
Group=caddy
ExecStart=/nix/store/9nf7n51c8y8h9pyjw4b8jclh5m85zqyq-caddy-2.11.4/bin/caddy run --environ --config /etc/caddy/Caddyfile
ExecReload=/nix/store/9nf7n51c8y8h9pyjw4b8jclh5m85zqyq-caddy-2.11.4/bin/caddy reload --config /etc/caddy/Caddyfile --force
TimeoutStopSec=5s
LimitNOFILE=1048576
PrivateTmp=true
ProtectSystem=full
AmbientCapabilities=CAP_NET_ADMIN CAP_NET_BIND_SERVICE

[Install]
WantedBy=multi-user.target

d. My complete Caddy config:

{
        layer4 {
                udp/:3478 {
                        route {
                                tls
                                proxy udp/127.0.0.1:13478
                        }
                }
        }
        servers :443 {
                protocols h1
        }
        servers {
                listener_wrappers {
                        layer4 {
                                @tls tls
                                route @tls {
                                        subroute {
                                                @clients tls sni mesh.ryz.dev
                                                route @clients {
                                                        tls
                                                        proxy {
                                                                upstream tcp/127.0.0.1:5443 {
                                                                        tls_server_name mesh.ryz.dev
                                                                        tls
                                                                }
                                                        }
                                                }
                                        }
                                }
                        }
                        tls
                }
        }

        log {
                level DEBUG
        }
}

mesh.ryz.dev {
        tls /var/lib/acme/mesh.ryz.dev/cert.pem /var/lib/acme/mesh.ryz.dev/key.pem
        log {
                output file /var/log/caddy/access-mesh.ryz.dev.log
        }

        reverse_proxy https://127.0.0.1:5443 {
                transport http {
                        tls_server_name mesh.ryz.dev
                }
        }
}

5. Links to relevant resources: