Help with layer4 & caddy2-ext/layer4 - Proxy both UDP and TCP

[Gymnae (Gunnar)]

1. The problem I’m having:

I’d like to utilize the layer4 app with caddy2-ext/layer4 to stick with the YAML Caddyfile, but be able to proxy UDP traffic to certain upstream servers.

Strech goal is to keep H3 running in the http app, so that certain upstream service can still be accessed via QUIC - so not all UDP requests should be covered by the Layer4 app, some still by http

It appears that I don’t quite get the concept or run into limitations of caddy2-ext/layer4. Is there a way to achieve my goal without going down the .json route?

In short, it should work like this:

----> Ingress 443
#Layer4
-------> vpn.domain.tld - proxied to wireguard:5820/udp
-------> turn.domain.tld - proxied to coturn:3389/udp & tls
-------> mumble.domain.tld - proxied to mumbleserver/udp & tls
[…]
#http with h1 h2 h3
-------> cloud.domain.tld - fast cgi to nextcloud:9000/udp&tcp & tls (this one would benefit from h3)
-------> mumble.domain.tld - proxied to mumblserver:/tcp & tls
-------> otherhttp.domain.tld - proxied to other_services:80/tcp & tls
[…]

2. Error messages and/or full log output:

In the caddyfile below the lines for layer4 are commented out. If I remove those comments and try to activate the, say, dot server, the error message below is shown:

"layer4 app module: start: listen udp 0.0.0.0:443: bind: address already in use"

3. Caddy version:

v2.6.4 h1:2hwYqiRwk1tf3VruhMpLcYTg+11fCdr8S3jhNAdnPy8=

4. How I installed and ran Caddy:

Built my own caddy image:

FROM caddy:builder AS builder

RUN xcaddy build \ 
  --with github.com/caddy-dns/cloudflare \
  --with github.com/greenpau/caddy-security \
  --with github.com/mholt/caddy-l4 \
  --with github.com/RussellLuo/caddy-ext/layer4

FROM caddy:latest
RUN apk add --no-cache nano

COPY --from=builder /usr/bin/caddy /usr/bin/caddy

a. System environment:

6.1.0-0.deb11.5-amd64 #1 SMP PREEMPT_DYNAMIC Debian 6.1.12-1~bpo11+1 (2023-03-05) x86_64 GNU/Linux
Debian Bullseye with backports for Bookworm Kernel

b. Command:

Docker run command

docker run -d \
	--restart always \
	-p 80:80 \
	-p 443:443 \
	-p 80:80/udp \
	-p 443:443/udp \
	--network=mynetwork \
	-v /home/user1/services/caddy/fs:/fs \
	-v /home/user1/services/caddy/data:/data \
	-v /home/user2/infra/caddy/Caddyfile:/etc/caddy/Caddyfile \
	-v /home/user1/services/caddy/config:/config \
        -v ~/webserver/nextcloud/pushsocket:/run/notify_push \
        -v nextcloud_install:/var/www/html:ro \
        -v storagebox_user1_crypt:/var/www/html/data/storagebox/user1:ro \
	-v storagebox_user3_crypt:/var/www/html/data/storagebox/user3:ro \
        -v ~/webserver/nextcloud/data:/var/www/html/data:ro \
        -v ~/webserver/nextcloud/apps:/var/www/html/custom_apps:ro \
	-v /home/user1/webserver/dl5gu.radio/hugo/output:/var/www/dl5gu.radio \
        -v /home/user1/webserver/productopia/hugo/output:/var/www/productopia \
	--name caddy \
	gymnae/caddy

d. My complete Caddy config:

{
	#https_port 443
	#default_bind 0.0.0.0
	#http_port 80
	servers tcp/0.0.0.0:443 {
		protocols h1 h2 h3
	}
	#layer4 {
	#	udp/0.0.0.0:443 {
	#		turn.grundstil.de {
	#			tls
	#			proxy {
	#				to udp/signaling_coturn:3389
	#			}
	#		}
	#       vpn.grundsti.de {
	#	tls
	#	proxy {
	#		to udp/wireguard:51820
	#	}
	#}
	#       dot.argonath.de, dot.grundstil.de, dot.amonsul.net, dot.wxbu.de {
	#	tls
	#	proxy {
	#		to udp/dnsproxy:853
	#	}
	#}
	#       dot.amonsul.net {
	#	tls
	#	proxy {
	#		to dnsproxy:853
	#	}
	#}
	#	}
	#}
}




isso.dl5gu.radio, isso.productopia.net {
	reverse_proxy isso:8080
}

tube.wxbu.de {
	reverse_proxy invidious:3000
}

(matrix-well-known-header) {
	# Headers
	header Access-Control-Allow-Origin "*"
	header Access-Control-Allow-Methods "GET, POST, PUT, DELETE, OPTIONS"
	header Access-Control-Allow-Headers "Origin, X-Requested-With, Content-Type, Accept, Authorization"
	header Content-Type "application/json"
}

amonsul.net {
	handle /.well-known/matrix/server {
		import matrix-well-known-header
		respond `{"m.server":"matrix.amonsul.net:443"}`
	}

	handle /.well-known/matrix/client {
		import matrix-well-known-header
		respond `{"m.homeserver":{"base_url":"https://matrix.amonsul.net"}}`
	}

	respond "hello world"
	encode gzip
	file_server
}

grundstil.de, www.grundstil.de {
	encode gzip
	root * /var/www/dl5gu.radio

	handle /.well-known/matrix/server {
		import matrix-well-known-header
		respond `{"m.server":"matrix.grundstil.de:443"}`
	}

	handle /.well-known/matrix/client {
		import matrix-well-known-header
		respond `{"m.homeserver":{"base_url":"https://matrix.grundstil.de"}}`
	}
	file_server

	# Begin - Security
	# deny all direct access for these folders
	rewrite /(\.git|cache|bin|logs|backups|tests)/.* /403

	# deny running scripts inside core system folders
	rewrite /(system|vendor)/.*\.(txt|xml|md|html|yaml|yml|php|pl|py|cgi|twig|sh|bat)$ /403

	# deny running scripts inside user folder
	rewrite /user/.*\.(txt|md|yaml|yml|php|pl|py|cgi|twig|sh|bat)$ /403

	# deny access to specific files in the root folder
	rewrite /(LICENSE\.txt|composer\.lock|composer\.json|nginx\.conf|web\.config|htaccess\.txt|\.htaccess) /403

	respond /403 403
	## End - Security

	# global rewrite should come last.
	try_files {path} {path}/ /index.php?_url={uri}&{query}
}

argonath.de {
	handle /.well-known/matrix/server {
		import matrix-well-known-header
		respond `{"m.server":"matrix.argonath.de:443"}`
	}

	handle /.well-known/matrix/client {
		import matrix-well-known-header
		respond `{"m.homeserver":{"base_url":"https://matrix.argonath.de"}}`
	}
	respond "hello world"
	encode gzip
	file_server
}

csgofastdl.amonsul.net {
	root * /fs/csgo
	# respond "hello world"
	encode gzip
	file_server browse
}

csgo.wxbu.de {
	reverse_proxy csgo-webcron-server:8080
}

search:80, search.local:80, search.argonath.de, search.grundstil.de, search.amonsul.net {
	reverse_proxy searx:8080
}

pihole.amonsul.net, pihole.argonath.de, pihole.grundstil.de {
	reverse_proxy pi-hole:80
}

doh.wxbu.de, doh.productopia.net, doh.grundstil.de, doh.amonsul.net {
	reverse_proxy * amonsul-doh-server:8053
}

vpn.grundstil.de, vpn.amonsul.net {
	reverse_proxy wireguard:51820
}

mumble.wxbu.de, mumble.grundstil.de, mumble.wxbu.de, mumble.amonsul.net {
	reverse_proxy mumble-server:64738
}

space.wxbu.de, space.grundstil.de, space.amonsul.net {
	handle_path /radio/* {
		reverse_proxy botamusique_space:8181
	}
	reverse_proxy gomumblesoundboard_space:3000
}

radio.wxbu.de {
	reverse_proxy botamusique_space:8181
}

freaks.grundstil.de, freaks.wxbu.de, freaks.amonsul.net {
	reverse_proxy gomumblesoundboard_freaks:3000
}

hw.wxbu.de, hw.grundstil.de, hw.amonsul.net {
	reverse_proxy gomumblesoundboard_hw:3000
}

http://cloud, cloud.wxbu.de, cloud.local, cloud.dl5gu.radio, cloud.argonath.de, cloud.productopia.net, cloud.grundstil.de, cloud.amonsul.net {
	encode gzip
	redir /.well-known/webfinger /index.php/.well-known/webfinger 301
	redir /.well-known/nodeinfo /index.php/.well-known/nodeinfo 301
	redir /.well-known/carddav /remote.php/dav 301
	redir /.well-known/caldav /remote.php/dav 301

	header {
		Strict-Transport-Security max-age=31536000;
	}
	handle_path /standalone-signaling/* {
		reverse_proxy signaling_spreed:8088
	}
	route /push/* {
		uri strip_prefix /push
		reverse_proxy * unix//run/notify_push/notify_push.sock {
			trusted_proxies private_ranges
		}
	}

	# .htaccess / data / config / ... shouldn't be accessible from outside
	@forbidden {
		path /.htaccess
		path /data/*
		path /config/*
		path /db_structure
		path /.xml
		path /README
		path /3rdparty/*
		path /lib/*
		path /templates/*
		path /occ
		path /console.php
	}
	respond @forbidden 404
	handle {
		root * /var/www/html
		php_fastcgi nextcloud:9000 {
			env front_controller_active true # Remove index.php form url
			trusted_proxies private_ranges
		}
		file_server
	}
}
notify.amonsul.net, notify.grundstil.de {
	uri strip_prefix /push
	reverse_proxy * unix//run/notify_push/notify_push.sock {
		trusted_proxies private_ranges
	}
}

mail.grundstil.de, mail.dl5gu.radio {
	respond "I am a mailserver"
}

turn.grundstil.de {
	reverse_proxy * signaling_coturn:3389 {
		trusted_proxies private_ranges
	}
}

spreed.amonsul.net, spreed.grundstil.de {
	handle_path /standalone-signaling/* {
		reverse_proxy signaling_spreed:8088
	}
}

dot.argonath.de, dot.grundstil.de, dot.amonsul.net, dot.wxbu.de {
	reverse_proxy dnsproxy:853
}

speedup.grundstil.de {
	reverse_proxy * docker-matomo-web-1:80
}

sftpgrav.grundstil.de {
	reverse_proxy * grundstil-grav-sftp:8080
}

calibre.grundstil.de {
	reverse_proxy * calibre-web:8083
}

backup.grundstil.de {
	reverse_proxy * kopia:51515
}

portainer.argonath.de, portainer.grundstil.de, portainer.productopia.net, portainer.amonsul.net {
	reverse_proxy * portainer:9000
}


matrix.wxbu.de, matrix.productopia.net, matrix.amonsul.net, matrix.grundstil.de, matrix.argonath.de {
	reverse_proxy /_matrix/* matrix-synapse-1:8008 {
	}
	reverse_proxy /_synapse/client/* matrix-synapse-1:8008 {
	}
}

element.amonsul.net, element.wxbu.de, element.productopia.net, element.grundstil.de, element.argonath.de {
	encode zstd gzip
	reverse_proxy matrix-element-1:80 {
	}
}

monitor.grundstil.de, monitor.amonsul.net {
	reverse_proxy netdata:19999
	basicauth /* {
		<redacted> <redacted>
	}
}

glances.wxbu.de {
	reverse_proxy glances:61208
	basicauth /* {
		<redacted> <redacted>
	}
}


dl5gu.radio {
	root * /var/www/dl5gu.radio
	#respond "und der seb darf mit in  mein raumschiff"
	encode gzip
	file_server
}

productopia.net, wxbu.de {
	handle /.well-known/matrix/server {
		import matrix-well-known-header
		respond `{"m.server":"matrix.productopia.net:443"}`
	}

	handle /.well-known/matrix/client {
		import matrix-well-known-header
		respond `{"m.homeserver":{"base_url":"https://matrix.productopia.net"}}`
	}

	#basicauth {
	#	<redacted> <redacted>
	#}
	#	root * /var/www/productopia
	redir https://cloud.wxbu.de/apps/collectives/wxbu.de permanent
	#	file_server
}

git.dl5gu.radio {
	reverse_proxy gitea:3000
}

drone.dl5gu.radio {
	reverse_proxy drone:80
}

rxresu.grundstil.de {
	handle_path /api/* {
		reverse_proxy rxresu_server:3100
	}
	handle /* {
		reverse_proxy rxresu_client:3000
	}
}

headscale.grundstil.de, headscale.productopia.net {
	reverse_proxy headscale:8080
}

mc.amonsul.net {
	reverse_proxy mc:8100
}

# Dashboard
https://dashboard.productopia.net {
	# Apply basic security headers
	header {
		# Enable cross origin access to *.productopia.net
		Access-Control-Allow-Origin *.productopia.net
		# Enable HTTP Strict Transport Security (HSTS)
		Strict-Transport-Security "max-age=31536000;"
		# Enable cross-site filter (XSS) and tell browser to block detected attacks
		X-XSS-Protection "1; mode=block"
		# Disallow the site to be rendered within a frame on a foreign domain (clickjacking protection)
		X-Frame-Options "SAMEORIGIN"
		# Prevent search engines from indexing
		X-Robots-Tag "none"
		# Remove the server name
		-Server
	}

	reverse_proxy http://netmaker-ui
}

# API
https://api.productopia.net {
	reverse_proxy http://netmaker:8081
}

# mq

wss://broker.productopia.net {
	reverse_proxy ws://mq:8883
}

office.productopia.net, office.grundstil.de {
	encode gzip
	reverse_proxy http://collabora:9980
}

reddit.wxbu.de, reddit.grundstil.de, reddit.productopia.net {
	encode gzip
	reverse_proxy libreddit:8080
}

Btw, I wanted to say that for a looong time now: Caddy is an absolute blast to use and made my life soooo much easier. Thank you.

[francislavoie]

@Gymnae

with the YAML Caddyfile

Caddyfile isn’t YAML, so I’m not sure what you mean there.

@Gymnae

	servers tcp/0.0.0.0:443 {
		protocols h1 h2 h3
	}

This doesn’t really make sense – h3 is UDP only. The default is already to enable all protocols, so this doesn’t do anything useful. Plus it probably doesn’t match any listeners because you don’t have any that match that exactly.

@Gymnae

wss://broker.productopia.net {
	reverse_proxy ws://mq:8883
}

wss:// is not a valid scheme in Caddy. Neither is ws:// for upstream addresses. Remove both of those schemes, they don’t do anything.

The scheme is just a shortcut in browsers, it’s not really a real thing.

@Gymnae

handle /* {

@Gymnae

basicauth /* {

Remove the /*, this is very slightly less efficient. Having no matcher is faster because it doesn’t need to do a string comparison.

@Gymnae

	reverse_proxy /_matrix/* matrix-synapse-1:8008 {
	}
	reverse_proxy /_synapse/client/* matrix-synapse-1:8008 {
	}

Remove the braces if you’re not setting any proxy options. Just cosmetic though.

@Gymnae

reverse_proxy * calibre-web:8083

You can remove the * for these as well, this is also just cosmetic.

@Gymnae

spreed.amonsul.net, spreed.grundstil.de {
	handle_path /standalone-signaling/* {
		reverse_proxy signaling_spreed:8088
	}
}

Keep in mind you aren’t actually handling requests for other paths, so Caddy will just respond with an empty body. Probably best if you put something so it’s less confusing.

@Gymnae

trusted_proxies private_ranges

You can set this in global options now, which is what we recommend, so it’s set for all proxies at once:

{
	servers {
		trusted_proxies static private_ranges
	}
}

@Gymnae

rewrite /(\.git|cache|bin|logs|backups|tests)/.* /403

This doesn’t work – Caddy’s path matcher is not a regexp matcher. If you want a regexp match, you need to use the path_regexp matcher (named matcher).

@deny-direct path_regexp /(\.git|cache|bin|logs|backups|tests)/.*
rewrite @deny-direct /403

@Gymnae

In the caddyfile below the lines for layer4 are commented out. If I remove those comments and try to activate the, say, dot server, the error message below is shown:

"layer4 app module: start: listen udp 0.0.0.0:443: bind: address already in use"

Yeah, that’s impossible. You can’t have two servers listening to the same port.

The HTTP app wants to own ports 80 and 443 for a variety of reasons. You could use layer4 for other ports, but not those.

[Gymnae (Gunnar)]

Thank you for taking the time to check my Caddyfile - I thought it was YAML, but something learned

Is there a way to use port 443 for TCP/UDP at the same time with Caddy?

[francislavoie]

@Gymnae

Is there a way to use port 443 for TCP/UDP at the same time with Caddy?

You already are – HTTP 1 & 2 use TCP and HTTP 3 uses UDP.

[Gymnae (Gunnar)]

Ok, sorry to rephrase my question.
Can I achieve my stated goal? Or would it be easier if I turn off h3?

----> Ingress 443
#Layer4
-------> vpn.domain.tld - proxied to wireguard:5820/udp
-------> turn.domain.tld - proxied to coturn:3389/udp & tls
-------> mumble.domain.tld - proxied to mumbleserver/udp & tls
[…]
#http with h1 h2 h3
-------> cloud.domain.tld - fast cgi to nextcloud:9000/udp&tcp & tls (this one would benefit from h3)
-------> mumble.domain.tld - proxied to mumblserver:/tcp & tls
-------> otherhttp.domain.tld - proxied to other_services:80/tcp & tls
[…]

[francislavoie]

Why can’t you use a different port for your non-HTTP UDP traffic?

[Gymnae (Gunnar)]

Fewer ports I need to open in my firewall and a sure fire way through firewalls, where only few ports like 443/80 are open.

[francislavoie]

You’re making it more complicated than it needs to be. Just use a different port. That’s the reason ports exist, to differentiate different types of traffic.

If you used the same port, you’d still need to match on something in the traffic to route it to the correct place.

With HTTP you have headers like the Host header, with HTTPS you have TLS-SNI (server name identification).

You need some mechanism in to determine the kind of traffic (by looking at the shape of the bytes in the traffic) and something in the data to identify where it should go.

If you can’t do that because you don’t know exactly what the data is, or caddy-l4 doesn’t have specific support for that kind of data, then it’s not possible to route it.

So the only thing you can do is use a different port, because that clearly separates the traffic away from the rest, so the server doesn’t get confused.

[Gymnae (Gunnar)]

I think my scenario is quite close to what this user here asked, but solved with layer4 & caddy-yaml:

How to combine layer4 module with tls and http modules?

Thanks! I’ve been finally able to implement the desired logic. In case readers of this issue are looking for a final result, my final result looks like this: logging: logs: default: level: INFO storage: "module": "file_system" "root": "/data" apps: tls: certificates: automate: - livekit01.spvw.de - livekit-turns.spvw.de layer4: servers: main: listen: [":443"] routes: - match: - tls: sn…

I know which subdomains bring which traffic, could that be enough to route?
Example:
vpn.domain.tld → always UDP wireguard
turn.domain.tld → can be TCP and UDP, but UDP preferred
mumble.domain.tld → main traffic is udp

For now I’m able to use port 8080 for turn, which already enables me passing by some firewalls.

[francislavoie]

@Gymnae

know which subdomains bring which traffic, could that be enough to route?

No, it’s not. You need to also know how to find that information from the traffic data coming in. If it’s a protocol Caddy doesn’t understand, then there’s no way to do that. Like I said, for HTTP you can use the Host header to find out the hostname the client requested, and for HTTPS you can use TLS-SNI. For wireguard…

[Gymnae (Gunnar)]

Thank you @francislavoie, for clarifying this for me.
I guess then I will bypass caddy for my coturn and wireguard traffic