Help understanding expected behavior for caddy-l4 and dynamic UDP ports

nameless · January 6, 2025, 11:05pm

So I’ve been heavily experimenting with Caddy-l4 and been successful in getting it set up to route a game server through a VPS, but there are some quirks I’m trying to understand about how to set up the reverse proxy fully or if I just don’t have a full understanding of the expected behaviors yet.

Primary example: I’m trying to query the status of my server through the reverse proxy. The proxy itself is configured to route TCP and UDP through 2 ports (game port and query port - and this seems to be working). I can connect to the game and play without issue, but when I use something like gamedig to query status, it always tries to use dynamic UDP ports (48k-65k range) and gets no response for its UDP transactions which time out. If I query with gamedig on the local network, it works fine.

A few guiding questions:

Could this be a configuration gap? I’m still learning caddy-l4 and attempting to use the new caddyfile support. Do I need to break out a route specifically for UDP? All I had to do to get it to work initially:

image566×196 3.33 KB
Is proxying UDP fully functional for caddy-l4? I’m seeing mixed reports (example), and wondering if this is just broken at the moment. I was able to get it working in nginx before deciding to switch to Caddy.
Am I missing something such as passing in a proxy protocol that would get this to auto config correctly? I don’t have a full understanding of where this applies yet, but was an avenue I was learning more about.

Thanks for any help in advance!

Mohammed90 · January 7, 2025, 8:27am

Howdy

The configuration doesn’t tell Caddy when to proxy to :9876 versus :9877. They’re all mashed up in the same pool, and the second proxy line is likely not called at all.

They’re talking about a very specific scenario involving use of PROXY protocol. I don’t believe it applies to your case.

In your case, if the ports serve different upstream services, split them into separate definitions, one for each port. This means you’ll define 2 layer4 services.

nameless · January 7, 2025, 10:15am

Thanks for the welcome and for chiming in to help me figure this out!

I tried this config and it seems to function similarly so far:

But it is still failing to respond to query requests from gamedig as they still seem to be going to the dynamic UDP ports. Even when I force calls on the right ports with flags, its not getting a response leading me to think I’m still messing something up with UDP. Do I also need to separate out the UDP and TCP traffic? In my quick test that didn’t seem to change the result.

I am using a Tailscale subnet to expose my local services to the VPS, but I don’t think that’s the issue here (my next avenue to explore if Caddy is working).

Mohammed90 · January 7, 2025, 10:50am

What do you mean by these? Can you be more specific? If you configured Caddy to listen on a specific port and to proxy to a specific port, it’ll use those ports. It won’t just use random ports.

Are you services behaving differently? When you list the upstreams in one line without specifying selection policty, Caddy proxies the requests randomly across the upstream options. Caddy will translate TCP/UDP across each other if the protocol differs.

Are you sure the proxy through Tailscale happens as expected? Do you see packets dropped between them? Can you reach Caddy from the VPS?

It helps if you share logs for your attempts

nameless · January 8, 2025, 4:06am

Yeah - I’ll try to give more detail and see if I can grab relevant logs. What I meant here is UDP ports in the 48k-65k range, chosen at random. Here’s an example of what a request from gamedig to the server responds with:

Q#0 DNS Lookup: my.domain.com
Q#0 Standard Resolve: my.domain.com
Q#0 Found address: XX.XX.XX.XX (home IP address)
Q#0 Requesting info ...
XX.XX.XX.XX:9877 UDP(50931)--> (responds with a random UDP port)
Buffer length: 25 bytes

As you can see above - its not retaining that UDP is only intended to be on 9876/9877.

I might be thinking about this wrong as its’s really one service. It’s a game server that connect over TCP then attempts to use the 2nd port as a query port. The TCP part is working so the game plays just fine, but the query part isn’t. How do I configure Caddy to do TCP and UDP on the same port in the same line? Is that not possible and only separate works?

Almost certain this is mostly working. I can tailscale ping between all devices with direct connections, and only be routed through DERP servers in niche scenarios (one being a potential longstanding bug where Windows machines choose a subnet over the local network). But you might be onto something here as I don’t know if I can ping back from the container running the game server since it is not running Tailscale and is dependent on the subnet router - I’ll check that next.

I’ll take another look and maybe come back with logs if I’m still stuck - thanks a ton for the guidance.

Mohammed90 · January 8, 2025, 9:03am

This is your (the client) port, not Caddy

The response from Caddy is from XX.XX.XX.XX:9877 to client port UDP(50931)

github.com

gamedig/node-gamedig/blob/0ad23aa1a49d9518c84f2fdf466eca4609daf505/lib/GlobalUdpSocket.js#L26


      
          if (!this.socket) {
            const udpSocket = createSocket({
              type: 'udp4',
              reuseAddr: true
            })
            udpSocket.unref()
            udpSocket.on('message', (buffer, rinfo) => {
              const fromAddress = rinfo.address
              const fromPort = rinfo.port
              this.logger.debug(log => {
                log(fromAddress + ':' + fromPort + ' <--UDP(' + this.port + ')')
                log(debugDump(buffer))
              })
              for (const callback of this.callbacks) {
                callback(fromAddress, fromPort, buffer)
              }
            })
            udpSocket.on('error', e => {
              this.logger.debug('UDP ERROR:', e)
            })
            await promisify(udpSocket.bind).bind(udpSocket)(this.port)

Those are 2 services, not one.

Roughly, this is what your config should be, because each port on each protocol is a different service:

{
	layer4 {
		tcp/:9876 {
			route {
				tls
				proxy 192.168.1.47:9876
			}
		}
		tcp/:9877 {
			route {
				tls
				proxy 192.168.1.47:9877
			}
		}
		udp/:9876 {
			route {
				tls
				proxy 192.168.1.47:9876
			}
		}
		udp/:9876 {
			route {
				tls
				proxy 192.168.1.47:9877
			}
		}
	}
}

nameless · January 8, 2025, 8:24pm

Thank you! I had done something similar to this and didn’t notice a difference, but the explanation makes sense and I’m sure it will matter once I figure out what’s going on.

In terms of what I’m seeing so far - I enabled layer4 logs and the requests coming through tend to complain about missing TLS handshakes.

{
  "level": "error",
  "ts": 1736367530.4751482,
  "logger": "layer4",
  "msg": "handling connection",
  "remote": "XX.XX.XXX.X:61015",
  "error": "tls: first record does not look like a TLS handshake"
}

nameless · January 9, 2025, 12:41am

Is it possible that because i have the acme_dns global flag set to Cloudflare for automatic HTTPS that it is causing this handshake to fail? I’m playing around with the secure/insecure routes and wondering if this is the wrong direction.

Mohammed90 · January 9, 2025, 6:02am

Are you sharing redacted config? Please don’t.

Are you sure your client expects TLS? Why do you have tls in your config?

nameless · January 9, 2025, 7:35am

May I kindly ask why? I manually took out identifiable IP addresses and keys - that’s all. My initial intent was to focus on understanding what caddy-l4 was doing so I just shared the global block as the rest is straightforward reverse proxies for my domain that all work great.

I thought it did, but everything still works without it - so you’re likely right that it does not so I’ve taken that out. I’m currently trying to use tcpdump to test between the VPS running Caddy and the local container running the server so I can get more of a look into whats going on, though haven’t had time to finish figuring that out yet.

For reference, the full Caddyfile on the VPS (with PII taken out still):

{
	debug
	log default {
		output file /var/log/caddy/log
		include layer4 http.handlers.reverse_proxy
	}

	acme_dns cloudflare <removed api key>

	layer4 {
		tcp/localhost:9876 {
			route {
				proxy 192.168.1.47:9876
			}
		}
		tcp/localhost:9877 {
			route {
				proxy 192.168.1.47:9877
			}
		}
		udp/localhost:9876 {
			route {
				proxy udp/192.168.1.47:9876
			}
		}
		udp/localhost:9877 {
			route {
				proxy udp/192.168.1.47:9877
			}
		}
	}
}

siteA.com {
	# Set path to site's directory.
	root * /var/www/html

	# Enable static file server.
	file_server
}

www.siteA.com {
	# Set path to site's directory.
	root * /var/www/html

	# Enable static file server.
	file_server
}

watch.siteA.com {
	reverse_proxy 192.168.1.43:8096
}

game.siteA.com {
	# Set path to site's directory.
	root * /var/www/html

	# Enable static file server.
	file_server
}

Greatly appreciate the help either way and been learning a lot about how Caddy works .

nameless · January 14, 2025, 2:14am

Finally managed to get it working after much fine tuning! My experiments with adding the source route were not needed, and going straight back to the simpler config and reverifying all my connections were working lit things up end to end. Thanks for the guidance!

Mohammed90 · January 14, 2025, 8:42am

Can you elaborate for future visitors?