L4 Reverse Proxy - JSON Config

grimportal · February 28, 2024, 8:45pm

1. The problem I’m having:

I am not sure if my caddy.json configuration is set up correctly for a UDP & TCP reverse proxy use case. I think my caddy.json must be incorrect, but I’m not sure how to fix. The documentation is not quite there yet for the layer4 app, but I imagine it must be very similar to the http json… which I find quite confusing TBH…

I have two virtualbox VMs running on my windows computer. One is running game server software (Enshrouded) and the other is running Caddy. I am able to connect to my game server through my windows steam client at 10.0.0.27:15637 and 10.0.0.27:15636 (the game uses two ports), but I am not able to reach the game server through the Caddy reverse proxy (10.0.0.29). I am able to get Caddy responses from my “hello world” and my admin api.

Please help me configure caddy correctly!

Diagram of my setup. Red = failure; green = success

2. Error messages and/or full log output:

Started caddy.service.
{"level":"info","ts":1709160833.3655574,"msg":"using provided configuration","config_file":"/etc/caddy/caddy.json","config_adapter":""}
{"level":"info","ts":1709160833.3666823,"logger":"admin","msg":"admin endpoint started","address":":2020","enforce_origin":false,"origins":["//:2020"]}
{"level":"warn","ts":1709160833.3667228,"logger":"admin","msg":"admin endpoint on open interface; host checking disabled","address":":2020"}
{"level":"info","ts":1709160833.367122,"logger":"http.log","msg":"server running","name":"hello_world","protocols":["h1","h2","h3"]}
{"level":"info","ts":1709160833.3679407,"logger":"tls.cache.maintenance","msg":"started background certificate maintenance","cache":"0xc000288a00"}
{"level":"info","ts":1709160833.3683507,"msg":"autosaved config (load with --resume flag)","file":"/etc/caddy/.config/caddy/autosave.json"}
{"level":"info","ts":1709160833.368376,"msg":"serving initial configuration"}
{"level":"warn","ts":1709160833.3695774,"logger":"tls","msg":"storage cleaning happened too recently; skipping for now","storage":"FileStorage:/etc/caddy/storage","instance":"b530c8bc-a031-4928-b8e6-81c2e1ec89d7","try_again":1709247233.3695767,"try_again_in":86399.999999825}
{"level":"info","ts":1709160833.369609,"logger":"tls","msg":"finished cleaning storage units"}

3. Caddy version:

v2.7.6 h1:w0NymbG2m9PcvKWsrX06EEkY9Ru4FJK8uQbYcev1p3A=

4. How I installed and ran Caddy:

This is on Nixos 23.11 on a virtualbox vm running in bridged networking mode.

a. System environment:

I ran this to create a caddy executable

xcaddy build --output /etc/caddy --with github.com/mholt/caddy-l4

And then set my configuration.nix is as follows:

{ config, lib, pkgs, ... };
{ 
  imports =
  [
    ./hardware-configuration.nix
  ];
  system.stateVersion = "23.11";

  boot.loader.systemd-boot.enable = true;
  boot.loader.efi.canTouchEfiVariables = true;

  networking.hostName = "caddy-vm";

  systemd.services.caddy = {
    enable = true;
    wanetdBy = [ "multi-user.target" ];
    after = [ "networking.target" ];
    serviceConfig = {
      ExecStart = "/etc/caddy/caddy run --config /etc/caddy/caddy.json";
      Restart = "on-failure";
    };
  };

  environment.systemPackages = [
    pkgs.xcaddy
    pkgs.go
  ];

  networking.firewall.enable = false;
}

b. Command:

systemctl start caddy

And I get the same behavior if I directly run

/etc/caddy/caddy run --config /etc/caddy/caddy.json

c. Service/unit/compose file:

n/a

d. My complete Caddy config:

{
    "admin": {
        "listen": ":2020"
    },
    "storage": {
        "module": "file_system",
        "root": "/etc/caddy/storage"
    },
    "apps": {
        "http": {
            "servers": {
                "hello_world": {
                    "listen": [":8080"],
                    "routes": [
                        {
                            "handle": [
                                {
                                    "body": "Hello, World!",
                                    "handler": "static_response"
                                }
                            ]
                        }
                    ]
                }
            }
        },
        "layer4": {
            "servers": {
                "game_tcp": {
                    "listen": [":15637"],
                    "routes": [
                        {
                            "handle": [
                                {
                                    "handler": "proxy",
                                    "proxy_protocol": "v1",
                                    "upstreams": [
                                        {"dial": ["10.0.0.27:25637"]}
                                    ]
                                }
                            ]
                        }
                    ]
                },
                "game_udp": {
                    "listen": [":15636"],
                    "routes": [
                        {
                            "handle": [
                                {
                                    "handler": "proxy",
                                    "proxy_protocol": "v2",
                                    "upstreams": [
                                        {"dial": ["10.0.0.27:25636"]}
                                    ]
                                }
                            ]
                        }
                    ]
                }
            }
        }
    }
}

5. Links to relevant resources:

matt · February 29, 2024, 11:07pm

Not sure if that’d cause a problem, but that’s spelled wrong

What errors are you getting when trying to connect?

francislavoie · March 1, 2024, 1:13am

Are you sure you want to enable this? Does your upstream app support PROXY protocol? If you’re not sure, it probably doesn’t, so remove that.

grimportal · March 1, 2024, 3:21am

Thanks for the replies.

The nix config wanetBy is a typo only in this thread… I hand typed that because I hadn’t done whatever it takes to copy paste over. So not the issue!
Thanks for the tip about protocol version. I was trying different things and thought that might help. But it made no difference. I’ll remove that

It sounds like there’s nothing glaringly wrong with my caddy config, so there’s probably something wrong with my setup outside caddy. I’m going to try setting up some testing tools as per ChatGPT’s advice to send and listen to tcp/udp messages. Until now I have been trying to test by directly using my game client and server which doesn’t provide really any feedback besides “Could not find server at that address”.

matt · March 1, 2024, 4:37am

That’s listening on TCP, btw. To listen on UDP: udp/:15636 (because TCP Is the default)

grimportal · March 1, 2024, 6:16am

Did some testing

Setup

On my NixOS VMs, I temporarily installed netcat with nix-shell -p netcat
On my Windows host, I opened powershell as an administrator and temporarily enabled scripts with Set-ExecutionPolicy -ExecutionPolicy Bypass -Scope Process

Test 1: Manually Send UDP from VM to Host

On my windows Host, I created a script file named udpListen.ps1 with the contents below and ran it in powershell with .\udpListen.ps1

# Define the port number to listen on
$port = 10001 # Change this to your desired port

# Create a UDP client bound to the given port
$endpoint = New-Object System.Net.IPEndPoint([System.Net.IPAddress]::Any, $port)
$udpClient = New-Object System.Net.Sockets.UdpClient $port

# Wait for a message
Write-Host "Listening on UDP port $port..."
$message = $udpClient.Receive([ref]$endpoint)

# Convert the message from bytes to a string and display it
$string = [System.Text.Encoding]::ASCII.GetString($message)
Write-Host "Received message: $string"

# Clean up
$udpClient.Close()

On my NixOS VM, I ran the following command:

echo "Hello Windows" | nc -u 10.0.0.111 10001

My windows powershell printed “Hello Windows”

Test 2: Manually Send UDP from Host to VM

On my NixOS VM, I ran the following command:

nc -ul 10001

On windows, I created a script file named udpSend.ps1 with the contents below and ran it in powershell with .\udpSend.ps1

# Define the target IP and port
$targetIP = "10.0.0.29"
$port = 10001

# Create a UDP client
$client = New-Object System.Net.Sockets.UdpClient

# Connect to the target
$client.Connect($targetIP, $port)

# Convert a message to bytes
$message = "Hello Nix"
$bytes = [Text.Encoding]::ASCII.GetBytes($message)

# Send the message
$client.Send($bytes, $bytes.Length)

# Close the client
$client.Close()

# Output to confirm operation
Write-Host "UDP message sent to $targetIP on port $port"

My NixOS CLI printed “Hello Nix”

Test 3: Proxying from Host through Caddy VM to Target VM

I updated my caddy json to the following:

{
    "admin": {
        "listen": ":2020"
    },
    "storage": {
        "module": "file_system",
        "root": "/etc/caddy/storage"
    },
    "apps": {
        "http": {
            "servers": {
                "hello_world": {
                    "listen": [":8080"],
                    "routes": [
                        {
                            "handle": [
                                {
                                    "body": "Hello, World!",
                                    "handler": "static_response"
                                }
                            ]
                        }
                    ]
                }
            }
        },
        "layer4": {
            "servers": {
                "game_tcp": {
                    "listen": [":15637"],
                    "routes": [
                        {
                            "handle": [
                                {
                                    "handler": "proxy",
                                    "upstreams": [
                                        {"dial": ["10.0.0.27:15637"]}
                                    ]
                                }
                            ]
                        }
                    ]
                },
                "game_udp": {
                    "listen": ["udp/:15636"],
                    "routes": [
                        {
                            "handle": [
                                {
                                    "handler": "proxy",
                                    "upstreams": [
                                        {"dial": ["udp/10.0.0.27:15636"]}
                                    ]
                                }
                            ]
                        }
                    ]
                },
                "manual_udp": {
                    "listen": ["udp/:10001"],
                    "routes": [
                        {
                            "handle": [
                                {
                                    "handler": "proxy",
                                    "upstreams": [
                                        {"dial": ["udp/10.0.0.27:10001"]}
                                    ]
                                }
                            ]
                        }
                    ]
                },
                "manual_tcp": {
                    "listen": [":10002"],
                    "routes": [
                        {
                            "handle": [
                                {
                                    "handler": "proxy",
                                    "upstreams": [
                                        {"dial": ["10.0.0.27:10002"]}
                                    ]
                                }
                            ]
                        }
                    ]
                }
            }
        }
    }
}

This time I ran nc -ul 10001 on my target vm. I ran .\udpListen.ps1

I also ran a new script .\tcpSend.ps1 defined below with the target vm listening with nc -l 10002.

# Define the proxy server address and port
$proxyIP = "10.0.0.29"
$port = 10002

# Create a TCP client and connect
$client = New-Object System.Net.Sockets.TcpClient
$client.Connect($proxyIP, $port)

# Get a stream for writing
$stream = $client.GetStream()
$writer = New-Object System.IO.StreamWriter($stream)

# Write a message to the stream
$message = "Hello, Proxy!"
$writer.WriteLine($message)
$writer.Flush() # Ensures the message is sent immediately

# Close the client connection
$writer.Close()
$stream.Close()
$client.Close()

# Output to confirm the operation
Write-Host "Message sent to $proxyIP on port $port"

Both tests printed successfully on the target VM

Therefore, Caddy is set up perfectly???

The issue must be with how I’m using my game software!

THANK YOU

I fixed it! Among any mistakes I may have made with my caddy json config, I had my game ports backward! The one they call the “query” port is the udp one and the “game” port is the tcp one! I assumed the opposite.

Thank you for all the responses and assistance, I really appreciate it. Such a great feeling to get something working. Loved the page in the docs that said “I can do hard things.” I’m SUPER excited to continue working on my gaming project and will continue to share my experiences with Caddy. I hope that this explanation of how I did my troubleshooting can be of some use to someone some day.

matt · March 1, 2024, 4:40pm

Yay Thanks for the detailed writeup!

system · March 31, 2024, 4:40pm

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.