SSH Multiplexing with layer-4

Help
1

Hello,

I switched from nginx to caddy and the experience is great, much easier to setup.
However, I am still using sslh for multiplexing ssh and https on port 443.
From time to time sslh fails and I need to restart the service because otherwise all my web services are offline.
So I would like to move this multiplexing task to caddy (with the caddy-l4 plugin) as well.
But I don’t know how to write the correct caddyfile.
I found a caddyfile sample for multiplexing http and tls, how would i have to change this to multiplex ssh and https?
Thanks for your help!
Ruediger
P.S. The backend servers should receive the source IP so that fail2ban and similar stuff can still work.

{
layer4 {
127.0.0.1:5000 {
@insecure http
route @insecure {
proxy localhost:80
}
@secure tls
route @secure {
proxy localhost:443
}
}
}
}

1 Like
2

This is what I use to multiplex ssh and the normal TLS terminated sites on the same port 443. SSH is restricted to internal IPs for me, otherwise the subroute can be deleted completely.

# Global Options
{
	servers {
		listener_wrappers {
			layer4 {
				@ssh_layer4 ssh
				route @ssh_layer4 {
					subroute {
						@allowed_ips remote_ip 192.168.0.0/16 172.16.0.0/12 10.0.0.0/8
						route @allowed_ips {
							proxy 10.0.0.10:22 {
							}
						}
					}
				}
				route
			}
			tls
		}
	}
}

#Site Block
site1.example.com {
        reverse_proxy 192.168.1.1
}

Using Layer4 in the listener_wrapper makes it match before the http app and that enables (TCP) protocols on the HTTP and HTTPS ports to match and get routed someplace else. Then the route and tls statement sends it to the site blocks where the TLS terminated handling can begin.

Edit: For source IP you have to add the proxy protocol. It can be added to the layer4 too (check the docs). Never used it so I dont know how it works.

3 Likes
3

Thank you so much for your help!
I’ve integrated your code into my Caddyfile (and removed the subroute) and now I can use SSH and TLS on port 443 with caddy only, sslh is gone :slight_smile:
This is really impressive, before Caddy had to listen on 8443 , because sslh was sitting on 443, but not its only caddy directly on 443 and taking care of redirecting ssh to the internal port of the ssh server!
I really hope that the official caddy docker will integrate l4 soon, currently I am using https://hub.docker.com/r/cybergeek369/caddy-l4

Thanks a lot for caddy!
Ruediger
P.S. The backend servers receive the source IP without adding some proxy_procotol config into the caddyfile, seems to be done all automatically. AMAZING! :star_struck:

4 Likes
4

Hello again,
I just realized that caddy-l4 does not forward the real IP to my ssh server.
I am using the following configuration.
How do I have to change it so that caddy forwards the external IP to my ssh server?
{
servers {
listener_wrappers {
layer4 {
@ssh_layer4 ssh
route @ssh_layer4 {
proxy :22 {
}
}
route
}
tls
}
}
}

Thanks a lot!
Ruediger

5

Please mind the formatting of your config by using code blocks. Here’s a cleaned format:

{
	servers {
		listener_wrappers {
			layer4 {
				@ssh_layer4 ssh
				route @ssh_layer4 {
					proxy :22 {
					}
				}
				route
			}
			tls
		}
	}
}

To get the real client IP address, you’ll have to use PROXY protocol. The solution found online for SSH is to use mmproxy, which merely unwarps the PROXY protocol header, which Caddy can already do via the proxy_protocol inside listener_wrapper, so you don’t really need to use mmproxy. However, it appears to require these iptable manipulation:

ip rule add from 127.0.0.1/8 iif lo table 123
ip route add local 0.0.0.0/0 dev lo table 123

ip -6 rule add from ::1/128 iif lo table 123
ip -6 route add local ::/0 dev lo table 123

The final config can be something like this:

{
	servers {
		listener_wrappers {
			proxy_protocol
			layer4 {
				@ssh_layer4 ssh
				route @ssh_layer4 {
					proxy :22
				}
				route
			}
			tls
		}
	}
}

However, DISCLAIMER, I have not tested myself. Manipulating iptables without understanding may cause network issue, to say the least. You’ll have to asses, test, and judge if the config and setup is correct.

6

Thanks for your reply.
Sorry for the formatting, this was a copy paste mistake, my original caddy file is correctly formatted.
I’ll try your proposal and will report back if it works.
Thanks,
Ruediger

1 Like
7

Please note that the empty route became obsolete and is not needed anymore to make the config work.

8

By empty route you mean the iptable changes?
Kind regards, Ruediger

9

No I mean the one in the Caddyfile.

{
	servers {
		listener_wrappers {
			layer4 {
				@ssh_layer4 ssh
				route @ssh_layer4 {
					proxy :22 {
                        proxy_protocol v2
                    }
				}
			}
			tls
		}
	}
}

I put the proxy protocol into the proxy directive.

I could be wrong with this proxy protocol position though.

10

This tells the proxy directive to use the proxy_protocol when connecting to upstream, and upstream has to know how to unwrap it. The proxy_protocol in the listener_wrapper is so Caddy unwraps requests wrapped by the PROXY protocol by downstream.

The flow is that Caddy receives the packets wrapped by the PROXY protocol to retain the original client IP address. Once it’s at Caddy, Caddy unwraps it, which takes the original client IP address from the PROXY protocol header and plugs it into the packet as seen by Caddy. However, when Caddy connects to upstream, the upstream will still see Caddy’s own IP address as the client. I think the iptable trick is to have the kernel move the unwrapped packets directly to the ssh server without having their source address header field touched.

I’m still parsing the thing in my head, but I don’t feel my understanding is solid. I feel like I need to see the packet dump to truly grok what’s going on.

11

Ah okay I see. I would have sent the proxy protocol header to the ssh server and processed it there. This requires awareness of this header though, as mentioned.

I dont understand the mentioned iptables trick at first glance.