Replicating what Stunnel does but with caddy-l4

alphajoza · June 3, 2023, 5:49pm

I want to tunnel SSH traffic over TLS with the help of caddy-l4 (I’m using a virtual machine (Ubuntu 20.04) which is hosted in a Windows 10 setup)

I want to SSH into Ubuntu virtual machine from Windows 10;

My config.json in Ubuntu virtual machine:
A TCP reverse proxy that terminates TLS on 443 and proxies to SSH server on port 22

{
	"apps": {
		"layer4": {
			"servers": {
				"example": {
					"listen": [
						"192.168.1.11:443"
					],
					"routes": [
						{
							"handle": [
								{
									"handler": "tls"
								},
								{
									"handler": "proxy",
									"upstreams": [
										{
											"dial": [
												"localhost:22"
											]
										}
									]
								}
							]
						}
					]
				}
			}
		},
		"tls": {
			"certificates": {
				"automate": ["192.168.1.11"]
			},
			"automation": {
				"policies": [
					{
						"issuers": [{"module": "internal"}]
					}
				]
			}
		}
	}
}

My Problem is in config.json of Windows 10 machine:
A config that tunnels the received SSH traffic on port 22000 over TLS and sends it to 192.168.1.11:443

{
	"apps": {
		"layer4": {
			"servers": {
				"example": {
					"listen": [
						"localhost:22000"
					],
					"routes": [
						{
							"handle": [
								{
									??? What should I put here ???
								}
							]
						}
					]
				}
			}
		},
		"tls": {
			"certificates": {
				"automate": [
					"localhost"
				]
			},
			"automation": {
				"policies": [
					{
						"issuers": [
							{
								"module": "internal"
							}
						]
					}
				]
			}
		}
	}
}

francislavoie · June 3, 2023, 6:17pm

Probably just the proxy handler with 192.168.1.11:443 as the dial address?

alphajoza · June 3, 2023, 7:35pm

If you mean this config:

{
	"apps": {
		"layer4": {
			"servers": {
				"example": {
					"listen": [
						"localhost:22000"
					],
					"routes": [
						{
							"handle": [
								{
									"handler": "proxy",
									"upstreams": [
										{
											"dial": [
												"192.168.1.11:443"
											]
										}
									]
								}
							]
						}
					]
				}
			}
		},
		"tls": {
			"certificates": {
				"automate": [
					"localhost"
				]
			},
			"automation": {
				"policies": [
					{
						"issuers": [
							{
								"module": "internal"
							}
						]
					}
				]
			}
		}
	}
}

Upon trying to SSH with ssh -p 22000 root@localhost, I’m getting this complain on the Ubuntu side:

2023/06/03 19:31:07.802 ERROR layer4 handling connection {"remote": "192.168.1.5:1819", "error": "tls: first record does not look like a TLS handshake"}

I think it’s obvious that the SSH traffic is not TLS encrypted but is passed rather without any change.

WeidiDeng · June 4, 2023, 2:40am

Are you just using ssh -p 22000 root@localhost to ssh to remote host? You’ll also need to use ProxyCommand to tunnel ssh over tls. It can be specified in .ssh/config or command line arguments. Look here for examples.

Also you should put tls and proxy in your config like this.

alphajoza · June 4, 2023, 5:50am

My aim is to use caddy-l4 to encrypt the incoming SSH traffic to TLS and then send it to remote machine (where the opposite will happen); I did this scenario with Stunnel without the need for any other application in between.

Using ProxyCommand /usr/local/bin/socat - OPENSSL:localhost:22000,verify=0 in ~/.ssh/config is relegation of tunneling SSH over TLS to another application (but I’m trying to use caddy-l4 for that)

The problem now is this:
Can caddy-l4 wrap SSH traffic in TLS? if yes, what should I put in my json config file to do that?

WeidiDeng · June 4, 2023, 3:24pm

Oh, you mean ssh to caddy-l4 like any other ssh server, then caddy-l4 encrypt this ssh connection with tls and forward it to another server which can descrypt this tls connection and handle the underlying ssh? Currently caddy-l4 doesn’t support this use case.

francislavoie · June 4, 2023, 4:20pm

I guess it’s worth mentioning, SSH is already encrypted so I’m not sure what the benefit of that would be.

Mohammed90 · June 4, 2023, 6:19pm

Yes, it can. The link shared by @WeidiDeng shows multiple working configs. The handler chain the layer4 app should start with a tls handler, followed by a proxy to your ssh backend server. Here’s reproduced working config with automatic TLS:

{
	"logging": {
		"logs": {
			"default": {
				"level": "DEBUG"
			}
		}
	},
	"apps": {
		"tls": {
			"certificates": {
				"automate": [
					"localhost"
				]
			},
			"automation": {
				"policies": [
					{
						"issuers": [
							{
								"module": "internal"
							}
						]
					}
				]
			}
		},
		"layer4": {
			"servers": {
				"example": {
					"listen": [
						"127.0.0.1:8443"
					],
					"routes": [
						{
							"handle": [
								{
									"handler": "tls"
								},
								{
									"handler": "proxy",
									"upstreams": [
										{"dial": ["localhost:2012"]}
									]
								}
							]
						}
					]
				}
			}
		}
	}
}

The logic for the handler chain is:
First terminate TLS, then proxy the enveloped bytes to the backend, which may be SSH or HTTP. The point is TLS is only a wrapper around those bytes.

alphajoza · June 4, 2023, 7:41pm

Some firewalls are very picky and only allow TLS traffic on port 443;

alphajoza · June 4, 2023, 7:47pm

Once again, I’m not talking about terminating TLS, I need TLS encapsulation.

Your config is for the server side (in my case, Ubuntu). As I stated in my original post, I have no issue in terminating TLS and proxying it to SSH server on port 22 (check the first config).

The problem is in client side (in my case, Windows 10); SSH traffic is coming on port 22000; How to encapsulate it into TLS and then proxy it to 192.168.1.11:443?

matt · June 4, 2023, 8:55pm

You should be able to proxy the incoming SSH to a TLS endpoint. I’m mobile right now but you can enable TLS on the proxy config, I think. If not, it’s a good feature request.

Mohammed90 · June 4, 2023, 9:54pm

Sorry, I misunderstood. You can do that as well, but you’ll have to run an instance of caddy-l4 on the Windows machine as a proxy server to your upstream. Here’s working config:

{
	"apps": {
		"layer4": {
			"servers": {
				"ssh-proxy": {
					"listen": ["127.0.0.1:8443"],
					"routes": [
						{
							"handle": [
								{
									"handler": "proxy",
									"upstreams": [
										{
											"dial": ["example.com:443"],
											"tls": {
												"server_name": "example.com"
											}
										}
									]
								}
							]
						}
					]
				}
			}
		}
	}
}

You can then run ssh -p 8443 username@localhost, and it’ll wrap the connection in TLS to upstream, which is then unwrapped into ssh again.

WeidiDeng · June 5, 2023, 1:14am

Never knew you can specify tls in upstreams before. Always assumed it was just in matcher and handler. Examples in README doesn’t really show this.

matt · June 5, 2023, 5:46am

Yeah, sorry. The L4 docs really need expansion. It was just kind of an experiment at first, but it’s really starting to prove itself so it’s probably time to flush out the rest of the docs.

In the meantime the JSON docs aren’t a bad start, at least for discovering features: JSON Config Structure - Caddy Documentation

alphajoza · June 5, 2023, 6:16am

Your solution seems to work; but I cannot fully acknowledge that because now I’m getting certificate errors;

Ubuntu side:
023/06/05 06:08:07.870 ERROR layer4 handling connection {"remote":"192.168.1.5:14508", "error": "remote error: tls: bad certificate"}

Windows side:
2023/06/05 06:08:07.278 ERROR layer4 handling connection {"remote": "127.0.0.1:14507", "error": "tls: failed to verify certificate: x509: certificate signed by unknown authority"}

Configs:

Ubuntu side:

{
	"apps": {
		"layer4": {
			"servers": {
				"example": {
					"listen": [
						"192.168.1.11:443"
					],
					"routes": [
						{
							"handle": [
								{
									"handler": "tls"
								},
								{
									"handler": "proxy",
									"upstreams": [
										{
											"dial": [
												"localhost:22"
											]
										}
									]
								}
							]
						}
					]
				}
			}
		},
		"tls": {
			"certificates": {
				"automate": ["192.168.1.11"]
			},
			"automation": {
				"policies": [
					{
						"issuers": [{"module": "internal"}]
					}
				]
			}
		}
	}
}

Windows side:

{
	"apps": {
		"layer4": {
			"servers": {
				"example": {
					"listen": [
						"localhost:2222"
					],
					"routes": [
						{
							"handle": [
								{
									"handler": "proxy",
									"upstreams": [
										{
											"dial": [
												"192.168.1.11:443"
											],
											"tls": {
												"server_name": "192.168.1.11"
											}
										}
									]
								}
							]
						}
					]
				}
			}
		},
		"tls": {
			"certificates": {
				"automate": [
					"localhost"
				]
			},
			"automation": {
				"policies": [
					{
						"issuers": [
							{
								"module": "internal"
							}
						]
					}
				]
			}
		}
	}
}

francislavoie · June 5, 2023, 7:29am

Did you install the upstream’s root cert on your front machine? You’ll need to do that. Caddy generates a CA when using internal that it uses to sign certs, and the connecting Caddy instance needs to trust the cert being served by the upstream, and the easiest way to establish trust is to install the upstream’s root cert on the client.

alphajoza · June 5, 2023, 7:31am

Yes (installed as Trusted Root Certificate) in both machines and still getting the certificate error.

I just let go of it. too much hassle.

BTW, thanks all for the help.

Mohammed90 · June 5, 2023, 10:35am

You can point Caddy to the CA files as part of the upstream TLS config

system · July 5, 2023, 10:35am

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.