Possible to share the same hostname between L4 and HTTP with listener_wrappers?

Help
1

1. The problem I’m having:

Is it possible to share the same site/hostname and the same port between a Layer 4 (L4) and a Layer 7 (L7/HTTP) application?

Specifically, when connecting to a specific hostname (e.g., vscode.portal.{env.ACME_DOMAIN}) via a browser, I need standard L7/HTTP handling to apply, including cookie and query string management within the Caddyfile. Conversely, when connecting via SSH over TLS to the same hostname, I want a simple L4 proxy to take effect.

I have no issues running the two setups separately on two different hostnames, but I cannot get them to work on the same one. For example, the following configuration works (vscode-ssh.portal.{env.ACME_DOMAIN} for ssh and vscode.portal.{env.ACME_DOMAIN} for http):

listener_wrappers {
			layer4 {
				matching_timeout 1s

				@vscode-ssh tls sni vscode-ssh.portal.{env.ACME_DOMAIN}

				route @vscode-ssh {
					tls
					proxy vscode:22 
				}
			}
			tls
		}

and also this

listener_wrappers {
			layer4 {
				matching_timeout 1s

				@vscode-ssh tls sni vscode-ssh.portal.{env.ACME_DOMAIN}
				route @vscode-ssh {
					tls
					subroute {
						@ssh ssh
						route @ssh {
							proxy vscode:22 
						}
					}
				}
			}
			tls
		}

but I don’t know if this could works

listener_wrappers {
			layer4 {
				matching_timeout 1s

				@vscode tls sni vscode.portal.{env.ACME_DOMAIN}

				route @vscode {
					tls
					subroute {
						@ssh ssh
						route @ssh {
							proxy vscode:22 
						}
						route {
							# How to send to normal L7 http handling?
						}
					}
				}
			}
			tls
		}

I also tried another approach, i.e. matching after tls

listener_wrappers {
			# tls <- implicit
			layer4 {
				matching_timeout 1s

				@vscode vars l4.tls.server_name vscode.portal.{env.ACME_DOMAIN}
				route @vscode {
					subroute {
						@ssh ssh
						route @ssh {
							proxy vscode:22 
						}
					}
				}
			}
		}

but this doesn’t works and raise the following error in log.

2. Error messages and/or full log output:

tls is enabled, but listener wrapper returns a connection that doesn’t implement connectionStater

3. Caddy version:

2.11.3

4. How I installed and ran Caddy:

FROM caddy:${CADDY_BUILDER_VERSION} AS builder

RUN xcaddy build ${CADDY_VERSION} \
    --with github.com/caddyserver/ntlm-transport \
    --with github.com/caddyserver/replace-response \
    --with github.com/caddy-dns/cloudflare \
    --with github.com/lucaslorentz/caddy-docker-proxy/v2 \
    --with github.com/mholt/caddy-l4

a. System environment:

Docker

b. Command:

c. Service/unit/compose file:

d. My complete Caddy config:

The full config is 1000 rows spread among 30 files. The http part for the same site is:

@vscode {
	host vscode.portal.{$ACME_DOMAIN}
	remote_ip private_ranges
}

handle @vscode {
    basic_auth {
        vscode "$2a$14$7nq..."
    }


    @inject_cookie {
        path /
        method GET
        not header Cookie *vscode-tkn=*
    }

    uri @inject_cookie query +tkn {env.VSCODE_WEB_TOKEN}

	reverse_proxy http://vscode:8000
}

5. Links to relevant resources:

2

Option 1 is to match after the tls listener wrapper. It should work fine for any domain Caddy serves, not only the specific one you mentioned.

{
  servers {
    listener_wrappers {
      tls
      layer4 {
        @ssh-in-tls ssh
        route @ssh-in-tls {
          proxy tcp/backend:22
        }
      }
    }
  }
}

If this is a problem for you, and you would like to serve SSH on one specific domain only, below is option 2.

{
  servers {
    listener_wrappers {
      layer4 {
        @tls-vscode tls sni vscode.example.com
        route @tls-vscode {
          tls
          subroute {
            @ssh-in-tls ssh
            route @ssh-in-tls {
              proxy tcp/backend:22
            }
            route {
              proxy {
                upstream tcp/backend:443 {
                  tls
                }
              }
            }
          }
        }
      }
      tls
    }
  }
}

Thus, either you terminate TLS with the HTTP app and serve SSH for any domain you have, or terminate TLS with the Layer 4 app for your specific domain and proxy both SSH and HTTP traffic.

3

Thanks for reply. I tried both options and they works both for the ssh part.

Neither works for the http part in this case because I need an L7 http proxy to manipulate query and cookies.

4

Then post your full config, so that we could see if there is anything wrong.

5

Thanks for your reply. I updated my original post to better explain my question. Just to clarify my previous comment, by “neither works” I did not mean that Caddy failed to start or threw an error, but rather that the backend HTTP application itself was not functioning properly.
From what I understand, I suspect the issue is that a Layer 4 fallback proxy—as suggested in those configurations—might not be sufficient in this specific case

6

Below is a full option 1 config that works fine for me. Tested on Windows with Chrome (navigate to https://localhost) and WinSCP (connect to 127.0.0.1:22443).

{
	debug
	layer4 {
		# SSH-in-TLS client setup, so that a standard SSH client could connect to localhost:22443
		tcp/:22443 {
			@ssh ssh
			route @ssh {
				proxy {
					upstream tcp/localhost:443 {
						# when `tls_insecure_skip_verify` is present, `tls` may be omitted
						tls
						# the option below is required, because self-signed certificates are used
						tls_insecure_skip_verify
					}
				}
			}
		}
	}
	servers {
		listener_wrappers {
			# no need to put `tls` below, if it's the first listener wrapper in the list
			# tls
			layer4 {
				@ssh-in-tls ssh
				route @ssh-in-tls {
					proxy tcp/backend:22
				}
			}
		}
		# switch off HTTP/3 for testing
		protocols h1 h2
	}
}

localhost {
	# use self-signed certificates for testing
	tls {
		issuer internal
	}

	respond "OK" 200
}
7

And this is a full option 2 config that works for me as well.

{
	debug
	layer4 {
		# SSH-in-TLS client setup, so that a standard SSH client could connect to localhost:22443
		tcp/:22443 {
			@ssh ssh
			route @ssh {
				proxy {
					upstream tcp/localhost:443 {
						# when `tls_insecure_skip_verify` is present, `tls` may be omitted
						tls
						# the option below is required, because self-signed certificates are used
						tls_insecure_skip_verify
					}
				}
			}
		}
	}
	servers {
		listener_wrappers {
			layer4 {
				@tls tls sni localhost
				route @tls {
					tls
					subroute {
						@ssh-in-tls ssh
						route @ssh-in-tls {
							proxy tcp/backend:22
						}
						route {
							proxy {
								upstream tcp/localhost:443 {
									# when `tls_insecure_skip_verify` is present, `tls` may be omitted
									tls
									# the option below is required, because self-signed certificates are used
									tls_insecure_skip_verify
									# the option below is required, if this is a non-existing domain name
									tls_server_name subdomain.example.com
								}
							}
						}
					}
				}
			}
			tls
		}
		protocols h1 h2
	}
}

localhost subdomain.example.com {
	# use self-signed certificates for testing
	tls {
		issuer internal
	}

	respond "OK" 200
}
8

I do wonder if we can make this easier, especially if we bring l4 into the standard distribution. (Discussion for another topic/issue.)

9

Yes, it would be a bit easier if we could pass a non-SSH decrypted stream after TLS termination by the Layer 4 app to the HTTP app. Then we wouldn’t have to proxy this non-SSH decrypted stream in option 2. Option 1 is already pretty simple.

By the way, I noticed that Caddy complains that the layer4 listener wrapper doesn’t satisfy connectionStater interface while running the option 1 config. I haven’t investigated this issue yet.

10

So it’s not possible right now?

This is what I also noticed.

I admit I’m not sure if I fully understand your options 1 and 2 above. Would you mind to clarify?

In Option 1, the first layer4{} section is useful? We are listening for normal SSH traffic, not for TLS handshaking. The listener_wrappers section is similar to something I tried, except for protocols h1 h2 part: for me it’s working for the SSH traffic but not for http(s).

In Option 2, again the first layer4{} section is listening for normal SSH traffic, not for TLS traffic. An then?

11

No, it isn’t. I’ve clearly stated all available options above. We may consider this feature when/if the layer 4 app gets merged into Caddy. No ETA.

Have you read my comment above the first layer4 block? It states this block is for testing purposes to simulate an SSH-in-TLS client. If something doesn’t work for you, compare the configs and analyze the differences you have.

See the comment above it. Or just drop it, as it isn’t required for the functionality you request.

Overall, why don’t you just copy-paste my configs and try them on your host. Otherwise, this discussion becomes a bit weird.