Can't locally proxy websockets through Caddy v2.7.6

Help
1

1. The problem I’m having:

I am trying to set up Caddy as reverse-proxy for my web-app (locally). It works fine mostly, but the problem I am stuck at, is reverse-proxying the WebSockets on my node.js/Fastify app.

I’ve built a very simplistic app to test this behaviour, it contains a small code-snippet and the Caddyfile I use:
GitHub repo

I would suspect the problem is in a library I use for WebSockets, but when I specifically connect to wss://example.localhost/ws/test (or ws://...) via Postman, Insomnia or even a Python script, it doesn’t seem to even trigger the logs.

Nevertheless, I can connect to the WebSocket directly, as well as access the reverse-proxied HTTP-endpoints.

2. Error messages and/or full log output:

2024/03/31 22:14:21.810 INFO    using provided configuration    {"config_file": "../Git/testfolder/Caddyfile", "config_adapter": ""}
2024/03/31 22:14:21.821 INFO    admin   admin endpoint started  {"address": "localhost:2019", "enforce_origin": false, "origins": ["//localhost:2019", "//[::1]:2019", "//127.0.0.1:2019"]}
2024/03/31 22:14:21.821 INFO    http.auto_https server is listening only on the HTTPS port but has no TLS connection policies; adding one to enable TLS {"server_name": "srv0", "https_port": 443}
2024/03/31 22:14:21.821 INFO    tls.cache.maintenance   started background certificate maintenance      {"cache": "0xc00040df00"}
2024/03/31 22:14:21.821 INFO    http.auto_https enabling automatic HTTP->HTTPS redirects        {"server_name": "srv0"}
2024/03/31 22:14:21.823 DEBUG   http.auto_https adjusted config {"tls": {"automation":{"policies":[{"subjects":["example.localhost"]},{}]}}, "http": {"servers":{"remaining_auto_https_redirects":{"listen":[":80"],"routes":[{},{}]},"srv0":{"listen":[":443"],"routes":[{"handle":[{"handler":"subroute","routes":[{"handle":[{"handler":"reverse_proxy","upstreams":[{"dial":"localhost:3000"}]}],"match":[{"path":["/api/*"]}]},{"handle":[{"handler":"reverse_proxy","upstreams":[{"dial":"localhost:3000"}]}],"match":[{"path":["/ws/*"]}]}]}],"terminal":true}],"tls_connection_policies":[{}],"automatic_https":{}}}}}
2024/03/31 22:14:21.830 WARN    tls     storage cleaning happened too recently; skipping for now        {"storage": "FileStorage:C:\\Users\\79251\\AppData\\Roaming\\Caddy", "instance": "7ebadf96-7be1-4cb4-9214-5f31abf20d24", "try_again": "2024/04/01 22:14:21.830", "try_again_in": 86400}
2024/03/31 22:14:21.831 INFO    tls     finished cleaning storage units
2024/03/31 22:14:21.840 INFO    pki.ca.local    root certificate is already trusted by system   {"path": "storage:pki/authorities/local/root.crt"}
2024/03/31 22:14:21.841 DEBUG   http    starting server loop    {"address": "[::]:80", "tls": false, "http3": false}
2024/03/31 22:14:21.841 INFO    http.log        server running  {"name": "remaining_auto_https_redirects", "protocols": ["h1", "h2", "h3"]}
2024/03/31 22:14:21.841 INFO    http    enabling HTTP/3 listener        {"addr": ":443"}
2024/03/31 22:14:21.842 DEBUG   http    starting server loop    {"address": "[::]:443", "tls": true, "http3": true}
2024/03/31 22:14:21.842 INFO    http.log        server running  {"name": "srv0", "protocols": ["h1", "h2", "h3"]}
2024/03/31 22:14:21.842 INFO    http    enabling automatic TLS certificate management   {"domains": ["example.localhost"]}
2024/03/31 22:14:21.844 WARN    tls     stapling OCSP   {"error": "no OCSP stapling for [example.localhost]: no OCSP server specified in certificate", "identifiers": ["example.localhost"]}
2024/03/31 22:14:21.844 DEBUG   tls.cache       added certificate to cache      {"subjects": ["example.localhost"], "expiration": "2024/04/01 06:50:21.000", "managed": true, "issuer_key": "local", "hash": "fffc8d5a81e2a877123be7d6e25ba07ce1b88cf3a2799a00b0d464f56ffd49ed", "cache_size": 1, "cache_capacity": 10000}
2024/03/31 22:14:21.844 DEBUG   events  event   {"name": "cached_managed_cert", "id": "92febdef-d2e9-42f9-b786-3096691fbdba", "origin": "tls", "data": {"sans":["example.localhost"]}}
2024/03/31 22:14:21.846 INFO    autosaved config (load with --resume flag)      {"file": "C:\\Users\\79251\\AppData\\Roaming\\Caddy\\autosave.json"}
2024/03/31 22:14:21.846 INFO    serving initial configuration

3. Caddy version:

Caddy v2.7.6, amd64, Windows

4. How I installed and ran Caddy:

Just ran the pre-build .exe file with caddy run

b. Command:

caddy run --config=./Caddyfile

d. My complete Caddy config:

{
	debug
}

example.localhost {
	reverse_proxy /api/* localhost:3000
	reverse_proxy /ws/* localhost:3000
}

5. Links to relevant resources:

2

I’m not sure I understand, what is the problem exactly? Are there no errors on both the client and the server?

3

Yep, that’s the strangest behaviour, caddy doesn’t seem to recognize websocket upgrade requests to “example.localhost” at all, which results in ECONNRESET. It doesn’t even trigger logs.

After testing on MacOS and Linux, we have found out that this is a problem specific only to Windows.
I can’t tell right now, whether it is Caddy, or the system itself. But I will try to keep in touch.

If anyone out there with Windows machine can try to reproduce I will greatly appreciate it :slight_smile:

4

It sounds like the request isn’t even getting to caddy. The front end didn’t have any errors either?

5

I suppose it is not getting to Caddy, based on logs.

It’s indeed very hard to debug. While localhost:3001/ws/... is exposed and can be reached through Postman and similar, the reverse-proxied example.localhost/ws/... can’t be.

As of the front end, it does not seem to produce any errors (except ECONNRESET in Postman, after timing out). Back-end doesn’t receive any proxied wss:// requests either.

Maybe the reason is how Windows treats the “.localhost” domains? Anything within http(s) protocol works perfectly.
I tried to add it to hosts but it didn’t change anything.

Nevertheless, I would appreciate if someone could replicate the issue on their machine.

6

Your Caddyfile has localhost:3000. Is that a mistake? Wrong port?

Either way, you should see reverse_proxy debug logs if it ever got to that point.

What do you see when you run curl -v https://example.localhost ? In theory you should see an empty 200 status response (because you didn’t configure Caddy to do anything for a request to the path /).

Unfortunately, Windows does some truly weird shit when it comes to networking. It’s very much a black-box to me. There might be somekind of Windows Firewall rules blocking networking on ports 80/443 for the Caddy process (though it was still able to start and bind to those ports??) You should look into that.

7

It’s just a typo, sorry.
I will try all of these, as well as turning off the firewall

Edit:
Curl to https://example.localhost/ results in:

*   Trying [::1]:443...
* Connected to example.localhost (::1) port 443
* schannel: disabled automatic use of client certificate
* ALPN: curl offers http/1.1
* ALPN: server accepted http/1.1
* using HTTP/1.1
> GET / HTTP/1.1
> Host: example.localhost
> User-Agent: curl/8.4.0
> Accept: */*
>
* schannel: remote party requests renegotiation
* schannel: renegotiating SSL/TLS connection
* schannel: SSL/TLS connection renegotiated
< HTTP/1.1 200 OK
< Alt-Svc: h3=":443"; ma=2592000
< Server: Caddy
< Date: Tue, 02 Apr 2024 08:46:27 GMT
< Content-Length: 0
<
* Connection #0 to host example.localhost left intact

As I mentioned, the only problem I experience, is the inability to connect to websockets, e.g.:

ws://example.localhost:3000/ws/test - connects just fine (Direct access)
https://example.localhost/api/test - works just as well (HTTP is proxied through Caddy)
wss://example.localhost/ws/test - the request seems not to pass to caddy at all, results in ECONNRESET
ws://example.localhost/ws/* (ws:// to any path) - results in error 308 for some reason (probably because the handshake is redirected to https?)

Turning the Firewall didn’t do much :frowning:

Anyways, I am okay to just deal with it, as it seems that it will still all work in the production environment

8

Very strange. And no logs show up when you use enable the debug global option?

Yeah, that’s Caddy’s HTTP->HTTPS redirects. That’s normal.

9

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.