Caddy reverse proxy to another domain

1. Caddy version (caddy version):

v2.1.1 h1:X9k1+ehZPYYrSqBvf/ocUgdLSRIuiNiMo7CvyGUQKeA=

2. How I run Caddy:

a. System environment:

Debian GNU/Linux v10

b. Command:

Caddy is run as a service (installed via apt)

sudo service caddy start

c. Service/unit/compose file:

N/A

d. My complete Caddyfile or JSON config:

(was giving me an error about port missing before adding 443 so that’s why that’s there)

chat.websocket.ws {
        reverse_proxy * 6only.chat.websocket.ws:443
}

3. The problem I’m having:

Ok, so I have an IPv6 only websocket server (running correctly [using caddy]) @ wss://6only.chat.websocket.ws

This is working fine (I’m using caddy on that server to handle SSL) - I can connect from my app and send messages.

The server that I’m having ths issue with is the one that has IPv4, I’m using it as a reverse proxy for the users who don’t have IPv6 (Like myself - I’m using a SOCKS proxy to test the IPv6 only server)

I did try sending my server with both IPv 4 & 6 directly to the backend avoiding caddy (no tls, so I don’t want to use it) and that works just fine, so I wonder if TLS might be part of the issue?

4. Error messages and/or full log output:

image

Aug 25 17:24:04 nftmart caddy[31337]: {"level":"error","ts":1598376244.2511349,"logger":"http.handlers.reverse_proxy","msg":"reading from backend","error":"read tcp [2a07:e01:3:2b::1]:50818->[2a07:e01:3:1d::1]:443: read: connection reset by peer"}
Aug 25 17:24:04 nftmart caddy[31337]: {"level":"error","ts":1598376244.2530627,"logger":"http.handlers.reverse_proxy","msg":"aborting with incomplete response","error":"read tcp [2a07:e01:3:2b::1]:50818->[2a07:e01:3:1d::1]:443: read: connection reset by peer"}
Aug 25 17:25:04 nftmart caddy[31337]: {"level":"error","ts":1598376304.5920196,"logger":"http.handlers.reverse_proxy","msg":"reading from backend","error":"read tcp [2a07:e01:3:2b::1]:50830->[2a07:e01:3:1d::1]:443: read: connection reset by peer"}
Aug 25 17:25:04 nftmart caddy[31337]: {"level":"error","ts":1598376304.5921538,"logger":"http.handlers.reverse_proxy","msg":"aborting with incomplete response","error":"read tcp [2a07:e01:3:2b::1]:50830->[2a07:e01:3:1d::1]:443: read: connection reset by peer"}
Aug 25 17:26:05 nftmart caddy[31337]: {"level":"error","ts":1598376365.4409103,"logger":"http.handlers.reverse_proxy","msg":"reading from backend","error":"read tcp [2a07:e01:3:2b::1]:50838->[2a07:e01:3:1d::1]:443: read: connection reset by peer"}
Aug 25 17:26:05 nftmart caddy[31337]: {"level":"error","ts":1598376365.4410937,"logger":"http.handlers.reverse_proxy","msg":"aborting with incomplete response","error":"read tcp [2a07:e01:3:2b::1]:50838->[2a07:e01:3:1d::1]:443: read: connection reset by peer"}
Aug 25 17:27:05 nftmart caddy[31337]: {"level":"error","ts":1598376425.9898794,"logger":"http.handlers.reverse_proxy","msg":"reading from backend","error":"read tcp [2a07:e01:3:2b::1]:50842->[2a07:e01:3:1d::1]:443: read: connection reset by peer"}
Aug 25 17:27:05 nftmart caddy[31337]: {"level":"error","ts":1598376425.9900362,"logger":"http.handlers.reverse_proxy","msg":"aborting with incomplete response","error":"read tcp [2a07:e01:3:2b::1]:50842->[2a07:e01:3:1d::1]:443: read: connection reset by peer"}
Aug 25 17:28:06 nftmart caddy[31337]: {"level":"error","ts":1598376486.6543398,"logger":"http.handlers.reverse_proxy","msg":"reading from backend","error":"read tcp [2a07:e01:3:2b::1]:50852->[2a07:e01:3:1d::1]:443: read: connection reset by peer"}
Aug 25 17:28:06 nftmart caddy[31337]: {"level":"error","ts":1598376486.6544511,"logger":"http.handlers.reverse_proxy","msg":"aborting with incomplete response","error":"read tcp [2a07:e01:3:2b::1]:50852->[2a07:e01:3:1d::1]:443: read: connection reset by peer"}

5. What I already tried:

So I have tried the following:
Connecting directly to the backend, which did work [Backend doesn’t have tls and I don’t want to send this data plaintext].
Removing the port from the above config

Aug 25 17:36:49 nftmart caddy[31720]: {"level":"info","ts":1598377009.8251972,"msg":"using provided configuration","config_file":"/etc/caddy/Caddyfile","config_adapter":""}
Aug 25 17:36:49 nftmart caddy[31720]: {"level":"info","ts":1598377009.8321679,"logger":"admin","msg":"admin endpoint started","address":"tcp/localhost:2019","enforce_origin":false,"origins":["localhost:2019","[::1]:2019","127.0.0.1:2019"]}
Aug 25 17:36:49 nftmart caddy[31720]: {"level":"info","ts":1598377009.8328273,"logger":"http","msg":"server is listening only on the HTTPS port but has no TLS connection policies; adding one to enable TLS","server_name":"srv0","https_port":443}
Aug 25 17:36:49 nftmart caddy[31720]: 2020/08/25 17:36:49 [INFO][cache:0xc00012bce0] Started certificate maintenance routine
Aug 25 17:36:49 nftmart caddy[31720]: {"level":"info","ts":1598377009.8328917,"logger":"http","msg":"enabling automatic HTTP->HTTPS redirects","server_name":"srv0"}
Aug 25 17:36:49 nftmart caddy[31720]: {"level":"info","ts":1598377009.84113,"logger":"tls","msg":"cleaned up storage units"}
Aug 25 17:36:49 nftmart caddy[31720]: {"level":"info","ts":1598377009.8414717,"logger":"http","msg":"enabling automatic TLS certificate management","domains":["phpmyadmin.ws2.nftm.art","nextcloud.nftm.art","chat.websocket.ws","phabricator.nftm.art","nftmart-files-phabricator.websocket.ws"]}
Aug 25 17:36:49 nftmart caddy[31720]: {"level":"info","ts":1598377009.8725863,"msg":"autosaved config","file":"/var/lib/caddy/.config/caddy/autosave.json"}
Aug 25 17:36:49 nftmart caddy[31720]: {"level":"info","ts":1598377009.8730967,"msg":"serving initial configuration"}
Aug 25 17:37:12 nftmart caddy[31720]: {"level":"error","ts":1598377032.8222232,"logger":"http.log.error","msg":"making dial info: upstream 6only.chat.websocket.ws:: invalid dial address 6only.chat.websocket.ws:: invalid start port: strconv.ParseUint: parsing \"\": invalid syntax","request":{"method":"GET","uri":"/","proto":"HTTP/1.1","remote_addr":"[2a07:e01:3:2b::1]:54494","host":"chat.websocket.ws","headers":{"Accept-Encoding":["gzip, deflate, br"],"Sec-Websocket-Version":["13"],"Sec-Websocket-Key":["/1C+h7QWNyuoNORAEXg/EA=="],"Dnt":["1"],"Connection":["keep-alive, Upgrade"],"Accept":["*/*"],"Sec-Websocket-Extensions":["permessage-deflate"],"Cache-Control":["no-cache"],"Upgrade":["websocket"],"Accept-Language":["en-GB,en;q=0.5"],"Origin":["https://hoppscotch.io"],"Pragma":["no-cache"],"User-Agent":["Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:79.0) Gecko/20100101 Firefox/79.0"]},"tls":{"resumed":true,"version":772,"ciphersuite":4865,"proto":"http/1.1","proto_mutual":true,"server_name":"chat.websocket.ws"}},"duration":0.000120127}

Connecting directly to the ipv6 ip to the backend (didn’t work) with or without port
Adding https:// to the address and taking away port (doesn’t work) - Leads To Similar Outcome:

image

Poorly drawn diagram because I don’t always explain well:

6. Links to relevant resources:

7. Footnotes

If you want to test it, only selected origins are allowed. Use: Hoppscotch - Open source API development ecosystem

If your backend uses https, then you should specify https:// on the proxy address so that Caddy enables TLS communication:

reverse_proxy https://6only.chat.websocket.ws

Thanks for the suggestion, I have already tried this and it leads to the wrong outcome:

image

Caddy Logs:

Aug 26 17:11:45 nftmart caddy[20964]: {"level":"info","ts":1598461905.802213,"msg":"autosaved config","file":"/var/lib/caddy/.config/caddy/autosave.json"}
Aug 26 17:11:45 nftmart caddy[20964]: {"level":"info","ts":1598461905.8022697,"msg":"serving initial configuration"}

(It doesn’t appear to be being logged :thinking:)

I also tried replacing https:// with wss:// (because it’s a websocket) and no luck either

And If I try :443, it returns 400 bad request (no log) in caddy

image

Hmm, I don’t see any problem in that first screenshot, actually. I’m not sure I understand.

FYI in Caddy, https:// essentially tells Caddy “enable TLS for this proxy”. Does your upstream have a 6only.chat.websocket.ws certificate that the one at the edge can trust? Looks like yes because it’s served by Caddy as well it seems.

It should be serving the following headers (because it’s a websocket):
Connection: Upgrade
Upgrade: websocket

Because these aren’t appearing (and my websocket can’t connect) - test at Hoppscotch - Open source API development ecosystem

I’ve tried throwing in the headers myself, but that doesn’t solve it as the websocket still isn’t there.

Yes, the upstream has a valid certificate (that was obtained from letsencypt by caddy)

It’s not being rejected by my backend either, because the backend is not seeing requests from it.

One request via 6only.chat.websocket.ws:

One request via chat.websocket.ws:

Nothing…

Heh, okay, I think I have an inkling of what’s going on…

This is the code that handles Upgrade, i.e. for websockets. Logging was turned off here because this code was copied directly from the Go stdlib, but when it was copied, the error handling was commented out to be handled later. I guess it was forgotten.

I’ll see if I can get you a build later this evening or sometime tomorrow, time permitting, that has this logging turned back on to see where it actually fails.

Thank you, that’d be very helpful!

Wrote a quick PR:

https://github.com/caddyserver/caddy/pull/3689

You can grab one of the CI artifacts from here: reverseproxy: Enable error logging for connection upgrades · caddyserver/caddy@708c420 · GitHub (you might need to wait a couple minutes for the tests to finish)

I’m not sure whether those errors will appear on the backend or on the edge Caddy instances, so maybe try it on both?

Thank you. I’ll try it tomorrow.

Ran into same problem: (same caddy version):

caddyfile:

https://foo.bar.com {
    reverse_proxy *  https://discord.com
    tls {
        dns cloudflare {env.CLOUDFLARE_API_TOKEN}
    }
}

when I visited foo.bar.com, it returns 403 forbidden.

@francislavoie Could you try it?

The only place I’ve been seeing errors is on the edge caddy instance. I now tried it with HTTPS and it still isn’t spitting out errors

Aug 27 18:25:03 nftmart caddy[12394]: {"level":"info","ts":1598552703.4456997,"logger":"tls","msg":"cleaned up storage units"}
Aug 27 18:25:03 nftmart caddy[12394]: {"level":"info","ts":1598552703.4635384,"msg":"autosaved config","file":"/var/lib/caddy/.config/caddy/autosave.json"}
Aug 27 18:25:03 nftmart caddy[12394]: {"level":"info","ts":1598552703.4635932,"msg":"serving initial configuration"}

Caddy seems to think it’s in the right though and everything is 200 OK, but it’s not doing what I expect still :confused:

Backend - No logs either

Aug 27 18:33:58 dmessages caddy[6009]: {"level":"info","ts":1598553238.3852093,"logger":"http","msg":"enabling automatic HTTP->HTTPS redirects","server_name":"srv0"}
Aug 27 18:33:58 dmessages caddy[6009]: {"level":"info","ts":1598553238.4156954,"logger":"http","msg":"enabling automatic TLS certificate management","domains":["6only.chat.websocket.ws"]}
Aug 27 18:33:58 dmessages caddy[6009]: {"level":"info","ts":1598553238.417535,"logger":"tls","msg":"cleaned up storage units"}
Aug 27 18:33:58 dmessages caddy[6009]: {"level":"info","ts":1598553238.4600666,"msg":"autosaved config","file":"/var/lib/caddy/.config/caddy/autosave.json"}
Aug 27 18:33:58 dmessages caddy[6009]: {"level":"info","ts":1598553238.4600976,"msg":"serving initial configuration"}

Hey again,

Sorry to bother you, but is there any update on this issue?

It could be useful to see what headers are going from hop to hop. If Caddy’s reverse proxy sees an Upgrade header for websockets, it will attempt to establish a websocket connection with the backend. Come to think of it, I’m not sure I’ve tried proxying websockets through two hops. Am I right in understand that’s what is happening here?

Would it be viable at all for you to proxy websockets directly to your backend, skipping the 2nd caddy, if it came from the ipv4/6 box?

Something like this might do, until we figure out what’s going on:

chat.websocket.ws {
	@websockets {
		header Connection *Upgrade*
		header Upgrade    websocket
	}
	reverse_proxy @websockets 6only.chat.websocket.ws:6001

	reverse_proxy https://6only.chat.websocket.ws
}

This assumes you also expose say, port 6001, for direct websocket access. Kinda gross, but I think it would do the trick for now?

@colachg I think you’d running into an entirely separate issue. Could you please open a new thread, and fill out the thread template? I don’t want to have two separate conversations at the same time in one thread. Thanks.

@SnaddyvitchDispenser I don’t think I yet asked you to enable debug logging yet. Could you add this to the top of both your Caddyfiles?

{
	debug
}

This should add more information to the logs, and we should be able to find out more of what’s going on at the proxy layer.

It happened to me with https, not sure if solution apply to websockets.

Front end caddy:

stg.example.com {
	reverse_proxy * https://int.example.com
}

Somehow the proxy connection complaint the backend is not stg and end up with empty reply. debug didn’t anything in my case. I enabled logging at site level on both front and back end caddy:

stg.example.com {
	log {
		format json
		output stdout
	}
	reverse_proxy * https://int.example.com

I did some cert trick to get it going during that time. But now there is a proper solution in the doc page example reverse_proxy (Caddyfile directive) — Caddy Documentation

stg.example.com {
	reverse_proxy * https://int.example.com  {
		header_up Host {http.reverse_proxy.upstream.hostport}
	}
}

header_up Host {http.reverse_proxy.upstream.hostport} replace the request header from front end to back end.

3 Likes

Thank you! It worked. And it was even in the docs :man_facepalming: my bad. Thank you for solving my issue! I thought I had tried that already but obviously I must’ve done something wrong when I tried it.

:grin:

1 Like

Sweet! Glad that worked out. Thanks so much @John_Siu for the idea :smiley:

I’m glad it wasn’t some deeper issue with websockets, that would’ve been a pain to debug further.

This topic was automatically closed after 30 days. New replies are no longer allowed.