Caddy reverse proxy to another domain

SnaddyvitchDispenser · August 25, 2020, 5:46pm

1. Caddy version (`caddy version`):

v2.1.1 h1:X9k1+ehZPYYrSqBvf/ocUgdLSRIuiNiMo7CvyGUQKeA=

2. How I run Caddy:

a. System environment:

Debian GNU/Linux v10

b. Command:

Caddy is run as a service (installed via apt)

sudo service caddy start

c. Service/unit/compose file:

N/A

d. My complete Caddyfile or JSON config:

(was giving me an error about port missing before adding 443 so that’s why that’s there)

chat.websocket.ws {
        reverse_proxy * 6only.chat.websocket.ws:443
}

3. The problem I’m having:

Ok, so I have an IPv6 only websocket server (running correctly [using caddy]) @ wss://6only.chat.websocket.ws

This is working fine (I’m using caddy on that server to handle SSL) - I can connect from my app and send messages.

The server that I’m having ths issue with is the one that has IPv4, I’m using it as a reverse proxy for the users who don’t have IPv6 (Like myself - I’m using a SOCKS proxy to test the IPv6 only server)

I did try sending my server with both IPv 4 & 6 directly to the backend avoiding caddy (no tls, so I don’t want to use it) and that works just fine, so I wonder if TLS might be part of the issue?

4. Error messages and/or full log output:

Aug 25 17:24:04 nftmart caddy[31337]: {"level":"error","ts":1598376244.2511349,"logger":"http.handlers.reverse_proxy","msg":"reading from backend","error":"read tcp [2a07:e01:3:2b::1]:50818->[2a07:e01:3:1d::1]:443: read: connection reset by peer"}
Aug 25 17:24:04 nftmart caddy[31337]: {"level":"error","ts":1598376244.2530627,"logger":"http.handlers.reverse_proxy","msg":"aborting with incomplete response","error":"read tcp [2a07:e01:3:2b::1]:50818->[2a07:e01:3:1d::1]:443: read: connection reset by peer"}
Aug 25 17:25:04 nftmart caddy[31337]: {"level":"error","ts":1598376304.5920196,"logger":"http.handlers.reverse_proxy","msg":"reading from backend","error":"read tcp [2a07:e01:3:2b::1]:50830->[2a07:e01:3:1d::1]:443: read: connection reset by peer"}
Aug 25 17:25:04 nftmart caddy[31337]: {"level":"error","ts":1598376304.5921538,"logger":"http.handlers.reverse_proxy","msg":"aborting with incomplete response","error":"read tcp [2a07:e01:3:2b::1]:50830->[2a07:e01:3:1d::1]:443: read: connection reset by peer"}
Aug 25 17:26:05 nftmart caddy[31337]: {"level":"error","ts":1598376365.4409103,"logger":"http.handlers.reverse_proxy","msg":"reading from backend","error":"read tcp [2a07:e01:3:2b::1]:50838->[2a07:e01:3:1d::1]:443: read: connection reset by peer"}
Aug 25 17:26:05 nftmart caddy[31337]: {"level":"error","ts":1598376365.4410937,"logger":"http.handlers.reverse_proxy","msg":"aborting with incomplete response","error":"read tcp [2a07:e01:3:2b::1]:50838->[2a07:e01:3:1d::1]:443: read: connection reset by peer"}
Aug 25 17:27:05 nftmart caddy[31337]: {"level":"error","ts":1598376425.9898794,"logger":"http.handlers.reverse_proxy","msg":"reading from backend","error":"read tcp [2a07:e01:3:2b::1]:50842->[2a07:e01:3:1d::1]:443: read: connection reset by peer"}
Aug 25 17:27:05 nftmart caddy[31337]: {"level":"error","ts":1598376425.9900362,"logger":"http.handlers.reverse_proxy","msg":"aborting with incomplete response","error":"read tcp [2a07:e01:3:2b::1]:50842->[2a07:e01:3:1d::1]:443: read: connection reset by peer"}
Aug 25 17:28:06 nftmart caddy[31337]: {"level":"error","ts":1598376486.6543398,"logger":"http.handlers.reverse_proxy","msg":"reading from backend","error":"read tcp [2a07:e01:3:2b::1]:50852->[2a07:e01:3:1d::1]:443: read: connection reset by peer"}
Aug 25 17:28:06 nftmart caddy[31337]: {"level":"error","ts":1598376486.6544511,"logger":"http.handlers.reverse_proxy","msg":"aborting with incomplete response","error":"read tcp [2a07:e01:3:2b::1]:50852->[2a07:e01:3:1d::1]:443: read: connection reset by peer"}

5. What I already tried:

So I have tried the following:
Connecting directly to the backend, which did work [Backend doesn’t have tls and I don’t want to send this data plaintext].
Removing the port from the above config

Aug 25 17:36:49 nftmart caddy[31720]: {"level":"info","ts":1598377009.8251972,"msg":"using provided configuration","config_file":"/etc/caddy/Caddyfile","config_adapter":""}
Aug 25 17:36:49 nftmart caddy[31720]: {"level":"info","ts":1598377009.8321679,"logger":"admin","msg":"admin endpoint started","address":"tcp/localhost:2019","enforce_origin":false,"origins":["localhost:2019","[::1]:2019","127.0.0.1:2019"]}
Aug 25 17:36:49 nftmart caddy[31720]: {"level":"info","ts":1598377009.8328273,"logger":"http","msg":"server is listening only on the HTTPS port but has no TLS connection policies; adding one to enable TLS","server_name":"srv0","https_port":443}
Aug 25 17:36:49 nftmart caddy[31720]: 2020/08/25 17:36:49 [INFO][cache:0xc00012bce0] Started certificate maintenance routine
Aug 25 17:36:49 nftmart caddy[31720]: {"level":"info","ts":1598377009.8328917,"logger":"http","msg":"enabling automatic HTTP->HTTPS redirects","server_name":"srv0"}
Aug 25 17:36:49 nftmart caddy[31720]: {"level":"info","ts":1598377009.84113,"logger":"tls","msg":"cleaned up storage units"}
Aug 25 17:36:49 nftmart caddy[31720]: {"level":"info","ts":1598377009.8414717,"logger":"http","msg":"enabling automatic TLS certificate management","domains":["phpmyadmin.ws2.nftm.art","nextcloud.nftm.art","chat.websocket.ws","phabricator.nftm.art","nftmart-files-phabricator.websocket.ws"]}
Aug 25 17:36:49 nftmart caddy[31720]: {"level":"info","ts":1598377009.8725863,"msg":"autosaved config","file":"/var/lib/caddy/.config/caddy/autosave.json"}
Aug 25 17:36:49 nftmart caddy[31720]: {"level":"info","ts":1598377009.8730967,"msg":"serving initial configuration"}
Aug 25 17:37:12 nftmart caddy[31720]: {"level":"error","ts":1598377032.8222232,"logger":"http.log.error","msg":"making dial info: upstream 6only.chat.websocket.ws:: invalid dial address 6only.chat.websocket.ws:: invalid start port: strconv.ParseUint: parsing \"\": invalid syntax","request":{"method":"GET","uri":"/","proto":"HTTP/1.1","remote_addr":"[2a07:e01:3:2b::1]:54494","host":"chat.websocket.ws","headers":{"Accept-Encoding":["gzip, deflate, br"],"Sec-Websocket-Version":["13"],"Sec-Websocket-Key":["/1C+h7QWNyuoNORAEXg/EA=="],"Dnt":["1"],"Connection":["keep-alive, Upgrade"],"Accept":["*/*"],"Sec-Websocket-Extensions":["permessage-deflate"],"Cache-Control":["no-cache"],"Upgrade":["websocket"],"Accept-Language":["en-GB,en;q=0.5"],"Origin":["https://hoppscotch.io"],"Pragma":["no-cache"],"User-Agent":["Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:79.0) Gecko/20100101 Firefox/79.0"]},"tls":{"resumed":true,"version":772,"ciphersuite":4865,"proto":"http/1.1","proto_mutual":true,"server_name":"chat.websocket.ws"}},"duration":0.000120127}

Connecting directly to the ipv6 ip to the backend (didn’t work) with or without port
Adding https:// to the address and taking away port (doesn’t work) - Leads To Similar Outcome:

Poorly drawn diagram because I don’t always explain well:

6. Links to relevant resources:

7. Footnotes

If you want to test it, only selected origins are allowed. Use: https://hoppscotch.io/realtime

francislavoie · August 25, 2020, 8:29pm

If your backend uses https, then you should specify https:// on the proxy address so that Caddy enables TLS communication:

reverse_proxy https://6only.chat.websocket.ws

SnaddyvitchDispenser · August 26, 2020, 5:17pm

Thanks for the suggestion, I have already tried this and it leads to the wrong outcome:

Caddy Logs:

Aug 26 17:11:45 nftmart caddy[20964]: {"level":"info","ts":1598461905.802213,"msg":"autosaved config","file":"/var/lib/caddy/.config/caddy/autosave.json"}
Aug 26 17:11:45 nftmart caddy[20964]: {"level":"info","ts":1598461905.8022697,"msg":"serving initial configuration"}

(It doesn’t appear to be being logged )

I also tried replacing https:// with wss:// (because it’s a websocket) and no luck either

And If I try :443, it returns 400 bad request (no log) in caddy

francislavoie · August 26, 2020, 8:03pm

Hmm, I don’t see any problem in that first screenshot, actually. I’m not sure I understand.

FYI in Caddy, https:// essentially tells Caddy “enable TLS for this proxy”. Does your upstream have a 6only.chat.websocket.ws certificate that the one at the edge can trust? Looks like yes because it’s served by Caddy as well it seems.

SnaddyvitchDispenser · August 26, 2020, 8:45pm

It should be serving the following headers (because it’s a websocket):
Connection: Upgrade
Upgrade: websocket

Because these aren’t appearing (and my websocket can’t connect) - test at https://hoppscotch.io/realtime

I’ve tried throwing in the headers myself, but that doesn’t solve it as the websocket still isn’t there.

Yes, the upstream has a valid certificate (that was obtained from letsencypt by caddy)

It’s not being rejected by my backend either, because the backend is not seeing requests from it.

One request via 6only.chat.websocket.ws:

One request via chat.websocket.ws:

Nothing…

francislavoie · August 26, 2020, 9:05pm

Heh, okay, I think I have an inkling of what’s going on…

github.com

caddyserver/caddy/blob/master/modules/caddyhttp/reverseproxy/streaming.go#L35

    
      
          	"net/http"
          	"sync"
          	"time"
          
          
	"go.uber.org/zap"
          )
          
          
func (h Handler) handleUpgradeResponse(logger *zap.Logger, rw http.ResponseWriter, req *http.Request, res *http.Response) {
          	reqUpType := upgradeType(req.Header)
          	resUpType := upgradeType(res.Header)
          	// TODO: Update to use "net/http/internal/ascii" once we bumped
          	// the minimum Go version to 1.17.
          	// See https://github.com/golang/go/commit/5c489514bc5e61ad9b5b07bd7d8ec65d66a0512a
          	if reqUpType != resUpType {
          		h.logger.Debug("backend tried to switch to unexpected protocol via Upgrade header",
          			zap.String("backend_upgrade", resUpType),
          			zap.String("requested_upgrade", reqUpType))
          		return
          	}
          
          
	hj, ok := rw.(http.Hijacker)

This is the code that handles Upgrade, i.e. for websockets. Logging was turned off here because this code was copied directly from the Go stdlib, but when it was copied, the error handling was commented out to be handled later. I guess it was forgotten.

I’ll see if I can get you a build later this evening or sometime tomorrow, time permitting, that has this logging turned back on to see where it actually fails.

SnaddyvitchDispenser · August 26, 2020, 9:05pm

Thank you, that’d be very helpful!

francislavoie · August 26, 2020, 9:18pm

Wrote a quick PR:

https://github.com/caddyserver/caddy/pull/3689

You can grab one of the CI artifacts from here: reverseproxy: Enable error logging for connection upgrades · caddyserver/caddy@708c420 · GitHub (you might need to wait a couple minutes for the tests to finish)

I’m not sure whether those errors will appear on the backend or on the edge Caddy instances, so maybe try it on both?

SnaddyvitchDispenser · August 26, 2020, 11:06pm

Thank you. I’ll try it tomorrow.

colachg · August 27, 2020, 7:35am

Ran into same problem: (same caddy version):

caddyfile:

https://foo.bar.com {
    reverse_proxy *  https://discord.com
    tls {
        dns cloudflare {env.CLOUDFLARE_API_TOKEN}
    }
}

when I visited foo.bar.com, it returns 403 forbidden.

@francislavoie Could you try it?

SnaddyvitchDispenser · August 27, 2020, 6:30pm

The only place I’ve been seeing errors is on the edge caddy instance. I now tried it with HTTPS and it still isn’t spitting out errors

Aug 27 18:25:03 nftmart caddy[12394]: {"level":"info","ts":1598552703.4456997,"logger":"tls","msg":"cleaned up storage units"}
Aug 27 18:25:03 nftmart caddy[12394]: {"level":"info","ts":1598552703.4635384,"msg":"autosaved config","file":"/var/lib/caddy/.config/caddy/autosave.json"}
Aug 27 18:25:03 nftmart caddy[12394]: {"level":"info","ts":1598552703.4635932,"msg":"serving initial configuration"}

Caddy seems to think it’s in the right though and everything is 200 OK, but it’s not doing what I expect still

Backend - No logs either

Aug 27 18:33:58 dmessages caddy[6009]: {"level":"info","ts":1598553238.3852093,"logger":"http","msg":"enabling automatic HTTP->HTTPS redirects","server_name":"srv0"}
Aug 27 18:33:58 dmessages caddy[6009]: {"level":"info","ts":1598553238.4156954,"logger":"http","msg":"enabling automatic TLS certificate management","domains":["6only.chat.websocket.ws"]}
Aug 27 18:33:58 dmessages caddy[6009]: {"level":"info","ts":1598553238.417535,"logger":"tls","msg":"cleaned up storage units"}
Aug 27 18:33:58 dmessages caddy[6009]: {"level":"info","ts":1598553238.4600666,"msg":"autosaved config","file":"/var/lib/caddy/.config/caddy/autosave.json"}
Aug 27 18:33:58 dmessages caddy[6009]: {"level":"info","ts":1598553238.4600976,"msg":"serving initial configuration"}

SnaddyvitchDispenser · September 1, 2020, 6:12pm

Hey again,

Sorry to bother you, but is there any update on this issue?

matt · September 1, 2020, 8:38pm

It could be useful to see what headers are going from hop to hop. If Caddy’s reverse proxy sees an Upgrade header for websockets, it will attempt to establish a websocket connection with the backend. Come to think of it, I’m not sure I’ve tried proxying websockets through two hops. Am I right in understand that’s what is happening here?

francislavoie · September 1, 2020, 10:52pm

Would it be viable at all for you to proxy websockets directly to your backend, skipping the 2nd caddy, if it came from the ipv4/6 box?

Something like this might do, until we figure out what’s going on:

chat.websocket.ws {
	@websockets {
		header Connection *Upgrade*
		header Upgrade    websocket
	}
	reverse_proxy @websockets 6only.chat.websocket.ws:6001

	reverse_proxy https://6only.chat.websocket.ws
}

This assumes you also expose say, port 6001, for direct websocket access. Kinda gross, but I think it would do the trick for now?

francislavoie · September 1, 2020, 10:53pm

@colachg I think you’d running into an entirely separate issue. Could you please open a new thread, and fill out the thread template? I don’t want to have two separate conversations at the same time in one thread. Thanks.

francislavoie · September 1, 2020, 10:57pm

@SnaddyvitchDispenser I don’t think I yet asked you to enable debug logging yet. Could you add this to the top of both your Caddyfiles?

{
	debug
}

This should add more information to the logs, and we should be able to find out more of what’s going on at the proxy layer.

John_Siu · September 2, 2020, 10:52am

It happened to me with https, not sure if solution apply to websockets.

Front end caddy:

stg.example.com {
	reverse_proxy * https://int.example.com
}

Somehow the proxy connection complaint the backend is not stg and end up with empty reply. debug didn’t anything in my case. I enabled logging at site level on both front and back end caddy:

stg.example.com {
	log {
		format json
		output stdout
	}
	reverse_proxy * https://int.example.com

I did some cert trick to get it going during that time. But now there is a proper solution in the doc page example reverse_proxy (Caddyfile directive) — Caddy Documentation

stg.example.com {
	reverse_proxy * https://int.example.com  {
		header_up Host {http.reverse_proxy.upstream.hostport}
	}
}

header_up Host {http.reverse_proxy.upstream.hostport} replace the request header from front end to back end.

SnaddyvitchDispenser · September 3, 2020, 11:38am

Thank you! It worked. And it was even in the docs my bad. Thank you for solving my issue! I thought I had tried that already but obviously I must’ve done something wrong when I tried it.

francislavoie · September 3, 2020, 5:27pm

Sweet! Glad that worked out. Thanks so much @John_Siu for the idea

I’m glad it wasn’t some deeper issue with websockets, that would’ve been a pain to debug further.

system · September 24, 2020, 5:46pm

This topic was automatically closed after 30 days. New replies are no longer allowed.