Trouble with websocket proxy

I’m having some trouble configuring a proxy to HomeAssistant, specifically with the websocket connection.

The problem is that the client never gets the response when the connection is upgraded.

I’ve included some snippits from tcpdump showing the issue. This is the view from the server running caddy, communicating with the client:

15:07:35.593092 IP remote.ip.address.14094 > 10.10.5.1.80: Flags [P.], seq 166503828:166504345, ack 1390314881, win 4117, options [nop,nop,TS val 670613962 ecr 3044404422], length 517
	0x0000:  4520 0239 7ef5 4000 3006 325f d041 b7fe  E..9~.@.0.2_.A..
	0x0010:  0a0a 0501 370e 0050 09ec a594 52de 8581  ....7..P....R...
	0x0020:  8018 1015 a858 0000 0101 080a 27f8 c1ca  .....X......'...
	0x0030:  b575 ecc6 4745 5420 2f61 7069 2f77 6562  .u..GET./api/web
	0x0040:  736f 636b 6574 2048 5454 502f 312e 310d  socket.HTTP/1.1.
	0x0050:  0a48 6f73 743a 2068 612e 6d61 7474 7669  .Host:.ha.mattvi
	0x0060:  7274 2e63 6f6d 0d0a 436f 6e6e 6563 7469  rt.com..Connecti
	0x0070:  6f6e 3a20 5570 6772 6164 650d 0a50 7261  on:.Upgrade..Pra
	0x0080:  676d 613a 206e 6f2d 6361 6368 650d 0a43  gma:.no-cache..C
	0x0090:  6163 6865 2d43 6f6e 7472 6f6c 3a20 6e6f  ache-Control:.no
	0x00a0:  2d63 6163 6865 0d0a 5570 6772 6164 653a  -cache..Upgrade:
	0x00b0:  2077 6562 736f 636b 6574 0d0a 4f72 6967  .websocket..Orig
	0x00c0:  696e 3a20 6874 7470 3a2f 2f68 612e 6d61  in:.http://ha.ma
	0x00d0:  7474 7669 7274 2e63 6f6d 0d0a 5365 632d  ttvirt.com..Sec-
	0x00e0:  5765 6253 6f63 6b65 742d 5665 7273 696f  WebSocket-Versio
	0x00f0:  6e3a 2031 330d 0a55 7365 722d 4167 656e  n:.13..User-Agen
	0x0100:  743a 204d 6f7a 696c 6c61 2f35 2e30 2028  t:.Mozilla/5.0.(
	0x0110:  4d61 6369 6e74 6f73 683b 2049 6e74 656c  Macintosh;.Intel
	0x0120:  204d 6163 204f 5320 5820 3130 5f31 315f  .Mac.OS.X.10_11_
	0x0130:  3629 2041 7070 6c65 5765 624b 6974 2f35  6).AppleWebKit/5
	0x0140:  3337 2e33 3620 284b 4854 4d4c 2c20 6c69  37.36.(KHTML,.li
	0x0150:  6b65 2047 6563 6b6f 2920 4368 726f 6d65  ke.Gecko).Chrome
	0x0160:  2f35 352e 302e 3238 3833 2e37 3520 5361  /55.0.2883.75.Sa
	0x0170:  6661 7269 2f35 3337 2e33 360d 0a41 6363  fari/537.36..Acc
	0x0180:  6570 742d 456e 636f 6469 6e67 3a20 677a  ept-Encoding:.gz
	0x0190:  6970 2c20 6465 666c 6174 652c 2073 6463  ip,.deflate,.sdc
	0x01a0:  680d 0a41 6363 6570 742d 4c61 6e67 7561  h..Accept-Langua
	0x01b0:  6765 3a20 656e 2d55 532c 656e 3b71 3d30  ge:.en-US,en;q=0
	0x01c0:  2e38 0d0a 5365 632d 5765 6253 6f63 6b65  .8..Sec-WebSocke
	0x01d0:  742d 4b65 793a 207a 7671 3566 4e79 5146  t-Key:.zvq5fNyQF
	0x01e0:  7734 4a64 2f43 4455 4276 5069 773d 3d0d  w4Jd/CDUBvPiw==.
	0x01f0:  0a53 6563 2d57 6562 536f 636b 6574 2d45  .Sec-WebSocket-E
	0x0200:  7874 656e 7369 6f6e 733a 2070 6572 6d65  xtensions:.perme
	0x0210:  7373 6167 652d 6465 666c 6174 653b 2063  ssage-deflate;.c
	0x0220:  6c69 656e 745f 6d61 785f 7769 6e64 6f77  lient_max_window
	0x0230:  5f62 6974 730d 0a0d 0a                   _bits....
15:07:35.593127 IP 10.10.5.1.80 > remote.ip.address.14094: Flags [.], ack 166504345, win 235, options [nop,nop,TS val 3044404427 ecr 670613962], length 0
	0x0000:  4500 0034 694a 4000 4006 3a2f 0a0a 0501  E..4iJ@.@.:/....
	0x0010:  d041 b7fe 0050 370e 52de 8581 09ec a799  .A...P7.R.......
	0x0020:  8010 00eb 9771 0000 0101 080a b575 eccb  .....q.......u..
	0x0030:  27f8 c1ca                                '...
15:07:44.353401 IP remote.ip.address.21464 > 10.10.5.1.80: Flags [.], ack 1569076164, win 4117, length 0
	0x0000:  4520 0028 5ae3 0000 3006 9882 d041 b7fe  E..(Z...0....A..
	0x0010:  0a0a 0501 53d8 0050 02c8 42c9 5d86 33c4  ....S..P..B.].3.
	0x0020:  5010 1015 dd70 0000 0000 0000 0000       P....p........
15:07:44.353456 IP 10.10.5.1.80 > remote.ip.address.21464: Flags [.], ack 46678730, win 235, options [nop,nop,TS val 3044406617 ecr 670487496], length 0
	0x0000:  4500 0034 1639 4000 4006 8d40 0a0a 0501  E..4.9@.@..@....
	0x0010:  d041 b7fe 0050 53d8 5d86 33c4 02c8 42ca  .A...PS.].3...B.
	0x0020:  8010 00eb 9771 0000 0101 080a b575 f559  .....q.......u.Y
	0x0030:  27f6 d3c8                                '...

This is the view from the server running caddy, communicating with the backend HomeAssistant server:

15:07:35.594968 IP 10.10.5.1.48472 > 10.10.4.44.8123: Flags [P.], seq 1313885066:1313885616, ack 652908959, win 229, options [nop,nop,TS val 3044404427 ecr 3611999789], length 550
	0x0000:  4500 025a a744 4000 4006 7419 0a0a 0501  E..Z.D@.@.t.....
	0x0010:  0a0a 042c bd58 1fbb 4e50 4b8a 26ea 999f  ...,.X..NPK.&...
	0x0020:  8018 00e5 1f8d 0000 0101 080a b575 eccb  .............u..
	0x0030:  d74a be2d 4745 5420 2f61 7069 2f77 6562  .J.-GET./api/web
	0x0040:  736f 636b 6574 2048 5454 502f 312e 310d  socket.HTTP/1.1.
	0x0050:  0a48 6f73 743a 2031 302e 3130 2e34 2e34  .Host:.10.10.4.4
	0x0060:  343a 3831 3233 0d0a 5573 6572 2d41 6765  4:8123..User-Age
	0x0070:  6e74 3a20 4d6f 7a69 6c6c 612f 352e 3020  nt:.Mozilla/5.0.
	0x0080:  284d 6163 696e 746f 7368 3b20 496e 7465  (Macintosh;.Inte
	0x0090:  6c20 4d61 6320 4f53 2058 2031 305f 3131  l.Mac.OS.X.10_11
	0x00a0:  5f36 2920 4170 706c 6557 6562 4b69 742f  _6).AppleWebKit/
	0x00b0:  3533 372e 3336 2028 4b48 544d 4c2c 206c  537.36.(KHTML,.l
	0x00c0:  696b 6520 4765 636b 6f29 2043 6872 6f6d  ike.Gecko).Chrom
	0x00d0:  652f 3535 2e30 2e32 3838 332e 3735 2053  e/55.0.2883.75.S
	0x00e0:  6166 6172 692f 3533 372e 3336 0d0a 4163  afari/537.36..Ac
	0x00f0:  6365 7074 2d45 6e63 6f64 696e 673a 2067  cept-Encoding:.g
	0x0100:  7a69 702c 2064 6566 6c61 7465 2c20 7364  zip,.deflate,.sd
	0x0110:  6368 0d0a 4163 6365 7074 2d4c 616e 6775  ch..Accept-Langu
	0x0120:  6167 653a 2065 6e2d 5553 2c65 6e3b 713d  age:.en-US,en;q=
	0x0130:  302e 380d 0a43 6163 6865 2d43 6f6e 7472  0.8..Cache-Contr
	0x0140:  6f6c 3a20 6e6f 2d63 6163 6865 0d0a 436f  ol:.no-cache..Co
	0x0150:  6e6e 6563 7469 6f6e 3a20 5570 6772 6164  nnection:.Upgrad
	0x0160:  650d 0a4f 7269 6769 6e3a 2068 7474 703a  e..Origin:.http:
	0x0170:  2f2f 6861 2e6d 6174 7476 6972 742e 636f  //ha.mattvirt.co
	0x0180:  6d0d 0a50 7261 676d 613a 206e 6f2d 6361  m..Pragma:.no-ca
	0x0190:  6368 650d 0a53 6563 2d57 6562 736f 636b  che..Sec-Websock
	0x01a0:  6574 2d45 7874 656e 7369 6f6e 733a 2070  et-Extensions:.p
	0x01b0:  6572 6d65 7373 6167 652d 6465 666c 6174  ermessage-deflat
	0x01c0:  653b 2063 6c69 656e 745f 6d61 785f 7769  e;.client_max_wi
	0x01d0:  6e64 6f77 5f62 6974 730d 0a53 6563 2d57  ndow_bits..Sec-W
	0x01e0:  6562 736f 636b 6574 2d4b 6579 3a20 7a76  ebsocket-Key:.zv
	0x01f0:  7135 664e 7951 4677 344a 642f 4344 5542  q5fNyQFw4Jd/CDUB
	0x0200:  7650 6977 3d3d 0d0a 5365 632d 5765 6273  vPiw==..Sec-Webs
	0x0210:  6f63 6b65 742d 5665 7273 696f 6e3a 2031  ocket-Version:.1
	0x0220:  330d 0a55 7067 7261 6465 3a20 7765 6273  3..Upgrade:.webs
	0x0230:  6f63 6b65 740d 0a58 2d46 6f72 7761 7264  ocket..X-Forward
	0x0240:  6564 2d46 6f72 3a20 3230                 ed-For....
15:07:35.595555 IP 10.10.4.44.8123 > 10.10.5.1.48472: Flags [.], ack 1313885616, win 235, options [nop,nop,TS val 3611999790 ecr 3044404427], length 0
	0x0000:  4500 0034 27b2 4000 4006 f5d1 0a0a 042c  E..4'.@.@......,
	0x0010:  0a0a 0501 1fbb bd58 26ea 999f 4e50 4db0  .......X&...NPM.
	0x0020:  8010 00eb e738 0000 0101 080a d74a be2e  .....8.......J..
	0x0030:  b575 eccb                                .u..
15:07:35.606838 IP 10.10.4.44.8123 > 10.10.5.1.48472: Flags [P.], seq 652908959:652909227, ack 1313885616, win 235, options [nop,nop,TS val 3611999792 ecr 3044404427], length 268
	0x0000:  4500 0140 27b3 4000 4006 f4c4 0a0a 042c  E..@'.@.@......,
	0x0010:  0a0a 0501 1fbb bd58 26ea 999f 4e50 4db0  .......X&...NPM.
	0x0020:  8018 00eb eaeb 0000 0101 080a d74a be30  .............J.0
	0x0030:  b575 eccb 4854 5450 2f31 2e31 2031 3031  .u..HTTP/1.1.101
	0x0040:  2053 7769 7463 6869 6e67 2050 726f 746f  .Switching.Proto
	0x0050:  636f 6c73 0d0a 436f 6e74 656e 742d 5479  cols..Content-Ty
	0x0060:  7065 3a20 6170 706c 6963 6174 696f 6e2f  pe:.application/
	0x0070:  6f63 7465 742d 7374 7265 616d 0d0a 5570  octet-stream..Up
	0x0080:  6772 6164 653a 2077 6562 736f 636b 6574  grade:.websocket
	0x0090:  0d0a 436f 6e6e 6563 7469 6f6e 3a20 7570  ..Connection:.up
	0x00a0:  6772 6164 650d 0a53 6563 2d57 6562 736f  grade..Sec-Webso
	0x00b0:  636b 6574 2d41 6363 6570 743a 2072 7853  cket-Accept:.rxS
	0x00c0:  4656 394c 4778 7036 5965 644a 3161 446d  FV9LGxp6YedJ1aDm
	0x00d0:  5747 3874 334e 794d 3d0d 0a44 6174 653a  WG8t3NyM=..Date:
	0x00e0:  2054 6875 2c20 3239 2044 6563 2032 3031  .Thu,.29.Dec.201
	0x00f0:  3620 3233 3a30 373a 3334 2047 4d54 0d0a  6.23:07:34.GMT..
	0x0100:  5365 7276 6572 3a20 5079 7468 6f6e 2f33  Server:.Python/3
	0x0110:  2e34 2061 696f 6874 7470 2f31 2e32 2e30  .4.aiohttp/1.2.0
	0x0120:  0d0a 5472 616e 7366 6572 2d45 6e63 6f64  ..Transfer-Encod
	0x0130:  696e 673a 2063 6875 6e6b 6564 0d0a 0d0a  ing:.chunked....

There is traffic going to the client after the websocket has been requested but nothing related as far as I can tell.

I’m running caddy 0.9.4:
$ caddy --version
Caddy 0.9.4

Here’s my configuration for the site:

http://ha.notmyrealdomain.com {
  proxy / http://10.10.4.44:8123 {
    transparent
  }
  proxy /api/websocket http://10.10.4.44:8123 {
    websocket
    header_upstream Sec-WebSocket-Extensions {>Sec-WebSocket-Extensions}
    header_upstream Sec-WebSocket-Key {>Sec-WebSocket-Key}
    header_upstream Sec-WebSocket-Version {>Sec-WebSocket-Version}
  }
  errors stdout
}

I’ve intentionally disabled HTTPS for the moment.

I’m also currently running Caddy with HTTP/2 disabled by adding -http2=false to my systemd service file and restarting caddy.

Is there anything I can do to make this work or is this a bug within Caddy?

From the looks of it, you’re trying to proxy Home Assistant? I had exactly the same problem and worked out a fix in code (see https://github.com/mholt/caddy/pull/1316). As the PR says: I don’t know what scenario was causing the problem to present in the first place and am in the process of trying to create a test case to reproduce the issue.

@joerocklin Good timing :slight_smile:

@mdcollins05 We just need to figure out if there’s much reason to keep the code the way it is, but if not, then after adding a test to Joe’s change, it should be fixed for you both.

@joerocklin Yup, attempting to proxy Home Assistant.

@matt Awesome. I hope the change can be implemented without breaking anything.

Thanks for your help!

Incidentally, if we can’t figure out how to reproduce the problem and put it under test (and thus merge the change you’re hoping for), it’s worth filing an issue with Home Assistant since it’s quite possible their use of websockets that’s causing the problems (if it’s not us).

This topic was automatically closed 90 days after the last reply. New replies are no longer allowed.