Issue with reverse_proxy and video stream

1. Caddy version (caddy version):

v2.2.1 h1:Q62GWHMtztnvyRU+KPOpw6fNfeCD3SkwH7SfT1Tgt2c=

2. How I run Caddy:

I run caddy in a docker container created with docker-compose from the latest official image in Docker Hub

a. System environment:

Ubuntu 18.04.5 LTS
systemd 237
Client: Docker Engine - Community
Version: 19.03.13
API version: 1.40

b. Command:

docker-compose up -d

c. Service/unit/compose file:

version: '2'
services:
 http-proxy:
  image: caddy:2.2.1
  container_name: http-proxy
  ports:
   - "80:80"
   - "443:443"
  volumes:
   - /home/user/containers/http-proxy/Caddyfile:/etc/caddy/Caddyfile
   - /home/user/containers/http-proxy/configs:/etc/caddy/configs
   - /home/user/containers/http-proxy/certs:/etc/caddy/certs
   - /home/user/containers/http-proxy/logs:/etc/caddy/logs
   - /etc/localtime:/etc/localtime:ro
   - /etc/timezone:/etc/timezone:ro
  tty: true
  stdin_open: true
  restart: unless-stopped

d. My complete Caddyfile or JSON config:

Caddyfile

######################
####    Snippets
######################

(cert_park.pt) {
        tls /etc/caddy/certs/ssl-cert-park.pt.crt /etc/caddy/certs/ssl-cert-park.pt.key
}

(header_upgradehttp) {
        header Content-Security-Policy "upgrade-insecure-requests"
}

(logging) {
        log {
                output file /etc/caddy/logs/{args.0}.log
        }
}

######################
####    TEST
######################

import configs/test.park.pt.conf

test.park.pt.conf

test.park.pt {
        import logging test.park.pt
        import cert_park.pt
        import header_upgradehttp
        reverse_proxy http://10.35.104.129
}

3. The problem I’m having:

I’m using caddy to reverse proxy several internal web applications, devices and services between two segregated networks, and the server running caddy has access to both. Users are on a network which only has access to the server running the caddy container, and through caddy’s reverse proxy can access those services on hosts in the other network. This has worked great for a long time, I have several of those services running without issues.

The issue is with a new device, an NVR server with several CCTV cameras connected to it and I can’t manage to set it up correctly through the reverse proxy. Everything in web interface works, but trying to stream any video feed from the cameras does not.

I think it’s an issue with websockets, but this is new to me and from what I understood of the documentation websockets should work with v2 without any additional arguments or configuration to reverse_proxy,.

Thanks in advance, caddy is awesome!

4. Error messages and/or full log output:

caddy log:

2020/10/21 10:19:29.258	error	http.log.access.log0	handled request	{"request": {"remote_addr": "10.10.211.35:17313", "proto": "HTTP/1.1", "method": "GET", "host": "test.park.pt", "uri": "/StreamingServer/uwa", "headers": {"Connection": ["Upgrade"], "Pragma": ["no-cache"], "User-Agent": ["Mozilla/5.0 (Windows NT 6.3; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/86.0.4240.75 Safari/537.36"], "Upgrade": ["websocket"], "Origin": ["https://test.park.pt"], "Accept-Language": ["en-US,en;q=0.9"], "Cookie": ["nvr_lang=11; DATA1=Y2VudHJhbA==; AUTH_KEY=8a29f4674de070dcc3faf68efc40dff8; ID=Y2VudHJhbA==; AUTH_HEADER_UWA={\"realm\":\"Wisenet NVR\",\"nonce\":\"47a2d8fd3d907787b5f65ba3ee2b0c1a\",\"qop\":\"auth\",\"cnonce\":\"9614c20bccc13fc5\",\"opaque\":null,\"scheme\":\"Digest\",\"nc\":1}"], "Sec-Websocket-Extensions": ["permessage-deflate; client_max_window_bits"], "Sec-Websocket-Version": ["13"], "Accept-Encoding": ["gzip, deflate, br"], "Sec-Websocket-Key": ["Ivozi1994dxq2L4aZsjiQQ=="], "Cache-Control": ["no-cache"]}, "tls": {"resumed": true, "version": 772, "cipher_suite": 4865, "proto": "", "proto_mutual": true, "server_name": "test.park.pt"}}, "common_log": "10.10.211.35 - - [21/Oct/2020:11:19:29 +0100] \"GET /StreamingServer/uwa HTTP/1.1\" 400 349", "duration": 1.051554179, "size": 349, "status": 400, "resp_headers": {"X-Ua-Compatible": ["requiresActiveX=true"], "Access-Control-Allow-Headers": ["Authorization, Content-Type, If-None-Match"], "Content-Security-Policy": ["upgrade-insecure-requests"], "Access-Control-Expose-Headers": ["WWW-Authenticate, Server-Authorization"], "Vary": ["Origin"], "Access-Control-Allow-Origin": ["https://test.park.pt"], "Access-Control-Allow-Credentials": ["true"], "Content-Type": ["text/html"], "Server": ["Caddy"], "Access-Control-Allow-Methods": ["GET, HEAD, POST, PUT, DELETE, OPTIONS"], "Date": ["Wed, 21 Oct 2020 10:19:27 GMT"], "Content-Length": ["349"]}}

caddy log without TLS:

2020/10/21 11:17:14.113
error
http.log.access.log0
handled request {
	"request": {
		"remote_addr": "10.10.211.35:21275",
		-"proto": "HTTP/1.1",
		-"method": "GET",
		-"host": "test.empark.pt",
		-"uri": "/StreamingServer/uwa",
		"headers": {
			-"Pragma": ["no-cache"],
			-"User-Agent": ["Mozilla/5.0 (Windows NT 6.3; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/86.0.4240.75 Safari/537.36"], 
			-"Upgrade": ["websocket"], 
			-"Cookie": ["nvr_lang=11; AUTH_KEY=8a29f4674de070dcc3faf68efc40dff8; ID=Y2VudHJhbA==; AUTH_HEADER_UWA={\"realm\":\"Wisenet NVR\",\"nonce\":\"47a2d8fd3d907787b5f65ba3ee2b0c1a\",\"qop\":\"auth\",\"cnonce\":\"9614c20bccc13fc5\",\"opaque\":null,\"scheme\":\"Digest\",\"nc\":1}"], 
			-"Sec-Websocket-Extensions": ["permessage-deflate; client_max_window_bits"], 
			-"Accept-Language": ["en-US,en;q=0.9"], 
			-"Sec-Websocket-Key": ["ydPGVsiwefhYk7mHpmofwA=="], 
			-"Connection": ["Upgrade"], 
			-"Cache-Control": ["no-cache"], 
			-"Origin": ["http://test.empark.pt"], 
			-"Sec-Websocket-Version": ["13"], 
			-"Accept-Encoding": ["gzip, deflate"]
			}
		}, 
	"common_log": "10.10.211.35 - - [21/Oct/2020:12:17:14 +0100] \"GET /StreamingServer/uwa HTTP/1.1\" 400 349", 
	"duration": 1.079819924, 
	"size": 349, 
	"status": 400, 
	"resp_headers": {"Access-Control-Allow-Credentials": ["true"], 
	"Access-Control-Allow-Headers": ["Authorization, Content-Type, If-None-Match"], 
	"Vary": ["Origin"], 
	"Content-Type": ["text/html"], 
	"Date": ["Wed, 21 Oct 2020 11:17:12 GMT"], 
	"X-Ua-Compatible": ["requiresActiveX=true"], 
	"Access-Control-Expose-Headers": ["WWW-Authenticate, Server-Authorization"], 
	"Server": ["Caddy"], 
	"Content-Length": ["349"], 
	"Access-Control-Allow-Origin": ["http://test.empark.pt"], 
	"Access-Control-Allow-Methods": ["GET, HEAD, POST, PUT, DELETE, OPTIONS"]
	}
}

browser:

ump-player.js:24579 WebSocket connection to 'wss://test.park.pt/StreamingServer/uwa' failed: Error during WebSocket handshake: Unexpected response code: 400
[Line#ump-player.js:24541:21] Oct 21 2020 11:19:29 ERROR - onclose: websocket closed, channelId [0], Raw Data [{"isTrusted":true}], websocket state [3], Error Code: [-1, Name: Unknown, Desc: Unknown Status code.]
[Line#ump-player.js:24452:25] Oct 21 2020 11:19:29 ERROR - error1: websocket closed, channelId [0], Raw Data [{"isTrusted":true}], websocket state [3], Error Code: [1006, Name: Abnormal Closure, Desc: Reserved. Used to indicate that a connection was closed abnormally (that is, with no close frame being sent) when a status code is expected.]

request headers directly to backend, without caddy, with stream working:

GET ws://10.35.104.129/StreamingServer/uwa HTTP/1.1
Host: 10.35.104.129
Connection: Upgrade
Pragma: no-cache
Cache-Control: no-cache
User-Agent: Mozilla/5.0 (Windows NT 6.3; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/86.0.4240.75 Safari/537.36
Upgrade: websocket
Origin: http://10.35.104.129
Sec-WebSocket-Version: 13
Accept-Encoding: gzip, deflate
Accept-Language: en-US,en;q=0.9
Cookie: nvr_lang=11; AUTH_KEY=8a29f4674de070dcc3faf68efc40dff8; ID=Y2VudHJhbA==; AUTH_HEADER_UWA={"realm":"Wisenet NVR","nonce":"8fae1b2e96eb607655e9c9783be12f1c","qop":"auth","cnonce":"22da358ce2b117fc","opaque":null,"scheme":"Digest","nc":1}
Sec-WebSocket-Key: fHbEP3GaQqP8BlY3b/Nkog==
Sec-WebSocket-Extensions: permessage-deflate; client_max_window_bits

response headers for the above

HTTP/1.1 101 Switching Protocols
Upgrade: websocket
Connection: Upgrade
Sec-WebSocket-Accept: gpHs02MuXh7d4KCUm5x1+guzPas=

5. What I already tried:

I honestly tried so many things, trying to manipulate headers adding the ones below, adding specific reverse_proxy directives to /StreamingServer/uwa, disabling buffering, forcing the site to HTTP and disabing “upgrade-insecure-requests”, nothing has made me closer to solving this. I have isolated the configuration for this service from the rest, to exclude any issues with the configuration for other services and the issue continues.
I found an issue in GitHub of a similar problem ,error 400 with websockets, and tried the suggested header suppression.
https://github.com/caddyserver/caddy/issues/3778

header_up Host {host}
header_up X-Real-IP {remote_host}
header_up X-Forwarded-For {remote_host}
header_up X-Forwarded-Proto {scheme}

6. Links to relevant resources:

Hmm.

You’ve tried setting flush_interval -1 to turn off buffering?

Could you try inspecting the data on the wire with something like tcpdump? Looks like a 400 is returned? Would be good to figure out where that’s from.

Also as an aside, I highly recommend you mount a volume for /data and /config for the Caddy service in your docker-compose file, because Caddy uses those paths to persist some information. Not relevant to your issue, but just best practice, to avoid potential data loss.

Thank you for taking the time to help.

Yes, also I tried that, just checked again and the issues continues.
One of the other services I proxy with caddy is Apache Guacamole and I had to use that setting for that (following your suggestion to another user in the forum, I believe), so I tried it in this case too but it didn’t help with the problem.

I tried to capture the traffic between caddy and the upstream with tcdump, this is the part for the requests regarding the 400 error:

02:12:49.252409 IP 172.22.0.2.40306 > 10.35.104.129.80: Flags [P.], seq 1:831, ack 1, win 502, options [nop,nop,TS val 33875893 ecr 184355417], length 830: HTTP: GET /StreamingServer/uwa HTTP/1.1
E..r..@.@.......
#h..r.P...e~<......"!.....
....
.
YGET /StreamingServer/uwa HTTP/1.1
Host: test.park.pt
User-Agent: Mozilla/5.0 (Windows NT 6.3; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/86.0.4240.111 Safari/537.36
Accept-Encoding: gzip, deflate
Accept-Language: en-US,en;q=0.9
Cache-Control: no-cache
Connection: Upgrade
Cookie: nvr_lang=11; DATA1=Y2VudHJhbA==; AUTH_KEY=8a29f4674de070dcc3faf68efc40dff8; ID=Y2VudHJhbA==; AUTH_HEADER_UWA={"realm":"Wisenet NVR","nonce":"89b506a04ba3a24dca3ffb2f689a8514","qop":"auth","cnonce":"28b6e7a825082017","opaque":null,"scheme":"Digest","nc":1}
Origin: http://test.park.pt
Pragma: no-cache
Sec-Websocket-Extensions: permessage-deflate; client_max_window_bits
Sec-Websocket-Key: gKvxgZKv46r3UjHa8pB2DA==
Sec-Websocket-Version: 13
Upgrade: websocket
X-Forwarded-For: 10.10.211.35
X-Forwarded-Proto: http


02:12:50.284000 IP 10.35.104.129.80 > 172.22.0.2.40306: Flags [P.], seq 1:820, ack 831, win 479, options [nop,nop,TS val 184355519 ecr 33875893], length 819: HTTP: HTTP/1.1 400 Bad Request
E..g..@.6.Z.
#h......P.r~<.................

.
.....HTTP/1.1 400 Bad Request
X-UA-Compatible: requiresActiveX=true
Access-Control-Allow-Origin: http://test.park.pt
Access-Control-Allow-Credentials: true
Access-Control-Allow-Headers: Authorization, Content-Type, If-None-Match
Access-Control-Expose-Headers: WWW-Authenticate, Server-Authorization
Access-Control-Allow-Methods: GET, HEAD, POST, PUT, DELETE, OPTIONS
Vary: Origin
Content-Type: text/html
Content-Length: 349
Date: Thu, 22 Oct 2020 01:12:48 GMT

<?xml version="1.0" encoding="iso-8859-1"?>
<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN"
         "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">
<html xmlns="http://www.w3.org/1999/xhtml" xml:lang="en" lang="en">
 <head>
  <title>400 - Bad Request</title>
 </head>
 <body>
  <h1>400 - Bad Request</h1>
 </body>
</html>

02:12:50.284077 IP 172.22.0.2.40306 > 10.35.104.129.80: Flags [.], ack 820, win 501, options [nop,nop,TS val 33876925 ecr 184355519], length 0
E..4..@.@.! ....
#h..r.P....~<.............
....
.
.

It might help, to compare I also tried to capture the traffic with WireShark between the browser and caddy, and between the browser and the upstream.

Frame 5354: 808 bytes on wire (6464 bits), 808 bytes captured (6464 bits) on interface \Device\NPF_{3717874A-55EB-4750-BE86-74CE134D0925}, id 0
Ethernet II, Src: Microsof_cb:66:53 (00:15:5d:cb:66:53), Dst: Microsof_cb:65:59 (00:15:5d:cb:65:59)
Internet Protocol Version 4, Src: 10.10.211.35, Dst: 10.10.211.24
Transmission Control Protocol, Src Port: 28963, Dst Port: 80, Seq: 1, Ack: 1, Len: 754
    Source Port: 28963
    Destination Port: 80
    [Stream index: 23]
    [TCP Segment Len: 754]
    Sequence number: 1    (relative sequence number)
    Sequence number (raw): 2539672045
    [Next sequence number: 755    (relative sequence number)]
    Acknowledgment number: 1    (relative ack number)
    Acknowledgment number (raw): 3378393377
    0101 .... = Header Length: 20 bytes (5)
    Flags: 0x018 (PSH, ACK)
        000. .... .... = Reserved: Not set
        ...0 .... .... = Nonce: Not set
        .... 0... .... = Congestion Window Reduced (CWR): Not set
        .... .0.. .... = ECN-Echo: Not set
        .... ..0. .... = Urgent: Not set
        .... ...1 .... = Acknowledgment: Set
        .... .... 1... = Push: Set
        .... .... .0.. = Reset: Not set
        .... .... ..0. = Syn: Not set
        .... .... ...0 = Fin: Not set
        [TCP Flags: ·······AP···]
    Window size value: 513
    [Calculated window size: 131328]
    [Window size scaling factor: 256]
    Checksum: 0xbd5c [unverified]
    [Checksum Status: Unverified]
    Urgent pointer: 0
    [SEQ/ACK analysis]
        [iRTT: 0.002615000 seconds]
        [Bytes in flight: 754]
        [Bytes sent since last PSH flag: 754]
    [Timestamps]
        [Time since first frame in this TCP stream: 0.002927000 seconds]
        [Time since previous frame in this TCP stream: 0.000312000 seconds]
    TCP payload (754 bytes)
Hypertext Transfer Protocol
    GET /StreamingServer/uwa HTTP/1.1
        [Expert Info (Chat/Sequence): GET /StreamingServer/uwa HTTP/1.1]
            [GET /StreamingServer/uwa HTTP/1.1]
            [Severity level: Chat]
            [Group: Sequence]
        Request Method: GET
        Request URI: /StreamingServer/uwa
        Request Version: HTTP/1.1
    Host: test.park.pt
    Connection: Upgrade
    Pragma: no-cache
    Cache-Control: no-cache
    User-Agent: Mozilla/5.0 (Windows NT 6.3; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/86.0.4240.111 Safari/537.36
    Upgrade: websocket
    Origin: http://test.park.pt
    Sec-WebSocket-Version: 13
    Accept-Encoding: gzip, deflate
    Accept-Language: en-US,en;q=0.9
     [truncated]Cookie: nvr_lang=11; AUTH_KEY=8a29f4674de070dcc3faf68efc40dff8; ID=Y2VudHJhbA==; AUTH_HEADER_UWA={"realm":"Wisenet NVR","nonce":"89b506a04ba3a24dca3ffb2f689a8514","qop":"auth","cnonce":"28b6e7a825082017","opaque":null,"scheme"
    Sec-WebSocket-Key: 0QdVQRxwhdzw7QkPQzriXw==
    Sec-WebSocket-Extensions: permessage-deflate; client_max_window_bits
    
    [Full request URI: http://test.park.pt/StreamingServer/uwa]
    [HTTP request 1/1]
    [Response in frame: 5473]



Frame 5473: 888 bytes on wire (7104 bits), 888 bytes captured (7104 bits) on interface \Device\NPF_{3717874A-55EB-4750-BE86-74CE134D0925}, id 0
Ethernet II, Src: Microsof_cb:65:59 (00:15:5d:cb:65:59), Dst: Microsof_cb:66:53 (00:15:5d:cb:66:53)
Internet Protocol Version 4, Src: 10.10.211.24, Dst: 10.10.211.35
Transmission Control Protocol, Src Port: 80, Dst Port: 28963, Seq: 1, Ack: 755, Len: 834
    Source Port: 80
    Destination Port: 28963
    [Stream index: 23]
    [TCP Segment Len: 834]
    Sequence number: 1    (relative sequence number)
    Sequence number (raw): 3378393377
    [Next sequence number: 835    (relative sequence number)]
    Acknowledgment number: 755    (relative ack number)
    Acknowledgment number (raw): 2539672799
    0101 .... = Header Length: 20 bytes (5)
    Flags: 0x018 (PSH, ACK)
        000. .... .... = Reserved: Not set
        ...0 .... .... = Nonce: Not set
        .... 0... .... = Congestion Window Reduced (CWR): Not set
        .... .0.. .... = ECN-Echo: Not set
        .... ..0. .... = Urgent: Not set
        .... ...1 .... = Acknowledgment: Set
        .... .... 1... = Push: Set
        .... .... .0.. = Reset: Not set
        .... .... ..0. = Syn: Not set
        .... .... ...0 = Fin: Not set
        [TCP Flags: ·······AP···]
    Window size value: 501
    [Calculated window size: 64128]
    [Window size scaling factor: 128]
    Checksum: 0x4043 [unverified]
    [Checksum Status: Unverified]
    Urgent pointer: 0
    [SEQ/ACK analysis]
        [iRTT: 0.002615000 seconds]
        [Bytes in flight: 834]
        [Bytes sent since last PSH flag: 834]
    [Timestamps]
        [Time since first frame in this TCP stream: 1.051144000 seconds]
        [Time since previous frame in this TCP stream: 1.046957000 seconds]
    TCP payload (834 bytes)
Hypertext Transfer Protocol
    HTTP/1.1 400 Bad Request
        [Expert Info (Chat/Sequence): HTTP/1.1 400 Bad Request]
            [HTTP/1.1 400 Bad Request]
            [Severity level: Chat]
            [Group: Sequence]
        Response Version: HTTP/1.1
        Status Code: 400
        [Status Code Description: Bad Request]
        Response Phrase: Bad Request
    Access-Control-Allow-Credentials: true
    Access-Control-Allow-Headers: Authorization, Content-Type, If-None-Match
    Access-Control-Allow-Methods: GET, HEAD, POST, PUT, DELETE, OPTIONS
    Access-Control-Allow-Origin: http://test.park.pt
    Access-Control-Expose-Headers: WWW-Authenticate, Server-Authorization
    Content-Length: 349
    Content-Type: text/html
    Date: Thu, 22 Oct 2020 01:31:27 GMT
    Server: Caddy
    Vary: Origin
    X-Ua-Compatible: requiresActiveX=true
    
    [HTTP response 1/1]
    [Time since request: 1.048217000 seconds]
    [Request in frame: 5354]
    [Request URI: http://test.park.pt/StreamingServer/uwa]

Capture between the browser and the backend, without caddy proxying:

Frame 8627: 826 bytes on wire (6608 bits), 826 bytes captured (6608 bits) on interface \Device\NPF_{3717874A-55EB-4750-BE86-74CE134D0925}, id 0
Ethernet II, Src: Microsof_cb:66:53 (00:15:5d:cb:66:53), Dst: Cisco_9f:f0:6f (00:00:0c:9f:f0:6f)
Internet Protocol Version 4, Src: 10.10.211.35, Dst: 10.35.104.129
Transmission Control Protocol, Src Port: 29296, Dst Port: 80, Seq: 1, Ack: 1, Len: 772
    Source Port: 29296
    Destination Port: 80
    [Stream index: 30]
    [TCP Segment Len: 772]
    Sequence number: 1    (relative sequence number)
    Sequence number (raw): 3748518604
    [Next sequence number: 773    (relative sequence number)]
    Acknowledgment number: 1    (relative ack number)
    Acknowledgment number (raw): 3184372418
    0101 .... = Header Length: 20 bytes (5)
    Flags: 0x018 (PSH, ACK)
        000. .... .... = Reserved: Not set
        ...0 .... .... = Nonce: Not set
        .... 0... .... = Congestion Window Reduced (CWR): Not set
        .... .0.. .... = ECN-Echo: Not set
        .... ..0. .... = Urgent: Not set
        .... ...1 .... = Acknowledgment: Set
        .... .... 1... = Push: Set
        .... .... .0.. = Reset: Not set
        .... .... ..0. = Syn: Not set
        .... .... ...0 = Fin: Not set
        [TCP Flags: ·······AP···]
    Window size value: 516
    [Calculated window size: 132096]
    [Window size scaling factor: 256]
    Checksum: 0x52f0 [unverified]
    [Checksum Status: Unverified]
    Urgent pointer: 0
    [SEQ/ACK analysis]
        [iRTT: 0.022700000 seconds]
        [Bytes in flight: 772]
        [Bytes sent since last PSH flag: 772]
    [Timestamps]
        [Time since first frame in this TCP stream: 0.023065000 seconds]
        [Time since previous frame in this TCP stream: 0.000365000 seconds]
    TCP payload (772 bytes)
Hypertext Transfer Protocol
    GET /StreamingServer/uwa HTTP/1.1
        [Expert Info (Chat/Sequence): GET /StreamingServer/uwa HTTP/1.1]
            [GET /StreamingServer/uwa HTTP/1.1]
            [Severity level: Chat]
            [Group: Sequence]
        Request Method: GET
        Request URI: /StreamingServer/uwa
        Request Version: HTTP/1.1
    Host: 10.35.104.129
    Connection: Upgrade
    Pragma: no-cache
    Cache-Control: no-cache
    User-Agent: Mozilla/5.0 (Windows NT 6.3; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/86.0.4240.111 Safari/537.36
    Upgrade: websocket
    Origin: http://10.35.104.129
    Sec-WebSocket-Version: 13
    Accept-Encoding: gzip, deflate
    Accept-Language: en-US,en;q=0.9
     [truncated]Cookie: nvr_lang=11; DATA1=Y2VudHJhbA==; AUTH_KEY=8a29f4674de070dcc3faf68efc40dff8; ID=Y2VudHJhbA==; AUTH_HEADER_UWA={"realm":"Wisenet NVR","nonce":"e05b1502e549845c8c0496bc55490116","qop":"auth","cnonce":"7de3e6e6c93087db","o
    Sec-WebSocket-Key: JZwlqfNAjIqF8OxgkQlxmQ==
    Sec-WebSocket-Extensions: permessage-deflate; client_max_window_bits
    
    [Full request URI: http://10.35.104.129/StreamingServer/uwa]
    [HTTP request 1/1]
    [Response in frame: 8630]
	
	
	
Frame 8630: 183 bytes on wire (1464 bits), 183 bytes captured (1464 bits) on interface \Device\NPF_{3717874A-55EB-4750-BE86-74CE134D0925}, id 0
Ethernet II, Src: Cisco_89:8d:50 (74:a0:2f:89:8d:50), Dst: Microsof_cb:66:53 (00:15:5d:cb:66:53)
Internet Protocol Version 4, Src: 10.35.104.129, Dst: 10.10.211.35
Transmission Control Protocol, Src Port: 80, Dst Port: 29296, Seq: 1, Ack: 773, Len: 129
    Source Port: 80
    Destination Port: 29296
    [Stream index: 30]
    [TCP Segment Len: 129]
    Sequence number: 1    (relative sequence number)
    Sequence number (raw): 3184372418
    [Next sequence number: 130    (relative sequence number)]
    Acknowledgment number: 773    (relative ack number)
    Acknowledgment number (raw): 3748519376
    0101 .... = Header Length: 20 bytes (5)
    Flags: 0x018 (PSH, ACK)
        000. .... .... = Reserved: Not set
        ...0 .... .... = Nonce: Not set
        .... 0... .... = Congestion Window Reduced (CWR): Not set
        .... .0.. .... = ECN-Echo: Not set
        .... ..0. .... = Urgent: Not set
        .... ...1 .... = Acknowledgment: Set
        .... .... 1... = Push: Set
        .... .... .0.. = Reset: Not set
        .... .... ..0. = Syn: Not set
        .... .... ...0 = Fin: Not set
        [TCP Flags: ·······AP···]
    Window size value: 481
    [Calculated window size: 30784]
    [Window size scaling factor: 64]
    Checksum: 0xcd2c [unverified]
    [Checksum Status: Unverified]
    Urgent pointer: 0
    [SEQ/ACK analysis]
        [iRTT: 0.022700000 seconds]
        [Bytes in flight: 129]
        [Bytes sent since last PSH flag: 129]
    [Timestamps]
        [Time since first frame in this TCP stream: 0.047568000 seconds]
        [Time since previous frame in this TCP stream: 0.004949000 seconds]
    TCP payload (129 bytes)
Hypertext Transfer Protocol
    HTTP/1.1 101 Switching Protocols
        [Expert Info (Chat/Sequence): HTTP/1.1 101 Switching Protocols]
            [HTTP/1.1 101 Switching Protocols]
            [Severity level: Chat]
            [Group: Sequence]
        Response Version: HTTP/1.1
        Status Code: 101
        [Status Code Description: Switching Protocols]
        Response Phrase: Switching Protocols
    Upgrade: websocket
    Connection: Upgrade
    Sec-WebSocket-Accept: llI3o9n9wfypUd5VXA12z0IhO0Q=
    
    [HTTP response 1/1]
    [Time since request: 0.024503000 seconds]
    [Request in frame: 8627]
    [Request URI: http://10.35.104.129/StreamingServer/uwa]

I followed you suggestion about the container volumes, thanks.

Continuing to investigate this, I created an nginx container and configured it as a reverse proxy to the same address, and was able to get it to work without the issue mentioned above with the following configuration:

events {
}
http {
        server {
                listen 9998;

                location / {
                        proxy_pass http://10.35.104.129:80;
                        proxy_buffering off;
                        proxy_set_header Host $http_host;
                }

                location /StreamingServer/uwa {
                        proxy_pass http://10.35.104.129:80;
                        proxy_buffering off;
                        proxy_set_header Host $http_host;
                        proxy_set_header Upgrade $http_upgrade;
                        proxy_set_header Connection "Upgrade";
                }
        }
}

I then tried to replicate this configuration with the Caddyfile below, but the issue remains

test.park.pt:80 {

        @NotStreamingServer {
                not path /StreamingServer/uwa*
        }

        @StreamingServer {
                path /StreamingServer/uwa*
        }

        reverse_proxy @NotStreamingServer http://10.35.104.129:80 {
                flush_interval -1
                header_up Host {http.request.hostport}
        }

        reverse_proxy @StreamingServer http://10.35.104.129:80 {
                flush_interval -1
                header_up Host {http.request.hostport}
                header_up Upgrade websocket
                header_up Connection Upgrade
        }
}

Then I converted the nginx config to a JSON config for caddy, made small adjustments, tried it on Caddy and had the same issues as before. Converted the Caddyfile to JSON to compare them too, can’t seen anything that would be a problem.
At this point I’m assuming this is some peculiar behaviour of Caddy dealing with websockets.

JSON from nginx config

{
   "apps":{
      "http":{
         "servers":{
            "server_0":{
               "listen":[
                  ":9998"
               ],
               "routes":[
                  {
                     "match":[
                        {
                           "path":[
                              "*"
                           ]
                        }
                     ],
                     "handle":[
                        {
                           "handler":"subroute",
                           "routes":[
                              {
                                 "handle":[
                                    {
                                       "handler":"reverse_proxy",
                                       "headers":{
                                          "request":{
                                             "set":{
                                                "Host":[
                                                   "{http.request.hostport}"
                                                ]
                                             }
                                          }
                                       },
                                       "upstreams":[
                                          {
                                             "dial":"tcp/10.35.104.129:80"
                                          }
                                       ]
                                    }
                                 ],
                                 "match":[
                                    {
                                       "not":[
                                          {
                                             "path":[
                                                "/StreamingServer/uwa*"
                                             ]
                                          }
                                       ]
                                    }

                                 ]
                              }
                           ]
                        }
                     ]
                  },
                  {
                     "match":[
                        {
                           "path":[
                              "/StreamingServer/uwa*"
                           ]
                        }
                     ],
                     "handle":[
                        {
                           "handler":"subroute",
                           "routes":[
                              {
                                 "handle":[
                                    {
                                       "handler":"reverse_proxy",
                                       "headers":{
                                          "request":{
                                             "set":{
                                                "Connection":[
                                                   "Upgrade"
                                                ],
                                                "Host":[
                                                   "{http.request.hostport}"
                                                ],
                                                "Upgrade":[
                                                   "websocket"
                                                ]
                                             }

                                          }
                                       },
                                       "upstreams":[
                                          {
                                             "dial":"tcp/10.35.104.129:80"
                                          }
                                       ]
                                    }
                                 ],
                                 "match":[
                                    {
                                       "path":[
                                          "/StreamingServer/uwa*"
                                       ]
                                    }
                                 ]
                              }
                           ]
                        }
                     ]
                  }
               ]
            }
         }
      }
   }
}

JSON from Caddyfile

{
   "apps":{
      "http":{
         "servers":{
            "srv0":{
               "listen":[
                  ":80"
               ],
               "routes":[
                  {
                     "match":[
                        {
                           "host":[
                              "test.park.pt"
                           ]
                        }
                     ],
                     "handle":[
                        {
                           "handler":"subroute",
                           "routes":[
                              {
                                 "handle":[
                                    {
                                       "flush_interval":-1,
                                       "handler":"reverse_proxy",
                                       "headers":{
                                          "request":{
                                             "set":{
                                                "Connection":[
                                                   "Upgrade"
                                                ],
                                                "Host":[
                                                   "{http.request.hostport}"
                                                ],
                                                "Upgrade":[
                                                   "websocket"
                                                ]
                                             }
                                          }
                                       },
                                       "upstreams":[
                                          {
                                             "dial":"10.35.104.129:80"
                                          }
                                       ]
                                    }
                                 ],
                                 "match":[
                                    {
                                       "path":[
                                          "/StreamingServer/uwa*"
                                       ]
                                    }
                                 ]
                              },
                              {
                                 "handle":[
                                    {
                                       "flush_interval":-1,
                                       "handler":"reverse_proxy",
                                       "headers":{
                                          "request":{
                                             "set":{
                                                "Host":[
                                                   "{http.request.hostport}"
                                                ]
                                             }
                                          }
                                       },
                                       "upstreams":[
                                          {
                                             "dial":"10.35.104.129:80"
                                          }
                                       ]
                                    }
                                 ],
                                 "match":[
                                    {
                                       "not":[
                                          {
                                             "path":[
                                                "/StreamingServer/uwa*"
                                             ]
                                          }
                                       ]
                                    }
                                 ]
                              }
                           ]
                        }
                     ],
                     "terminal":true
                  }
               ]
            }
         }
      }
   }
}

I don’t think any of those header_up lines are necessary – Caddy automatically forwards any websocket headers and automatically sets Host to the appropriate value.

Caddy disables proxy buffering by default (but it has an option buffer_requests to enable it – not helpful for you here though).

I’d simplify your Caddyfile to this:

http://test.park.pt {
	reverse_proxy http://10.35.104.129 {
		flush_interval -1
	}
}

There’s ultimately no difference between the two proxies you defined in Caddy, because Caddy automatically forwards headers.

I know they should be set by Caddy, but I was following the headers that worked with nginx and trying to find something unusual. That configuration is what I started with and have issues, so I’m trying something different.

I ran tcpdump to catch requests between both each proxy and the upstream, to compare requests and with nginx the headers X-Forwarded-For and X-Forwarded-Proto weren’t added and it’s HTTP/1.0, so I tried to reproduce that in Caddy with the below and it didn’t made a difference.
I don’t know what else to try, so for now I’ll have to find another way without Caddy (already tried caddy-l4 too). Thanks for your help!

            header_up -X-Forwarded-For
            header_up -X-Forwarded-Proto
            transport http {
                    versions 1.0
            }

From Caddy:

GET /StreamingServer/uwa HTTP/1.1
Host: test.park.pt
User-Agent: Mozilla/5.0 (Windows NT 6.3; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/86.0.4240.193 Safari/537.36
Accept-Encoding: gzip, deflate
Accept-Language: en-US,en;q=0.9
Cache-Control: no-cache
Connection: Upgrade
Cookie: nvr_lang=11; DATA1=Y2VudHJhbA==; AUTH_KEY=***; ID=Y2VudHJhbA==; AUTH_HEADER_UWA={"realm":"Wisenet NVR","nonce":"a47590eb55bf746589a6d595dda3051a","qop":"auth","cnonce":"46709cb2581271ee","opaque":null,"scheme":"Digest","nc":1}
Origin: http://test.park.pt
Pragma: no-cache
Sec-Websocket-Extensions: permessage-deflate; client_max_window_bits
Sec-Websocket-Key: 1Jy1B/IZHQTvq4cG1dZU+w==
Sec-Websocket-Version: 13
Upgrade: websocket
X-Forwarded-For: 10.10.211.35
X-Forwarded-Proto: http

From nginx:

GET /StreamingServer/uwa HTTP/1.0
Host: test.park.pt:9998
Upgrade: websocket
Connection: Upgrade
Pragma: no-cache
Cache-Control: no-cache
User-Agent: Mozilla/5.0 (Windows NT 6.3; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/86.0.4240.193 Safari/537.36
Origin: http://test.park.pt:9998
Sec-WebSocket-Version: 13
Accept-Encoding: gzip, deflate
Accept-Language: en-US,en;q=0.9
Cookie: nvr_lang=11; AUTH_KEY=***; ID=Y2VudHJhbA==; AUTH_HEADER_UWA={"realm":"Wisenet NVR","nonce":"6ae7b14c4678373fa4db661761bf2206","qop":"auth","cnonce":"d988f6ab1b066b08","opaque":null,"scheme":"Digest","nc":1}
Sec-WebSocket-Key: Hk/ijqaHqj01M9c/ViN1ZA==
Sec-WebSocket-Extensions: permessage-deflate; client_max_window_bits

This topic was automatically closed after 30 days. New replies are no longer allowed.