1. Caddy version (caddy version
):
v2.2.1 h1:Q62GWHMtztnvyRU+KPOpw6fNfeCD3SkwH7SfT1Tgt2c=
2. How I run Caddy:
I run caddy in a docker container created with docker-compose from the latest official image in Docker Hub
a. System environment:
Ubuntu 18.04.5 LTS
systemd 237
Client: Docker Engine - Community
Version: 19.03.13
API version: 1.40
b. Command:
docker-compose up -d
c. Service/unit/compose file:
version: '2'
services:
http-proxy:
image: caddy:2.2.1
container_name: http-proxy
ports:
- "80:80"
- "443:443"
volumes:
- /home/user/containers/http-proxy/Caddyfile:/etc/caddy/Caddyfile
- /home/user/containers/http-proxy/configs:/etc/caddy/configs
- /home/user/containers/http-proxy/certs:/etc/caddy/certs
- /home/user/containers/http-proxy/logs:/etc/caddy/logs
- /etc/localtime:/etc/localtime:ro
- /etc/timezone:/etc/timezone:ro
tty: true
stdin_open: true
restart: unless-stopped
d. My complete Caddyfile or JSON config:
Caddyfile
######################
#### Snippets
######################
(cert_park.pt) {
tls /etc/caddy/certs/ssl-cert-park.pt.crt /etc/caddy/certs/ssl-cert-park.pt.key
}
(header_upgradehttp) {
header Content-Security-Policy "upgrade-insecure-requests"
}
(logging) {
log {
output file /etc/caddy/logs/{args.0}.log
}
}
######################
#### TEST
######################
import configs/test.park.pt.conf
test.park.pt.conf
test.park.pt {
import logging test.park.pt
import cert_park.pt
import header_upgradehttp
reverse_proxy http://10.35.104.129
}
3. The problem I’m having:
I’m using caddy to reverse proxy several internal web applications, devices and services between two segregated networks, and the server running caddy has access to both. Users are on a network which only has access to the server running the caddy container, and through caddy’s reverse proxy can access those services on hosts in the other network. This has worked great for a long time, I have several of those services running without issues.
The issue is with a new device, an NVR server with several CCTV cameras connected to it and I can’t manage to set it up correctly through the reverse proxy. Everything in web interface works, but trying to stream any video feed from the cameras does not.
I think it’s an issue with websockets, but this is new to me and from what I understood of the documentation websockets should work with v2 without any additional arguments or configuration to reverse_proxy,.
Thanks in advance, caddy is awesome!
4. Error messages and/or full log output:
caddy log:
2020/10/21 10:19:29.258 error http.log.access.log0 handled request {"request": {"remote_addr": "10.10.211.35:17313", "proto": "HTTP/1.1", "method": "GET", "host": "test.park.pt", "uri": "/StreamingServer/uwa", "headers": {"Connection": ["Upgrade"], "Pragma": ["no-cache"], "User-Agent": ["Mozilla/5.0 (Windows NT 6.3; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/86.0.4240.75 Safari/537.36"], "Upgrade": ["websocket"], "Origin": ["https://test.park.pt"], "Accept-Language": ["en-US,en;q=0.9"], "Cookie": ["nvr_lang=11; DATA1=Y2VudHJhbA==; AUTH_KEY=8a29f4674de070dcc3faf68efc40dff8; ID=Y2VudHJhbA==; AUTH_HEADER_UWA={\"realm\":\"Wisenet NVR\",\"nonce\":\"47a2d8fd3d907787b5f65ba3ee2b0c1a\",\"qop\":\"auth\",\"cnonce\":\"9614c20bccc13fc5\",\"opaque\":null,\"scheme\":\"Digest\",\"nc\":1}"], "Sec-Websocket-Extensions": ["permessage-deflate; client_max_window_bits"], "Sec-Websocket-Version": ["13"], "Accept-Encoding": ["gzip, deflate, br"], "Sec-Websocket-Key": ["Ivozi1994dxq2L4aZsjiQQ=="], "Cache-Control": ["no-cache"]}, "tls": {"resumed": true, "version": 772, "cipher_suite": 4865, "proto": "", "proto_mutual": true, "server_name": "test.park.pt"}}, "common_log": "10.10.211.35 - - [21/Oct/2020:11:19:29 +0100] \"GET /StreamingServer/uwa HTTP/1.1\" 400 349", "duration": 1.051554179, "size": 349, "status": 400, "resp_headers": {"X-Ua-Compatible": ["requiresActiveX=true"], "Access-Control-Allow-Headers": ["Authorization, Content-Type, If-None-Match"], "Content-Security-Policy": ["upgrade-insecure-requests"], "Access-Control-Expose-Headers": ["WWW-Authenticate, Server-Authorization"], "Vary": ["Origin"], "Access-Control-Allow-Origin": ["https://test.park.pt"], "Access-Control-Allow-Credentials": ["true"], "Content-Type": ["text/html"], "Server": ["Caddy"], "Access-Control-Allow-Methods": ["GET, HEAD, POST, PUT, DELETE, OPTIONS"], "Date": ["Wed, 21 Oct 2020 10:19:27 GMT"], "Content-Length": ["349"]}}
caddy log without TLS:
2020/10/21 11:17:14.113
error
http.log.access.log0
handled request {
"request": {
"remote_addr": "10.10.211.35:21275",
-"proto": "HTTP/1.1",
-"method": "GET",
-"host": "test.empark.pt",
-"uri": "/StreamingServer/uwa",
"headers": {
-"Pragma": ["no-cache"],
-"User-Agent": ["Mozilla/5.0 (Windows NT 6.3; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/86.0.4240.75 Safari/537.36"],
-"Upgrade": ["websocket"],
-"Cookie": ["nvr_lang=11; AUTH_KEY=8a29f4674de070dcc3faf68efc40dff8; ID=Y2VudHJhbA==; AUTH_HEADER_UWA={\"realm\":\"Wisenet NVR\",\"nonce\":\"47a2d8fd3d907787b5f65ba3ee2b0c1a\",\"qop\":\"auth\",\"cnonce\":\"9614c20bccc13fc5\",\"opaque\":null,\"scheme\":\"Digest\",\"nc\":1}"],
-"Sec-Websocket-Extensions": ["permessage-deflate; client_max_window_bits"],
-"Accept-Language": ["en-US,en;q=0.9"],
-"Sec-Websocket-Key": ["ydPGVsiwefhYk7mHpmofwA=="],
-"Connection": ["Upgrade"],
-"Cache-Control": ["no-cache"],
-"Origin": ["http://test.empark.pt"],
-"Sec-Websocket-Version": ["13"],
-"Accept-Encoding": ["gzip, deflate"]
}
},
"common_log": "10.10.211.35 - - [21/Oct/2020:12:17:14 +0100] \"GET /StreamingServer/uwa HTTP/1.1\" 400 349",
"duration": 1.079819924,
"size": 349,
"status": 400,
"resp_headers": {"Access-Control-Allow-Credentials": ["true"],
"Access-Control-Allow-Headers": ["Authorization, Content-Type, If-None-Match"],
"Vary": ["Origin"],
"Content-Type": ["text/html"],
"Date": ["Wed, 21 Oct 2020 11:17:12 GMT"],
"X-Ua-Compatible": ["requiresActiveX=true"],
"Access-Control-Expose-Headers": ["WWW-Authenticate, Server-Authorization"],
"Server": ["Caddy"],
"Content-Length": ["349"],
"Access-Control-Allow-Origin": ["http://test.empark.pt"],
"Access-Control-Allow-Methods": ["GET, HEAD, POST, PUT, DELETE, OPTIONS"]
}
}
browser:
ump-player.js:24579 WebSocket connection to 'wss://test.park.pt/StreamingServer/uwa' failed: Error during WebSocket handshake: Unexpected response code: 400
[Line#ump-player.js:24541:21] Oct 21 2020 11:19:29 ERROR - onclose: websocket closed, channelId [0], Raw Data [{"isTrusted":true}], websocket state [3], Error Code: [-1, Name: Unknown, Desc: Unknown Status code.]
[Line#ump-player.js:24452:25] Oct 21 2020 11:19:29 ERROR - error1: websocket closed, channelId [0], Raw Data [{"isTrusted":true}], websocket state [3], Error Code: [1006, Name: Abnormal Closure, Desc: Reserved. Used to indicate that a connection was closed abnormally (that is, with no close frame being sent) when a status code is expected.]
request headers directly to backend, without caddy, with stream working:
GET ws://10.35.104.129/StreamingServer/uwa HTTP/1.1
Host: 10.35.104.129
Connection: Upgrade
Pragma: no-cache
Cache-Control: no-cache
User-Agent: Mozilla/5.0 (Windows NT 6.3; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/86.0.4240.75 Safari/537.36
Upgrade: websocket
Origin: http://10.35.104.129
Sec-WebSocket-Version: 13
Accept-Encoding: gzip, deflate
Accept-Language: en-US,en;q=0.9
Cookie: nvr_lang=11; AUTH_KEY=8a29f4674de070dcc3faf68efc40dff8; ID=Y2VudHJhbA==; AUTH_HEADER_UWA={"realm":"Wisenet NVR","nonce":"8fae1b2e96eb607655e9c9783be12f1c","qop":"auth","cnonce":"22da358ce2b117fc","opaque":null,"scheme":"Digest","nc":1}
Sec-WebSocket-Key: fHbEP3GaQqP8BlY3b/Nkog==
Sec-WebSocket-Extensions: permessage-deflate; client_max_window_bits
response headers for the above
HTTP/1.1 101 Switching Protocols
Upgrade: websocket
Connection: Upgrade
Sec-WebSocket-Accept: gpHs02MuXh7d4KCUm5x1+guzPas=
5. What I already tried:
I honestly tried so many things, trying to manipulate headers adding the ones below, adding specific reverse_proxy directives to /StreamingServer/uwa, disabling buffering, forcing the site to HTTP and disabing “upgrade-insecure-requests”, nothing has made me closer to solving this. I have isolated the configuration for this service from the rest, to exclude any issues with the configuration for other services and the issue continues.
I found an issue in GitHub of a similar problem ,error 400 with websockets, and tried the suggested header suppression.
https://github.com/caddyserver/caddy/issues/3778
header_up Host {host}
header_up X-Real-IP {remote_host}
header_up X-Forwarded-For {remote_host}
header_up X-Forwarded-Proto {scheme}