Proxmox Websocket Issues

inamati · February 5, 2024, 6:27pm

Hello,

I am trying to use caddy as a kind of central proxy in my homelab but I am having some issues with proxmox. In proxmox I am now unable to use the embedded VNC viewer to access the guest VMs. If I connect to proxmox directly via IP it works.

The error I am getting is shown below.

This seems to be an issue with the websocket I think.

In the logs everything seems ok I think.

{"level":"debug","ts":1707157009.500832,"logger":"http.handlers.reverse_proxy","msg":"selected upstream","dial":"10.0.20.2:8006","total_upstreams":1}
{"level":"debug","ts":1707157009.526553,"logger":"http.handlers.reverse_proxy","msg":"upstream roundtrip","upstream":"10.0.20.2:8006","duration":0.025688574,"request":{"remote_ip":"10.0.10.2","remote_port":"50482","client_ip":"10.0.10.2","proto":"HTTP/2.0","method":"POST","host":"odin.undebug.org","uri":"/api2/json/nodes/odin/lxc/104/termproxy","headers":{"Sec-Ch-Ua-Platform":["\"Linux\""],"Referer":["https://odin.undebug.org/?console=lxc&vmid=104&node=odin&resize=scale&xtermjs=1"],"X-Forwarded-Proto":["https"],"Sec-Ch-Ua":["\"Not_A Brand\";v=\"8\", \"Chromium\";v=\"120\", \"Brave\";v=\"120\""],"Origin":["https://odin.undebug.org"],"Sec-Fetch-Site":["same-origin"],"Sec-Fetch-Dest":["empty"],"Cache-Control":["no-cache"],"Csrfpreventiontoken":["65C12611:zPxBXkPW2GXPWWIReWArA58sRqIVNFdS0O6+E/YyLE8"],"Accept":["*/*"],"Sec-Ch-Ua-Mobile":["?0"],"Sec-Fetch-Mode":["cors"],"Cookie":[],"User-Agent":["Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/120.0.0.0 Safari/537.36"],"X-Forwarded-Host":["odin.undebug.org"],"X-Forwarded-For":["10.0.10.2"],"Accept-Language":["en-US,en;q=0.9"],"Content-Length":["0"],"Accept-Encoding":["gzip, deflate, br"],"Sec-Gpc":["1"],"Content-Type":["application/x-www-form-urlencoded"]},"tls":{"resumed":false,"version":772,"cipher_suite":4865,"proto":"h2","server_name":"swlr.undebug.org"}},"headers":{"Pragma":["no-cache"],"Server":["pve-api-daemon/3.0"],"Content-Length":["490"],"Expires":["Mon, 05 Feb 2024 18:16:49 GMT"],"Cache-Control":["max-age=0"],"Date":["Mon, 05 Feb 2024 18:16:49 GMT"],"Content-Type":["application/json;charset=UTF-8"]},"status":200}
{"level":"debug","ts":1707157009.5618865,"logger":"events","msg":"event","name":"tls_get_certificate","id":"a7db6528-8eea-4ef3-8db1-ae1421186dc2","origin":"tls","data":{"client_hello":{"CipherSuites":[14906,4865,4866,4867,49195,49199,49196,49200,52393,52392,49171,49172,156,157,47,53],"ServerName":"odin.undebug.org","SupportedCurves":[47802,29,23,24],"SupportedPoints":"AA==","SignatureSchemes":[1027,2052,1025,1283,2053,1281,2054,1537],"SupportedProtos":["http/1.1"],"SupportedVersions":[64250,772,771],"RemoteAddr":{"IP":"10.0.10.2","Port":44234,"Zone":""},"LocalAddr":{"IP":"10.0.60.6","Port":443,"Zone":""}}}}
{"level":"debug","ts":1707157009.5619512,"logger":"tls.handshake","msg":"no matching certificates and no custom selection logic","identifier":"odin.undebug.org"}
{"level":"debug","ts":1707157009.56196,"logger":"tls.handshake","msg":"choosing certificate","identifier":"*.undebug.org","num_choices":1}
{"level":"debug","ts":1707157009.5619714,"logger":"tls.handshake","msg":"default certificate selection results","identifier":"*.undebug.org","subjects":["*.undebug.org"],"managed":true,"issuer_key":"acme-v02.api.letsencrypt.org-directory","hash":"a2615d42fd0043ed96c3654122339fe2a60aee312189f60abbbb7bf8537fc051"}
{"level":"debug","ts":1707157009.5619788,"logger":"tls.handshake","msg":"matched certificate in cache","remote_ip":"10.0.10.2","remote_port":"44234","subjects":["*.undebug.org"],"managed":true,"expiration":1713556708,"hash":"a2615d42fd0043ed96c3654122339fe2a60aee312189f60abbbb7bf8537fc051"}
{"level":"debug","ts":1707157009.5684803,"logger":"http.handlers.reverse_proxy","msg":"selected upstream","dial":"10.0.20.2:8006","total_upstreams":1}
{"level":"debug","ts":1707157009.5723453,"logger":"http.handlers.reverse_proxy","msg":"upstream roundtrip","upstream":"10.0.20.2:8006","duration":0.003829169,"request":{"remote_ip":"10.0.10.2","remote_port":"44234","client_ip":"10.0.10.2","proto":"HTTP/1.1","method":"GET","host":"odin.undebug.org","uri":"/api2/json/nodes/odin/lxc/104/vncwebsocket?port=5900&vncticket=PVEVNC%3A65C12611%3A%3AVbIdJuQC2kA1t6x2MHt0P5PLtFeg3FRdf4Rq%2FufjZOIr6tPzb77cYxmz94L008BZOir%2Beuwh%2BHkmVVykTuq4Nx4ghtUhd46uTwTNVLp2%2Fxc4KHeGyNeX4P4LKbbKBYjl8UDyZCWhTqB2iS0mEd1C8kCn6%2FxbIBI3qRcS3wkJig56vqRQRj3DzhatUOHXeQJnNXimMmL7MgB8BUEDpS4PCH%2B4G2TFoSX7n3RAq8yl1EZtMBdvW6rqhw1%2F%2FTYB%2FXczEarlV7TKuk9FXViBUEtQJkf%2BVfCMk%2ByaG4egxKabXieKB26CcGwJ1kThlMbyp18p3QWAr5fFV5bwrMAndxM%2Bqg%3D%3D","headers":{"Sec-Websocket-Version":["13"],"X-Forwarded-For":["10.0.10.2"],"X-Forwarded-Host":["odin.undebug.org"],"Origin":["https://odin.undebug.org"],"Accept-Encoding":["gzip, deflate, br"],"Accept-Language":["en-US,en;q=0.9"],"Connection":["Upgrade"],"Cache-Control":["no-cache"],"Sec-Websocket-Extensions":["permessage-deflate; client_max_window_bits"],"Upgrade":["websocket"],"X-Forwarded-Proto":["https"],"Pragma":["no-cache"],"Cookie":[],"Sec-Websocket-Key":["yLkvavMBsWZrh9eTWpBoqw=="],"User-Agent":["Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/120.0.0.0 Safari/537.36"],"Sec-Websocket-Protocol":["binary"]},"tls":{"resumed":false,"version":772,"cipher_suite":4865,"proto":"http/1.1","server_name":"odin.undebug.org"}},"headers":{"Upgrade":["websocket"],"Connection":["upgrade"],"Sec-Websocket-Accept":["9aDokSbIouJQJ906rFF/Mvpt5qo="],"Sec-Websocket-Protocol":["binary"]},"status":101}
{"level":"debug","ts":1707157009.572405,"logger":"http.handlers.reverse_proxy","msg":"upgrading connection","upstream":"10.0.20.2:8006","duration":0.003829169,"request":{"remote_ip":"10.0.10.2","remote_port":"44234","client_ip":"10.0.10.2","proto":"HTTP/1.1","method":"GET","host":"odin.undebug.org","uri":"/api2/json/nodes/odin/lxc/104/vncwebsocket?port=5900&vncticket=PVEVNC%3A65C12611%3A%3AVbIdJuQC2kA1t6x2MHt0P5PLtFeg3FRdf4Rq%2FufjZOIr6tPzb77cYxmz94L008BZOir%2Beuwh%2BHkmVVykTuq4Nx4ghtUhd46uTwTNVLp2%2Fxc4KHeGyNeX4P4LKbbKBYjl8UDyZCWhTqB2iS0mEd1C8kCn6%2FxbIBI3qRcS3wkJig56vqRQRj3DzhatUOHXeQJnNXimMmL7MgB8BUEDpS4PCH%2B4G2TFoSX7n3RAq8yl1EZtMBdvW6rqhw1%2F%2FTYB%2FXczEarlV7TKuk9FXViBUEtQJkf%2BVfCMk%2ByaG4egxKabXieKB26CcGwJ1kThlMbyp18p3QWAr5fFV5bwrMAndxM%2Bqg%3D%3D","headers":{"Sec-Websocket-Version":["13"],"X-Forwarded-For":["10.0.10.2"],"X-Forwarded-Host":["odin.undebug.org"],"Origin":["https://odin.undebug.org"],"Accept-Encoding":["gzip, deflate, br"],"Accept-Language":["en-US,en;q=0.9"],"Connection":["Upgrade"],"Cache-Control":["no-cache"],"Sec-Websocket-Extensions":["permessage-deflate; client_max_window_bits"],"Upgrade":["websocket"],"X-Forwarded-Proto":["https"],"Pragma":["no-cache"],"Cookie":[],"Sec-Websocket-Key":["yLkvavMBsWZrh9eTWpBoqw=="],"User-Agent":["Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/120.0.0.0 Safari/537.36"],"Sec-Websocket-Protocol":["binary"]},"tls":{"resumed":false,"version":772,"cipher_suite":4865,"proto":"http/1.1","server_name":"odin.undebug.org"}}}
{"level":"debug","ts":1707157010.270217,"logger":"http.handlers.reverse_proxy","msg":"selected upstream","dial":"10.0.20.2:8006","total_upstreams":1}
{"level":"debug","ts":1707157010.2840507,"logger":"http.handlers.reverse_proxy","msg":"upstream roundtrip","upstream":"10.0.20.2:8006","duration":0.013760844,"request":{"remote_ip":"10.0.10.2","remote_port":"50482","client_ip":"10.0.10.2","proto":"HTTP/2.0","method":"GET","host":"odin.undebug.org","uri":"/api2/json/nodes/odin/lxc/104/status/current","headers":{"Sec-Ch-Ua":["\"Not_A Brand\";v=\"8\", \"Chromium\";v=\"120\", \"Brave\";v=\"120\""],"X-Forwarded-Proto":["https"],"Cookie":[],"User-Agent":["Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/120.0.0.0 Safari/537.36"],"Sec-Fetch-Dest":["empty"],"Accept-Language":["en-US,en;q=0.9"],"Accept":["*/*"],"Sec-Fetch-Mode":["cors"],"Accept-Encoding":["gzip, deflate, br"],"Csrfpreventiontoken":["65C12348:RuqDIAmAuq8YTipX6+0oxhHfgD6AuchdT1+SNhjbpnw"],"X-Forwarded-For":["10.0.10.2"],"Sec-Ch-Ua-Mobile":["?0"],"Referer":["https://odin.undebug.org/"],"X-Requested-With":["XMLHttpRequest"],"Sec-Fetch-Site":["same-origin"],"Sec-Ch-Ua-Platform":["\"Linux\""],"Sec-Gpc":["1"],"X-Forwarded-Host":["odin.undebug.org"]},"tls":{"resumed":false,"version":772,"cipher_suite":4865,"proto":"h2","server_name":"swlr.undebug.org"}},"headers":{"Date":["Mon, 05 Feb 2024 18:16:50 GMT"],"Pragma":["no-cache"],"Expires":["Mon, 05 Feb 2024 18:16:50 GMT"],"Connection":["Keep-Alive","Keep-Alive"],"Server":["pve-api-daemon/3.0"],"Content-Length":["330"],"Content-Type":["application/json;charset=UTF-8"],"Cache-Control":["max-age=0"]},"status":200}
{"level":"debug","ts":1707157011.386197,"logger":"http.handlers.reverse_proxy","msg":"selected upstream","dial":"10.0.20.2:8006","total_upstreams":1}
{"level":"debug","ts":1707157011.3974354,"logger":"http.handlers.reverse_proxy","msg":"upstream roundtrip","upstream":"10.0.20.2:8006","duration":0.011179593,"request":{"remote_ip":"10.0.10.2","remote_port":"50482","client_ip":"10.0.10.2","proto":"HTTP/2.0","method":"GET","host":"odin.undebug.org","uri":"/api2/json/nodes/odin/lxc/104/status/current","headers":{"Sec-Fetch-Dest":["empty"],"Accept-Encoding":["gzip, deflate, br"],"Sec-Fetch-Mode":["cors"],"Cookie":[],"Csrfpreventiontoken":["65C12348:RuqDIAmAuq8YTipX6+0oxhHfgD6AuchdT1+SNhjbpnw"],"Sec-Ch-Ua-Mobile":["?0"],"Accept":["*/*"],"X-Forwarded-Proto":["https"],"Accept-Language":["en-US,en;q=0.9"],"Referer":["https://odin.undebug.org/"],"X-Requested-With":["XMLHttpRequest"],"X-Forwarded-Host":["odin.undebug.org"],"Sec-Ch-Ua-Platform":["\"Linux\""],"X-Forwarded-For":["10.0.10.2"],"Sec-Gpc":["1"],"Sec-Fetch-Site":["same-origin"],"Sec-Ch-Ua":["\"Not_A Brand\";v=\"8\", \"Chromium\";v=\"120\", \"Brave\";v=\"120\""],"User-Agent":["Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/120.0.0.0 Safari/537.36"]},"tls":{"resumed":false,"version":772,"cipher_suite":4865,"proto":"h2","server_name":"swlr.undebug.org"}},"headers":{"Cache-Control":["max-age=0"],"Connection":["Keep-Alive","Keep-Alive"],"Date":["Mon, 05 Feb 2024 18:16:51 GMT"],"Pragma":["no-cache"],"Content-Type":["application/json;charset=UTF-8"],"Expires":["Mon, 05 Feb 2024 18:16:51 GMT"],"Server":["pve-api-daemon/3.0"],"Content-Length":["312"]},"status":200}
{"level":"debug","ts":1707157011.5684938,"logger":"http.handlers.reverse_proxy","msg":"selected upstream","dial":"10.0.20.2:8006","total_upstreams":1}
{"level":"debug","ts":1707157011.5703118,"logger":"http.handlers.reverse_proxy","msg":"selected upstream","dial":"10.0.20.2:8006","total_upstreams":1}
{"level":"debug","ts":1707157011.570479,"logger":"http.handlers.reverse_proxy","msg":"upstream roundtrip","upstream":"10.0.20.2:8006","duration":0.001954489,"request":{"remote_ip":"10.0.10.2","remote_port":"50482","client_ip":"10.0.10.2","proto":"HTTP/2.0","method":"GET","host":"odin.undebug.org","uri":"/api2/json/cluster/resources","headers":{"X-Requested-With":["XMLHttpRequest"],"Referer":["https://odin.undebug.org/"],"Sec-Gpc":["1"],"Csrfpreventiontoken":["65C12348:RuqDIAmAuq8YTipX6+0oxhHfgD6AuchdT1+SNhjbpnw"],"Sec-Fetch-Site":["same-origin"],"X-Forwarded-Proto":["https"],"Accept":["*/*"],"User-Agent":["Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/120.0.0.0 Safari/537.36"],"Cookie":[],"Sec-Ch-Ua":["\"Not_A Brand\";v=\"8\", \"Chromium\";v=\"120\", \"Brave\";v=\"120\""],"Accept-Encoding":["gzip, deflate, br"],"Sec-Fetch-Mode":["cors"],"Accept-Language":["en-US,en;q=0.9"],"Sec-Ch-Ua-Platform":["\"Linux\""],"Sec-Ch-Ua-Mobile":["?0"],"X-Forwarded-For":["10.0.10.2"],"X-Forwarded-Host":["odin.undebug.org"],"Sec-Fetch-Dest":["empty"]},"tls":{"resumed":false,"version":772,"cipher_suite":4865,"proto":"h2","server_name":"swlr.undebug.org"}},"headers":{"Date":["Mon, 05 Feb 2024 18:16:51 GMT"],"Server":["pve-api-daemon/3.0"],"Content-Encoding":["gzip"],"Expires":["Mon, 05 Feb 2024 18:16:51 GMT"],"Cache-Control":["max-age=0"],"Connection":["Keep-Alive"],"Pragma":["no-cache"],"Content-Length":["985"],"Content-Type":["application/json;charset=UTF-8"]},"status":200}
{"level":"debug","ts":1707157011.5716717,"logger":"http.handlers.reverse_proxy","msg":"upstream roundtrip","upstream":"10.0.20.2:8006","duration":0.001333997,"request":{"remote_ip":"10.0.10.2","remote_port":"50482","client_ip":"10.0.10.2","proto":"HTTP/2.0","method":"GET","host":"odin.undebug.org","uri":"/api2/json/cluster/tasks","headers":{"Sec-Fetch-Dest":["empty"],"Csrfpreventiontoken":["65C12348:RuqDIAmAuq8YTipX6+0oxhHfgD6AuchdT1+SNhjbpnw"],"X-Forwarded-Host":["odin.undebug.org"],"Sec-Ch-Ua":["\"Not_A Brand\";v=\"8\", \"Chromium\";v=\"120\", \"Brave\";v=\"120\""],"X-Requested-With":["XMLHttpRequest"],"Accept-Language":["en-US,en;q=0.9"],"Sec-Fetch-Mode":["cors"],"Referer":["https://odin.undebug.org/"],"Accept-Encoding":["gzip, deflate, br"],"Cookie":[],"User-Agent":["Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/120.0.0.0 Safari/537.36"],"Accept":["*/*"],"Sec-Fetch-Site":["same-origin"],"X-Forwarded-For":["10.0.10.2"],"Sec-Ch-Ua-Mobile":["?0"],"Sec-Ch-Ua-Platform":["\"Linux\""],"Sec-Gpc":["1"],"X-Forwarded-Proto":["https"]},"tls":{"resumed":false,"version":772,"cipher_suite":4865,"proto":"h2","server_name":"swlr.undebug.org"}},"headers":{"Content-Length":["1023"],"Connection":["Keep-Alive"],"Date":["Mon, 05 Feb 2024 18:16:51 GMT"],"Pragma":["no-cache"],"Content-Type":["application/json;charset=UTF-8"],"Expires":["Mon, 05 Feb 2024 18:16:51 GMT"],"Cache-Control":["max-age=0"],"Server":["pve-api-daemon/3.0"],"Content-Encoding":["gzip"]},"status":200}
{"level":"debug","ts":1707157012.486501,"logger":"http.handlers.reverse_proxy","msg":"selected upstream","dial":"10.0.20.2:8006","total_upstreams":1}
{"level":"debug","ts":1707157012.498334,"logger":"http.handlers.reverse_proxy","msg":"upstream roundtrip","upstream":"10.0.20.2:8006","duration":0.011778606,"request":{"remote_ip":"10.0.10.2","remote_port":"50482","client_ip":"10.0.10.2","proto":"HTTP/2.0","method":"GET","host":"odin.undebug.org","uri":"/api2/json/nodes/odin/lxc/104/status/current","headers":{"Csrfpreventiontoken":["65C12348:RuqDIAmAuq8YTipX6+0oxhHfgD6AuchdT1+SNhjbpnw"],"Referer":["https://odin.undebug.org/"],"Accept-Encoding":["gzip, deflate, br"],"Sec-Fetch-Mode":["cors"],"X-Requested-With":["XMLHttpRequest"],"X-Forwarded-Proto":["https"],"User-Agent":["Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/120.0.0.0 Safari/537.36"],"Sec-Gpc":["1"],"Cookie":[],"Sec-Ch-Ua-Mobile":["?0"],"Sec-Ch-Ua-Platform":["\"Linux\""],"Sec-Ch-Ua":["\"Not_A Brand\";v=\"8\", \"Chromium\";v=\"120\", \"Brave\";v=\"120\""],"Accept":["*/*"],"Sec-Fetch-Site":["same-origin"],"X-Forwarded-For":["10.0.10.2"],"X-Forwarded-Host":["odin.undebug.org"],"Sec-Fetch-Dest":["empty"],"Accept-Language":["en-US,en;q=0.9"]},"tls":{"resumed":false,"version":772,"cipher_suite":4865,"proto":"h2","server_name":"swlr.undebug.org"}},"headers":{"Connection":["Keep-Alive","Keep-Alive"],"Date":["Mon, 05 Feb 2024 18:16:52 GMT"],"Content-Length":["331"],"Expires":["Mon, 05 Feb 2024 18:16:52 GMT"],"Cache-Control":["max-age=0"],"Server":["pve-api-daemon/3.0"],"Content-Type":["application/json;charset=UTF-8"],"Pragma":["no-cache"]},"status":200}

I am using v2.7.6 running in a LXC container and my currect config is as follows.

{
        # Global Options
        # debug
        acme_dns cloudflare D)dm1)-ABTC!Nu(R!5:(F*JbWaX$vH
        email MY_EMAIL
        log {
                output file /var/log/caddy/caddy.log
                level DEBUG
        }
}

*.undebug.org {
        @denied not client_ip private_ranges
        abort @denied

        @odin {
                host odin.undebug.org
                client_ip 10.0.0.0/8
        }

        handle @odin {
                reverse_proxy 10.0.20.2:8006 {
                        transport http {
                                tls
                                tls_insecure_skip_verify
                        }
                }
        }

        @all {
                client_ip 10.0.0.0/8
        }

        # Fallback for otherwise unhandled domains
        handle @all {
                reverse_proxy 10.0.10.29:443
        }
}

Any help is appreciated!!!

francislavoie · February 6, 2024, 1:20am

I hope that’s not your real API key. You must keep that secret. If it is your real one, you should revoke it as soon as possible and regenerate it.

This log suggests the websocket connection was upgraded successfully.

I’m not seeing any error logs.

inamati · February 6, 2024, 9:00am

No it’s not. I changed it to a random string.

Same here. I also don’t see anything on the logs which means that I am probably looking in the wrong place hence posting here.

francislavoie · February 6, 2024, 9:39am

Check your browser’s network tab (open it before loading the page). Look for the websocket request. What status code do you see? What’s in the Response tab? Is it empty or was there some traffic?

My hunch is that it’s an error that occurs after the connection is established.

Do you have somekind of proxy in front of Caddy? If so, maybe the connection is failing there.

inamati · February 6, 2024, 11:40am

This is what I see on the browser.

inamati · February 6, 2024, 11:40am

inamati · February 6, 2024, 11:43am

In Proxmox the error is a connection timeout so it definitely seems that Caddy is not forwarding something. There is nothing behind the Caddy proxy.

francislavoie · February 6, 2024, 11:53am

Expand the “Response Headers” section in the network log. Click on the “Messages” tab. Anything in there?

francislavoie · February 6, 2024, 11:55am

You should probably ask for help from the Proxmox community.

Right now, there’s no evidence of a problem with Caddy specifically.

inamati · February 6, 2024, 2:10pm

It only works when I remove Caddy. When I put Caddy in the middle it stops working. How is this NOT a Caddy issue?

inamati · February 6, 2024, 2:33pm

Just tried with a small instance running haproxy and it works. Seems to be caddy issue I think.

inamati · February 6, 2024, 2:53pm

Just found out that it also causes issues with Home Assistant

Has to be some wrong or missing configuration that I don’t know about.

Whitestrake · February 6, 2024, 11:39pm

Just chiming in to say that, anecdotally, I haven’t seen these websocket issues from Caddy’s end.

I’m using it on multiple Proxmox systems - you can check out the kind of simple config I use there, mostly just to get the GUI on default ports, see: Caddy as a Proxmox Reverse Proxy

Additionally, I use it on a pair of TrueNAS systems (one Scale and one Core), both of which use websockets for its GUI. In that case, I use Caddy to allow the GUI to virtual host my internal Tailscale hostname as well as its LAN hostname, since such a configuration isn’t achievable through the TrueNAS GUI itself. It uses NGINX under the hood, but throwing Caddy in front (especially with its built-in Tailscale Certificates integration) is simpler and more repeatable than hacking on conf files; it’s more reliable to use TrueNAS GUI to shift the web UI port and then reverse proxy it. Here’s one such Caddyfile:

Caddyfile for TrueNAS

{
	acme_dns cloudflare [snip]
}

triton.fell-monitor.ts.net, triton.lab.whitestrake.net {
	reverse_proxy https://:444 {
		transport http {
			tls_server_name triton.lab.whitestrake.net
		}
	}
	tls {
		# In practice, this seems necessary or Caddy won't acquire
		# and serve the TS cert simultaneously with the ACME cert
		get_certificate tailscale
	}
}

In both of these cases, websockets are functioning just fine. Websockets in Caddy v2, for me, have always been rock solid with zero config out of the box; I would be more inclined to investigate either end (server or client) for blocking issues - and in your case, since two backends seem to be showing this issue, I’d be looking at your browser. Does this fail on other browsers, in incognito/private with no extensions, on other computers, etc?

inamati · February 7, 2024, 2:12pm

You are absolutely right and I was pretty sure that this used to work so I started debugging everything in my homelab (even the roomba). Turns out Zenarmor wasn’t liking my domain for some reason. Nothing to do with Caddy at all.

Thank you for your help

system · March 8, 2024, 2:13pm

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.