Websockets fail - Venus OS - Cerbo GX Victron

Help
21

I did not definitively find the issue, but I was suspecting it was to do with the http header casing, which I can see causes some issues with certain proxy targets, e.g. Different capitalization on Response Headers - but that is just a wild guess, might be something totally different.

I saw that the websocket was connecting, but it was then being disconnected soon afterwards. The response headers were also missing the Sec-Websocket-Accept header… which is present when it is working correctly, so perhaps caddy is not handling that somehow?

As it worked with nginx-proxy-manager, I had wanted to inspect the headers, but that seemed too much effort.

I setup traefik to see how that would work, and it proxied to the cerbo fine, I can use the virtual console, etc…

So, sadly, I will switch to traefik. I did really enjoy the simplicity of the Caddyfile configuration, and I find traefik a bit more fiddly, verbose, and feels more complicated, but it works for this scenario, so it is the winner.

Unfortunately for your case, it looks like traefik is not a supported reverse proxy in OPNsense (Reverse Proxy and Webserver — OPNsense documentation).

22

As alternative you can create a TLS SNI route in the Layer 4 module included in Caddy on the OPNsense. If that victron thing has a self signed certificate installed, that will work just nicely.

1 Like
23

Can you share the url for your cerbo gx? If you don’t mind, you can host your app using nginx-proxy-manager, while I create a caddy instance to reverse proxy to your site to help debug this issue?

1 Like
24

Never mind, I think the problem is with websockify. I deployed it locally and websockets also fail.

1 Like
25

Also, it only happens for tls enabled websockets. Plain http succeeds.

1 Like
26

Thank you for the Test.
As you see in my pictures, plain http doesnt work for me :frowning:

I cant share my cerbo-url, I’m sorry. Its LAN-only because of security reasons (caddy plugin, HTTP access, allow only local IPs)

27

Can you show a screenshort of the devtools? Only the console part is ok, since I want to know which urls the websockets are for?

Like this one.

image
image795×652 109 KB

28

Please see the first Posting in this thread. There you find my F12 from Google Chrome.

Do you need another Pictures?

29

Yeah, this is definitely the case of header normalization with golang stdlib.

According to MDN, headers should use WebSocket, S is in uppercase.

However, after normalization, related headers becomes Websocket.

Cerbo GX is using websockify-c , which check for the exact case, and it will fail.

Interestingly, while testing websocket, I used gobwas/ws, which writes the exact header.

I suppose traefik also does raw bytes copying with like caddy-l4, so there is no problem there.

30

Here is the test code:

	const r = `GET / HTTP/1.1
Host: 127.0.0.1
Upgrade: websocket
Connection: Upgrade
Sec-WebSocket-Version: 13
Sec-WebSocket-Key: yXYfVqfxZIl1Bbi/Z/6DhA==
Sec-WebSocket-Protocol: binary, base64
Origin: 127.0.0.1

`

	req, _ := http.ReadRequest(bufio.NewReader(strings.NewReader(r)))
	fmt.Println("before replacing")
	_ = req.Write(os.Stderr)

	fmt.Println("after replacing")
	var replacer = strings.NewReplacer("Websocket", "WebSocket")
	for k, v := range req.Header {
		if strings.Contains(k, "Sec-Websocket") {
			req.Header.Del(k)
			req.Header[replacer.Replace(k)] = v
		}
	}
	_ = req.Write(os.Stderr)

The result is

before replacing
GET / HTTP/1.1
Host: 127.0.0.1
User-Agent: Go-http-client/1.1
Connection: Upgrade
Origin: 127.0.0.1
Sec-Websocket-Key: yXYfVqfxZIl1Bbi/Z/6DhA==
Sec-Websocket-Protocol: binary, base64
Sec-Websocket-Version: 13
Upgrade: websocket

after replacing
GET / HTTP/1.1
Host: 127.0.0.1
User-Agent: Go-http-client/1.1
Connection: Upgrade
Origin: 127.0.0.1
Sec-WebSocket-Key: yXYfVqfxZIl1Bbi/Z/6DhA==
Sec-WebSocket-Protocol: binary, base64
Sec-WebSocket-Version: 13
Upgrade: websocket

Notice the websocket header name is the correct case. And fortunately, this function is also used when stdlib send a request, so it will work if I added this to caddy.

2 Likes
31

@solear
Can you try using the included Layer4 module as an alternative?

Go to “General Settings - Advanced Settings” and check “Enable Layer4”. Apply and go to “Reverse Proxy - Layer4 Routes”

Add a new route with
Matchers: TLS (SNI Client Hello)
Domain: e.g., venus.example.com
Upstream Domain: Internal IP of your solar thingy
Upstream Port: Port of your solar thingy
Remote IP: The internal IPs you want allow access (e.g., 192.168.0.0/16)

Apply and then open https://venus.example.com and it will be reverse proxied without TLS termination, meaning you should see the self signed certificate (if it has one).

1 Like
32

Hello Monviech,
No luck with layer 4. There is the same error (“Failed to connect. Make sure to enable Remote Console for LAN, and reboot the device.”).

noluck1
noluck1844×1228 143 KB

edit: wrong picture, now there is the right picture

33

How did you install caddy? You seem to use OPNSense, and I’m not familiar with that.

34

@solear I will provide you with a new caddy binary to test and instructions later today or tomorrow based on this:

2 Likes
35

@solear I’ve build a caddy version with the cherry picked commit from above.

  1. Connect to your OPNsense via SSH with the root user
  2. Go to the Shell (Option 8)

Enter the following commands in order, each line is one command to copy and paste. Do not copy paste all of them at once, one line each and press enter:

cd /usr/local/bin/

service caddy stop

mv caddy caddy.backup

fetch https://os-caddy-plugin.pischem.com/repo-config/caddy

chmod +x caddy

service caddy start

Make sure you disable the Layer 4 module again if you havent by unchecking the checkbox.

Then try the domain + handler configuration from the beginning again where you first had the issue.

2 Likes
36

It works!
Awesome work, @Monviech !! Thank you very much!

(no need to do the domain + handler configuration from the beginning again - it worked after service caddy start)

2 Likes
37

@WeidiDeng did most of the work. Real mvp <3

Thank you too for testing and confirming.

Just be cautious with OPNsense updates, it might take a while to downstream this fix into the main repos. [probably after caddy 2.9.x released] If in doubt repeat these steps from above after an update, I will host this for a while.

2 Likes
38

Ah nice, thanks for the work. I somehow imagined this thread would die, and no fix emerge, so happy to see that :star:

Maybe I’ll switch back from traefik… although I found another benefit of traffic, the tcp proxying is built in, which I need for another endpoint that uses client certificates. It also let’s me handle caddy-terminated tls and tcp proxy with tls passthrough on the same port, not sure if the layer4 caddy module supports that?

39

It does, yes.

2 Likes
40

@Monviech I think Chrome is also sending Websocket instead of WebSocket, so caddy-l4 doesn’t work in this case.

1 Like