Help with Caddy reverse proxy configuration involving wss

Help
1

Hello Caddy community. Let me preface all of this by saying that I am a complete beginner, so please bear with me if write something that does not make sense. I would kindly ask for your suggestions in overcoming the issue I am facing.

Let me first describe the environment. There are two networks, each under a separate Active Directory domain, and those tho are corporate.local and industrial.local. Each of the networks has several web applications that the clients in the other network need access to. Caddy is running in DMZ in order to enable that. hosts files on clients in one of domains were modified by to include FQDNs of the servers in other domain, and each host record is pointing to the machine running Caddy. hosts file on the machine running Caddy was modified to point to the servers running the web applications in another domain.

1. The problem I’m having:

I was tasked with configuring Caddy as a reverse proxy for a web app but I am failing to do so.

It all started with the most basic configuration:

https://sns01mb01-sfhub.corporate.local {
	tls /etc/caddy/RevProxy.crt /etc/caddy/RevProxy.key
	reverse_proxy {
		to https://sns01mb01-sfhub.corporate.local
	}
}

With this configuration, login page loaded successfully. Upon logging in, I observed an error in browser’s console:

WebSocket connection to 'wss://sns01mb01-sfhub.corporate.local:18666/service/ebgojzh1rizxspso' failed: Error in connection establishment: net::ERR_CONNECTION_REFUSED

That error message prompted me to add the following to the configuration:

https://sns01mb01-sfhub.corporate.local:18666 {
	tls /etc/caddy/RevProxy.crt /etc/caddy/RevProxy.key
	reverse_proxy {
		to https://sns01mb01-sfhub.corporate.local:18666
	}
}

That change in the configuration file produced different output in browser’s console:

WebSocket connection to 'wss://sns01mb01-sfhub.corporate.local:18666/service/lzwagadca8ft5tia' failed: Error during WebSocket handshake: Unexpected response code: 426

Caddy logs the following errors:

{
    "level": "error",
    "ts": 1696316012.6178885,
    "logger": "http.handlers.reverse_proxy",
    "msg": "aborting with incomplete response",
    "upstream": "sns01mb01-sfhub.corporate.local:18666",
    "duration": 0.033175606,
    "request": {
        "remote_ip": "10.100.103.21",
        "remote_port": "53643",
        "client_ip": "10.100.103.21",
        "proto": "HTTP/1.1",
        "method": "GET",
        "host": "sns01mb01-sfhub.corporate.local:18666",
        "uri": "/service/lzwagadca8ft5tia",
        "headers": {
            "Cache-Control": [
                "no-cache"
            ],
            "Origin": [
                "https://sns01mb01-sfhub.corporate.local"
            ],
            "X-Forwarded-Proto": [
                "https"
            ],
            "Sec-Websocket-Extensions": [
                "permessage-deflate; client_max_window_bits"
            ],
            "Pragma": [
                "no-cache"
            ],
            "Sec-Websocket-Key": [
                "ZaJkn19lgPRGEdVXk28WHA=="
            ],
            "X-Forwarded-Host": [
                "sns01mb01-sfhub.corporate.local:18666"
            ],
            "Accept-Encoding": [
                "gzip, deflate, br"
            ],
            "Upgrade": [
                "websocket"
            ],
            "X-Forwarded-For": [
                "10.100.103.21"
            ],
            "Connection": [
                "Upgrade"
            ],
            "Sec-Websocket-Version": [
                "13"
            ],
            "User-Agent": [
                "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/101.0.4951.54 Safari/537.36 Edg/101.0.1210.39"
            ],
            "Accept-Language": [
                "en-US,en;q=0.9"
            ]
        },
        "tls": {
            "resumed": false,
            "version": 772,
            "cipher_suite": 4865,
            "proto": "http/1.1",
            "server_name": "sns01mb01-sfhub.corporate.local"
        }
    },
    "error": "reading: context canceled"
}

{
    "level": "error",
    "ts": 1696316171.8012946,
    "logger": "http.log.error",
    "msg": "EOF",
    "request": {
        "remote_ip": "10.100.103.21",
        "remote_port": "54203",
        "client_ip": "10.100.103.21",
        "proto": "HTTP/1.1",
        "method": "GET",
        "host": "sns01mb01-sfhub.corporate.local:18666",
        "uri": "/service/3459befb-d422-41e7-8b0d-09e6c1bc45e8",
        "headers": {
            "Cache-Control": [
                "no-cache"
            ],
            "User-Agent": [
                "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/101.0.4951.54 Safari/537.36 Edg/101.0.1210.39"
            ],
            "Sec-Websocket-Version": [
                "13"
            ],
            "Accept-Language": [
                "en-US,en;q=0.9"
            ],
            "Connection": [
                "Upgrade"
            ],
            "Origin": [
                "https://sns01mb01-sfhub.corporate.local"
            ],
            "Accept-Encoding": [
                "gzip, deflate, br"
            ],
            "Sec-Websocket-Key": [
                "TsByTSOfzFEOxUxKb2JLqQ=="
            ],
            "Sec-Websocket-Extensions": [
                "permessage-deflate; client_max_window_bits"
            ],
            "Pragma": [
                "no-cache"
            ],
            "Upgrade": [
                "websocket"
            ]
        },
        "tls": {
            "resumed": true,
            "version": 772,
            "cipher_suite": 4865,
            "proto": "http/1.1",
            "server_name": "sns01mb01-sfhub.corporate.local"
        }
    },
    "duration": 0.036415461,
    "status": 502,
    "err_id": "f9qa0kdd8",
    "err_trace": "reverseproxy.statusError (reverseproxy.go:1248)"
}

When accessing the web app directly from the client in corporate.local domain I can see that initial wss request succeeds with status code 101 Switching Protocols.

2. Error messages and/or full log output:

Due to post character limit, full logs are accessible here.

3. Caddy version:

v2.7.4 h1:J8nisjdOxnYHXlorUKXY75Gr6iBfudfoGhrJ8t7/flI=

4. How I installed and ran Caddy:

Caddy was installed by obtaining a Caddy binary from the download page a manually configuring it to run as a systemd service using the official caddy.service systemd unit file.

a. System environment:

Ubuntu 22.04.3 LTS

systemd 249 (249.11-0ubuntu3.10)

b. Command:

sudo systemctl start caddy

c. Service/unit/compose file:

[Unit]
Description=Caddy
Documentation=https://caddyserver.com/docs/
After=network.target network-online.target
Requires=network-online.target

[Service]
Type=notif
User=caddy
Group=caddy
ExecStart=/usr/bin/caddy run --environ --config /etc/caddy/Caddyfile
ExecReload=/usr/bin/caddy reload --config /etc/caddy/Caddyfile --force
TimeoutStopSec=5s
LimitNOFILE=1048576
LimitNPROC=512
PrivateTmp=true
ProtectSystem=full
AmbientCapabilities=CAP_NET_ADMIN CAP_NET_BIND_SERVICE

[Install]
WantedBy=multi-user.target

d. My complete Caddy config:

{
	debug
	log {
		output file /var/log/caddy/access.log
	}
	http_port 80
	https_port 443
}

https://sns01mb01-sfhub.corporate.local {
	tls /etc/caddy/RevProxy.crt /etc/caddy/RevProxy.key
	reverse_proxy {
		to https://sns01mb01-sfhub.corporate.local
	}
}
https://sns01mb01-sfhub.corporate.local:18666 {
	tls /etc/caddy/RevProxy.crt /etc/caddy/RevProxy.key
	reverse_proxy {
		to https://sns01mb01-sfhub.corporate.local:18666
	}
}

Can anyone spot something that I might have missed when configuring Caddy and point me in the right direction? Thank you!

2

I’m confused because that essentially reads like an infinite loop. Why are you proxying to the same domain that Caddy is serving?

Anyway, this will listen on port 80/443 (for HTTP and HTTPS), but you made a request to port 18666 instead. You should remove the port number from your connection address in your JS client, so that it uses port 443 for HTTPS (WSS is WebSockets over HTTPS, it’s a special scheme only supported in browsers).

Is your actual app running on the same machine as Caddy? If so you can probably just do reverse_proxy localhost:18666 or something to that effect.

You can remove this from your config, it’s not useful to re-state the defaults.

3

Yes, it does look confusing, however, I believe that it does not represent an infinite loop. Caddy resides in DMZ, and the clients that are accessing it reside in industrial.local domain. These clients cannot resolve corporate.local addresses. That is why hosts file on these clients have the following record (which points to the Caddy):

10.99.62.20 sns01mb01-sfhub.corporate.local

Caddy itself also cannot resolve corporate.local addresses, so it has the following record in its hosts file which points to the server hosting the app:

10.100.0.38 sns01mb01-sfhub.corporate.local

Unfortunately, I do not have possibility to modify anything in the app itself. It is a 3rd party app, and, as per the vendor, it supports reverse proxies. The following nginx configuration works as expected on the same system:

server {
    server_name sns01mb01-sfhub.corporate.local;
    listen 443 ssl;
    ssl_certificate	/etc/ssl/certs/RevProxy.crt;
    ssl_certificate_key	/etc/ssl/private/RevProxy.key;
    location / {
        proxy_pass https://sns01mb01-sfhub.corporate.local:443;
        proxy_set_header Host $host;
        proxy_ssl_name $host;
    }
}

server {
    server_name sns01mb01-sfhub.corporate.local;
    listen 18666 ssl;
    ssl_certificate	/etc/ssl/certs/RevProxy.crt;
    ssl_certificate_key	/etc/ssl/private/RevProxy.key;
    location / {
        proxy_pass https://sns01mb01-sfhub.corporate.local:18666;
        proxy_set_header Host $host;
        proxy_ssl_name $host;
        proxy_http_version 1.1;
        proxy_set_header Upgrade $http_upgrade;
        proxy_set_header Connection "upgrade";
    }
}

I hope that it is clearer now that the app isn’t running on the same machine as Caddy.

Thank you for the tip!

4

I’m seeing stuff like this in 502 responses coming from your upstream app:

"Errortype":["Error"],"Errormessage":["Protocol violation 2"]

You’ll need to dig into what this means in your upstream app.

I think you’ll need to contact them for help, then.

5

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.