Using Web Sockets

[Tal (Vitaliy Zvonarov)]

1. Caddy version (caddy version):

v2.1.1 h1:X9k1+ehZPYYrSqBvf/ocUgdLSRIuiNiMo7CvyGUQKeA=

2. How I run Caddy:

a. System environment:

CentOS Linux release 7.7.1908 (Core)

b. Command:

systemctl start caddy

c. Service/unit/compose file:

[Unit]
Description=Caddy
Documentation=https://caddyserver.com/docs/
After=network.target

[Service]
User=nginx
Group=nginx
ExecStart=/usr/local/bin/caddy run --environ --config /etc/caddy/caddy.conf
ExecReload=/usr/local/bin/caddy reload --config /etc/caddy/caddy.conf
TimeoutStopSec=5s
LimitNOFILE=1048576
LimitNPROC=512
PrivateTmp=true
ProtectSystem=full
AmbientCapabilities=CAP_NET_BIND_SERVICE
Restart=always
RestartSec=30

[Install]
WantedBy=multi-user.target

d. My complete Caddyfile or JSON config:

{
    "apps": {
        "http": {
            "servers": {
                "my_server": {
                    "listen": [":60000"],
                    "automatic_https": {
                        "disable": true
                    },
                    "routes": [
                        {
                            "handle": [{
                                "handler": "reverse_proxy",
                                "upstreams": [
                                    {
                                        "dial": "192.168.185.70:80"
                                    }
                                ]
                            }]
                        }
                    ],
                    "tls_connection_policies": [{
                        "match": {
                            "sni": ["SUBDOMAIN.DOMAIN.COM"]
                        }
                    }]
                }
            }
        },
        "tls": {
            "certificates": {
                "load_files": [{
                    "certificate": "/etc/caddy/ssl/SUBDOMAIN.DOMAIN.COM/fullchain.pem",
                    "key": "/etc/caddy/ssl/SUBDOMAIN.DOMAIN.COM/privkey.pem"
                }]
            }
        }
    }   
}

3. The problem I’m having:

While this seems to work great for transparently adding encryption to my HTTP upstream server, I am having issues with websockets.

My upstream server is running Flask, with flask_socketio, which sets up a websocket between the browser and Flask. When I use caddy without all the SSL stuff (just to proxy HTTP connections), everything works fine. The web socket connects, and I have no issues.

With the caddy setup above, my websocket connects, and almost immediately disconnects. I see this:

From what I understand, the websocket connects, and since my server is configured to send a message to the browser immediately upon connection, that goes through (the Object { status: "SCRIPT_STATUS_COMPLETE"... line). Then, for some unknown reason, the web socket is dropped almost immediately after that. No idea why.

It then tries to reconnect, reconnects, gets the message from the server, and disconnects again.

4. Error messages and/or full log output:

For errors on the browser, see above.
journalctl -u caddy shows no errors. Neither does Flask.
Normally the all requests hit nginx on the upstream server before they get to flask, but I removed Nginx from the equation by making gunicorn, which is what runs my flask, bind directly to port 80, rather than a unix socket that talks to Nginx.

5. What I already tried:

I’m at a loss for what I can even look at.

Am I right to assume that in this scenario, the browser tries to establish the web socket using wss (rather than ws) because the web server is being provided as an HTTPS service, which caddy will than translate to an HTTP (ws) request before passing it on to the upstream server? It seems like if this was the problem, I wouldn’t get that initial connection and message from the server at all.

[francislavoie]

Are you sure that the websocket server is served on the same port as your regular HTTP flask server?

Add this to your config (at the top level) to turn on debug logs, it should reveal some more info about the proxying.

"logging":{"logs":{"default":{"level":"DEBUG"}}}

Also, check the network tab of your browser debugger, what does the websocket request/response look like?

[Tal (Vitaliy Zvonarov)]

The web browser shows this:

The caddy logs, set to level DEBUG, show this:

https://gist.github.com/terminator14/d0689da67345e720149365b2883dda64

Caddy seems to be returning an HTTP 400 response code. Not sure why.

Edit: Upon further inspection, tcpdump on the upstream server shows that it is the upstream server returning the HTTP 400 code - not Caddy. Still don’t know why. It works perfectly fine without SSL enabled

Edit 2: This does not seem to be a Caddy-specific problem. Disabling Caddy, and setting up nginx to do the same thing I have Caddy doing here, I get the same result (web sockets disconnecting with an HTTP 400). My Nginx config is this:

server {
    listen 60000 ssl default_server;
    server_name ___;

    ssl_certificate         /etc/caddy/ssl/SUBDOMAIN.DOMAIN.COM/fullchain.pem;
    ssl_certificate_key     /etc/caddy/ssl/SUBDOMAIN.DOMAIN.COM/privkey.pem;

    proxy_set_header Upgrade $http_upgrade;
    proxy_set_header Connection "upgrade";
    proxy_http_version 1.1;

    location / { 
        proxy_pass http://192.168.185.70;
    }   
}

[Tal (Vitaliy Zvonarov)]

Found the issue.
On the Flask SocketIO page it says:

Cross-Origin Controls

For security reasons, this server enforces a same-origin policy by default. In practical terms, this means the following:

• If an incoming HTTP or WebSocket request includes the Origin header, this header must match the scheme and host of the connection URL. In case of a mismatch, a 400 status code response is returned and the connection is rejected.
• No restrictions are imposed on incoming requests that do not include the Origin header.

If necessary, the cors_allowed_origins option can be used to allow other origins.

Since this sounded exactly like what I was seeing (HTTP 400), I quickly added cors_allowed_origins to my Flask config, which fixed the issue using either nginx or caddy. I’m not exactly sure what the issue is, and leaving cros_allowed_origins set to allow all origins sounds like some sort of security issue, but at least this gives me an idea of where the problem lies, and what to research.

Thanks

[matt (Matt Holt)]

Thanks for following up with the solution! (And yes, allowing all origins could potentially be a security issue depending on… things. Try to be rigorous about which ones are approved.)

[francislavoie]

I think it might be ignoring the X-Forwarded-Proto header which would tell Flask what the originating request’s scheme was. Caddy will set that to https when the originating request was over HTTPS. The proxied request is HTTP though, because Caddy terminates TLS.

Or maybe it’s expecting to see X-Forwarded-Proto set to http?

I think you should reach out to them and see how they determine the “the scheme and host of the connection URL” so that you can configure Caddy to send what they expect to see.

[Tal (Vitaliy Zvonarov)]

Yes - that was part of the problem. It turns out that:

1. Since Caddy was sending the request to the Nginx process on the upstream server, Nginx needed to pass along the X-Forwarded-Proto that Caddy was setting to Flask (actually to Gunicorn, which passed it to Flask)
2. Flask needs to be told to not ignore the X-Forwarded-Proto

Everything seems to work now.

Thanks