Config for venueless reverse_proxy - giving me "socket timeout" errors

1. Caddy version (caddy version):

caddy version
v2.3.0

2. How I run Caddy:

a. System environment:

lsb_release -a
No LSB modules are available.
Distributor ID: Ubuntu
Description:    Ubuntu 20.04.1 LTS
Release:        20.04
Codename:       focal

No docker

b. Command:

paste command here

c. Service/unit/compose file:

    cat /etc/systemd/system/caddy.service

    [Unit]
    Description=Caddy
    Documentation=https://caddyserver.com/docs/
    After=network.target network-online.target
    Requires=network-online.target

    [Service]
    User=caddy
    Group=caddy
    ExecStart=/usr/local/bin/caddy run --environ --config /etc/caddy/Caddyfile
    ExecReload=/usr/local/bin/caddy reload --config /etc/caddy/Caddyfile
    TimeoutStopSec=5s
    LimitNOFILE=1048576
    LimitNPROC=512
    PrivateTmp=true
    ProtectSystem=full
    AmbientCapabilities=CAP_NET_BIND_SERVICE

    [Install]
    WantedBy=multi-user.target

d. My complete Caddyfile or JSON config:

testevent.x.de {
        log {
                output file /home/x/domains/x.de/logs/access.log
                format single_field common_log
        }

        reverse_proxy 172.16.0.101:8002 {
                header_up Host {host}
                header_up X-Real-IP {remote_host}
                header_up X-Forwarded-For {remote_host}
                header_up X-Forwarded-Proto {scheme}
                #                header_up Upgrade {upgrade} (commented out, didn't work either)
                #                header_up Connection upgrade
                #                header_up X-Forwarded-Ssl on
        }

        encode gzip
}

3. The problem I’m having:

Trying to setup “venueless”. It mentions an nginx config, that seems pretty basic. The site itself is loading, but the websocket connection is not working properly.
Image of the errors in my local client console

This is their default config taken from the official documentation:

server {
    listen 80 default_server;
    listen [::]:80 ipv6only=on default_server;
    server_name venueless.mydomain.com;
}
server {
    listen 443 default_server;
    listen [::]:443 ipv6only=on default_server;
    server_name venueless.mydomain.com;

    ssl on;
    ssl_certificate /path/to/cert.chain.pem;
    ssl_certificate_key /path/to/key.pem;

    location / {
        proxy_set_header    Host $host;
        proxy_set_header    X-Real-IP $remote_addr;
        proxy_set_header    X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header    X-Forwarded-Proto $scheme;
        proxy_http_version 1.1;
        proxy_set_header    Upgrade $http_upgrade;
        proxy_set_header    Connection "upgrade";
        proxy_set_header    X-Forwarded-Ssl on;
        proxy_read_timeout  300s;
        proxy_redirect              http:// https://;
        proxy_pass                  http://localhost:8002;
    }
}

4. Error messages and/or full log output:

Getting “socket closed” errors on the client side. Every request/response for the given websocket looks like this:

5. What I already tried:

  • Tried to add and remove all kinds of headers
  • Tried to catch the /ws/ url and adding all kinds of headers (see the commented sections, these were the headers I was playing with) - no success

6. Links to relevant resources:

https://venueless.readthedocs.io/en/latest/admin/setup.html#id1

Thanks for your time and help!

These lines should are not necessary:

Caddy passes through all headers by default, unlike Nginx, so those aren’t necessary either.

This one depends on your app, whether it needs to know this or not. But if your app looks for X-Forwarded-Proto already, it won’t be needed.

As for the “socket closed” issue though, I couldn’t say. That seems like a message that’s coming from the frontend JS code for that app. I think you should ask on their issue board and cross-link them to here – if they have any questions about Caddy I’d be happy to help

Hi @francislavoie,

thanks for your response. Indeed, I have read that they should not be needed but figured out they don’t hurt either. I have removed them by now and see the same thing happening.

The issue is open over there:
https://github.com/venueless/venueless/issues/188

Let me know in case you have any questions or have any pointers on what I could try.

1 Like

I would enable debug mode and see what is printed in Caddy’s logs:

{
    debug
}
1 Like

Thanks Matt. The Developer over at venueless stated “Nah, “socket closed” just means the websocket (which should normally stay active for the whole usage duration) has been closed by the server or something in between. I don’t think it’s related to HTTP headers, sounds more like a problem with websocket or timeout configuration in caddy.” - this is what I got out of the debug log:

    Feb  2 19:30:02 web2 caddy[3085456]: {"level":"debug","ts":1612290602.610858,"logger":"http.handlers.reverse_proxy","msg":"upstream roundtrip","upstream":"172.16.0.101:8002","request":{"remote_addr":"ipxxx:59148","proto":"HTTP/1.1","method":"GET","host":"testevent.x.de","uri":"/ws/world/homtesting2021/","headers":{"Cookie":["__cfduid=x"],"User-Agent":["Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/88.0.4324.104 Safari/537.36"],"Sec-Websocket-Key":["TYzUBeKR/hAg7k1P4k3qRQ=="],"X-Forwarded-Ssl":["on"],"Cache-Control":["no-cache"],"Sec-Websocket-Extensions":["permessage-deflate; client_max_window_bits"],"X-Forwarded-Proto":["https"],"Connection":["Upgrade"],"X-Forwarded-For":["ipxxx"],"Accept-Language":["en-US,en;q=0.9,de;q=0.8"],"Sec-Websocket-Version":["13"],"Upgrade":["websocket"],"Pragma":["no-cache"],"Origin":["https://testevent.x.de"],"Accept-Encoding":["gzip, deflate, br"]},"tls":{"resumed":true,"version":772,"cipher_suite":4867,"proto":"","proto_mutual":true,"server_name":"testevent.x.de"}},"duration":0.015971005,"headers":{"Connection":["upgrade"],"Upgrade":["websocket"],"Sec-Websocket-Accept":["/TcQOsIlyUIH55rPq9jB429j/FY="],"Server":["nginx"],"Date":["Tue, 02 Feb 2021 18:30:02 GMT"]},"status":101}
    Feb  2 19:30:02 web2 caddy[3085456]: {"level":"info","ts":1612290602.630685,"logger":"http.log.access.log7","msg":"handled request","request":{"remote_addr":"ipxxx:59148","proto":"HTTP/1.1","method":"GET","host":"testevent.x.de","uri":"/ws/world/homtesting2021/","headers":{"Cache-Control":["no-cache"],"Upgrade":["websocket"],"Sec-Websocket-Version":["13"],"Cookie":["__cfduid=x"],"Accept-Language":["en-US,en;q=0.9,de;q=0.8"],"Sec-Websocket-Key":["TYzUBeKR/hAg7k1P4k3qRQ=="],"Sec-Websocket-Extensions":["permessage-deflate; client_max_window_bits"],"Connection":["Upgrade"],"Pragma":["no-cache"],"User-Agent":["Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/88.0.4324.104 Safari/537.36"],"Origin":["https://testevent.x.de"],"Accept-Encoding":["gzip, deflate, br"]},"tls":{"resumed":true,"version":772,"cipher_suite":4867,"proto":"","proto_mutual":true,"server_name":"testevent.x.de"}},"common_log":"ipxxx - - [02/Feb/2021:19:30:02 +0100] \"GET /ws/world/homtesting2021/ HTTP/1.1\" 0 0","duration":0.035942311,"size":0,"status":0,"resp_headers":{"Server":["Caddy"]}}
    Feb  2 19:30:03 web2 caddy[3085456]: {"level":"debug","ts":1612290603.7193534,"logger":"http.handlers.reverse_proxy","msg":"upstream roundtrip","upstream":"172.16.0.101:8002","request":{"remote_addr":"ipxxx:59150","proto":"HTTP/1.1","method":"GET","host":"testevent.x.de","uri":"/ws/world/homtesting2021/","headers":{"X-Forwarded-Ssl":["on"],"Sec-Websocket-Key":["7oKw1Sgw8GSYodgFTDBGWA=="],"Connection":["Upgrade"],"User-Agent":["Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/88.0.4324.104 Safari/537.36"],"Upgrade":["websocket"],"Origin":["https://testevent.x.de"],"X-Forwarded-Proto":["https"],"Pragma":["no-cache"],"Sec-Websocket-Version":["13"],"Cookie":["__cfduid=x"],"Accept-Language":["en-US,en;q=0.9,de;q=0.8"],"Sec-Websocket-Extensions":["permessage-deflate; client_max_window_bits"],"Cache-Control":["no-cache"],"X-Forwarded-For":["ipxxx"],"Accept-Encoding":["gzip, deflate, br"]},"tls":{"resumed":true,"version":772,"cipher_suite":4867,"proto":"","proto_mutual":true,"server_name":"testevent.x.de"}},"duration":0.020374258,"headers":{"Server":["nginx"],"Date":["Tue, 02 Feb 2021 18:30:03 GMT"],"Connection":["upgrade"],"Upgrade":["websocket"],"Sec-Websocket-Accept":["mX1bdw4lZ0wuacPB104jgcFl3NM="]},"status":101}
    Feb  2 19:30:03 web2 caddy[3085456]: {"level":"info","ts":1612290603.7275274,"logger":"http.log.access.log7","msg":"handled request","request":{"remote_addr":"ipxxx:59150","proto":"HTTP/1.1","method":"GET","host":"testevent.x.de","uri":"/ws/world/homtesting2021/","headers":{"Cache-Control":["no-cache"],"User-Agent":["Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/88.0.4324.104 Safari/537.36"],"Upgrade":["websocket"],"Accept-Language":["en-US,en;q=0.9,de;q=0.8"],"Sec-Websocket-Key":["7oKw1Sgw8GSYodgFTDBGWA=="],"Connection":["Upgrade"],"Pragma":["no-cache"],"Origin":["https://testevent.x.de"],"Sec-Websocket-Version":["13"],"Accept-Encoding":["gzip, deflate, br"],"Cookie":["__cfduid=x"],"Sec-Websocket-Extensions":["permessage-deflate; client_max_window_bits"]},"tls":{"resumed":true,"version":772,"cipher_suite":4867,"proto":"","proto_mutual":true,"server_name":"testevent.x.de"}},"common_log":"ipxxx - - [02/Feb/2021:19:30:03 +0100] \"GET /ws/world/homtesting2021/ HTTP/1.1\" 0 0","duration":0.028677879,"size":0,"status":0,"resp_headers":{"Server":["Caddy"]}}

I can’t seem to find any apparent errors :slightly_frowning_face:

Edit:
I have also tried playing around with the transport http options. But I doubt this is the issue, as there seems to be a response. Doesn’t seem to be a client issue either, tried a couple of clients.

1 Like

Hmm, yeah, I’m not sure what to tell you. I know websockets work (I use them myself, as do many others) – but I’m not sure from a glance what the difference is in your case.

You can prove it works by using a Caddyfile like this:

{
	debug
}

localhost

reverse_proxy https://echo.websocket.org {
	header_up Host echo.websocket.org
}

This should connect to the public demo at https://www.websocket.org/echo.html.

They have a sample HTML file you can save and then load in your browser. I simply used that and changed var wsUri to be "wss://localhost/".

Output:

WebSocket Test

CONNECTED

SENT: WebSocket rocks

RESPONSE: WebSocket rocks

DISCONNECTED

Websockets – and secure websockets – clearly are functioning in Caddy. There’s something odd going on with either the JS client or the backend.

1 Like

Thanks for your response. Websockets definitely work - I am reverse_proxy’ing a node-red installation which heavily relies on websockets and works flawlessly. I guess there is nothing I can do other than wait for it to be fixed in venueless.

1 Like

I’m realizing that venueless uses gunicorn under the hood, I think.

You could try adding buffer_requests and/or flush_interval -1 to your reverse_proxy; I’ve seen that gunicorn applications have had strange behaviour requiring these (they should not be required, but gunicorn does weird things…)

Note in the code:

2 Likes

Tried it with the following config:

        reverse_proxy 172.16.0.101:8002 {
                header_up X-Forwarded-Ssl on
                buffer_requests
                flush_interval -1
        }

And a combination of just buffer_requests and just flush_interval. Unfortunately that didn’t do the trick for me. Thanks for the idea, though!

Edit:
Indeed, pretty much looks like it’s using gunicorn:

1 Like

This topic was automatically closed after 30 days. New replies are no longer allowed.