Mattermost with Caddy reverse_proxy Websocket error - 403 Forbidden

[chrisd]

1. The problem I’m having:

Mattermost websocket error with Caddy reverse proxy. Initial connection in Firefox works fine and then after about 10 seconds the error occurs. Same error with other browsers. The error message on the website is:

Please check connection, Mattermost unreachable. If issue persists, ask administrator to check WebSocket port.

Accessing Mattermost running on EC2 Amazon Linux 2023 instance at:
http://localhost:8065

with Caddy reverse proxy returns websocket error.

Server is:
https://chat.inseytel.com.

Port 443 is open but not Port 80 on EC2 instance.

Mattermost provide Nginx reverse proxy (at the bottom) and from my research the the upgrade option is required. I understood that in Caddy-v2, websockets just worked. Do I need to add anything to my Caddyfile to make this work?

Thanks

2. Error messages and/or full log output:

Error log from Firefox:

>> GET wss://chat.inseytel.com/api/v4/websocket?connection_id=&sequence_number=0&posted_ack=true

Status 403 Forbidden
VersionHTTP/1.1
Transferred432 B (0 B size)
DNS ResolutionDNS over HTTPS

Firefox can’t establish a connection to the server at wss://chat.inseytel.com/api/v4/websocket?connection_id=&sequence_number=0&posted_ack=true.

Firefox debugger highlights:

websocket.js:

  if (this.connectFailCount === 0) {
            console.log('websocket connecting to ' + connectionUrl); //eslint-disable-line no-console
        }

3. Caddy version:

# caddy version
v2.8.1 h1:UVWB6J5f/GwHPyvdTrm0uM7YhfaWb4Ztdrp/z6ROHsM=`

4. How I installed and ran Caddy:

a. System environment:

Added the Red Hat repo to EC2 and used # dnf install caddy

c. Service/unit/compose file:

I use systemd to run Caddy
# cat /usr/lib/systemd/system/caddy.service

[Unit]
Description=Caddy
Documentation=https://caddyserver.com/docs/
After=network.target network-online.target
Requires=network-online.target

[Service]
Type=notify
User=caddy
Group=caddy
ExecStart=/usr/bin/caddy run --environ --config /etc/caddy/Caddyfile
ExecReload=/usr/bin/caddy reload --config /etc/caddy/Caddyfile --force
TimeoutStopSec=5s
LimitNOFILE=1048576
PrivateTmp=true
ProtectSystem=full
AmbientCapabilities=CAP_NET_ADMIN CAP_NET_BIND_SERVICE

[Install]
WantedBy=multi-user.target

d. My complete Caddy config:

cat /etc/caddy/Caddyfile

{
email	username@inseytel.com
}

chat.inseytel.com { 
	reverse_proxy localhost:8065
}

The NGINX reverse proxy from Mattermost is (docker/docker-compose.nginx.yml at main · mattermost/docker · GitHub):

version: "2.4"

services:
  nginx:
    depends_on:
      - mattermost
    container_name: nginx_mattermost
    image: nginx:${NGINX_IMAGE_TAG}
    restart: ${RESTART_POLICY}
    security_opt:
      - no-new-privileges:true
    pids_limit: 100
    read_only: true
    tmpfs:
      - /var/run
      - /var/cache
      - /var/log/nginx
    volumes:
      - ${NGINX_CONFIG_PATH}:/etc/nginx/conf.d:ro
      - ${NGINX_DHPARAMS_FILE}:/dhparams4096.pem
      - ${CERT_PATH}:/cert.pem:ro
      - ${KEY_PATH}:/key.pem:ro
      - shared-webroot:/usr/share/nginx/html
    environment:
      # timezone inside container
      - TZ
    ports:
      - ${HTTPS_PORT}:443
      - ${HTTP_PORT}:80
  mattermost:
    ports:
      - ${CALLS_PORT}:${CALLS_PORT}/udp
      - ${CALLS_PORT}:${CALLS_PORT}/tcp

[TheRettom (Riley M)]

I don’t know anything about Mattermost, but wanted to try to help anyway. Can you upload Caddy’s log as well?

I’d start with ensuring that Mattermost allows WebSocket connections from the specified domain, and that firewalls on the server and client are allowing WebSocket traffic.