Websockets over https

CarbonNanotubes · August 18, 2022, 1:40pm

1. Output of `caddy version`:

v2.4.6 h1:HGkGICFGvyrodcqOOclHKfvJC0qTU7vny/7FhYp9hNw=

2. How I run Caddy:

caddy:
image: lucaslorentz/caddy-docker-proxy:ci-alpine
ports:
- 443:443
- 80:80
- 6001:6001
environment:
- CADDY_INGRESS_NETWORKS=routable
networks:
- routable
volumes:
- /var/run/docker.sock:/var/run/docker.sock
- ./caddy_data:/data
- ./certs:/certs
restart: unless-stopped

a. System environment:

RockyLinux 8.6/Docker

b. Command:

docker-compose up -d

c. Service/unit/compose file:

version: "3.7"
services:
  caddy:
    image: lucaslorentz/caddy-docker-proxy:ci-alpine
    ports:
      - 443:443
      - 80:80
      - 6001:6001
    environment:
      - CADDY_INGRESS_NETWORKS=routable
    networks:
      - routable
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock
      - ./caddy_data:/data
      - ./certs:/certs
    restart: unless-stopped

networks:
  routable:
    external: true

d. My complete Caddy config:

dev.traefik.listech.on.ca:80 {
    reverse_proxy
}

http://gitlab.listech.on.ca {
    reverse_proxy http://172.17.63.3:80
    tls internal
}

staging.h1.listech.on.ca {
    reverse_proxy 172.17.63.5
    tls /certs/h1/certificate.crt /certs/h1/private.key
}

staging.h1.listech.on.ca:6001 {
    @ws {
    header Connection *Upgrade*
    header Upgrade websocket
}

reverse_proxy @ws https://172.17.63.2:6001
    tls /certs/h1/certificate.crt /certs/h1/private.key
}

3. The problem I’m having:

I am trying to serve my internal (no outside access) websocket server over tls. The caddy configuration was working fine without tls on it or the main container (:80). When my client attempts to connect to the server via port 6001 it gets a 502 Bad Gateway response. TLS is working for the main container on port 443.

4. Error messages and/or full log output:

Log was too long for this post, link to log on Pastebin

5. What I already tried:

Lots of googling. I’m pretty new to using caddy so I have not been too sure what to try. I have been trying to find any nugget of information that could lead me downt he right path to a solution

6. Links to relevant resources:

This is how I am using Caddy in docker

francislavoie · August 18, 2022, 2:06pm

I’m going to guess your issue is that you used https:// here, but your upstream app isn’t actually using HTTPS.

Why are you using a different port for websockets?

You can simplify your setup by multiplexing your app and websockets in the same site, using a matcher to determine if the request is WS or simply normal HTTP.

Try this:

staging.h1.listech.on.ca {
	tls /certs/h1/certificate.crt /certs/h1/private.key

	# Websockets, if headers match
	@ws {
		header Connection *Upgrade*
		header Upgrade websocket
	}
	reverse_proxy 172.17.63.2:6001
	
	# Main app
	reverse_proxy 172.17.63.5:80
}

Of course, you’ll need to update your frontend config to use wss://staging.h1.listech.on.ca to connect (remove the :6001 port from the connection address)

CarbonNanotubes · August 18, 2022, 3:59pm

So I made changes as you suggested but I’m still getting an error from caddy

{
    "level": "error",
    "ts": 1660835989.096334,
    "logger": "http.log.error",
    "msg": "EOF",
    "request": {
        "remote_addr": "10.2.32.207:58041",
        "proto": "HTTP/1.1",
        "method": "GET",
        "host": "staging.h1.listech.on.ca",
        "uri": "/",
        "headers": {
            "Upgrade": [
                "websocket"
            ],
            "Sec-Websocket-Key": [
                "qp/gAHEq0lmlghZMep6kdQ=="
            ],
            "Connection": [
                "Upgrade"
            ],
            "Origin": [
                "https://staging.h1.listech.on.ca"
            ],
            "Sec-Websocket-Version": [
                "13"
            ]
        },
        "tls": {
            "resumed": false,
            "version": 771,
            "cipher_suite": 49199,
            "proto": "",
            "proto_mutual": true,
            "server_name": "staging.h1.listech.on.ca"
        }
    },
    "duration": 0.001523141,
    "status": 502,
    "err_id": "d8uyaqa5j",
    "err_trace": "reverseproxy.statusError (reverseproxy.go:886)"
}

Here is my up to date caddyfile. I will note

{
    debug True
}

dev.traefik.listech.on.ca:80 {
    reverse_proxy
}

http://gitlab.listech.on.ca {
    reverse_proxy http://172.17.63.3:80
    tls internal
}

staging.h1.listech.on.ca {
    @ws {
        header Connection *Upgrade*
        header Upgrade websocket
    }

    reverse_proxy 172.17.63.5 172.17.63.4:6001
    tls /certs/h1/certificate.crt /certs/h1/private.key
}

matt · August 18, 2022, 5:00pm

This should just be debug

This reverse proxies to nothing, is that intended?

You can remove http:// here

Doesn’t make sense to use this for an http:// site.

This matcher is never used, so you can remove it.

This load balances between 172.17.63.5 and 172.17.63.4:6001 – is that intended? It’s basically a random selection.

CarbonNanotubes · August 18, 2022, 7:27pm

Not my intention, let me explain further what I’m doing. app is just a normal webserver and I also need a websockets server also available at the same url, I had this working before I attempted tls by using caddy to expose port 6001 but when I added tls to the app container I could no longer access my websocket server. At this point though as per the first responders suggestion removing the additional port and just routing the traffic based on whether it’s https or ws/wss.

I have these two containers that I want exposed via the staging.h1.listech.on.ca url.

app:
    image: lti-alpine:1.21.4-alpine
    build:
      context: ./dockerfiles
      dockerfile: nginx.dockerfile
      args:
        - UID=${UID:-1000}
        - GID=${GID:-1000}
    labels:
      caddy: "staging.h1.listech.on.ca"
      caddy.tls: /certs/h1/certificate.crt /certs/h1/private.key
      caddy.0_reverse_proxy: "{{upstreams}}"
    volumes:
      - ./cert:/cert
      - ./web:/var/www/html:delegated
    depends_on:
      - php
      - redis
      - mailhog
      - websockets
    networks:
      - h1project
      - routable

websockets:
    image: lti-php:8.1-fpm-alpine
    volumes:
      - ./web:/var/www/html:delegated
      - ./cert:/cert
    working_dir: /var/www/html
    expose: [6001]
    labels:
      caddy: staging.h1.listech.on.ca
      # caddy.tls: /certs/h1/certificate.crt /certs/h1/private.key
      caddy.@ws.0_header: Connection *Upgrade*
      caddy.@ws.1_header: Upgrade websocket
      caddy.1_reverse_proxy: "@ws {{upstreams 6001}}"
    entrypoint: ["php", "/var/www/html/artisan", "websockets:serve"]
    networks:
      - routable
      - h1project

I made some changes to my docker-compose file to fix issues with my in-memory caddyfile that the caddy docker image generates

{
    debug
}

http://gitlab.listech.on.ca:80 {
    reverse_proxy http://172.17.63.3:80
    tls internal
}

staging.h1.listech.on.ca {
    @ws {
        header Connection *Upgrade*
        header Upgrade websocket
    }
    reverse_proxy 172.17.63.5
    reverse_proxy @ws 172.17.63.4:6001
    tls /certs/h1/certificate.crt /certs/h1/private.key
}

francislavoie · August 18, 2022, 10:16pm

So is that working now? That looks good to me.

CarbonNanotubes · August 22, 2022, 3:52pm

Still wasn’t working but I was able to get it going, in a slightly different way. The docs for my websocket server were updated with some more information on using TLS.

I didn’t enable TLS on the websocket server but left it on caddy and proxied as needed, here is my updated caddy file. At this point

{
    debug
}

http://gitlab.listech.on.ca:80 {
    reverse_proxy http://172.17.63.3:80
    tls internal
}

staging.h1.listech.on.ca {
    @ws {
        header Connection *Upgrade*
        header Upgrade websocket
    }
    reverse_proxy 172.17.63.4
    reverse_proxy @ws http://172.17.63.2:6001
    tls /certs/h1/certificate.crt /certs/h1/private.key
}

system · September 17, 2022, 1:40pm

This topic was automatically closed after 30 days. New replies are no longer allowed.