Caddy 502 on reverse proxy to socketio - main app in same AKS cluster proxies

Alex_Alex · October 9, 2024, 9:53am

1. The problem I’m having:

Caddy able to proxy TLS to main service running in AKS pod but fails to connect to socket.io server within same pod. Main app is running on port 8000 and socketio is running on port 8009. Works locally.

2. Error messages and/or full log output:

{"level":"error","ts":1728467116.444659,"logger":"http.log.error.log0","msg":"EOF","request":{"remote_ip":"10.224.0.8","remote_port":"21681","proto":"HTTP/1.1","method":"GET","host":"api-dev.fulfilledwealth.co","uri":"/socket.io/?EIO=4&transport=websocket","headers":{"Cookie":[],"Upgrade":["websocket"],"Sec-Websocket-Key":["RkFtwjRxlxXOz4snB+6tGw=="],"Accept-Encoding":["gzip, deflate, br, zstd"],"Accept-Language":["en-CA,en-GB;q=0.9,en-US;q=0.8,en;q=0.7"],"Sec-Websocket-Extensions":["permessage-deflate; client_max_window_bits"],"Cache-Control":["no-cache"],"User-Agent":["Mozilla/5.0 (Linux; Android 6.0; Nexus 5 Build/MRA58N) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/129.0.0.0 Mobile Safari/537.36"],"Sec-Websocket-Version":["13"],"Connection":["Upgrade"],"Pragma":["no-cache"],"Origin":["https://dev.fulfilledwealth.co"]},"tls":{"resumed":true,"version":772,"cipher_suite":4865,"proto":"http/1.1","server_name":"api-dev.fulfilledwealth.co"}},"duration":1.003459221,"status":502,"err_id":"e6b2qzfjw","err_trace":"reverseproxy.statusError (reverseproxy.go:1272)"}
2024/10/09 09:45:16.444 ←[31mERROR←[0m  http.log.access.log0    handled request {"request": {"remote_ip": "10.224.0.8", "remote_port": "21681", "proto": "HTTP/1.1", "method": "GET", "host": "api-dev.fulfilledwealth.co", "uri": "/socket.io/?EIO=4&transport=websocket", "headers": {"Upgrade": ["websocket"], "Sec-Websocket-Key": ["RkFtwjRxlxXOz4snB+6tGw=="], "Cache-Control": ["no-cache"], "User-Agent": ["Mozilla/5.0 (Linux; Android 6.0; Nexus 5 Build/MRA58N) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/129.0.0.0 Mobile Safari/537.36"], "Accept-Encoding": ["gzip, deflate, br, zstd"], "Accept-Language": ["en-CA,en-GB;q=0.9,en-US;q=0.8,en;q=0.7"], "Sec-Websocket-Extensions": ["permessage-deflate; client_max_window_bits"], "Connection": ["Upgrade"], "Pragma": ["no-cache"], "Origin": ["https://dev.fulfilledwealth.co"], "Sec-Websocket-Version": ["13"], "Cookie": []}, "tls": {"resumed": true, "version": 772, "cipher_suite": 4865, "proto": "http/1.1", "server_name": "api-dev.fulfilledwealth.co"}}, "user_id": "", "duration": 1.003459221, "size": 0, "status": 502, "resp_headers": {"Server": ["Caddy"], "Alt-Svc": ["h3=\":443\"; ma=2592000"]}}
{"level":"debug","ts":1728467119.2724895,"logger":"http.stdlib","msg":"http: TLS handshake error from 10.224.0.4:22163: EOF"}

3. Caddy version:

4. How I installed and ran Caddy:

a. System environment:

AKS is using AMD64, application is nestjs.

b. Command:

I apply the ymls & assign a static_ip to the loadbalancer

c. Service/unit/compose file:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: caddy-proxy
  namespace: default
spec:
  replicas: 1
  selector:
    matchLabels:
      app: caddy-proxy
  template:
    metadata:
      labels:
        app: caddy-proxy
    spec:
      containers:
      - name: caddy
        image: caddy:2.6.2
        ports:
        - containerPort: 80
        - containerPort: 443
        - containerPort: 8001
        - containerPort: 8009
        volumeMounts:
        - name: caddy-config
          mountPath: /etc/caddy/Caddyfile
          subPath: Caddyfile
      volumes:
      - name: caddy-config
        configMap:
          name: caddy-config

apiVersion: v1
kind: Service
metadata:
  name: caddy-proxy-service
  namespace: default
  annotations:
    service.beta.kubernetes.io/azure-load-balancer-resource-group: fulfilledwealth-dev-rg
spec:
  type: LoadBalancer
  loadBalancerIP: STATIC_IP_PLACEHOLDER
  ports:
  - name: http
    port: 80
    targetPort: 80
  - name: https
    port: 443
    targetPort: 443
  - name: websocket
    port: 8001
    targetPort: 8001
  - name: socketio
    port: 8009
    targetPort: 8009
  selector:
    app: caddy-proxy

d. My complete Caddy config:

apiVersion: v1
kind: ConfigMap
metadata:
  name: caddy-config
  namespace: default
data:
  Caddyfile: |
    {
      email **
      acme_ca https://acme-v02.api.letsencrypt.org/directory
      debug
    }

    http://api-dev.fulfilledwealth.co {
      # Handle ACME HTTP-01 challenge
      handle /.well-known/acme-challenge/* {
        root * /var/lib/caddy
        file_server
      }

      # Redirect all other HTTP traffic to HTTPS
      redir https://{host}{uri} permanent
    }

    https://api-dev.fulfilledwealth.co {
      tls {
        protocols tls1.2 tls1.3
      }

      log {
        output stdout
        format console
        level DEBUG
      }
      
      # Socket.IO
      handle_path /socket.io/* {
        rewrite * /socket.io{path}
        reverse_proxy http://fulfilledwealth-dev-service:8009 {
          header Host {host}
          header X-Forwarded-For {remote}
          header X-Real-IP {remote}
          header Connection *Upgrade*
          header Connection *upgrade*
          header Upgrade websocket
      }

      # Main API (fallback)
      handle {
        reverse_proxy http://fulfilledwealth-dev-service:8000
      }
    }

5. Links to relevant resources:

Not seeing any failure logs in the server layer. Get the web socket failure in the client:

7555-816c37973b9984ed.js:1 WebSocket connection to ‘wss://api-dev.fulfilledwealth.co/socket.io/?EIO=4&transport=websocket’ failed:

francislavoie · October 9, 2024, 10:48am

Please upgrade to the latest version, v2.8.4.

That’s unnecessary, Caddy already handles ACME challenges automatically, it inserts a route to handle that path in front of your own config.

This is also not needed, Caddy already does HTTP->HTTPS redirects. You can just remove that entire site block, it does nothing.

Don’t re-state the defaults. If for example a tls1.4 was added later, then this config would prevent you from using the latest modern security protocols.

Alex_Alex:

          header Host {host}
          header X-Forwarded-For {remote}
          header X-Real-IP {remote}
          header Connection *Upgrade*
          header Connection *upgrade*
          header Upgrade websocket

Remove all this. Caddy already handles headers correctly by default. See reverse_proxy (Caddyfile directive) — Caddy Documentation

Don’t use handle_path if you’re just going to re-add the same path prefix with rewrite. Use handle instead, and delete the rewrite.

Your entire config can be simplified to just this:

{
	email <email>
	debug
}

api-dev.fulfilledwealth.co {
	log

	reverse_proxy /socket.io/* fulfilledwealth-dev-service:8009
	reverse_proxy fulfilledwealth-dev-service:8000
}

That’s it. The rest is unnecessary.

Alex_Alex · October 9, 2024, 4:41pm

Thanks for the reply, good to clean up the config (many were added trying to fix this issue), however this ends up with the same error unfortunately.

Seems like no other issue here?

system · November 8, 2024, 4:41pm

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.