Load balanced reverse proxy with TLS

1. The problem I’m having:

I’m trying to use Caddy to act as both a reverse proxy and load balancer for some sites with TLS. Right now I’m just using the containous/whoami container on two different hosts to test and running into problems with the load balanced entry, but the non load balanced entries work fine.

The issue is very clearly with my load balanced entry, but I’m not sure exactly why.

2. Error messages and/or full log output:

ERR ts=1684189504.9152136 logger=http.log.error.log0 msg=tls: first record does not look like a TLS handshake request={"remote_ip":"","remote_port":"46772","proto":"HTTP/2.0","method":"GET","host":"lb.foreveratroll.com","uri":"/","headers":{"Upgrade-Insecure-Requests":["1"],"Sec-Fetch-Dest":["document"],"Sec-Fetch-Site":["none"],"Sec-Fetch-User":["?1"],"Te":["trailers"],"User-Agent":["Mozilla/5.0 (X11; Linux x86_64; rv:109.0) Gecko/20100101 Firefox/113.0"],"Accept":["text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,*/*;q=0.8"],"Accept-Language":["en-US,en;q=0.5"],"Dnt":["1"],"Accept-Encoding":["gzip, deflate, br"],"Sec-Fetch-Mode":["navigate"],"Sec-Gpc":["1"]},"tls":{"resumed":false,"version":772,"cipher_suite":4867,"proto":"h2","server_name":"lb.foreveratroll.com"}} duration=1.046067821 status=502 err_id=b9e66z5ij err_trace=reverseproxy.statusError (reverseproxy.go:1299)

2023/05/15 22:25:04.915	ERROR	http.log.access.log0	handled request	{"request": {"remote_ip": "", "remote_port": "46772", "proto": "HTTP/2.0", "method": "GET", "host": "lb.foreveratroll.com", "uri": "/", "headers": {"Accept-Encoding": ["gzip, deflate, br"], "Sec-Fetch-Mode": ["navigate"], "Sec-Gpc": ["1"], "Dnt": ["1"], "Sec-Fetch-Site": ["none"], "Sec-Fetch-User": ["?1"], "Te": ["trailers"], "User-Agent": ["Mozilla/5.0 (X11; Linux x86_64; rv:109.0) Gecko/20100101 Firefox/113.0"], "Accept": ["text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,*/*;q=0.8"], "Accept-Language": ["en-US,en;q=0.5"], "Upgrade-Insecure-Requests": ["1"], "Sec-Fetch-Dest": ["document"]}, "tls": {"resumed": false, "version": 772, "cipher_suite": 4867, "proto": "h2", "server_name": "lb.foreveratroll.com"}}, "user_id": "", "duration": 1.046067821, "size": 0, "status": 502, "resp_headers": {"Server": ["Caddy"], "Alt-Svc": ["h3=\":443\"; ma=2592000"]}}

3. Caddy version:


4. How I installed and ran Caddy:

Custom docker image hosted on an internal container registry.
Dockerfile used to build it:

FROM caddy:2.6.4-builder-alpine AS builder

RUN xcaddy build v2.6.4 --with github.com/caddy-dns/cloudflare@latest

FROM caddy:2.6.4

COPY --from=builder /usr/bin/caddy /usr/bin/caddy

a. System environment:

Debian 11 host
Docker version 23.0.6, build ef23cbc

b. Command:

Container is deployed as a Portainer stack with a cloudflare API token for TLS. It monitors an internal git repo. My Caddy file path is where Portainer sticks the repo, it exists in the top level of the project and works fine when I push to the monitored branch and update from Portainer.

c. Service/unit/compose file:

version: "3.7"

    container_name: caddy
    hostname: caddy
    image: flcr.foreveratroll.com/caddy:2.6.4
    restart: unless-stopped
      - "80:80"
      - "443:443"
      - "443:443/udp"
      - /opt/docker/portainer/data/compose/15/Caddyfile:/etc/caddy/Caddyfile
      - /opt/docker/caddy/data/site:/srv
      - /opt/docker/caddy/data/caddy-data:/data
      - /opt/docker/caddy/data/config:/config

    external: true

d. My complete Caddy config:

lb.foreveratroll.com {
    tls {
        dns cloudflare "{env.CF_API_TOKEN}"
    log {
        output stdout
        format console
    respond /health 200
    reverse_proxy / fish.foreveratroll.com:32768 tiger.foreveratroll.com:32768 {
        lb_policy round_robin
        lb_try_duration 1s
        lb_try_interval 250ms

        transport http { 

lb1.foreveratroll.com {
    reverse_proxy fish.foreveratroll.com:32768
    tls {
        dns cloudflare "{env.CF_API_TOKEN}"

lb2.foreveratroll.com {
    reverse_proxy tiger.foreveratroll.com:32768
    tls {
        dns cloudflare "{env.CF_API_TOKEN}"

5. Links to relevant resources:


I figured it out, this portion of my Caddyfile was the culprit. Removing that and leaving everything else the same solved it.

You have a / matcher here. This means only requests to exactly / would be proxied. Make sure to remove that to allow all requests to be proxied.

Good looking out. That was one of the many things I had tried as part of getting this up at the most basic level. I’ll pull that out and build from here. Thanks!

1 Like