1. The problem I’m having:
I’m trying to use Caddy to act as both a reverse proxy and load balancer for some sites with TLS. Right now I’m just using the containous/whoami container on two different hosts to test and running into problems with the load balanced entry, but the non load balanced entries work fine.
The issue is very clearly with my load balanced entry, but I’m not sure exactly why.
2. Error messages and/or full log output:
ERR ts=1684189504.9152136 logger=http.log.error.log0 msg=tls: first record does not look like a TLS handshake request={"remote_ip":"10.13.37.38","remote_port":"46772","proto":"HTTP/2.0","method":"GET","host":"lb.foreveratroll.com","uri":"/","headers":{"Upgrade-Insecure-Requests":["1"],"Sec-Fetch-Dest":["document"],"Sec-Fetch-Site":["none"],"Sec-Fetch-User":["?1"],"Te":["trailers"],"User-Agent":["Mozilla/5.0 (X11; Linux x86_64; rv:109.0) Gecko/20100101 Firefox/113.0"],"Accept":["text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,*/*;q=0.8"],"Accept-Language":["en-US,en;q=0.5"],"Dnt":["1"],"Accept-Encoding":["gzip, deflate, br"],"Sec-Fetch-Mode":["navigate"],"Sec-Gpc":["1"]},"tls":{"resumed":false,"version":772,"cipher_suite":4867,"proto":"h2","server_name":"lb.foreveratroll.com"}} duration=1.046067821 status=502 err_id=b9e66z5ij err_trace=reverseproxy.statusError (reverseproxy.go:1299)
2023/05/15 22:25:04.915 ERROR http.log.access.log0 handled request {"request": {"remote_ip": "10.13.37.38", "remote_port": "46772", "proto": "HTTP/2.0", "method": "GET", "host": "lb.foreveratroll.com", "uri": "/", "headers": {"Accept-Encoding": ["gzip, deflate, br"], "Sec-Fetch-Mode": ["navigate"], "Sec-Gpc": ["1"], "Dnt": ["1"], "Sec-Fetch-Site": ["none"], "Sec-Fetch-User": ["?1"], "Te": ["trailers"], "User-Agent": ["Mozilla/5.0 (X11; Linux x86_64; rv:109.0) Gecko/20100101 Firefox/113.0"], "Accept": ["text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,*/*;q=0.8"], "Accept-Language": ["en-US,en;q=0.5"], "Upgrade-Insecure-Requests": ["1"], "Sec-Fetch-Dest": ["document"]}, "tls": {"resumed": false, "version": 772, "cipher_suite": 4867, "proto": "h2", "server_name": "lb.foreveratroll.com"}}, "user_id": "", "duration": 1.046067821, "size": 0, "status": 502, "resp_headers": {"Server": ["Caddy"], "Alt-Svc": ["h3=\":443\"; ma=2592000"]}}
3. Caddy version:
2.6.4
4. How I installed and ran Caddy:
Custom docker image hosted on an internal container registry.
Dockerfile used to build it:
FROM caddy:2.6.4-builder-alpine AS builder
RUN xcaddy build v2.6.4 --with github.com/caddy-dns/cloudflare@latest
FROM caddy:2.6.4
COPY --from=builder /usr/bin/caddy /usr/bin/caddy
a. System environment:
Debian 11 host
Docker version 23.0.6, build ef23cbc
b. Command:
Container is deployed as a Portainer stack with a cloudflare API token for TLS. It monitors an internal git repo. My Caddy file path is where Portainer sticks the repo, it exists in the top level of the project and works fine when I push to the monitored branch and update from Portainer.
c. Service/unit/compose file:
version: "3.7"
services:
caddy:
container_name: caddy
hostname: caddy
image: flcr.foreveratroll.com/caddy:2.6.4
restart: unless-stopped
environment:
- CF_API_TOKEN=${CF_API_TOKEN}
ports:
- "80:80"
- "443:443"
- "443:443/udp"
volumes:
- /opt/docker/portainer/data/compose/15/Caddyfile:/etc/caddy/Caddyfile
- /opt/docker/caddy/data/site:/srv
- /opt/docker/caddy/data/caddy-data:/data
- /opt/docker/caddy/data/config:/config
volumes:
caddy_data:
external: true
caddy_config:
d. My complete Caddy config:
lb.foreveratroll.com {
tls {
dns cloudflare "{env.CF_API_TOKEN}"
}
log {
output stdout
format console
}
respond /health 200
reverse_proxy / fish.foreveratroll.com:32768 tiger.foreveratroll.com:32768 {
lb_policy round_robin
lb_try_duration 1s
lb_try_interval 250ms
transport http {
tls
}
}
}
lb1.foreveratroll.com {
reverse_proxy fish.foreveratroll.com:32768
tls {
dns cloudflare "{env.CF_API_TOKEN}"
}
}
lb2.foreveratroll.com {
reverse_proxy tiger.foreveratroll.com:32768
tls {
dns cloudflare "{env.CF_API_TOKEN}"
}
}
5. Links to relevant resources:
https://caddy.community/t/v2-load-balancer-example-with-caddyfile/6903/5