Intermittent connection resets on config reloads

Help
1

1. The problem I’m having:

I’m seeing connection failures during TLS establishment when Caddy reloads its configuration.

Is this behavior:

  1. Expected / a technical constraint in Caddy’s architecture
  2. A bug in Caddy
  3. Something that can be fixed with configuration

Detailed Description

I’m using Caddy Docker Proxy for several services in Docker Swarm. I noticed one service was repeatedly restarting every few seconds and indirectly affecting the availability of my other services. Each restart removes or changes its upstream task IP, so Docker-proxy regenerates the configuration and reloads Caddy which I think is causing the issue,

Note: I know I could just fix the crashlooping service and this would probably end up being not too noticeable in the grand scheme of things, but I see it as a good opportunity to expose a failure mode that may normally happen so infrequently that it could never be reproduced.

I observed the impact on blobse.us.to (GitHub, if curious), whose upstream stayed at 10.0.6.19:7330. I ran an external monitor that made repeated GET requests using curl. Over nine minutes I saw:

  • 17/1,000 requests failed, curl exit 35
    • Failed after sending ClientHello but before receiving ServerHello or sending HTTP.
  • Each failure overlapped with a Caddy reload. There were a total of 62 reloads during the run.
  • Failed requests lasted ~300 ms; they reset rather than timed out
  • Immediately after the failed requests, they succeed.

Is this behavior is expected and how I can prevent these failures so reloads don’t trigger connection resets impacting all my other services when one service restarts?

2. Error messages and/or full log output:

Caddy/Docker-proxy logs:

2026-09-30T09:08:59.218935968Z {"level":"warn","ts":1790759339.2169304,"logger":"docker-proxy","msg":"Service has no tasks in running state","service":"machazat-deployments_voxellum-backend-app","serviceId":"n5nv11pwbhuvws1aqs825602q"}
2026-09-30T09:08:59.238970640Z {"level":"info","ts":1790759339.237504,"logger":"docker-proxy","msg":"Sending configuration to","server":"localhost"}
2026-09-30T09:08:59.238975900Z {"level":"info","ts":1790759339.2385414,"logger":"admin.api","msg":"received request","method":"POST","host":"localhost:2019","uri":"/load","remote_ip":"127.0.0.1","remote_port":"38382","headers":{"Accept-Encoding":["gzip"],"Content-Length":["2320"],"Content-Type":["application/json"],"User-Agent":["Go-http-client/1.1"]}}
2026-09-30T09:08:59.260123875Z {"level":"info","ts":1790759339.243778,"logger":"http","msg":"servers shutting down with eternal grace period"}
2026-09-30T09:08:59.260128757Z {"level":"info","ts":1790759339.245707,"logger":"admin.api","msg":"load complete"}

My client’s complete failed curl trace, followed by the monitor’s retry result:

sample=979 attempt=primary started=2026-09-30T09:08:59.056309906Z ended=2026-09-30T09:08:59.375143406Z curl_exit=35 outcome=transport_error
09:08:59.073557 * Host blobse.us.to:443 was resolved.
09:08:59.073631 * IPv6: (none)
09:08:59.073656 * IPv4: 153.75.235.115
09:08:59.073719 *   Trying 153.75.235.115:443...
09:08:59.224378 * ALPN: curl offers h2,http/1.1
09:08:59.224770 } [5 bytes data]
09:08:59.224825 * TLSv1.3 (OUT), TLS handshake, Client hello (1):
09:08:59.224850 } [1571 bytes data]
09:08:59.224944 * SSL Trust Anchors:
09:08:59.232172 *   CAfile: /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem
09:08:59.369959 * Recv failure: Connection reset by peer
09:08:59.370046 * TLS connect error: error:00000000:lib(0)::reason(0)
09:08:59.370084 * OpenSSL SSL_connect: Connection reset by peer in connection to blobse.us.to:443 
09:08:59.370125 * closing connection #0
curl: (35) Recv failure: Connection reset by peer
sample=979 attempt=retry started=2026-09-30T09:08:59.382226861Z ended=2026-09-30T09:08:59.872977261Z curl_exit=0 outcome=ok
http=200

3. Caddy version:

Currently using 2.11.4. I also was previously on 2.8.4 where I observed the exact same behavior.

4. How I installed and ran Caddy:

I deployed lucaslorentz/caddy-docker-proxy as a Docker Swarm service with one replica. I publish HTTPS in host mode and use service labels to route requests to upstream task IPs on an overlay network.

a. System environment:

  • Server: Alpine Linux, kernel 6.18.53-0-virt, Docker Engine 29.5.3.
  • Client: Linux, curl 8.18.0, OpenSSL 3.5.8; approximately 148 ms TCP round-trip time.

b. Command:

Swarm starts this process in the container:

/bin/caddy docker-proxy

c. Service/unit/compose file:

I have the relevant Swarm settings below as a Compose excerpt, using the image I ran during the measurements. I have not deployed this standalone using this: I configure routing through labels on the other services. I’ve included the complete generated Caddyfile in section d.

services:
  caddy:
    image: lucaslorentz/caddy-docker-proxy:2.9.1-alpine
    environment:
      CADDY_INGRESS_NETWORKS: machazat-network
    networks:
      - machazat-network
    ports:
      - target: 443
        published: 443
        protocol: tcp
        mode: host
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock
      - caddy-data-volume:/data
      - caddy-config-volume:/config
      - /opt/crashloop-service-frontend:/opt/crashloop-service-frontend
      - /opt/other-service-3-website:/opt/other-service-3-website
    deploy:
      replicas: 1

networks:
  machazat-network:
    external: true

volumes:
  caddy-data-volume:
  caddy-config-volume:

I set these routing labels under deploy.labels on the service I probed:

caddy: https://blobse.us.to
caddy.header: "/* { -Server }"
caddy.reverse_proxy: "{{upstreams 7330}}"

d. My complete Caddy config:

Docker-proxy generated this Caddyfile during the failure above. It left reverse_proxy empty for the restarting service because that service had no running tasks; I haven’t omitted an upstream here.

app.other-service-1.hs.vc {
	header /* -Server
	reverse_proxy 10.0.6.55:8080
}
other-service-2.hs.vc {
	header /* -Server
	reverse_proxy 10.0.6.10:80
}
https://api.crashloop-service.com {
	header /* -Server
	reverse_proxy
}
https://blobse.us.to {
	header /* -Server
	reverse_proxy 10.0.6.19:7330
}
https://other-service-3 {
	file_server
	root * /opt/other-service-3-website
	try_files {path} /index.html
}
https://crashloop-service.com {
	file_server
	root * /opt/crashloop-service-frontend
	try_files {path} /index.html
}
other-service-4.hs.vc {
	header /* -Server
	reverse_proxy 10.0.6.9:80
}

5. Links to relevant resources:

2

Your log shows each /load replacing the running servers:

"msg":"servers shutting down with eternal grace period"

Reloads are meant to be graceful (the listening socket is kept across config loads), so a reset between ClientHello and ServerHello is worth opening as an issue on the Caddy GitHub repo. Your curl timing and the 17/62 correlation make a good repro. I’m not aware of a config option that changes it.

What you can fix is how often it reloads. The config only changes because {{upstreams 7330}} renders task IPs, so every crash rewrites the Caddyfile. Let Caddy resolve Swarm’s DNS at request time instead:

caddy.reverse_proxy.dynamic: "a tasks.<service_name> <port>"

which generates:

reverse_proxy {
	dynamic a tasks.<service_name> <port>
}

Now the generated config stays identical when tasks come and go, and Caddy skips a /load whose config is unchanged. While the service has no tasks, only its own site returns 502 instead of every site going through a reload. For services where you don’t need per-task balancing, reverse_proxy <service_name>:<port> (the service VIP) is the simpler static option.