1. The problem I’m having:
I have 3 services (plex, jellyfin and nextcloud) running in docker and using caddy as a reverse proxy. I’ve had this setup for a few months now and it worked fine until about ~2 weeks ago.
Now if I try any of them I just get a Connection timed out ( Error 552) -
curl vL:
* Host jelly.domain.xyz:80 was resolved.
* IPv6:2606:23232:446b, 2606:23232::ac43:c248
* IPv4: 104.99.99.107, 172.00.00.72
* Trying [2606:22222815:446b]:80...
* Connected to jelly.domain.xyz (2606:4700:2323:446b port 80
> GET / HTTP/1.1
> Host: jelly.domain.xyz
> User-Agent: curl/8.5.0
> Accept: */*
>
* Empty reply from server
* Closing connection
curl: (52) Empty reply from server
This is mostly the same response for all 3 of them, only difference is the IP based on if I have Cloudflare proxy ON or OFF
Caddy log shows no hits, so for some reason even if I can see the correct IP in the response it does not reach Caddy.
2. Error messages and/or full log output:
This is basically the only error I get, in a bowser.
Connection timed out Error code 522
Startup log:
Jan 07 17:18:47 HostName caddy[95377]: {"level":"info","ts":1704647927.622949,"logger":"http.log","msg":"server running","name":"srv0","protocols":["h1","h2","h3"]}
Jan 07 17:18:47 HostName caddy[95377]: {"level":"info","ts":1704647927.6231134,"logger":"http.log","msg":"server running","name":"remaining_auto_https_redirects","protocols":["h1","h2","h3"]}
Jan 07 17:18:47 HostName caddy[95377]: {"level":"info","ts":1704647927.6231434,"logger":"http","msg":"enabling automatic TLS certificate management","domains":["plex.domainname.xyz","cloud.domainname.xyz","jelly.domainname.xyz"]}
Jan 07 17:18:47 HostName caddy[95377]: {"level":"info","ts":1704647927.6232333,"logger":"http","msg":"servers shutting down with eternal grace period"}
Jan 07 17:18:47 HostName caddy[95377]: {"level":"info","ts":1704647927.628054,"msg":"autosaved config (load with --resume flag)","file":"/var/lib/caddy/.config/caddy/autosave.json"}
Jan 07 17:18:47 HostName caddy[95377]: {"level":"info","ts":1704647927.628848,"logger":"admin.api","msg":"load complete"}
Jan 07 17:18:47 HostName caddy[95377]: {"level":"info","ts":1704647927.6305325,"logger":"admin","msg":"stopped previous server","address":"localhost:2019"}
Jan 07 17:21:16 HostName caddy[95377]: {"level":"info","ts":1704648076.6227636,"logger":"admin.api","msg":"received request","method":"POST","host":"localhost:2019","uri":"/load","remote_ip":"127.0.0.1","remote_port":"38946","headers":{"Accept-Encoding":["gzip"],"Content-Length":["1696"],"Content-Type":["application/json"],"Origin":["http://localhost:2019"],"User-Agent":["Go-http-client/1.1"]}}
Jan 07 17:21:16 HostName caddy[95377]: {"level":"info","ts":1704648076.6237512,"msg":"config is unchanged"}
Jan 07 17:21:16 HostName caddy[95377]: {"level":"info","ts":1704648076.6239383,"logger":"admin.api","msg":"load complete"}
3. Caddy version:
v2.7.6 h1:w0NymbG2m9PcvKWsrXO6EEkY9Ru4FJK8uQbYcev1p3A=
4. How I installed and ran Caddy:
apt install caddy
a. System environment:
Running on Raspberry Pi 4
Operating System: Debian GNU/Linux 12 (bookworm)
Kernel: Linux 6.1.0-rpi7-rpi-v8
Architecture: arm64
b. Command:
caddy start / sudo systemctl start caddy
c. Service/unit/compose file:
[Unit]
Description=Caddy
Documentation=https://caddyserver.com/docs/
After=network.target network-online.target
Requires=network-online.target
[Service]
Type=notify
User=caddy
Group=caddy
ExecStart=/usr/bin/caddy run --environ --config /etc/caddy/Caddyfile
ExecReload=/usr/bin/caddy reload --config /etc/caddy/Caddyfile --force
TimeoutStopSec=5s
LimitNOFILE=1048576
LimitNPROC=512
PrivateTmp=true
ProtectSystem=full
AmbientCapabilities=CAP_NET_ADMIN CAP_NET_BIND_SERVICE
[Install]
WantedBy=multi-user.target
d. My complete Caddy config:
https://cloud.domain.xyz {
log {
level Debug
}
header Strict-Transport-Security max-age=31536000;
reverse_proxy localhost:11443
tls {
dns cloudflare cloudflare-key
resolvers 1.1.1.1
}
}
https://plex.domain.xyz {
log {
level Debug
}
header Strict-Transport-Security max-age=31536000;
reverse_proxy localhost:32400
tls {
dns cloudflare cloudflare-key
resolvers 1.1.1.1
}
}
https://jelly.domain.xyz {
header Strict-Transport-Security max-age=31536000;
reverse_proxy localhost:8096
tls {
dns cloudflare cloudflare-key
resolvers 1.1.1.1
}
}
Some extra stuff:
All 3 services(plex, nextcloud and jellyfin) run in docker, while caddy is run a normal systemd service.
First time I noticed they were down the issue was that ddclient was trying to set ipv6 as a A record on cloudflare. So I assume I received an ipv6 from network provider. Once I fixed that everything was back to normal for a few days.
IP Tables:
ACCEPT tcp -- anywhere 172.25.0.2 tcp dpt:19080
ACCEPT tcp -- anywhere 172.28.0.2 tcp dpt:https
ACCEPT tcp -- anywhere 172.25.0.2 tcp dpt:9117
ACCEPT tcp -- anywhere 172.28.0.2 tcp dpt:http
ACCEPT tcp -- anywhere 172.25.0.2 tcp dpt:8888
ACCEPT tcp -- anywhere 192.168.224.2 tcp dpt:3001
ACCEPT tcp -- anywhere 172.25.0.2 tcp dpt:8388
ACCEPT tcp -- anywhere 172.30.0.2 tcp dpt:5230
ACCEPT tcp -- anywhere 172.26.0.2 tcp dpt:9443
ACCEPT tcp -- anywhere 192.168.192.2 tcp dpt:7575
ACCEPT udp -- anywhere 172.25.0.2 udp dpt:8388
ACCEPT tcp -- anywhere 172.25.0.2 tcp dpt:6881
ACCEPT udp -- anywhere 172.25.0.2 udp dpt:6881
ACCEPT tcp -- anywhere 192.168.144.2 tcp dpt:2468
ACCEPT tcp -- anywhere 172.22.0.7 tcp dpt:11443
ACCEPT tcp -- anywhere 172.19.0.2 tcp dpt:http-alt
Ports 443 and 80 forwarded in router and UPnP enabled (these have not been changed in a long time, long before using caddy and everything worked as intended)
sudo lsof -nP -iTCP -sTCP:LISTEN:
docker-pr 2207 root 4u IPv4 22603 0t0 TCP *:19080 (LISTEN)
docker-pr 2212 root 4u IPv4 20915 0t0 TCP *:4443 (LISTEN)
docker-pr 2236 root 4u IPv6 21752 0t0 TCP *:19080 (LISTEN)
docker-pr 2240 root 4u IPv6 22612 0t0 TCP *:4443 (LISTEN)
docker-pr 2287 root 4u IPv4 20000 0t0 TCP *:9117 (LISTEN)
docker-pr 2289 root 4u IPv4 22634 0t0 TCP *:4080 (LISTEN)
docker-pr 2295 root 4u IPv6 20003 0t0 TCP *:9117 (LISTEN)
docker-pr 2305 root 4u IPv6 20943 0t0 TCP *:4080 (LISTEN)
docker-pr 2324 root 4u IPv4 21801 0t0 TCP *:8888 (LISTEN)
docker-pr 2332 root 4u IPv6 21804 0t0 TCP *:8888 (LISTEN)
docker-pr 2355 root 4u IPv4 20972 0t0 TCP *:9081 (LISTEN)
docker-pr 2366 root 4u IPv6 22658 0t0 TCP *:9081 (LISTEN)
docker-pr 2382 root 4u IPv4 21846 0t0 TCP *:8388 (LISTEN)
docker-pr 2384 root 4u IPv4 20994 0t0 TCP *:5230 (LISTEN)
docker-pr 2399 root 4u IPv6 22682 0t0 TCP *:8388 (LISTEN)
docker-pr 2406 root 4u IPv6 22687 0t0 TCP *:5230 (LISTEN)
docker-pr 2423 root 4u IPv4 22696 0t0 TCP *:9443 (LISTEN)
docker-pr 2442 root 4u IPv4 21867 0t0 TCP *:7575 (LISTEN)
docker-pr 2456 root 4u IPv6 21008 0t0 TCP *:9443 (LISTEN)
docker-pr 2459 root 4u IPv6 20082 0t0 TCP *:7575 (LISTEN)
docker-pr 2507 root 4u IPv4 22731 0t0 TCP *:6881 (LISTEN)
docker-pr 2513 root 4u IPv6 21916 0t0 TCP *:6881 (LISTEN)
jellyfin 3126 justme 313u IPv4 34989 0t0 TCP *:8096 (LISTEN)
Plex\x20M 4141 justme 10u IPv6 27274 0t0 TCP *:32400 (LISTEN)
Plex\x20M 4141 justme 11u IPv4 27276 0t0 TCP 127.0.0.1:32401 (LISTEN)
Plex\x20S 5093 justme 4u IPv4 32837 0t0 TCP 127.0.0.1:41451 (LISTEN)
docker-pr 5498 root 4u IPv4 30259 0t0 TCP *:2468 (LISTEN)
docker-pr 5505 root 4u IPv6 29378 0t0 TCP *:2468 (LISTEN)
Plex\x20T 5929 justme 10u IPv4 32987 0t0 TCP 127.0.0.1:32600 (LISTEN)
docker-pr 77973 root 4u IPv4 302746 0t0 TCP 127.0.0.1:11443 (LISTEN)
docker-pr 87261 root 4u IPv4 337110 0t0 TCP *:8080 (LISTEN)
docker-pr 87271 root 4u IPv6 338007 0t0 TCP *:8080 (LISTEN)
caddy 95377 caddy 3u IPv6 409135 0t0 TCP *:443 (LISTEN)
caddy 95377 caddy 11u IPv4 409134 0t0 TCP 127.0.0.1:2019 (LISTEN)
caddy 95377 caddy 12u IPv6 409136 0t0 TCP *:80 (LISTEN)