1. The problem I’m having:
I have a setup with two servers, the first running Caddy and the second running tomcat and nginx.
The Caddyfile contains two reverse proxy routes both leading to the second server, one to tomcat (running as docker container) and second to nginx (running as docker container).
Approximately once a week, I am facing the following situation:
- Both tomcat and nginx are running at the second server.
- Reverse proxy route to nginx is working.
- Problem: Reverse proxy route to tomcat stops working and is returning http status 0 and response of size 0 to all requests.
After restarting Caddy, the route to tomcat starts to work again.
The Caddy server is up and running but one route stops forwarding the requests.
What can be the reason for this behaviour?
How can I get more detailed logs to resolve the situation?
2. Error messages and/or full log output:
I have not found anything exceptional, journalctl contains just the log of manual restart.
May 02 20:01:37 gw systemd[1]: Stopped Caddy.
May 02 20:01:37 gw systemd[1]: caddy.service: Consumed 40min 13.067s CPU time.
May 02 20:01:40 gw systemd[1]: Starting Caddy...
May 02 20:01:40 gw caddy[2584635]: caddy.HomeDir=/var/lib/caddy
May 02 20:01:40 gw caddy[2584635]: caddy.AppDataDir=/var/lib/caddy/.local/share/caddy
May 02 20:01:40 gw caddy[2584635]: caddy.AppConfigDir=/var/lib/caddy/.config/caddy
May 02 20:01:40 gw caddy[2584635]: caddy.ConfigAutosavePath=/var/lib/caddy/.config/caddy/autosave.json
May 02 20:01:40 gw caddy[2584635]: caddy.Version=v2.6.4 h1:2hwYqiRwk1tf3VruhMpLcYTg+11fCdr8S3jhNAdnPy8=
May 02 20:01:40 gw caddy[2584635]: runtime.GOOS=linux
May 02 20:01:40 gw caddy[2584635]: runtime.GOARCH=amd64
May 02 20:01:40 gw caddy[2584635]: runtime.Compiler=gc
May 02 20:01:40 gw caddy[2584635]: runtime.NumCPU=3
May 02 20:01:40 gw caddy[2584635]: runtime.GOMAXPROCS=3
May 02 20:01:40 gw caddy[2584635]: runtime.Version=go1.20
May 02 20:01:40 gw caddy[2584635]: os.Getwd=/
May 02 20:01:40 gw caddy[2584635]: LANG=en_US.UTF-8
May 02 20:01:40 gw caddy[2584635]: PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
May 02 20:01:40 gw caddy[2584635]: NOTIFY_SOCKET=/run/systemd/notify
May 02 20:01:40 gw caddy[2584635]: HOME=/var/lib/caddy
May 02 20:01:40 gw caddy[2584635]: LOGNAME=caddy
May 02 20:01:40 gw caddy[2584635]: USER=caddy
May 02 20:01:40 gw caddy[2584635]: INVOCATION_ID=36c5b39c13f44298862cf4c83163ef86
May 02 20:01:40 gw caddy[2584635]: JOURNAL_STREAM=8:26524555
May 02 20:01:40 gw caddy[2584635]: {"level":"info","ts":1683057700.3525376,"msg":"using provided configuration","config_file":"/etc/caddy/Caddyfile","config_adapter":""}
May 02 20:01:40 gw caddy[2584635]: {"level":"info","ts":1683057700.3541627,"msg":"redirected default logger","from":"stderr","to":"/var/log/caddy/default.log"}
May 02 20:01:40 gw systemd[1]: Started Caddy.
3. Caddy version:
v2.6.4 h1:2hwYqiRwk1tf3VruhMpLcYTg+11fCdr8S3jhNAdnPy8=
4. How I installed and ran Caddy:
sudo apt install caddy
a. System environment:
Debian GNU/Linux 11 (bullseye)
Linux gw 5.10.0-21-amd64 #1 SMP Debian 5.10.162-1 (2023-01-21) x86_64 GNU/Linux
b. Command:
no command
c. Service/unit/compose file:
[Unit]
Description=Caddy
Documentation=https://caddyserver.com/docs/
After=network.target network-online.target
Requires=network-online.target
[Service]
Type=notify
User=caddy
Group=caddy
ExecStart=/usr/bin/caddy run --environ --config /etc/caddy/Caddyfile
ExecReload=/usr/bin/caddy reload --config /etc/caddy/Caddyfile --force
TimeoutStopSec=5s
LimitNOFILE=1048576
LimitNPROC=512
PrivateDevices=yes
PrivateTmp=true
ProtectSystem=full
AmbientCapabilities=CAP_NET_BIND_SERVICE
[Install]
WantedBy=multi-user.target
d. My complete Caddy config:
## first route to tomcat
tomcat.mycompany.net {
reverse_proxy https://second.net
log {
format json
output file /var/log/caddy/tomcat.log {
roll_size 1gb
roll_keep 30
roll_local_time
}
}
}
## second route to nginx
nginx.mycompany.net {
reverse_proxy http://second.net:9999
log {
format json
output file /var/log/caddy/nginx.log
}
}
5. Links to relevant resources:
no resources