Caddy stops serving webpage after a few days

Help
1

1. Output of caddy version:

v2.6.2 h1:wKoFIxpmOJLGl3QXoo6PNbYvGW4xLEgo32GPBEjWL8o=

2. How I run Caddy:

a. System environment:

Ubuntu 22.04.1 LTS - 5.15.0-56-generic

b. Command:

caddy start

c. Service/unit/compose file:

# caddy.service
#
# For using Caddy with a config file.
#
# Make sure the ExecStart and ExecReload commands are correct
# for your installation.
#
# See https://caddyserver.com/docs/install for instructions.
#
# WARNING: This service does not use the --resume flag, so if you
# use the API to make changes, they will be overwritten by the
# Caddyfile next time the service is restarted. If you intend to
# use Caddy's API to configure it, add the --resume flag to the
# `caddy run` command or use the caddy-api.service file instead.

[Unit]
Description=Caddy
Documentation=https://caddyserver.com/docs/
After=network.target network-online.target
Requires=network-online.target

[Service]
Type=notify
User=caddy
Group=caddy
ExecStart=/usr/bin/caddy run --environ --config /etc/caddy/Caddyfile
ExecReload=/usr/bin/caddy reload --config /etc/caddy/Caddyfile --force
TimeoutStopSec=5s
LimitNOFILE=1048576
LimitNPROC=512
PrivateTmp=true
ProtectSystem=full
AmbientCapabilities=CAP_NET_BIND_SERVICE

[Install]
WantedBy=multi-user.target

d. My complete Caddy config:

just-vibing.net {
        root * /var/www/Vibing
        file_server
}

7DTD.just-vibing.net {
        reverse_proxy 10.0.0.81:27016
}

amp.just-vibing.net {
        reverse_proxy 10.0.0.80:8080
}

ark.just-vibing.net {
        reverse_proxy 10.0.0.81:27015
}

factorio.just-vibing.net {
        reverse_proxy 10.0.0.81:34197
}

rust.just-vibing.net {
        reverse_proxy 10.0.0.81:28015
}

valheim.just-vibing.net {
        reverse_proxy 10.0.0.81:2457
}

3. The problem I’m having:

After a few days, Caddy just stops serving any webpage at all. However, using curl -v shows that the web server is at least trying to serve the default “Caddy works!” webpage. Restarting caddy makes everything work again without error until it breaks again.

4. Error messages and/or full log output:

-- Boot 2a073e719fbc40ab84f7d3d171a92f72 --
Dec 12 05:00:29 webvm systemd[1]: Starting Caddy...
Dec 12 05:00:30 webvm caddy[814]: caddy.HomeDir=/var/lib/caddy
Dec 12 05:00:30 webvm caddy[814]: caddy.AppDataDir=/var/lib/caddy/.local/share/caddy
Dec 12 05:00:30 webvm caddy[814]: caddy.AppConfigDir=/var/lib/caddy/.config/caddy
Dec 12 05:00:30 webvm caddy[814]: caddy.ConfigAutosavePath=/var/lib/caddy/.config/caddy/autosave.json
Dec 12 05:00:30 webvm caddy[814]: caddy.Version=v2.6.2 h1:wKoFIxpmOJLGl3QXoo6PNbYvGW4xLEgo32GPBEjWL8o=
Dec 12 05:00:30 webvm caddy[814]: runtime.GOOS=linux
Dec 12 05:00:30 webvm caddy[814]: runtime.GOARCH=amd64
Dec 12 05:00:30 webvm caddy[814]: runtime.Compiler=gc
Dec 12 05:00:30 webvm caddy[814]: runtime.NumCPU=2
Dec 12 05:00:30 webvm caddy[814]: runtime.GOMAXPROCS=2
Dec 12 05:00:30 webvm caddy[814]: runtime.Version=go1.19.4
Dec 12 05:00:30 webvm caddy[814]: os.Getwd=/
Dec 12 05:00:30 webvm caddy[814]: LANG=en_US.UTF-8
Dec 12 05:00:30 webvm caddy[814]: PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/snap/bin
Dec 12 05:00:30 webvm caddy[814]: NOTIFY_SOCKET=/run/systemd/notify
Dec 12 05:00:30 webvm caddy[814]: HOME=/var/lib/caddy
Dec 12 05:00:30 webvm caddy[814]: LOGNAME=caddy
Dec 12 05:00:30 webvm caddy[814]: USER=caddy
Dec 12 05:00:30 webvm caddy[814]: INVOCATION_ID=3d7f62f7fc5740818e632422734cd46d
Dec 12 05:00:30 webvm caddy[814]: JOURNAL_STREAM=8:22551
Dec 12 05:00:30 webvm caddy[814]: SYSTEMD_EXEC_PID=814
Dec 12 05:00:30 webvm caddy[814]: {"level":"info","ts":1670821230.4000623,"msg":"using provided configuration","config_file":"/etc/caddy/Caddyfile","config_adapter":""}
Dec 12 05:00:30 webvm caddy[814]: {"level":"info","ts":1670821230.4061868,"logger":"admin","msg":"admin endpoint started","address":"localhost:2019","enforce_origin":false,"origins":["//localhost:2019","//[::1]:2019","//127.0.0.1:2019"]}
Dec 12 05:00:30 webvm caddy[814]: {"level":"warn","ts":1670821230.4126754,"logger":"http","msg":"server is listening only on the HTTP port, so no automatic HTTPS will be applied to this server","server_name":"srv0","http_port":80}
Dec 12 05:00:30 webvm caddy[814]: {"level":"info","ts":1670821230.4133155,"logger":"tls.cache.maintenance","msg":"started background certificate maintenance","cache":"0xc0003fba40"}
Dec 12 05:00:30 webvm caddy[814]: {"level":"info","ts":1670821230.4159014,"logger":"tls","msg":"cleaning storage unit","description":"FileStorage:/var/lib/caddy/.local/share/caddy"}
Dec 12 05:00:30 webvm caddy[814]: {"level":"info","ts":1670821230.4161127,"logger":"http.log","msg":"server running","name":"srv0","protocols":["h1","h2","h3"]}
Dec 12 05:00:30 webvm caddy[814]: {"level":"info","ts":1670821230.417206,"msg":"autosaved config (load with --resume flag)","file":"/var/lib/caddy/.config/caddy/autosave.json"}
Dec 12 05:00:30 webvm caddy[814]: {"level":"info","ts":1670821230.4173903,"msg":"serving initial configuration"}
Dec 12 05:00:30 webvm systemd[1]: Started Caddy.
Dec 12 05:00:30 webvm caddy[814]: {"level":"info","ts":1670821230.440295,"logger":"tls","msg":"finished cleaning storage units"}
Dec 13 05:00:30 webvm caddy[814]: {"level":"info","ts":1670907630.8956876,"logger":"tls","msg":"cleaning storage unit","description":"FileStorage:/var/lib/caddy/.local/share/caddy"}
Dec 13 05:00:30 webvm caddy[814]: {"level":"info","ts":1670907630.8968818,"logger":"tls","msg":"finished cleaning storage units"}
Dec 14 05:00:30 webvm caddy[814]: {"level":"info","ts":1670994030.956278,"logger":"tls","msg":"cleaning storage unit","description":"FileStorage:/var/lib/caddy/.local/share/caddy"}
Dec 14 05:00:30 webvm caddy[814]: {"level":"info","ts":1670994030.9573529,"logger":"tls","msg":"finished cleaning storage units"}
Dec 15 05:00:30 webvm caddy[814]: {"level":"info","ts":1671080430.895727,"logger":"tls","msg":"cleaning storage unit","description":"FileStorage:/var/lib/caddy/.local/share/caddy"}
Dec 15 05:00:30 webvm caddy[814]: {"level":"info","ts":1671080430.897098,"logger":"tls","msg":"finished cleaning storage units"}
Dec 16 05:00:30 webvm caddy[814]: {"level":"info","ts":1671166830.8956997,"logger":"tls","msg":"cleaning storage unit","description":"FileStorage:/var/lib/caddy/.local/share/caddy"}
Dec 16 05:00:30 webvm caddy[814]: {"level":"info","ts":1671166830.8967397,"logger":"tls","msg":"finished cleaning storage units"}
Dec 17 05:00:30 webvm caddy[814]: {"level":"info","ts":1671253230.8957176,"logger":"tls","msg":"cleaning storage unit","description":"FileStorage:/var/lib/caddy/.local/share/caddy"}
Dec 17 05:00:30 webvm caddy[814]: {"level":"info","ts":1671253230.8967993,"logger":"tls","msg":"finished cleaning storage units"}
Dec 18 05:00:30 webvm caddy[814]: {"level":"info","ts":1671339630.9532027,"logger":"tls","msg":"cleaning storage unit","description":"FileStorage:/var/lib/caddy/.local/share/caddy"}
Dec 18 05:00:30 webvm caddy[814]: {"level":"info","ts":1671339630.9543676,"logger":"tls","msg":"finished cleaning storage units"}

● caddy.service - Caddy
     Loaded: loaded (/lib/systemd/system/caddy.service; enabled; vendor preset: enabled)
     Active: active (running) since Mon 2022-12-12 05:00:30 UTC; 6 days ago
       Docs: https://caddyserver.com/docs/
   Main PID: 814 (caddy)
      Tasks: 9 (limit: 4574)
     Memory: 46.0M
        CPU: 12.883s
     CGroup: /system.slice/caddy.service
             └─814 /usr/bin/caddy run --environ --config /etc/caddy/Caddyfile

Dec 18 06:59:45 webvm caddy[814]: {"level":"info","ts":1671346785.029071,"logger":"http","msg":"enabling HTTP/3 listener","addr":":443"}
Dec 18 06:59:45 webvm caddy[814]: {"level":"info","ts":1671346785.0295017,"logger":"tls.cache.maintenance","msg":"started background certificate maintenance","cache":"0xc0000eb7a0"}
Dec 18 06:59:45 webvm caddy[814]: {"level":"info","ts":1671346785.0295217,"msg":"failed to sufficiently increase receive buffer size (was: 208 kiB, wanted: 2048 kiB, got: 416 kiB). See https://github.com/luca>
Dec 18 06:59:45 webvm caddy[814]: {"level":"info","ts":1671346785.029592,"logger":"http.log","msg":"server running","name":"srv0","protocols":["h1","h2","h3"]}
Dec 18 06:59:45 webvm caddy[814]: {"level":"info","ts":1671346785.0296388,"logger":"http.log","msg":"server running","name":"remaining_auto_https_redirects","protocols":["h1","h2","h3"]}
Dec 18 06:59:45 webvm caddy[814]: {"level":"info","ts":1671346785.0296443,"logger":"http","msg":"enabling automatic TLS certificate management","domains":["rust.just-vibing.net","amp.just-vibing.net","ark.jus>
Dec 18 06:59:45 webvm caddy[814]: {"level":"info","ts":1671346785.8089614,"logger":"tls.cache.maintenance","msg":"stopped background certificate maintenance","cache":"0xc0003fba40"}
Dec 18 06:59:45 webvm caddy[814]: {"level":"info","ts":1671346785.8090935,"msg":"autosaved config (load with --resume flag)","file":"/var/lib/caddy/.config/caddy/autosave.json"}
Dec 18 06:59:45 webvm caddy[814]: {"level":"info","ts":1671346785.8094208,"logger":"admin.api","msg":"load complete"}
Dec 18 06:59:46 webvm caddy[814]: {"level":"info","ts":1671346786.093588,"logger":"admin","msg":"stopped previous server","address":"localhost:2019"}

5. What I already tried:

I’ve tried recreating my config file from scratch, purged and reinstalled caddy. I’ve also tried having it just serve the website (no reverse proxy) and it still eventually stops responding.

At the time of this post the VM has been restarted and so for now it works properly.

6. Links to relevant resources:

2

Don’t think this is the culprit but it does not harm to change the default values of the maximum socket receive buffer of your OS to something greater like this:

Lookup default:

# sysctl net.core.rmem_max
net.core.rmem_max = 212992

Change e.g.

# echo '2097152' > /proc/sys/net/core/rmem_max

Persist in /etc/sysctl.conf or better /etc/sysctl.d/local.conf

And I would take a look at: reverse_proxy (Caddyfile directive) — Caddy Documentation (https://caddyserver.com/docs/caddyfile/directives/reverse_proxy#the-http-transport)
the http transport options and on every option that says “Default: No timeout.” I would set one. Depending what it is that you proxy for example 30s to 3 minutes (depending on if it relates to uploads or downloads).

3

Alright, I’ll go ahead and set that.

I’m really only using it so my friends don’t have to bother with port numbers when connecting for a game.

Now with those set, just have to play the waiting game to see if it bonks out.

4

This makes me think you’re running Caddy in such a way that it breaks when it gets restarted.

Where are you placing/editing your Caddyfile? What commands are you running exactly, when changing your config etc?

Make sure to follow these instructions:

5

This topic was automatically closed after 30 days. New replies are no longer allowed.