Caddy crashes when rebooting

Help
1

1. The problem I’m having:

When rebooting my system, caddy seems to crash. After doing ‘sudo caddy start’ it works perfectly fine until I reboot and then it crashes again.

2. Error messages and/or full log output:

erver_name":"srv0","https_port":443}
Nov 13 11:41:53 comet caddy[215]: {"level":"info","ts":1731498113.9888456,"logger":"http.auto_https","msg":"enabling automatic HTTP->HTTPS redirects","server_name":"srv0"}
Nov 13 11:41:53 comet caddy[215]: {"level":"info","ts":1731498113.9906592,"logger":"tls.cache.maintenance","msg":"started background certificate maintenance","cache":"0xc000409e00"}
Nov 13 11:41:53 comet caddy[215]: {"level":"info","ts":1731498113.9952269,"logger":"http","msg":"enabling HTTP/3 listener","addr":":443"}
Nov 13 11:41:53 comet caddy[215]: {"level":"info","ts":1731498113.9995847,"logger":"http.log","msg":"server running","name":"srv0","protocols":["h1","h2","h3"]}
Nov 13 11:41:53 comet caddy[215]: {"level":"info","ts":1731498113.9996219,"logger":"http.log","msg":"server running","name":"srv1","protocols":["h1","h2","h3"]}
Nov 13 11:41:53 comet caddy[215]: {"level":"info","ts":1731498113.9996266,"logger":"http","msg":"enabling automatic TLS certificate management","domains":["home2.aaronrodrigues.com","home.aaronrodrigues.com","mail.aaronrodrigues.com"]}
Nov 13 11:41:54 comet caddy[215]: {"level":"info","ts":1731498114.00825,"logger":"tls.cache.maintenance","msg":"stopped background certificate maintenance","cache":"0xc000409e00"}
Nov 13 11:41:54 comet caddy[215]: Error: loading initial config: loading new config: http app module: start: finalizing automatic HTTPS: managing certificates for [home2.aaronrodrigues.com home.aaronrodrigues.com mail.aaronrodrigues.com]: automate: manage [home2.aaronrodrigues.com home.aaronrodrigues.com mail.aaronrodrigues.com]: home2.aaronrodrigues.com: caching certificate: decoding certificate metadata: unexpected end of JSON input
Nov 13 11:41:54 comet systemd[1]: caddy.service: Main process exited, code=exited, status=1/FAILURE
Nov 13 11:41:54 comet systemd[1]: caddy.service: Failed with result 'exit-code'.
Nov 13 11:41:54 comet systemd[1]: Failed to start caddy.service - Caddy.
-- Boot 3f40a8dcf5d84e67b1f09192a7046a44 --
Nov 13 12:11:41 comet systemd[1]: Starting caddy.service - Caddy...
Nov 13 12:11:42 comet caddy[217]: caddy.HomeDir=/var/lib/caddy
Nov 13 12:11:42 comet caddy[217]: caddy.AppDataDir=/var/lib/caddy/.local/share/caddy
Nov 13 12:11:42 comet caddy[217]: caddy.AppConfigDir=/var/lib/caddy/.config/caddy
Nov 13 12:11:42 comet caddy[217]: caddy.ConfigAutosavePath=/var/lib/caddy/.config/caddy/autosave.json
Nov 13 12:11:42 comet caddy[217]: caddy.Version=v2.8.4 h1:q3pe0wpBj1OcHFZ3n/1nl4V4bxBrYoSoab7rL9BMYNk=
Nov 13 12:11:42 comet caddy[217]: runtime.GOOS=linux
Nov 13 12:11:42 comet caddy[217]: runtime.GOARCH=amd64
Nov 13 12:11:42 comet caddy[217]: runtime.Compiler=gc
Nov 13 12:11:42 comet caddy[217]: runtime.NumCPU=6
Nov 13 12:11:42 comet caddy[217]: runtime.GOMAXPROCS=6
Nov 13 12:11:42 comet caddy[217]: runtime.Version=go1.22.3
Nov 13 12:11:42 comet caddy[217]: os.Getwd=/
Nov 13 12:11:42 comet caddy[217]: LANG=C
Nov 13 12:11:42 comet caddy[217]: PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin
Nov 13 12:11:42 comet caddy[217]: NOTIFY_SOCKET=/run/systemd/notify
Nov 13 12:11:42 comet caddy[217]: USER=caddy
Nov 13 12:11:42 comet caddy[217]: LOGNAME=caddy
Nov 13 12:11:42 comet caddy[217]: HOME=/var/lib/caddy
Nov 13 12:11:42 comet caddy[217]: INVOCATION_ID=be4c8c108d3f48c8bb1fcc938969b6e3
Nov 13 12:11:42 comet caddy[217]: JOURNAL_STREAM=8:464758
Nov 13 12:11:42 comet caddy[217]: SYSTEMD_EXEC_PID=217
Nov 13 12:11:42 comet caddy[217]: MEMORY_PRESSURE_WATCH=/sys/fs/cgroup/system.slice/caddy.service/memory.pressure
Nov 13 12:11:42 comet caddy[217]: MEMORY_PRESSURE_WRITE=c29tZSAyMDAwMDAgMjAwMDAwMAA=
Nov 13 12:11:42 comet caddy[217]: {"level":"info","ts":1731499902.4986823,"msg":"using config from file","file":"/etc/caddy/Caddyfile"}
Nov 13 12:11:42 comet caddy[217]: {"level":"info","ts":1731499902.5010014,"msg":"adapted config to JSON","adapter":"caddyfile"}
Nov 13 12:11:42 comet caddy[217]: {"level":"info","ts":1731499902.501544,"logger":"admin","msg":"admin endpoint started","address":"localhost:2019","enforce_origin":false,"origins":["//localhost:2019","//[::1]:2019","//127.0.0.1:2019"]}
Nov 13 12:11:42 comet caddy[217]: {"level":"info","ts":1731499902.501648,"logger":"http.auto_https","msg":"server is listening only on the HTTPS port but has no TLS connection policies; adding one to enable TLS","server_name":"srv0","https_port":443}
Nov 13 12:11:42 comet caddy[217]: {"level":"info","ts":1731499902.5016582,"logger":"http.auto_https","msg":"enabling automatic HTTP->HTTPS redirects","server_name":"srv0"}
Nov 13 12:11:42 comet caddy[217]: {"level":"warn","ts":1731499902.5016687,"logger":"http.auto_https","msg":"server is listening only on the HTTP port, so no automatic HTTPS will be applied to this server","server_name":"srv1","http_port":80}
Nov 13 12:11:42 comet caddy[217]: {"level":"info","ts":1731499902.5036993,"logger":"tls.cache.maintenance","msg":"started background certificate maintenance","cache":"0xc0005f4380"}
Nov 13 12:11:42 comet caddy[217]: {"level":"info","ts":1731499902.5128977,"logger":"http","msg":"enabling HTTP/3 listener","addr":":443"}
Nov 13 12:11:42 comet caddy[217]: {"level":"info","ts":1731499902.5174947,"logger":"http.log","msg":"server running","name":"srv0","protocols":["h1","h2","h3"]}
Nov 13 12:11:42 comet caddy[217]: {"level":"info","ts":1731499902.5175462,"logger":"http.log","msg":"server running","name":"srv1","protocols":["h1","h2","h3"]}
Nov 13 12:11:42 comet caddy[217]: {"level":"info","ts":1731499902.5175514,"logger":"http","msg":"enabling automatic TLS certificate management","domains":["home.aaronrodrigues.com","mail.aaronrodrigues.com","home2.aaronrodrigues.com"]}
Nov 13 12:11:42 comet caddy[217]: {"level":"info","ts":1731499902.5192642,"logger":"tls.cache.maintenance","msg":"stopped background certificate maintenance","cache":"0xc0005f4380"}
Nov 13 12:11:42 comet caddy[217]: {"level":"error","ts":1731499902.51927,"logger":"tls","msg":"could not clean default/global storage","error":"unable to acquire storage_clean lock: context canceled"}
Nov 13 12:11:42 comet caddy[217]: {"level":"info","ts":1731499902.5193012,"logger":"tls","msg":"finished cleaning storage units"}
Nov 13 12:11:42 comet caddy[217]: Error: loading initial config: loading new config: http app module: start: finalizing automatic HTTPS: managing certificates for [home.aaronrodrigues.com mail.aaronrodrigues.com home2.aaronrodrigues.com]: automate: manage [home.aaronrodrigues.com mail.aaronrodrigues.com home2.aaronrodrigues.com]: home.aaronrodrigues.com: caching certificate: decoding certificate metadata: unexpected end of JSON input
Nov 13 12:11:42 comet systemd[1]: caddy.service: Main process exited, code=exited, status=1/FAILURE
Nov 13 12:11:42 comet systemd[1]: caddy.service: Failed with result 'exit-code'.
Nov 13 12:11:42 comet systemd[1]: Failed to start caddy.service - Caddy.

3. Caddy version:

v2.8.4 h1:q3pe0wpBj1OcHFZ3n/1nl4V4bxBrYoSoab7rL9BMYNk=

4. How I installed and ran Caddy:

sudo apt install -y debian-keyring debian-archive-keyring apt-transport-https curl
curl -1sLf 'https://dl.cloudsmith.io/public/caddy/stable/gpg.key' | sudo gpg --dearmor -o /usr/share/keyrings/caddy-stable-archive-keyring.gpg
curl -1sLf 'https://dl.cloudsmith.io/public/caddy/stable/debian.deb.txt' | sudo tee /etc/apt/sources.list.d/caddy-stable.list
sudo apt update
sudo apt install caddy

a. System environment:

Ubuntu Server 24.04, Intel i5 8400, 32GB RAM

b. Command:

sudo caddy start

d. My complete Caddy config:

home.aaronrodrigues.com {
        reverse_proxy 192.168.1.116:8123
}

home2.aaronrodrigues.com {
        reverse_proxy 192.168.1.135:8123
}

mail.aaronrodrigues.com {
        reverse_proxy localhost:8081
}

# The Caddyfile is an easy way to configure your Caddy web server.
#
# Unless the file starts with a global options block, the first
# uncommented line is always the address of your site.
#
# To use your own domain name (with automatic HTTPS), first make
# sure your domain's A/AAAA DNS records are properly pointed to
# this machine's public IP, then replace ":80" below with your
# domain name.

:80 {
        # Set this path to your site's directory.
        root * /home/aaronrodrigues/website

        # Enable the static file server.
        file_server

        # Another common task is to set up a reverse proxy:
        # reverse_proxy localhost:8080

        # Or serve a PHP site through php-fpm:
        # php_fastcgi localhost:9000
}

# Refer to the Caddy docs for more information:
# https://caddyserver.com/docs/caddyfile

5. Links to relevant resources:

I’ve also attached the output of ‘sudo systemctl status caddy.service’

x caddy.service - Caddy
     Loaded: loaded (/usr/lib/systemd/system/caddy.service; enabled; preset: enabled)
     Active: failed (Result: exit-code) since Wed 2024-11-13 12:11:42 UTC; 1min 43s ago
       Docs: https://caddyserver.com/docs/
    Process: 217 ExecStart=/usr/bin/caddy run --environ --config /etc/caddy/Caddyfile (code=exited, status=1/FAILURE)
   Main PID: 217 (code=exited, status=1/FAILURE)
     Status: "loading new config: http app module: start: finalizing automatic HTTPS: managing certificates for [home.aaronrodrigues.com mail.aaronrodrigues.com home2.aaronrodrigues.com]: automate: manage [home.aa for [home.aaronrodrigues.com mail.aaronrodrigues.com home2.aaronrodrigues.com]: automate: manage [home.aaronrodrigues.com mail.aaronrodrigues.com home2.aaronrodrigues.com]: home.aaronrodrigues.com: caching certifonrodrigues.com mail.aaronrodrigues.com home2.aaronrodrigues.com]: home.aaronrodrigues.com: caching certificate: decoding certificate metadata: unexpected end of JSON input"


        CPU: 87ms

Nov 13 12:11:42 comet caddy[217]: {"level":"info","ts":1731499902.5174947,"logger":"http.log","msg":"server running","name":"srv0","protocols":["h1","h2","h3"]}
Nov 13 12:11:42 comet caddy[217]: {"level":"info","ts":1731499902.5175462,"logger":"http.log","msg":"server running","name":"srv1","protocols":["h1","h2","h3"]}
Nov 13 12:11:42 comet caddy[217]: {"level":"info","ts":1731499902.5175514,"logger":"http","msg":"enabling automatic TLS certificate management","domains":["home.aaronrodrigues.com","mail.aaronrodrigues.com","home2>
Nov 13 12:11:42 comet caddy[217]: {"level":"info","ts":1731499902.5192642,"logger":"tls.cache.maintenance","msg":"stopped background certificate maintenance","cache":"0xc0005f4380"}
Nov 13 12:11:42 comet caddy[217]: {"level":"error","ts":1731499902.51927,"logger":"tls","msg":"could not clean default/global storage","error":"unable to acquire storage_clean lock: context canceled"}
Nov 13 12:11:42 comet caddy[217]: {"level":"info","ts":1731499902.5193012,"logger":"tls","msg":"finished cleaning storage units"}
Nov 13 12:11:42 comet caddy[217]: Error: loading initial config: loading new config: http app module: start: finalizing automatic HTTPS: managing certificates for [home.aaronrodrigues.com mail.aaronrodrigues.com h: finalizing automatic HTTPS: managing certificates for [home.aaronrodrigues.com mail.aaronrodrigues.com home2.aaronrodrigues.com]: automate: manage [home.aaronrodrigues.com mail.aaronrodrigues.com home2.aaronrodrme2.aaronrodrigues.com]: automate: manage [home.aaronrodrigues.com mail.aaronrodrigues.com home2.aaronrodrigues.com]: home.aaronrodrigues.com: caching certificate: decoding certificate metadata: unexpected end of gues.com]: home.aaronrodrigues.com: caching certificate: decoding certificate metadata: unexpected end of JSON input



Nov 13 12:11:42 comet systemd[1]: caddy.service: Main process exited, code=exited, status=1/FAILURE
Nov 13 12:11:42 comet systemd[1]: caddy.service: Failed with result 'exit-code'.
Nov 13 12:11:42 comet systemd[1]: Failed to start caddy.service - Caddy.
2

Fixed! If anyone else has this issue, the following worked for me:

Uninstall caddy using ‘sudo apt purge caddy’
Reboot
Try installing caddy again using ‘sudo apt install caddy’

If it doesn’t work, try one more time (for some reason I had to do it two times)

3

Unfortunately this is a known issue with versions before 2.9.0, he storage driver sometimes failed to correctly write to disk and caused corruption. The workaround is to wipe the storage and let Caddy reissue certs.

1 Like
4

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.