Caddy is not starting up

Help
1

1. The problem I’m having:

I have a few services running on a VPS (e.g., subdomain1.domain.com, subdomain2.domain.com and subdomain3.domain.com and Caddy v2.8.4 is the reverse proxy. When I am trying to go to these websites, I’m getting an error “The site can’t be reached”. However, when I open the specific port of the service in firewall and connect directly with the IP address:port, the service page loads just fine. Things were working fine for me until about 1 hour ago when I got a notification from Uptime Kuma of a downtime for these domain names.

Did docker ps and it shows all the docker containers running as expected. When I caddy version, I get v2.8.4 h1:q3pe0wpBj1OcHFZ3n/1nl4V4bxBrYoSoab7rL9BMYNk= as response. However, when I do sudo systemctl status caddy, I get the following.

× caddy.service - Caddy
     Loaded: loaded (/lib/systemd/system/caddy.service; enabled; vendor preset: enabled)
     Active: failed (Result: exit-code) since Wed 2024-06-19 20:58:07 UTC; 58s ago
       Docs: https://caddyserver.com/docs/
    Process: 801 ExecStart=/usr/bin/caddy run --environ --config /etc/caddy/Caddyfile (code=exited, status=1/FAILURE)
   Main PID: 801 (code=exited, status=1/FAILURE)
     Status: "loading new config: http app module: start: finalizing automatic HTTPS: managing certificates for [subdomain1.domain.tld subdomain2.domain.tld subdomain3.domain.tld subdomain4.domain.tld subdomain5.domain.tld subdomain6.domain.tld subdomain7.domain.tld]: automate: manage [cha>
        CPU: 307ms

Jun 19 20:58:07 my-media caddy[801]: {"level":"info","ts":1718830687.9394069,"logger":"http","msg":"enabling HTTP/3 listener","addr":":443"}
Jun 19 20:58:07 my-media caddy[801]: {"level":"info","ts":1718830687.9396617,"logger":"tls.cache.maintenance","msg":"started background certificate maintenance","cache":"0x40009ab100"}
Jun 19 20:58:07 my-media caddy[801]: {"level":"info","ts":1718830687.9406016,"logger":"http.log","msg":"server running","name":"srv0","protocols":["h1","h2","h3"]}
Jun 19 20:58:07 my-media caddy[801]: {"level":"info","ts":1718830687.9407897,"logger":"http.log","msg":"server running","name":"remaining_auto_https_redirects","protocols":["h1","h2","h3"]}
Jun 19 20:58:07 my-media caddy[801]: {"level":"info","ts":1718830687.9409275,"logger":"http","msg":"enabling automatic TLS certificate management","domains":["subdomain1.domain.tld","subdomain2.domain.tld","subdomain3.domain.tld","subdomain4.domain.tld","subdomain5.domain.tld","b>
Jun 19 20:58:07 my-media caddy[801]: {"level":"info","ts":1718830687.967151,"logger":"tls.cache.maintenance","msg":"stopped background certificate maintenance","cache":"0x40009ab100"}
Jun 19 20:58:07 my-media caddy[801]: Error: loading initial config: loading new config: http app module: start: finalizing automatic HTTPS: managing certificates for [subdomain1.domain.tld subdomain2.domain.tld subdomain3.domain.tld subdomain4.domain.tld subdomain5.domain.tld boo>
Jun 19 20:58:07 my-media systemd[1]: caddy.service: Main process exited, code=exited, status=1/FAILURE
Jun 19 20:58:07 my-media systemd[1]: caddy.service: Failed with result 'exit-code'.
Jun 19 20:58:07 my-media systemd[1]: Failed to start Caddy.

Running sudo systemctl start caddy freezes terminal and restarting the server doesn’t start Caddy either. Cloudflare is handling my DNS and when I go to my dashboard, I’m seeing about 5.69M total requests in the last 24 hours, out of which there are 5.45M cached requests and 241.08k uncached requests. 99.99% of the requests seem to be from my country only.

How to fix Caddy?

2. Error messages and/or full log output:

After typing journalctl -u caddy --no-pager | less +G in the terminal, the output was was so large that I couldn’t go to the top no matter how much I scrolled. Please let me know if you need more from the log.

Jun 19 07:21:32 my-media caddy[800]: {"level":"error","ts":1718781692.73709,"logger":"http.log.error","msg":"dial tcp 10.0.0.58:3000: i/o timeout","request":{"remote_ip":"162.158.90.122","remote_port":"21726","client_ip":"162.158.90.122","proto":"HTTP/2.0
","method":"GET","host":"subdomain1.domain.tld","uri":"/api/tags","headers":{"Cdn-Loop":["cloudflare"],"X-Forwarded-For":["129.154.xxx.xxx"],"X-Forwarded-Proto":["https"],"Cf-Visitor":["{\"scheme\":\"https\"}"],"User-Agent":["Python/3.11 aiohttp/3.9.5"],"Cf-Conn
ecting-Ip":["129.154.xxx.xxx"],"Cf-Ipcountry":["IN"],"Accept-Encoding":["gzip, br"],"Cf-Ray":["8961c3380cf02b75-LAX"],"Accept":["*/*"]},"tls":{"resumed":false,"version":772,"cipher_suite":4865,"proto":"h2","server_name":"subdomain1.domain.tld"}},"duration":3.001
416213,"status":502,"err_id":"wefiktz42","err_trace":"reverseproxy.statusError (reverseproxy.go:1269)"}
-- Boot f29d5f305a6a40c7982edfeafc1b3206 --
Jun 19 20:25:08 my-media systemd[1]: Starting Caddy...
Jun 19 20:25:10 my-media caddy[801]: caddy.HomeDir=/var/lib/caddy
Jun 19 20:25:10 my-media caddy[801]: caddy.AppDataDir=/var/lib/caddy/.local/share/caddy
Jun 19 20:25:10 my-media caddy[801]: caddy.AppConfigDir=/var/lib/caddy/.config/caddy
Jun 19 20:25:10 my-media caddy[801]: caddy.ConfigAutosavePath=/var/lib/caddy/.config/caddy/autosave.json
Jun 19 20:25:10 my-media caddy[801]: caddy.Version=v2.8.4 h1:q3pe0wpBj1OcHFZ3n/1nl4V4bxBrYoSoab7rL9BMYNk=
Jun 19 20:25:10 my-media caddy[801]: runtime.GOOS=linux
Jun 19 20:25:10 my-media caddy[801]: runtime.GOARCH=arm64
Jun 19 20:25:10 my-media caddy[801]: runtime.Compiler=gc
Jun 19 20:25:10 my-media caddy[801]: runtime.NumCPU=4
Jun 19 20:25:10 my-media caddy[801]: runtime.GOMAXPROCS=4
Jun 19 20:25:10 my-media caddy[801]: runtime.Version=go1.22.3
Jun 19 20:25:10 my-media caddy[801]: os.Getwd=/
Jun 19 20:25:10 my-media caddy[801]: LANG=C.UTF-8
Jun 19 20:25:10 my-media caddy[801]: PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/snap/bin
Jun 19 20:25:10 my-media caddy[801]: NOTIFY_SOCKET=/run/systemd/notify
Jun 19 20:25:10 my-media caddy[801]: HOME=/var/lib/caddy
Jun 19 20:25:10 my-media caddy[801]: LOGNAME=caddy
Jun 19 20:25:10 my-media caddy[801]: USER=caddy
Jun 19 20:25:10 my-media caddy[801]: INVOCATION_ID=e78b15167e7a4845bb7d338e1defd2f9
Jun 19 20:25:10 my-media caddy[801]: JOURNAL_STREAM=8:18980
Jun 19 20:25:10 my-media caddy[801]: SYSTEMD_EXEC_PID=801
Jun 19 20:25:10 my-media caddy[801]: {"level":"info","ts":1718828710.3668466,"msg":"using config from file","file":"/etc/caddy/Caddyfile"}
Jun 19 20:25:10 my-media caddy[801]: {"level":"info","ts":1718828710.3740854,"msg":"adapted config to JSON","adapter":"caddyfile"}
Jun 19 20:25:10 my-media caddy[801]: {"level":"warn","ts":1718828710.3770466,"msg":"Caddyfile input is not formatted; run 'caddy fmt --overwrite' to fix inconsistencies","adapter":"caddyfile","file":"/etc/caddy/Caddyfile","line":2}
Jun 19 20:25:10 my-media caddy[801]: {"level":"info","ts":1718828710.3845356,"logger":"admin","msg":"admin endpoint started","address":"localhost:2019","enforce_origin":false,"origins":["//localhost:2019","//[::1]:2019","//127.0.0.1:2019"]}
Jun 19 20:25:10 my-media caddy[801]: {"level":"info","ts":1718828710.38942,"logger":"http.auto_https","msg":"server is listening only on the HTTPS port but has no TLS connection policies; adding one to enable TLS","server_name":"srv0","https_port":443}
Jun 19 20:25:10 my-media caddy[801]: {"level":"info","ts":1718828710.3896527,"logger":"http.auto_https","msg":"enabling automatic HTTP->HTTPS redirects","server_name":"srv0"}
Jun 19 20:25:10 my-media caddy[801]: {"level":"info","ts":1718828710.3914015,"logger":"crowdsec","msg":"initializing streaming bouncer","instance_id":"1e9bcf3c"}
Jun 19 20:25:10 my-media caddy[801]: {"level":"info","ts":1718828710.39667,"logger":"crowdsec","msg":"using API key auth","instance_id":"1e9bcf3c","address":"http://127.0.0.1:8080/"}
Jun 19 20:25:10 my-media caddy[801]: {"level":"info","ts":1718828710.3976383,"logger":"crowdsec","msg":"started","instance_id":"1e9bcf3c"}
Jun 19 20:25:10 my-media caddy[801]: {"level":"info","ts":1718828710.3980856,"logger":"http","msg":"enabling HTTP/3 listener","addr":":443"}
Jun 19 20:25:10 my-media caddy[801]: {"level":"info","ts":1718828710.398622,"logger":"http.log","msg":"server running","name":"srv0","protocols":["h1","h2","h3"]}
Jun 19 20:25:10 my-media caddy[801]: {"level":"info","ts":1718828710.3988543,"logger":"http.log","msg":"server running","name":"remaining_auto_https_redirects","protocols":["h1","h2","h3"]}
Jun 19 20:25:10 my-media caddy[801]: {"level":"info","ts":1718828710.3989809,"logger":"http","msg":"enabling automatic TLS certificate management","domains":["subdomain2.domain.tld","subdomain3.domain.tld","subdomain4.domain.tld","subdomain5.domain.tld","subdomain6.domain.tld","subdomain7.domain.tld","subdomain1.domain.tld"]}
Jun 19 20:25:10 my-media caddy[801]: {"level":"info","ts":1718828710.3974667,"logger":"tls.cache.maintenance","msg":"started background certificate maintenance","cache":"0x40005fb080"}
Jun 19 20:25:10 my-media caddy[801]: {"level":"error","ts":1718828710.4005394,"logger":"crowdsec","msg":"auth-api: auth with api key failed return nil response, error: dial tcp 127.0.0.1:8080: connect: connection refused","instance_id":"1e9bcf3c","address":"http://127.0.0.1:8080/","error":"auth-api: auth with api key failed return nil response, error: dial tcp 127.0.0.1:8080: connect: connection refused"}
Jun 19 20:25:10 my-media caddy[801]: {"level":"error","ts":1718828710.4007406,"logger":"crowdsec","msg":"failed to connect to LAPI, retrying in 10s: Get \"http://127.0.0.1:8080/v1/decisions/stream?startup=true\": dial tcp 127.0.0.1:8080: connect: connection refused","instance_id":"1e9bcf3c","address":"http://127.0.0.1:8080/","error":"failed to connect to LAPI, retrying in 10s: Get \"http://127.0.0.1:8080/v1/decisions/stream?startup=true\": dial tcp 127.0.0.1:8080: connect: connection refused"}
Jun 19 20:25:10 my-media caddy[801]: {"level":"info","ts":1718828710.4183342,"logger":"crowdsec","msg":"stopping","instance_id":"1e9bcf3c"}
Jun 19 20:25:10 my-media caddy[801]: {"level":"info","ts":1718828710.418386,"logger":"crowdsec","msg":"processing new and deleted decisions stopped","instance_id":"1e9bcf3c"}
Jun 19 20:25:10 my-media caddy[801]: {"level":"info","ts":1718828710.418394,"logger":"crowdsec","msg":"finished","instance_id":"1e9bcf3c"}
Jun 19 20:25:10 my-media caddy[801]: {"level":"info","ts":1718828710.4184306,"logger":"tls.cache.maintenance","msg":"stopped background certificate maintenance","cache":"0x40005fb080"}
Jun 19 20:25:10 my-media caddy[801]: {"level":"error","ts":1718828710.41872,"logger":"tls","msg":"could not clean default/global storage","error":"unable to acquire storage_clean lock: context canceled"}
Jun 19 20:25:10 my-media caddy[801]: {"level":"info","ts":1718828710.4187453,"logger":"tls","msg":"finished cleaning storage units"}
Jun 19 20:25:10 my-media caddy[801]: Error: loading initial config: loading new config: http app module: start: finalizing automatic HTTPS: managing certificates for [subdomain2.domain.tld subdomain3.domain.tld subdomain4.domain.tld subdomain5.domain.tld subdomain6.domain.tld subdomain7.domain.tld subdomain1.domain.tld]: automate: manage [subdomain2.domain.tld subdomain3.domain.tld subdomain4.domain.tld subdomain5.domain.tld subdomain6.domain.tld subdomain7.domain.tld subdomain1.domain.tld]: subdomain4.domain.tld: caching certificate: decoding certificate metadata: unexpected end of JSON input
Jun 19 20:25:10 my-media systemd[1]: caddy.service: Main process exited, code=exited, status=1/FAILURE
Jun 19 20:25:10 my-media systemd[1]: caddy.service: Failed with result 'exit-code'.
Jun 19 20:25:10 my-media systemd[1]: Failed to start Caddy.
Jun 19 20:28:34 my-media systemd[1]: caddy.service: Unit cannot be reloaded because it is inactive.
Jun 19 20:28:42 my-media systemd[1]: Starting Caddy...
Jun 19 20:28:42 my-media caddy[5256]: caddy.HomeDir=/var/lib/caddy
Jun 19 20:28:42 my-media caddy[5256]: caddy.AppDataDir=/var/lib/caddy/.local/share/caddy
Jun 19 20:28:42 my-media caddy[5256]: caddy.AppConfigDir=/var/lib/caddy/.config/caddy
Jun 19 20:28:42 my-media caddy[5256]: caddy.ConfigAutosavePath=/var/lib/caddy/.config/caddy/autosave.json
Jun 19 20:28:42 my-media caddy[5256]: caddy.Version=v2.8.4 h1:q3pe0wpBj1OcHFZ3n/1nl4V4bxBrYoSoab7rL9BMYNk=
Jun 19 20:28:42 my-media caddy[5256]: runtime.GOOS=linux
Jun 19 20:28:42 my-media caddy[5256]: runtime.GOARCH=arm64
Jun 19 20:28:42 my-media caddy[5256]: runtime.Compiler=gc
Jun 19 20:28:42 my-media caddy[5256]: runtime.NumCPU=4
Jun 19 20:28:42 my-media caddy[5256]: runtime.GOMAXPROCS=4
Jun 19 20:28:42 my-media caddy[5256]: runtime.Version=go1.22.3
Jun 19 20:28:42 my-media caddy[5256]: os.Getwd=/
Jun 19 20:28:42 my-media caddy[5256]: LANG=C.UTF-8
Jun 19 20:28:42 my-media caddy[5256]: PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/snap/bin
Jun 19 20:28:42 my-media caddy[5256]: NOTIFY_SOCKET=/run/systemd/notify
Jun 19 20:28:42 my-media caddy[5256]: HOME=/var/lib/caddy
Jun 19 20:28:42 my-media caddy[5256]: LOGNAME=caddy
Jun 19 20:28:42 my-media caddy[5256]: USER=caddy
Jun 19 20:28:42 my-media caddy[5256]: INVOCATION_ID=b352a310fa6143b6a0cec40d698685f5
Jun 19 20:28:42 my-media caddy[5256]: JOURNAL_STREAM=8:36734
Jun 19 20:28:42 my-media caddy[5256]: SYSTEMD_EXEC_PID=5256
Jun 19 20:28:42 my-media caddy[5256]: {"level":"info","ts":1718828922.7855647,"msg":"using config from file","file":"/etc/caddy/Caddyfile"}
Jun 19 20:28:42 my-media caddy[5256]: {"level":"info","ts":1718828922.7875128,"msg":"adapted config to JSON","adapter":"caddyfile"}
Jun 19 20:28:42 my-media caddy[5256]: {"level":"warn","ts":1718828922.7875512,"msg":"Caddyfile input is not formatted; run 'caddy fmt --overwrite' to fix inconsistencies","adapter":"caddyfile","file":"/etc/caddy/Caddyfile","line":2}
Jun 19 20:28:42 my-media caddy[5256]: {"level":"info","ts":1718828922.7887905,"logger":"admin","msg":"admin endpoint started","address":"localhost:2019","enforce_origin":false,"origins":["//localhost:2019","//[::1]:2019","//127.0.0.1:2019"]}
Jun 19 20:28:42 my-media caddy[5256]: {"level":"info","ts":1718828922.7890842,"logger":"tls.cache.maintenance","msg":"started background certificate maintenance","cache":"0x40001c0680"}
Jun 19 20:28:42 my-media caddy[5256]: {"level":"info","ts":1718828922.7891684,"logger":"http.auto_https","msg":"server is listening only on the HTTPS port but has no TLS connection policies; adding one to enable TLS","server_name":"srv0","https_port":443}
Jun 19 20:28:42 my-media caddy[5256]: {"level":"info","ts":1718828922.7891908,"logger":"http.auto_https","msg":"enabling automatic HTTP->HTTPS redirects","server_name":"srv0"}
Jun 19 20:28:42 my-media caddy[5256]: {"level":"info","ts":1718828922.7896883,"logger":"crowdsec","msg":"initializing streaming bouncer","instance_id":"14463f97"}
Jun 19 20:28:42 my-media caddy[5256]: {"level":"info","ts":1718828922.7897213,"logger":"crowdsec","msg":"using API key auth","instance_id":"14463f97","address":"http://127.0.0.1:8080/"}
Jun 19 20:28:42 my-media caddy[5256]: {"level":"info","ts":1718828922.7897372,"logger":"crowdsec","msg":"started","instance_id":"14463f97"}
Jun 19 20:28:42 my-media caddy[5256]: {"level":"info","ts":1718828922.7898254,"logger":"http","msg":"enabling HTTP/3 listener","addr":":443"}
Jun 19 20:28:42 my-media caddy[5256]: {"level":"info","ts":1718828922.7899964,"logger":"http.log","msg":"server running","name":"srv0","protocols":["h1","h2","h3"]}
Jun 19 20:28:42 my-media caddy[5256]: {"level":"info","ts":1718828922.790064,"logger":"http.log","msg":"server running","name":"remaining_auto_https_redirects","protocols":["h1","h2","h3"]}
Jun 19 20:28:42 my-media caddy[5256]: {"level":"info","ts":1718828922.7900696,"logger":"http","msg":"enabling automatic TLS certificate management","domains":["subdomain6.domain.tld","subdomain7.domain.tld","subdomain1.domain.tld","subdomain2.domain.tld","subdomain3.domain.tld","subdomain4.domain.tld","subdomain5.domain.tld"]}
Jun 19 20:28:42 my-media caddy[5256]: {"level":"info","ts":1718828922.8007321,"logger":"crowdsec","msg":"stopping","instance_id":"14463f97"}
Jun 19 20:28:42 my-media caddy[5256]: {"level":"info","ts":1718828922.80078,"logger":"crowdsec","msg":"processing new and deleted decisions stopped","instance_id":"14463f97"}
Jun 19 20:28:43 my-media caddy[5256]: {"level":"info","ts":1718828923.865127,"logger":"tls.issuance.acme.acme_client","msg":"got renewal info","names":["subdomain1.domain.tld"],"window_start":1723103556,"window_end":1723276356,"selected_time":1723167269,"recheck_after":1718850523.8651228,"explanation_url":""}
Jun 19 20:28:43 my-media caddy[5256]: {"level":"info","ts":1718828923.8656101,"logger":"tls","msg":"updated ACME renewal information","identifiers":["subdomain1.domain.tld"],"cert_hash":"586457f1cecd8de6734dffa71a1f6269fee96071e3b5c757e7e42228f6f264b8","ari_unique_id":"nytfzzwhT50Et-0rLMTGcIvS1w0.A2CYMwcvhrSlMotgjBkazGR3","cert_expiry":1725780756,"selected_time":1723239847,"next_update":1718850523.8651228,"explanation_url":""}
Jun 19 20:29:55 my-media caddy[5256]: {"level":"info","ts":1718828995.8649776,"msg":"shutting down apps, then terminating","signal":"SIGTERM"}
Jun 19 20:29:55 my-media caddy[5256]: {"level":"warn","ts":1718828995.8651161,"msg":"exiting; byeee!! 👋","signal":"SIGTERM"}
Jun 19 20:29:56 my-media caddy[5256]: {"level":"error","ts":1718828996.2627873,"logger":"http.log.error","msg":"dial tcp 10.0.0.58:3000: connect: connection refused","request":{"remote_ip":"172.71.122.234","remote_port":"44280","client_ip":"172.71.122.234","proto":"HTTP/2.0","method":"GET","host":"subdomain1.domain.tld","uri":"/api/tags","headers":{"Cf-Ipcountry":["IN"],"X-Forwarded-Proto":["https"],"Cf-Visitor":["{\"scheme\":\"https\"}"],"Accept":["*/*"],"Cdn-Loop":["cloudflare"],"Accept-Encoding":["gzip, br"],"X-Forwarded-For":["129.154.xxx.xxx"],"Cf-Ray":["8966462a1a7d9ec7-CDG"],"User-Agent":["Python/3.11 aiohttp/3.9.5"],"Cf-Connecting-Ip":["129.154.xxx.xxx"]},"tls":{"resumed":false,"version":772,"cipher_suite":4865,"proto":"h2","server_name":"subdomain1.domain.tld"}},"duration":0.000392043,"status":502,"err_id":"ussazpdx8","err_trace":"reverseproxy.statusError (reverseproxy.go:1269)"}
Jun 19 20:29:56 my-media caddy[5256]: {"level":"error","ts":1718828996.3812368,"logger":"http.log.error","msg":"dial tcp 10.0.0.58:3000: connect: connection refused","request":{"remote_ip":"141.101.95.109","remote_port":"15652","client_ip":"141.101.95.109","proto":"HTTP/2.0","method":"GET","host":"subdomain1.domain.tld","uri":"/api/tags","headers":{"Accept":["*/*"],"User-Agent":["Python/3.11 aiohttp/3.9.5"],"Cf-Connecting-Ip":["129.154.xxx.xxx"],"Cdn-Loop":["cloudflare"],"Cf-Ipcountry":["IN"],"Cf-Visitor":["{\"scheme\":\"https\"}"],"X-Forwarded-Proto":["https"],"Accept-Encoding":["gzip, br"],"X-Forwarded-For":["129.154.xxx.xxx"],"Cf-Ray":["8966462aef742a4b-CDG"]},"tls":{"resumed":false,"version":772,"cipher_suite":4865,"proto":"h2","server_name":"subdomain1.domain.tld"}},"duration":0.000304003,"status":502,"err_id":"2dfu7pvgb","err_trace":"reverseproxy.statusError (reverseproxy.go:1269)"}
Jun 19 20:30:00 my-media systemd[1]: caddy.service: State 'stop-sigterm' timed out. Killing.
Jun 19 20:30:00 my-media systemd[1]: caddy.service: Killing process 5256 (caddy) with signal SIGKILL.
Jun 19 20:30:00 my-media systemd[1]: caddy.service: Killing process 5258 (caddy) with signal SIGKILL.
Jun 19 20:30:00 my-media systemd[1]: caddy.service: Killing process 5259 (caddy) with signal SIGKILL.
Jun 19 20:30:00 my-media systemd[1]: caddy.service: Killing process 5260 (n/a) with signal SIGKILL.
Jun 19 20:30:00 my-media systemd[1]: caddy.service: Killing process 5261 (n/a) with signal SIGKILL.
Jun 19 20:30:00 my-media systemd[1]: caddy.service: Killing process 5265 (n/a) with signal SIGKILL.
Jun 19 20:30:00 my-media systemd[1]: caddy.service: Main process exited, code=killed, status=9/KILL
Jun 19 20:30:00 my-media systemd[1]: caddy.service: Failed with result 'timeout'.
Jun 19 20:30:00 my-media systemd[1]: Stopped Caddy.
-- Boot 3b879133c78248a4957533313e8034f5 --
Jun 19 20:30:20 my-media systemd[1]: Starting Caddy...
Jun 19 20:30:21 my-media caddy[804]: caddy.HomeDir=/var/lib/caddy
Jun 19 20:30:21 my-media caddy[804]: caddy.AppDataDir=/var/lib/caddy/.local/share/caddy
Jun 19 20:30:21 my-media caddy[804]: caddy.AppConfigDir=/var/lib/caddy/.config/caddy
Jun 19 20:30:21 my-media caddy[804]: caddy.ConfigAutosavePath=/var/lib/caddy/.config/caddy/autosave.json
Jun 19 20:30:21 my-media caddy[804]: caddy.Version=v2.8.4 h1:q3pe0wpBj1OcHFZ3n/1nl4V4bxBrYoSoab7rL9BMYNk=
Jun 19 20:30:21 my-media caddy[804]: runtime.GOOS=linux
Jun 19 20:30:21 my-media caddy[804]: runtime.GOARCH=arm64
Jun 19 20:30:21 my-media caddy[804]: runtime.Compiler=gc
Jun 19 20:30:21 my-media caddy[804]: runtime.NumCPU=4
Jun 19 20:30:21 my-media caddy[804]: runtime.GOMAXPROCS=4
Jun 19 20:30:21 my-media caddy[804]: runtime.Version=go1.22.3
Jun 19 20:30:21 my-media caddy[804]: os.Getwd=/
Jun 19 20:30:21 my-media caddy[804]: LANG=C.UTF-8
Jun 19 20:30:21 my-media caddy[804]: PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/snap/bin
Jun 19 20:30:21 my-media caddy[804]: NOTIFY_SOCKET=/run/systemd/notify
Jun 19 20:30:21 my-media caddy[804]: HOME=/var/lib/caddy
Jun 19 20:30:21 my-media caddy[804]: LOGNAME=caddy
Jun 19 20:30:21 my-media caddy[804]: USER=caddy
Jun 19 20:30:21 my-media caddy[804]: INVOCATION_ID=477e272c04784c27bd5730e934e66c56
Jun 19 20:30:21 my-media caddy[804]: JOURNAL_STREAM=8:19909
Jun 19 20:30:21 my-media caddy[804]: SYSTEMD_EXEC_PID=804
Jun 19 20:30:21 my-media caddy[804]: {"level":"info","ts":1718829021.7965825,"msg":"using config from file","file":"/etc/caddy/Caddyfile"}
Jun 19 20:30:21 my-media caddy[804]: {"level":"info","ts":1718829021.8007112,"msg":"adapted config to JSON","adapter":"caddyfile"}
Jun 19 20:30:21 my-media caddy[804]: {"level":"warn","ts":1718829021.8009968,"msg":"Caddyfile input is not formatted; run 'caddy fmt --overwrite' to fix inconsistencies","adapter":"caddyfile","file":"/etc/caddy/Caddyfile","line":2}
Jun 19 20:30:21 my-media caddy[804]: {"level":"info","ts":1718829021.80641,"logger":"admin","msg":"admin endpoint started","address":"localhost:2019","enforce_origin":false,"origins":["//127.0.0.1:2019","//localhost:2019","//[::1]:2019"]}
Jun 19 20:30:21 my-media caddy[804]: {"level":"info","ts":1718829021.8077645,"logger":"tls.cache.maintenance","msg":"started background certificate maintenance","cache":"0x4000508f00"}
Jun 19 20:30:21 my-media caddy[804]: {"level":"info","ts":1718829021.808072,"logger":"http.auto_https","msg":"server is listening only on the HTTPS port but has no TLS connection policies; adding one to enable TLS","server_name":"srv0","https_port":443}
Jun 19 20:30:21 my-media caddy[804]: {"level":"info","ts":1718829021.8080919,"logger":"http.auto_https","msg":"enabling automatic HTTP->HTTPS redirects","server_name":"srv0"}
Jun 19 20:30:21 my-media caddy[804]: {"level":"info","ts":1718829021.8157105,"logger":"crowdsec","msg":"initializing streaming bouncer","instance_id":"93b7bf68"}
Jun 19 20:30:21 my-media caddy[804]: {"level":"info","ts":1718829021.8158858,"logger":"crowdsec","msg":"using API key auth","instance_id":"93b7bf68","address":"http://127.0.0.1:8080/"}
Jun 19 20:30:21 my-media caddy[804]: {"level":"info","ts":1718829021.8160203,"logger":"crowdsec","msg":"started","instance_id":"93b7bf68"}
Jun 19 20:30:21 my-media caddy[804]: {"level":"info","ts":1718829021.816362,"logger":"http","msg":"enabling HTTP/3 listener","addr":":443"}
Jun 19 20:30:21 my-media caddy[804]: {"level":"info","ts":1718829021.821182,"logger":"http.log","msg":"server running","name":"srv0","protocols":["h1","h2","h3"]}
Jun 19 20:30:21 my-media caddy[804]: {"level":"info","ts":1718829021.8212965,"logger":"http.log","msg":"server running","name":"remaining_auto_https_redirects","protocols":["h1","h2","h3"]}
Jun 19 20:30:21 my-media caddy[804]: {"level":"info","ts":1718829021.8213124,"logger":"http","msg":"enabling automatic TLS certificate management","domains":["subdomain2.domain.tld","subdomain3.domain.tld","subdomain4.domain.tld","subdomain5.domain.tld","subdomain6.domain.tld","subdomain7.domain.tld","subdomain1.domain.tld"]}
Jun 19 20:30:21 my-media caddy[804]: {"level":"error","ts":1718829021.8223078,"logger":"crowdsec","msg":"auth-api: auth with api key failed return nil response, error: dial tcp 127.0.0.1:8080: connect: connection refused","instance_id":"93b7bf68","address":"http://127.0.0.1:8080/","error":"auth-api: auth with api key failed return nil response, error: dial tcp 127.0.0.1:8080: connect: connection refused"}
Jun 19 20:30:21 my-media caddy[804]: {"level":"error","ts":1718829021.8225327,"logger":"crowdsec","msg":"failed to connect to LAPI, retrying in 10s: Get \"http://127.0.0.1:8080/v1/decisions/stream?startup=true\": dial tcp 127.0.0.1:8080: connect: connection refused","instance_id":"93b7bf68","address":"http://127.0.0.1:8080/","error":"failed to connect to LAPI, retrying in 10s: Get \"http://127.0.0.1:8080/v1/decisions/stream?startup=true\": dial tcp 127.0.0.1:8080: connect: connection refused"}
Jun 19 20:30:21 my-media caddy[804]: {"level":"info","ts":1718829021.8462546,"logger":"crowdsec","msg":"stopping","instance_id":"93b7bf68"}
Jun 19 20:30:21 my-media caddy[804]: {"level":"info","ts":1718829021.846337,"logger":"crowdsec","msg":"processing new and deleted decisions stopped","instance_id":"93b7bf68"}
Jun 19 20:30:21 my-media caddy[804]: {"level":"info","ts":1718829021.8463519,"logger":"crowdsec","msg":"finished","instance_id":"93b7bf68"}
Jun 19 20:30:21 my-media caddy[804]: {"level":"info","ts":1718829021.8463771,"logger":"tls.cache.maintenance","msg":"stopped background certificate maintenance","cache":"0x4000508f00"}
Jun 19 20:30:21 my-media caddy[804]: {"level":"error","ts":1718829021.8463824,"logger":"tls","msg":"could not clean default/global storage","error":"unable to acquire storage_clean lock: context canceled"}
Jun 19 20:30:21 my-media caddy[804]: {"level":"info","ts":1718829021.8464105,"logger":"tls","msg":"finished cleaning storage units"}
Jun 19 20:30:21 my-media caddy[804]: Error: loading initial config: loading new config: http app module: start: finalizing automatic HTTPS: managing certificates for [subdomain2.domain.tld subdomain3.domain.tld subdomain4.domain.tld subdomain5.domain.tld subdomain6.domain.tld subdomain7.domain.tld subdomain1.domain.tld]: automate: manage [subdomain2.domain.tld subdomain3.domain.tld subdomain4.domain.tld subdomain5.domain.tld subdomain6.domain.tld subdomain7.domain.tld subdomain1.domain.tld]: subdomain4.domain.tld: caching certificate: decoding certificate metadata: unexpected end of JSON input
Jun 19 20:30:21 my-media systemd[1]: caddy.service: Main process exited, code=exited, status=1/FAILURE
Jun 19 20:30:21 my-media systemd[1]: caddy.service: Failed with result 'exit-code'.
Jun 19 20:30:21 my-media systemd[1]: Failed to start Caddy.
-- Boot 895d1ecb3d5f4df4aa78cef01f8640c6 --
Jun 19 20:58:06 my-media systemd[1]: Starting Caddy...
Jun 19 20:58:07 my-media caddy[801]: caddy.HomeDir=/var/lib/caddy
Jun 19 20:58:07 my-media caddy[801]: caddy.AppDataDir=/var/lib/caddy/.local/share/caddy
Jun 19 20:58:07 my-media caddy[801]: caddy.AppConfigDir=/var/lib/caddy/.config/caddy
Jun 19 20:58:07 my-media caddy[801]: caddy.ConfigAutosavePath=/var/lib/caddy/.config/caddy/autosave.json
Jun 19 20:58:07 my-media caddy[801]: caddy.Version=v2.8.4 h1:q3pe0wpBj1OcHFZ3n/1nl4V4bxBrYoSoab7rL9BMYNk=
Jun 19 20:58:07 my-media caddy[801]: runtime.GOOS=linux
Jun 19 20:58:07 my-media caddy[801]: runtime.GOARCH=arm64
Jun 19 20:58:07 my-media caddy[801]: runtime.Compiler=gc
Jun 19 20:58:07 my-media caddy[801]: runtime.NumCPU=4
Jun 19 20:58:07 my-media caddy[801]: runtime.GOMAXPROCS=4
Jun 19 20:58:07 my-media caddy[801]: runtime.Version=go1.22.3
Jun 19 20:58:07 my-media caddy[801]: os.Getwd=/
Jun 19 20:58:07 my-media caddy[801]: LANG=C.UTF-8
Jun 19 20:58:07 my-media caddy[801]: PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/snap/bin
Jun 19 20:58:07 my-media caddy[801]: NOTIFY_SOCKET=/run/systemd/notify
Jun 19 20:58:07 my-media caddy[801]: HOME=/var/lib/caddy
Jun 19 20:58:07 my-media caddy[801]: LOGNAME=caddy
Jun 19 20:58:07 my-media caddy[801]: USER=caddy
Jun 19 20:58:07 my-media caddy[801]: INVOCATION_ID=5f6a90ebf8f04ed098fea8371fc82b75
Jun 19 20:58:07 my-media caddy[801]: JOURNAL_STREAM=8:20321
Jun 19 20:58:07 my-media caddy[801]: SYSTEMD_EXEC_PID=801
Jun 19 20:58:07 my-media caddy[801]: {"level":"info","ts":1718830687.9044018,"msg":"using config from file","file":"/etc/caddy/Caddyfile"}
Jun 19 20:58:07 my-media caddy[801]: {"level":"info","ts":1718830687.9274287,"msg":"adapted config to JSON","adapter":"caddyfile"}
Jun 19 20:58:07 my-media caddy[801]: {"level":"warn","ts":1718830687.9276414,"msg":"Caddyfile input is not formatted; run 'caddy fmt --overwrite' to fix inconsistencies","adapter":"caddyfile","file":"/etc/caddy/Caddyfile","line":2}
Jun 19 20:58:07 my-media caddy[801]: {"level":"info","ts":1718830687.9296575,"logger":"admin","msg":"admin endpoint started","address":"localhost:2019","enforce_origin":false,"origins":["//[::1]:2019","//127.0.0.1:2019","//localhost:2019"]}
Jun 19 20:58:07 my-media caddy[801]: {"level":"info","ts":1718830687.9327786,"logger":"http.auto_https","msg":"server is listening only on the HTTPS port but has no TLS connection policies; adding one to enable TLS","server_name":"srv0","https_port":443}
Jun 19 20:58:07 my-media caddy[801]: {"level":"info","ts":1718830687.9329631,"logger":"http.auto_https","msg":"enabling automatic HTTP->HTTPS redirects","server_name":"srv0"}
Jun 19 20:58:07 my-media caddy[801]: {"level":"info","ts":1718830687.9394069,"logger":"http","msg":"enabling HTTP/3 listener","addr":":443"}
Jun 19 20:58:07 my-media caddy[801]: {"level":"info","ts":1718830687.9396617,"logger":"tls.cache.maintenance","msg":"started background certificate maintenance","cache":"0x40009ab100"}
Jun 19 20:58:07 my-media caddy[801]: {"level":"info","ts":1718830687.9406016,"logger":"http.log","msg":"server running","name":"srv0","protocols":["h1","h2","h3"]}
Jun 19 20:58:07 my-media caddy[801]: {"level":"info","ts":1718830687.9407897,"logger":"http.log","msg":"server running","name":"remaining_auto_https_redirects","protocols":["h1","h2","h3"]}
Jun 19 20:58:07 my-media caddy[801]: {"level":"info","ts":1718830687.9409275,"logger":"http","msg":"enabling automatic TLS certificate management","domains":["subdomain1.domain.tld","subdomain2.domain.tld","subdomain3.domain.tld","subdomain4.domain.tld","subdomain5.domain.tld","subdomain6.domain.tld","subdomain7.domain.tld"]}
Jun 19 20:58:07 my-media caddy[801]: {"level":"info","ts":1718830687.967151,"logger":"tls.cache.maintenance","msg":"stopped background certificate maintenance","cache":"0x40009ab100"}
Jun 19 20:58:07 my-media caddy[801]: Error: loading initial config: loading new config: http app module: start: finalizing automatic HTTPS: managing certificates for [subdomain1.domain.tld subdomain2.domain.tld subdomain3.domain.tld subdomain4.domain.tld subdomain5.domain.tld subdomain6.domain.tld subdomain7.domain.tld]: automate: manage [subdomain1.domain.tld subdomain2.domain.tld subdomain3.domain.tld subdomain4.domain.tld subdomain5.domain.tld subdomain6.domain.tld subdomain7.domain.tld]: subdomain4.domain.tld: caching certificate: decoding certificate metadata: unexpected end of JSON input
Jun 19 20:58:07 my-media systemd[1]: caddy.service: Main process exited, code=exited, status=1/FAILURE
Jun 19 20:58:07 my-media systemd[1]: caddy.service: Failed with result 'exit-code'.
Jun 19 20:58:07 my-media systemd[1]: Failed to start Caddy.
Jun 20 06:08:26 my-media systemd[1]: Starting Caddy...
Jun 20 06:08:26 my-media caddy[223300]: caddy.HomeDir=/var/lib/caddy
Jun 20 06:08:26 my-media caddy[223300]: caddy.AppDataDir=/var/lib/caddy/.local/share/caddy
Jun 20 06:08:26 my-media caddy[223300]: caddy.AppConfigDir=/var/lib/caddy/.config/caddy
Jun 20 06:08:26 my-media caddy[223300]: caddy.ConfigAutosavePath=/var/lib/caddy/.config/caddy/autosave.json
Jun 20 06:08:26 my-media caddy[223300]: caddy.Version=v2.8.4 h1:q3pe0wpBj1OcHFZ3n/1nl4V4bxBrYoSoab7rL9BMYNk=
Jun 20 06:08:26 my-media caddy[223300]: runtime.GOOS=linux
Jun 20 06:08:26 my-media caddy[223300]: runtime.GOARCH=arm64
Jun 20 06:08:26 my-media caddy[223300]: runtime.Compiler=gc
Jun 20 06:08:26 my-media caddy[223300]: runtime.NumCPU=4
Jun 20 06:08:26 my-media caddy[223300]: runtime.GOMAXPROCS=4
Jun 20 06:08:26 my-media caddy[223300]: runtime.Version=go1.22.3
Jun 20 06:08:26 my-media caddy[223300]: os.Getwd=/
Jun 20 06:08:26 my-media caddy[223300]: LANG=C.UTF-8
Jun 20 06:08:26 my-media caddy[223300]: PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/snap/bin
Jun 20 06:08:26 my-media caddy[223300]: NOTIFY_SOCKET=/run/systemd/notify
Jun 20 06:08:26 my-media caddy[223300]: HOME=/var/lib/caddy
Jun 20 06:08:26 my-media caddy[223300]: LOGNAME=caddy
Jun 20 06:08:26 my-media caddy[223300]: USER=caddy
Jun 20 06:08:26 my-media caddy[223300]: INVOCATION_ID=93bcd928593042b28b108e395ee999a8
Jun 20 06:08:26 my-media caddy[223300]: JOURNAL_STREAM=8:913674
Jun 20 06:08:26 my-media caddy[223300]: SYSTEMD_EXEC_PID=223300
Jun 20 06:08:26 my-media caddy[223300]: {"level":"info","ts":1718863706.6501842,"msg":"using config from file","file":"/etc/caddy/Caddyfile"}
Jun 20 06:08:26 my-media caddy[223300]: {"level":"info","ts":1718863706.654139,"msg":"adapted config to JSON","adapter":"caddyfile"}
Jun 20 06:08:26 my-media caddy[223300]: {"level":"warn","ts":1718863706.6541696,"msg":"Caddyfile input is not formatted; run 'caddy fmt --overwrite' to fix inconsistencies","adapter":"caddyfile","file":"/etc/caddy/Caddyfile","line":2}
Jun 20 06:08:26 my-media caddy[223300]: {"level":"info","ts":1718863706.6553712,"logger":"admin","msg":"admin endpoint started","address":"localhost:2019","enforce_origin":false,"origins":["//localhost:2019","//[::1]:2019","//127.0.0.1:2019"]}
Jun 20 06:08:26 my-media caddy[223300]: {"level":"info","ts":1718863706.656039,"logger":"tls.cache.maintenance","msg":"started background certificate maintenance","cache":"0x400049cd00"}
Jun 20 06:08:26 my-media caddy[223300]: {"level":"info","ts":1718863706.6562264,"logger":"http.auto_https","msg":"server is listening only on the HTTPS port but has no TLS connection policies; adding one to enable TLS","server_name":"srv0","https_port":443}
Jun 20 06:08:26 my-media caddy[223300]: {"level":"info","ts":1718863706.6562598,"logger":"http.auto_https","msg":"enabling automatic HTTP->HTTPS redirects","server_name":"srv0"}
Jun 20 06:08:26 my-media caddy[223300]: {"level":"info","ts":1718863706.6571238,"logger":"crowdsec","msg":"initializing streaming bouncer","instance_id":"a81df50e"}
Jun 20 06:08:26 my-media caddy[223300]: {"level":"info","ts":1718863706.6571512,"logger":"crowdsec","msg":"using API key auth","instance_id":"a81df50e","address":"http://127.0.0.1:8080/"}
Jun 20 06:08:26 my-media caddy[223300]: {"level":"info","ts":1718863706.6571677,"logger":"crowdsec","msg":"started","instance_id":"a81df50e"}
Jun 20 06:08:26 my-media caddy[223300]: {"level":"info","ts":1718863706.6574585,"logger":"http","msg":"enabling HTTP/3 listener","addr":":443"}
Jun 20 06:08:26 my-media caddy[223300]: {"level":"info","ts":1718863706.6582403,"logger":"http.log","msg":"server running","name":"srv0","protocols":["h1","h2","h3"]}
Jun 20 06:08:26 my-media caddy[223300]: {"level":"info","ts":1718863706.658282,"logger":"http.log","msg":"server running","name":"remaining_auto_https_redirects","protocols":["h1","h2","h3"]}
Jun 20 06:08:26 my-media caddy[223300]: {"level":"info","ts":1718863706.6582873,"logger":"http","msg":"enabling automatic TLS certificate management","domains":["subdomain2.domain.tld","subdomain3.domain.tld","subdomain4.domain.tld","subdomain5.domain.tld","subdomain6.domain.tld","subdomain7.domain.tld","subdomain1.domain.tld"]}
Jun 20 06:08:26 my-media caddy[223300]: {"level":"info","ts":1718863706.6610878,"logger":"crowdsec","msg":"stopping","instance_id":"a81df50e"}
Jun 20 06:08:26 my-media caddy[223300]: {"level":"info","ts":1718863706.6611419,"logger":"crowdsec","msg":"processing new and deleted decisions stopped","instance_id":"a81df50e"}
Jun 20 06:08:27 my-media caddy[223300]: {"level":"info","ts":1718863707.8231642,"logger":"tls.issuance.acme.acme_client","msg":"got renewal info","names":["subdomain2.domain.tld"],"window_start":1722786976,"window_end":1722959776,"selected_time":1722798392,"recheck_after":1718885307.8231595,"explanation_url":""}
Jun 20 06:08:27 my-media caddy[223300]: {"level":"info","ts":1718863707.8239114,"logger":"tls","msg":"updated ACME renewal information","identifiers":["subdomain2.domain.tld"],"cert_hash":"c0236b1a5f0bb5c588b0d36fa81b3caf4873e08af63a448ff33990dbde7b2180","ari_unique_id":"FC6zF7dYVsuuUAlA5h-vnYsUwsY.BAfOo1qZEDgBd4LbdG7qYt-G","cert_expiry":1725464176,"selected_time":1722822073,"next_update":1718885307.8231595,"explanation_url":""}
Jun 20 06:08:27 my-media caddy[223300]: {"level":"info","ts":1718863707.8674655,"logger":"tls.issuance.acme.acme_client","msg":"got renewal info","names":["subdomain3.domain.tld"],"window_start":1722804055,"window_end":1722976855,"selected_time":1722887496,"recheck_after":1718885307.8674605,"explanation_url":""}
Jun 20 06:08:27 my-media caddy[223300]: {"level":"info","ts":1718863707.8680503,"logger":"tls","msg":"updated ACME renewal information","identifiers":["subdomain3.domain.tld"],"cert_hash":"01902c1dbfb9bcec7401404409d5f7855f6023c183d200bae710afcb3474950c","ari_unique_id":"kydGmAOpUWiOmNbEQkjbI79YlNI.A6Yho7hZyG39yPm4f1yTSCfb","cert_expiry":1725481255,"selected_time":1722810820,"next_update":1718885307.8674605,"explanation_url":""}
Jun 20 06:08:28 my-media caddy[223300]: {"level":"info","ts":1718863708.4119291,"msg":"[INFO][FileStorage:/var/lib/caddy/.local/share/caddy] /var/lib/caddy/.local/share/caddy/locks/storage_clean.lock: Empty lockfile (EOF) - likely previous process crashed or storage medium failure; treating as stale"}
Jun 20 06:08:28 my-media caddy[223300]: {"level":"info","ts":1718863708.4119842,"msg":"[INFO][FileStorage:/var/lib/caddy/.local/share/caddy] Lock for 'storage_clean' is stale (created: 0001-01-01 00:00:00 +0000 UTC, last update: 0001-01-01 00:00:00 +0000 UTC); removing then retrying: /var/lib/caddy/.local/share/caddy/locks/storage_clean.lock"}
Jun 20 06:08:28 my-media caddy[223300]: {"level":"info","ts":1718863708.4164512,"logger":"tls","msg":"cleaning storage unit","storage":"FileStorage:/var/lib/caddy/.local/share/caddy"}
Jun 20 06:08:28 my-media caddy[223300]: {"level":"info","ts":1718863708.4392,"logger":"tls","msg":"finished cleaning storage units"}
Jun 20 06:08:39 my-media caddy[223300]: {"level":"info","ts":1718863719.1348546,"msg":"shutting down apps, then terminating","signal":"SIGTERM"}
Jun 20 06:08:39 my-media caddy[223300]: {"level":"warn","ts":1718863719.1350133,"msg":"exiting; byeee!! 👋","signal":"SIGTERM"}
Jun 20 06:08:44 my-media systemd[1]: caddy.service: State 'stop-sigterm' timed out. Killing.
Jun 20 06:08:44 my-media systemd[1]: caddy.service: Killing process 223300 (caddy) with signal SIGKILL.
Jun 20 06:08:44 my-media systemd[1]: caddy.service: Killing process 223301 (caddy) with signal SIGKILL.
Jun 20 06:08:44 my-media systemd[1]: caddy.service: Killing process 223302 (caddy) with signal SIGKILL.
Jun 20 06:08:44 my-media systemd[1]: caddy.service: Killing process 223303 (caddy) with signal SIGKILL.
Jun 20 06:08:44 my-media systemd[1]: caddy.service: Killing process 223304 (caddy) with signal SIGKILL.
Jun 20 06:08:44 my-media systemd[1]: caddy.service: Killing process 223305 (caddy) with signal SIGKILL.
Jun 20 06:08:44 my-media systemd[1]: caddy.service: Killing process 223306 (caddy) with signal SIGKILL.
Jun 20 06:08:44 my-media systemd[1]: caddy.service: Killing process 223307 (caddy) with signal SIGKILL.
Jun 20 06:08:44 my-media systemd[1]: caddy.service: Killing process 223308 (caddy) with signal SIGKILL.
Jun 20 06:08:44 my-media systemd[1]: caddy.service: Killing process 223309 (caddy) with signal SIGKILL.
Jun 20 06:08:44 my-media systemd[1]: caddy.service: Main process exited, code=killed, status=9/KILL
Jun 20 06:08:44 my-media systemd[1]: caddy.service: Failed with result 'timeout'.
Jun 20 06:08:44 my-media systemd[1]: Stopped Caddy.
-- Boot 7328c5bfb39548ecb3260a09562240de --
Jun 20 06:09:05 my-media systemd[1]: Starting Caddy...
Jun 20 06:09:06 my-media caddy[806]: caddy.HomeDir=/var/lib/caddy
Jun 20 06:09:06 my-media caddy[806]: caddy.AppDataDir=/var/lib/caddy/.local/share/caddy
Jun 20 06:09:06 my-media caddy[806]: caddy.AppConfigDir=/var/lib/caddy/.config/caddy
Jun 20 06:09:06 my-media caddy[806]: caddy.ConfigAutosavePath=/var/lib/caddy/.config/caddy/autosave.json
Jun 20 06:09:06 my-media caddy[806]: caddy.Version=v2.8.4 h1:q3pe0wpBj1OcHFZ3n/1nl4V4bxBrYoSoab7rL9BMYNk=
Jun 20 06:09:06 my-media caddy[806]: runtime.GOOS=linux
Jun 20 06:09:06 my-media caddy[806]: runtime.GOARCH=arm64
Jun 20 06:09:06 my-media caddy[806]: runtime.Compiler=gc
Jun 20 06:09:06 my-media caddy[806]: runtime.NumCPU=4
Jun 20 06:09:06 my-media caddy[806]: runtime.GOMAXPROCS=4
Jun 20 06:09:06 my-media caddy[806]: runtime.Version=go1.22.3
Jun 20 06:09:06 my-media caddy[806]: os.Getwd=/
Jun 20 06:09:06 my-media caddy[806]: LANG=C.UTF-8
Jun 20 06:09:06 my-media caddy[806]: PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/snap/bin
Jun 20 06:09:06 my-media caddy[806]: NOTIFY_SOCKET=/run/systemd/notify
Jun 20 06:09:06 my-media caddy[806]: HOME=/var/lib/caddy
Jun 20 06:09:06 my-media caddy[806]: LOGNAME=caddy
Jun 20 06:09:06 my-media caddy[806]: USER=caddy
Jun 20 06:09:06 my-media caddy[806]: INVOCATION_ID=06aa9e75ed0f4258a7093dfdda9d48f5
Jun 20 06:09:06 my-media caddy[806]: JOURNAL_STREAM=8:20428
Jun 20 06:09:06 my-media caddy[806]: SYSTEMD_EXEC_PID=806
Jun 20 06:09:06 my-media caddy[806]: {"level":"info","ts":1718863746.6911073,"msg":"using config from file","file":"/etc/caddy/Caddyfile"}
Jun 20 06:09:06 my-media caddy[806]: {"level":"info","ts":1718863746.7021832,"msg":"adapted config to JSON","adapter":"caddyfile"}
Jun 20 06:09:06 my-media caddy[806]: {"level":"warn","ts":1718863746.7024295,"msg":"Caddyfile input is not formatted; run 'caddy fmt --overwrite' to fix inconsistencies","adapter":"caddyfile","file":"/etc/caddy/Caddyfile","line":2}
Jun 20 06:09:06 my-media caddy[806]: {"level":"info","ts":1718863746.7088192,"logger":"admin","msg":"admin endpoint started","address":"localhost:2019","enforce_origin":false,"origins":["//localhost:2019","//[::1]:2019","//127.0.0.1:2019"]}
Jun 20 06:09:06 my-media caddy[806]: {"level":"info","ts":1718863746.709976,"logger":"http.auto_https","msg":"server is listening only on the HTTPS port but has no TLS connection policies; adding one to enable TLS","server_name":"srv0","https_port":443}
Jun 20 06:09:06 my-media caddy[806]: {"level":"info","ts":1718863746.710592,"logger":"http.auto_https","msg":"enabling automatic HTTP->HTTPS redirects","server_name":"srv0"}
Jun 20 06:09:06 my-media caddy[806]: {"level":"info","ts":1718863746.7138646,"logger":"crowdsec","msg":"initializing streaming bouncer","instance_id":"4cf761d2"}
Jun 20 06:09:06 my-media caddy[806]: {"level":"info","ts":1718863746.7143095,"logger":"crowdsec","msg":"using API key auth","instance_id":"4cf761d2","address":"http://127.0.0.1:8080/"}
Jun 20 06:09:06 my-media caddy[806]: {"level":"info","ts":1718863746.7144635,"logger":"crowdsec","msg":"started","instance_id":"4cf761d2"}
Jun 20 06:09:06 my-media caddy[806]: {"level":"info","ts":1718863746.7151191,"logger":"http.log","msg":"server running","name":"remaining_auto_https_redirects","protocols":["h1","h2","h3"]}
Jun 20 06:09:06 my-media caddy[806]: {"level":"info","ts":1718863746.715351,"logger":"http","msg":"enabling HTTP/3 listener","addr":":443"}
Jun 20 06:09:06 my-media caddy[806]: {"level":"info","ts":1718863746.7111907,"logger":"tls.cache.maintenance","msg":"started background certificate maintenance","cache":"0x4000487a00"}
Jun 20 06:09:06 my-media caddy[806]: {"level":"error","ts":1718863746.7183206,"logger":"crowdsec","msg":"auth-api: auth with api key failed return nil response, error: dial tcp 127.0.0.1:8080: connect: connection refused","instance_id":"4cf761d2","address":"http://127.0.0.1:8080/","error":"auth-api: auth with api key failed return nil response, error: dial tcp 127.0.0.1:8080: connect: connection refused"}
Jun 20 06:09:06 my-media caddy[806]: {"level":"error","ts":1718863746.718376,"logger":"crowdsec","msg":"failed to connect to LAPI, retrying in 10s: Get \"http://127.0.0.1:8080/v1/decisions/stream?startup=true\": dial tcp 127.0.0.1:8080: connect: connection refused","instance_id":"4cf761d2","address":"http://127.0.0.1:8080/","error":"failed to connect to LAPI, retrying in 10s: Get \"http://127.0.0.1:8080/v1/decisions/stream?startup=true\": dial tcp 127.0.0.1:8080: connect: connection refused"}
Jun 20 06:09:06 my-media caddy[806]: {"level":"info","ts":1718863746.7250314,"logger":"http.log","msg":"server running","name":"srv0","protocols":["h1","h2","h3"]}
Jun 20 06:09:06 my-media caddy[806]: {"level":"info","ts":1718863746.7252622,"logger":"http","msg":"enabling automatic TLS certificate management","domains":["subdomain1.domain.tld","subdomain2.domain.tld","subdomain3.domain.tld","subdomain4.domain.tld","subdomain5.domain.tld","subdomain6.domain.tld","subdomain7.domain.tld"]}
Jun 20 06:09:06 my-media caddy[806]: {"level":"info","ts":1718863746.7354658,"logger":"tls","msg":"storage cleaning happened too recently; skipping for now","storage":"FileStorage:/var/lib/caddy/.local/share/caddy","instance":"fe7c124c-05c9-4f61-b066-59eb8ad690e3","try_again":1718950146.7354622,"try_again_in":86399.99999932}
Jun 20 06:09:06 my-media caddy[806]: {"level":"info","ts":1718863746.7357516,"logger":"tls","msg":"finished cleaning storage units"}
Jun 20 06:09:06 my-media caddy[806]: {"level":"info","ts":1718863746.7859063,"logger":"crowdsec","msg":"stopping","instance_id":"4cf761d2"}
Jun 20 06:09:06 my-media caddy[806]: {"level":"info","ts":1718863746.786194,"logger":"crowdsec","msg":"processing new and deleted decisions stopped","instance_id":"4cf761d2"}
Jun 20 06:09:06 my-media caddy[806]: {"level":"info","ts":1718863746.7862115,"logger":"crowdsec","msg":"finished","instance_id":"4cf761d2"}
Jun 20 06:09:06 my-media caddy[806]: {"level":"info","ts":1718863746.7862403,"logger":"tls.cache.maintenance","msg":"stopped background certificate maintenance","cache":"0x4000487a00"}
Jun 20 06:09:06 my-media caddy[806]: Error: loading initial config: loading new config: http app module: start: finalizing automatic HTTPS: managing certificates for [subdomain1.domain.tld subdomain2.domain.tld subdomain3.domain.tld subdomain4.domain.tld subdomain5.domain.tld subdomain6.domain.tld subdomain7.domain.tld]: automate: manage [subdomain1.domain.tld subdomain2.domain.tld subdomain3.domain.tld subdomain4.domain.tld subdomain5.domain.tld subdomain6.domain.tld subdomain7.domain.tld]: subdomain4.domain.tld: caching certificate: decoding certificate metadata: unexpected end of JSON input
Jun 20 06:09:06 my-media systemd[1]: caddy.service: Main process exited, code=exited, status=1/FAILURE
Jun 20 06:09:06 my-media systemd[1]: caddy.service: Failed with result 'exit-code'.
Jun 20 06:09:06 my-media systemd[1]: Failed to start Caddy.
-- Boot 31afa80d57244314987f2baeec7883f2 --
Jun 20 06:32:02 my-media systemd[1]: Starting Caddy...
Jun 20 06:32:04 my-media caddy[807]: caddy.HomeDir=/var/lib/caddy
Jun 20 06:32:04 my-media caddy[807]: caddy.AppDataDir=/var/lib/caddy/.local/share/caddy
Jun 20 06:32:04 my-media caddy[807]: caddy.AppConfigDir=/var/lib/caddy/.config/caddy
Jun 20 06:32:04 my-media caddy[807]: caddy.ConfigAutosavePath=/var/lib/caddy/.config/caddy/autosave.json
Jun 20 06:32:04 my-media caddy[807]: caddy.Version=v2.8.4 h1:q3pe0wpBj1OcHFZ3n/1nl4V4bxBrYoSoab7rL9BMYNk=
Jun 20 06:32:04 my-media caddy[807]: runtime.GOOS=linux
Jun 20 06:32:04 my-media caddy[807]: runtime.GOARCH=arm64
Jun 20 06:32:04 my-media caddy[807]: runtime.Compiler=gc
Jun 20 06:32:04 my-media caddy[807]: runtime.NumCPU=4
Jun 20 06:32:04 my-media caddy[807]: runtime.GOMAXPROCS=4
Jun 20 06:32:04 my-media caddy[807]: runtime.Version=go1.22.3
Jun 20 06:32:04 my-media caddy[807]: os.Getwd=/
Jun 20 06:32:04 my-media caddy[807]: LANG=C.UTF-8
Jun 20 06:32:04 my-media caddy[807]: PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/snap/bin
Jun 20 06:32:04 my-media caddy[807]: NOTIFY_SOCKET=/run/systemd/notify
Jun 20 06:32:04 my-media caddy[807]: HOME=/var/lib/caddy
Jun 20 06:32:04 my-media caddy[807]: LOGNAME=caddy
Jun 20 06:32:04 my-media caddy[807]: USER=caddy
Jun 20 06:32:04 my-media caddy[807]: INVOCATION_ID=c06ace22d1fc492b8348528eae4e49d9
Jun 20 06:32:04 my-media caddy[807]: JOURNAL_STREAM=8:17373
Jun 20 06:32:04 my-media caddy[807]: SYSTEMD_EXEC_PID=807
Jun 20 06:32:04 my-media caddy[807]: {"level":"info","ts":1718865124.5426047,"msg":"using config from file","file":"/etc/caddy/Caddyfile"}
Jun 20 06:32:04 my-media caddy[807]: {"level":"info","ts":1718865124.5573823,"msg":"adapted config to JSON","adapter":"caddyfile"}
Jun 20 06:32:04 my-media caddy[807]: {"level":"warn","ts":1718865124.5574307,"msg":"Caddyfile input is not formatted; run 'caddy fmt --overwrite' to fix inconsistencies","adapter":"caddyfile","file":"/etc/caddy/Caddyfile","line":2}
Jun 20 06:32:04 my-media caddy[807]: {"level":"info","ts":1718865124.560594,"logger":"admin","msg":"admin endpoint started","address":"localhost:2019","enforce_origin":false,"origins":["//localhost:2019","//[::1]:2019","//127.0.0.1:2019"]}
Jun 20 06:32:04 my-media caddy[807]: {"level":"info","ts":1718865124.5610535,"logger":"http.auto_https","msg":"server is listening only on the HTTPS port but has no TLS connection policies; adding one to enable TLS","server_name":"srv0","https_port":443}
Jun 20 06:32:04 my-media caddy[807]: {"level":"info","ts":1718865124.5616932,"logger":"http.auto_https","msg":"enabling automatic HTTP->HTTPS redirects","server_name":"srv0"}
Jun 20 06:32:04 my-media caddy[807]: {"level":"info","ts":1718865124.5634298,"logger":"http","msg":"enabling HTTP/3 listener","addr":":443"}
Jun 20 06:32:04 my-media caddy[807]: {"level":"info","ts":1718865124.5635078,"logger":"tls.cache.maintenance","msg":"started background certificate maintenance","cache":"0x40000a2780"}
Jun 20 06:32:04 my-media caddy[807]: {"level":"info","ts":1718865124.5644627,"logger":"http.log","msg":"server running","name":"srv0","protocols":["h1","h2","h3"]}
Jun 20 06:32:04 my-media caddy[807]: {"level":"info","ts":1718865124.5736084,"logger":"http.log","msg":"server running","name":"remaining_auto_https_redirects","protocols":["h1","h2","h3"]}
Jun 20 06:32:04 my-media caddy[807]: {"level":"info","ts":1718865124.5738313,"logger":"http","msg":"enabling automatic TLS certificate management","domains":["subdomain7.domain.tld","subdomain1.domain.tld","subdomain2.domain.tld","subdomain3.domain.tld","subdomain4.domain.tld","subdomain5.domain.tld","subdomain6.domain.tld"]}
Jun 20 06:32:04 my-media caddy[807]: {"level":"info","ts":1718865124.6537483,"logger":"tls.cache.maintenance","msg":"stopped background certificate maintenance","cache":"0x40000a2780"}
Jun 20 06:32:04 my-media caddy[807]: {"level":"warn","ts":1718865124.6537678,"logger":"tls.issuance.acme.acme_client","msg":"HTTP request failed; retrying","url":"https://acme-v02.api.letsencrypt.org/directory","error":"performing request: Get \"https://acme-v02.api.letsencrypt.org/directory\": context canceled"}
Jun 20 06:32:04 my-media caddy[807]: {"level":"error","ts":1718865124.6538122,"logger":"tls","msg":"failed updating renewal info from ACME CA","identifiers":["subdomain7.domain.tld"],"cert_hash":"633341a4295e1d5821f8a4b0475d3017ceaa0df44eab6be6401fdbd4d4d3e39f","ari_unique_id":"nytfzzwhT50Et-0rLMTGcIvS1w0.BGJVgDTyZOD77LcXd2hFJ1Xd","cert_expiry":1725651685,"issuer":"acme-v02.api.letsencrypt.org-directory","error":"provisioning client: context canceled"}
Jun 20 06:32:04 my-media caddy[807]: {"level":"warn","ts":1718865124.6538508,"logger":"tls.issuance.acme.acme_client","msg":"HTTP request failed; retrying","url":"https://acme.zerossl.com/v2/DV90","error":"performing request: Get \"https://acme.zerossl.com/v2/DV90\": context canceled"}
Jun 20 06:32:04 my-media caddy[807]: {"level":"error","ts":1718865124.653858,"logger":"tls","msg":"failed updating renewal info from ACME CA","identifiers":["subdomain7.domain.tld"],"cert_hash":"633341a4295e1d5821f8a4b0475d3017ceaa0df44eab6be6401fdbd4d4d3e39f","ari_unique_id":"nytfzzwhT50Et-0rLMTGcIvS1w0.BGJVgDTyZOD77LcXd2hFJ1Xd","cert_expiry":1725651685,"issuer":"acme.zerossl.com-v2-DV90","error":"provisioning client: context canceled"}
Jun 20 06:32:04 my-media caddy[807]: {"level":"warn","ts":1718865124.6538656,"logger":"tls","msg":"ARI window or selected renewal time changed","identifiers":["subdomain7.domain.tld"],"cert_hash":"633341a4295e1d5821f8a4b0475d3017ceaa0df44eab6be6401fdbd4d4d3e39f","ari_unique_id":"nytfzzwhT50Et-0rLMTGcIvS1w0.BGJVgDTyZOD77LcXd2hFJ1Xd","cert_expiry":1725651685,"prev_start":1722974485,"next_start":-6795364578.8713455,"prev_end":1723147285,"next_end":-6795364578.8713455,"prev_selected_time":1723024176,"next_selected_time":-6795364578.8713455,"explanation_url":""}
Jun 20 06:32:04 my-media caddy[807]: {"level":"error","ts":1718865124.653871,"logger":"tls","msg":"updating ARI upon managing","error":"could not fully update ACME renewal info: either no issuer supporting ARI is configured for certificate, or all such failed (make sure the ACME CA that issued the certificate is configured)"}
Jun 20 06:32:04 my-media caddy[807]: Error: loading initial config: loading new config: http app module: start: finalizing automatic HTTPS: managing certificates for [subdomain7.domain.tld subdomain1.domain.tld subdomain2.domain.tld subdomain3.domain.tld subdomain4.domain.tld subdomain5.domain.tld subdomain6.domain.tld]: automate: manage [subdomain7.domain.tld subdomain1.domain.tld subdomain2.domain.tld subdomain3.domain.tld subdomain4.domain.tld subdomain5.domain.tld subdomain6.domain.tld]: subdomain4.domain.tld: caching certificate: decoding certificate metadata: unexpected end of JSON input
Jun 20 06:32:04 my-media systemd[1]: caddy.service: Main process exited, code=exited, status=1/FAILURE
Jun 20 06:32:04 my-media systemd[1]: caddy.service: Failed with result 'exit-code'.
Jun 20 06:32:04 my-media systemd[1]: Failed to start Caddy.

3. Caddy version:

v2.8.4 h1:q3pe0wpBj1OcHFZ3n/1nl4V4bxBrYoSoab7rL9BMYNk=

4. How I installed and ran Caddy:

a. System environment:

Ubuntu 22.04.4 LTS on Oracle Cloud with Oracle Cloud Ampere CPU (free tier), Docker Engine Community version 26.1.3, Docker Compose version 2.27.0

b. Command:

sudo apt install -y debian-keyring debian-archive-keyring apt-transport-https curl
curl -1sLf 'https://dl.cloudsmith.io/public/caddy/stable/gpg.key' | sudo gpg --dearmor -o /usr/share/keyrings/caddy-stable-archive-keyring.gpg
curl -1sLf 'https://dl.cloudsmith.io/public/caddy/stable/debian.deb.txt' | sudo tee /etc/apt/sources.list.d/caddy-stable.list
sudo apt update
sudo apt install caddy

c. Service/unit/compose file:

d. My complete Caddy config:

{
acme_dns cloudflare cloudflare_api_token
email email_address@to.use

crowdsec {
    api_key crowdsec_api_key
  }
}

subdomain1.domain.com {
        route {
                crowdsec
    }
        reverse_proxy 10.0.0.58:2283
}

subdomain2.domain.com {
        route {
                crowdsec
    }
        reverse_proxy 10.0.0.58:4000
}

subdomain3.domain.com {
        route {
                crowdsec
    }
        reverse_proxy 10.0.0.58:8096
}

subdomain4.domain.com {
        route {
                crowdsec
    }
        reverse_proxy 10.0.0.58:5001
}

subdomain5.domain.com {
        route {
                crowdsec
    }
        reverse_proxy 10.0.0.58:3001
}

subdomain6.domain.com {
        route {
                crowdsec
    }
        reverse_proxy 10.0.0.58:3010
}

subdomain7.domain.com {
#        route {
#                crowdsec
#    }
        reverse_proxy 10.0.0.58:3000
}

#subdomain8.domain.com {
#        route {
#                crowdsec
#    }
#        reverse_proxy 10.0.0.58:8945
#}

5. Links to relevant resources:

3

Looks like when Caddy tried to read its data from storage, it encountered bad data, so it couldn’t start.

You can wipe out Caddy’s storage to start fresh, then reboot Caddy. By default, Caddy’s storage is at /var/lib/caddy/.local/share/caddy.

1 Like
4

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.