Caddy fails to start after temporary storage exhaustion

Help
1

Hello!

1. The problem I’m having:

My caddy installation worked fine for some time. Last night a had an issue with a backup-job, which filled the disk which also holds caddy 100%. Immediately in the morning I fixed the situation and restarted the server. Caddy now fails to start.

2. Error messages and/or full log output:

Error: loading initial config: loading new config: http app module: start: finalizing automatic HTTPS: managing certificates for [share.fp.on.at web.fp.on.at]: automate: manage [share.fp.on.at web.fp.on.at]: share.fp.on.at: caching certificate: decoding certificate metadata: unexpected end of JSON input
Error: caddy process exited with error: exit status 1

3. Caddy version:

root@webserver:/var/log/caddy # caddy version
v2.8.4

4. How I installed and ran Caddy:

via the FreeBSD package manager “pkg”.

a. System environment:

FreeBSD fractal.fp.on.at 13.1-RELEASE-p9 FreeBSD 13.1-RELEASE-p9 n245431-b8ec9bde091 TRUENAS amd64

Caddy runs in a container (iocage).

b. Command:

service caddy start

d. What I already tried:

The logmessage quite obviously points to a corrupted certificate cache file. I’m only struggling to find where to find that.

I found the directory

root@webserver:/var/log/caddy # ls -lsha /var/db/caddy/data/caddy/
total 16
9 drwx------  6 root  wheel     8B Sep 23 15:55 .
1 drwx------  3 root  wheel     3B Jul 25 16:51 ..
1 drwx------  3 root  wheel     3B Jul 25 16:51 acme
1 drwx------  3 root  wheel     3B Jul 25 16:51 certificates
1 -rw-------  1 root  wheel    36B Jul 25 16:51 instance.uuid
5 -rw-------  1 root  wheel   112B Oct 11 11:24 last_clean.json
1 drwx------  2 root  wheel     2B Oct 12 10:38 locks
1 drwx------  2 root  wheel     4B Oct  9 11:24 ocsp

Should I just delete the contents of “certificates”?

Any advice welcome. Thanks!

2

Yeah you can wipe out Caddy’s storage, it’ll reissue your certs just fine.

We have a known issue with our storage writer (now fixed, in the next release) which can fail to write files completely in some cases.

1 Like
3

This solved the problem, thanks. I wasn’t sure if /var/db/caddy is the right directory containing the files, but it was.

Well in my case the block device didn’t have any capacity left for a brief period of time, so I think it’s not your storage writer to blame. =)

4

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.