V2: can't use local CA certs, storage misconfigured

skyfaller · May 7, 2020, 3:06pm

1. Caddy version (`caddy version`):

v2.0.0 h1:pQSaIJGFluFvu8KDGDODV8u4/QRED/OPyIR+MWYYse8=

2. How I run Caddy:

a. System environment:

Arch Linux, Linux 5.4.10-x86_64-linode132, x86-64. Running in a Linode “nanode” VM. Manually installed binary https://github.com/caddyserver/caddy/releases/download/v2.0.0/caddy_2.0.0_linux_amd64.tar.gz into /usr/local/bin/caddy

b. Command:

/usr/local/bin/caddy run --environ --config /etc/caddy/Caddyfile

c. Service/unit/compose file:

Created using Ansible+Jinja2: devops/caddy.service.j2 at master · sunrisemovement/devops · GitHub

# caddy.service
#
# For using Caddy with a config file.
#
# Make sure the ExecStart and ExecReload commands are correct
# for your installation.
#
# See https://caddyserver.com/docs/install for instructions.
#
# WARNING: This service does not use the --resume flag, so if you
# use the API to make changes, they will be overwritten by the
# Caddyfile next time the service is restarted. If you intend to
# use Caddy's API to configure it, add the --resume flag to the
# `caddy run` command or use the caddy-api.service file instead.

[Unit]
Description=Caddy
Documentation=https://caddyserver.com/docs/
After=network.target

[Service]
; User and group the process will run as.
User=http
Group=http
; caddy command assumes the caddyfile adapter if filename starts with Caddyfile
ExecStart=/usr/local/bin/caddy run --environ --config /etc/caddy/Caddyfile
ExecReload=/usr/local/bin/caddy reload --config /etc/caddy/Caddyfile
TimeoutStopSec=5s
; Raise limit for file descriptors; see `man systemd.exec` for more limit settings.
LimitNOFILE=1048576
LimitNPROC=512
PrivateTmp=true
ProtectSystem=full
; Note that you may have to add capabilities required by any plugins in use.
AmbientCapabilities=CAP_NET_BIND_SERVICE

[Install]
WantedBy=multi-user.target

d. My complete Caddyfile or JSON config:

This is a Caddyfile for testing using a local CA: devops/LocalCaddyfile.j2 at master · sunrisemovement/devops · GitHub

# global options block: https://caddyserver.com/docs/caddyfile/options
{
	storage file_system {
		root	/etc/caddy/storage
	}
	experimental_http3
	local_certs
}

# reusable snippets: https://caddyserver.com/docs/caddyfile/concepts#snippets
(boilerplate) {
	encode gzip zstd
	file_server
}

# start site blocks

# public test page
brockovich.sunrisemovement.dev {
	root * /srv/sunrisemovement.dev/www/public/
	import boilerplate
	# https://caddyserver.com/docs/caddyfile/directives/templates
	# can't use default double curly braces b/c Ansible uses them
	templates {
		between (( ))
	}
	tls	{{ admin_email }}
}

# local test page
www.sunrisemovement.dev {
	root * /srv/sunrisemovement.dev/www/public/
	import boilerplate
	templates {
		between (( ))
	}
}
# redirect no-www to www
sunrisemovement.dev {
	redir https://www.sunrisemovement.dev
}

# handcoded Rhode Island site
ri.sunrisemovement.dev {
	root * /srv/sunrisemovement.dev/ri/public
	import boilerplate
	php_fastcgi unix/{{ php_fpm_unix_socket }}
}

# Hugo SF Bay website
sfbay.sunrisemovement.org {
	root * /srv/sunrisemovement.org/sfbay/public
	import boilerplate
}

# Hugo sites
philadelphia.sunrisemovement.dev {
	root * /srv/sunrisemovement.dev/philadelphia/public
	import boilerplate
}

houston.sunrisemovement.dev {
	root * /srv/sunrisemovement.dev/houston/public
	import boilerplate
}

3. The problem I’m having:

When I try to view a site using my local CA, Firefox tells me “The certificate for www.sunrisemovement.dev expired on 5/6/2020.” This doesn’t seem right, since when I view the certificate, the Validity section says Not Before 5/3/2020, 2:42:26 AM (Eastern Daylight Time) and Not After 5/10/2020, 2:42:26 AM (Eastern Daylight Time).

I suspect this is some sort of problem with my Caddy storage, since when I sudo journalctl --boot -u caddy.service I get a bunch of failed cert renewals and failed storage checks. I don’t see anything wrong with the permissions on the relevant directories, however, so I don’t know what to do next.

I believe the problem started when I upgraded from the release candidate to the stable v2 release.

4. Error messages and/or full log output:

Representative sections of journalctl log:

May 06 15:29:33 brockovich systemd[1]: Reloaded Caddy.
May 06 15:33:10 brockovich caddy[365478]: {"level":"info","ts":1588793590.6481147,"msg":"shutting down apps then terminating","signal":"SIGTERM"}
May 06 15:33:10 brockovich caddy[365478]: 2020/05/06 15:33:10 [DEBUG] Fake-closing underlying packet conn
May 06 15:33:10 brockovich caddy[365478]: 2020/05/06 15:33:10 [INFO][cache:0xc0007c2690] Stopped certificate maintenance routine
May 06 15:33:10 brockovich caddy[365478]: {"level":"info","ts":1588793590.6485255,"logger":"admin","msg":"stopped previous server"}
May 06 15:33:10 brockovich caddy[365478]: {"level":"info","ts":1588793590.648537,"msg":"shutdown done","signal":"SIGTERM"}
May 06 15:33:10 brockovich systemd[1]: Stopping Caddy...
May 06 15:33:10 brockovich systemd[1]: caddy.service: Succeeded.
May 06 15:33:10 brockovich systemd[1]: Stopped Caddy.
May 06 15:33:10 brockovich systemd[1]: Started Caddy.
May 06 15:33:10 brockovich caddy[366379]: caddy.HomeDir=/srv/http
May 06 15:33:10 brockovich caddy[366379]: caddy.AppDataDir=/srv/http/.local/share/caddy
May 06 15:33:10 brockovich caddy[366379]: caddy.AppConfigDir=/srv/http/.config/caddy
May 06 15:33:10 brockovich caddy[366379]: caddy.ConfigAutosavePath=/srv/http/.config/caddy/autosave.json
May 06 15:33:10 brockovich caddy[366379]: runtime.GOOS=linux
May 06 15:33:10 brockovich caddy[366379]: runtime.GOARCH=amd64
May 06 15:33:10 brockovich caddy[366379]: runtime.Compiler=gc
May 06 15:33:10 brockovich caddy[366379]: runtime.NumCPU=1
May 06 15:33:10 brockovich caddy[366379]: runtime.GOMAXPROCS=1
May 06 15:33:10 brockovich caddy[366379]: runtime.Version=go1.14.2
May 06 15:33:10 brockovich caddy[366379]: os.Getwd=/
May 06 15:33:10 brockovich caddy[366379]: LANG=en_US.UTF-8
May 06 15:33:10 brockovich caddy[366379]: PATH=/usr/local/sbin:/usr/local/bin:/usr/bin
May 06 15:33:10 brockovich caddy[366379]: HOME=/srv/http
May 06 15:33:10 brockovich caddy[366379]: LOGNAME=http
May 06 15:33:10 brockovich caddy[366379]: USER=http
May 06 15:33:10 brockovich caddy[366379]: INVOCATION_ID=ac0508753c25480788dcf086d1624a8d
May 06 15:33:10 brockovich caddy[366379]: JOURNAL_STREAM=9:17654494
May 06 15:33:10 brockovich caddy[366379]: {"level":"info","ts":1588793590.7335124,"msg":"using provided configuration","config_file":"/etc/caddy/Caddyfile","config_adapter":""}
May 06 15:33:10 brockovich caddy[366379]: {"level":"info","ts":1588793590.7390516,"logger":"admin","msg":"admin endpoint started","address":"tcp/localhost:2019","enforce_origin":false,"origins":["localhost:2019","[::1]:2019","127.0.0.1:2019"]}
May 06 15:33:10 brockovich caddy[366379]: 2020/05/06 15:33:10 [INFO][cache:0xc000394730] Started certificate maintenance routine
May 06 15:33:10 brockovich caddy[366379]: {"level":"info","ts":1588793590.7511714,"logger":"http","msg":"enabling automatic HTTP->HTTPS redirects","server_name":"srv0"}
May 06 15:33:10 brockovich caddy[366379]: {"level":"warn","ts":1588793590.8411036,"logger":"pki.ca.local","msg":"installing root certificate (you might be prompted for password)","path":"storage:pki/authorities/local/root.crt"}
May 06 15:33:10 brockovich caddy[366379]: 2020/05/06 15:33:10 Warning: "certutil" is not available, install "certutil" with "apt install libnss3-tools" or "yum install nss-tools" and try again
May 06 15:33:10 brockovich caddy[366379]: 2020/05/06 15:33:10 define JAVA_HOME environment variable to use the Java trust
May 06 15:33:10 brockovich sudo[366386]: pam_unix(sudo:auth): conversation failed
May 06 15:33:10 brockovich sudo[366386]: pam_unix(sudo:auth): auth could not identify password for [http]
May 06 15:33:10 brockovich sudo[366386]:     http : user NOT in sudoers ; TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/usr/bin/tee /etc/ca-certificates/trust-source/anchors/Caddy_Local_Authority_-_2020_ECC_Root_23010221280656044929999170162283841857.crt
May 06 15:33:10 brockovich caddy[366379]: {"level":"error","ts":1588793590.8522694,"logger":"pki.ca.local","msg":"failed to install root certificate","error":"failed to execute sudo: exit status 1","certificate_file":"storage:pki/authorities/local/root.crt"}
May 06 15:33:10 brockovich caddy[366379]: {"level":"info","ts":1588793590.854445,"logger":"tls","msg":"cleaned up storage units"}
May 06 15:33:10 brockovich caddy[366379]: {"level":"info","ts":1588793590.854759,"logger":"http","msg":"enabling experimental HTTP/3 listener","addr":":443"}
May 06 15:33:10 brockovich caddy[366379]: {"level":"info","ts":1588793590.8548963,"logger":"http","msg":"enabling automatic TLS certificate management","domains":["philadelphia.sunrisemovement.dev","brockovich.sunrisemovement.dev","houston.sunrisemovement.dev","sfbay.sunrisemovement.org","www.sunrisemovement.dev","ri.sunrisemovement.dev","sunrisemovement.dev"]}
May 06 15:33:10 brockovich caddy[366379]: 2020/05/06 15:33:10 [WARNING] Stapling OCSP: no OCSP stapling for [philadelphia.sunrisemovement.dev]: no OCSP server specified in certificate
May 06 15:33:10 brockovich caddy[366379]: 2020/05/06 15:33:10 [WARNING] Stapling OCSP: no OCSP stapling for [houston.sunrisemovement.dev]: no OCSP server specified in certificate
May 06 15:33:10 brockovich caddy[366379]: 2020/05/06 15:33:10 [WARNING] Stapling OCSP: no OCSP stapling for [sfbay.sunrisemovement.org]: no OCSP server specified in certificate
May 06 15:33:10 brockovich caddy[366379]: 2020/05/06 15:33:10 [WARNING] Stapling OCSP: no OCSP stapling for [www.sunrisemovement.dev]: no OCSP server specified in certificate
May 06 15:33:10 brockovich caddy[366379]: 2020/05/06 15:33:10 [WARNING] Stapling OCSP: no OCSP stapling for [ri.sunrisemovement.dev]: no OCSP server specified in certificate
May 06 15:33:10 brockovich caddy[366379]: 2020/05/06 15:33:10 [WARNING] Stapling OCSP: no OCSP stapling for [sunrisemovement.dev]: no OCSP server specified in certificate
May 06 15:33:10 brockovich caddy[366379]: {"level":"info","ts":1588793590.8593447,"msg":"autosaved config","file":"/srv/http/.config/caddy/autosave.json"}
May 06 15:33:10 brockovich caddy[366379]: {"level":"info","ts":1588793590.8594263,"msg":"serving initial configuration"}
May 06 15:36:27 brockovich caddy[366379]: 2020/05/06 15:36:27 http: TLS handshake error from 146.88.240.26:50204: no certificate available for '45.33.69.233'
May 06 16:08:20 brockovich caddy[366379]: 2020/05/06 16:08:20 http: TLS handshake error from 128.14.134.170:41550: no certificate available for '45.33.69.233'
May 06 16:33:10 brockovich caddy[366379]: 2020/05/06 16:33:10 [INFO] [sfbay.sunrisemovement.org] Maintenance routine: certificate expires in 3h59m15.249534545s; queueing for renewal
May 06 16:33:10 brockovich caddy[366379]: 2020/05/06 16:33:10 [INFO] [sfbay.sunrisemovement.org] Maintenance routine: attempting renewal with 3h59m15.248586623s remaining
May 06 16:33:10 brockovich caddy[366379]: 2020/05/06 16:33:10 [ERROR] [sfbay.sunrisemovement.org] failed storage check: open /etc/caddy/storage/rw_test_6402366579442096566: read-only file system - storage is probably misconfigured
May 06 16:43:10 brockovich caddy[366379]: 2020/05/06 16:43:10 [INFO] [sfbay.sunrisemovement.org] Maintenance routine: certificate expires in 3h49m15.249134587s; queueing for renewal
May 06 16:43:10 brockovich caddy[366379]: 2020/05/06 16:43:10 [INFO] [sfbay.sunrisemovement.org] Maintenance routine: attempting renewal with 3h49m15.248493066s remaining
May 06 16:43:10 brockovich caddy[366379]: 2020/05/06 16:43:10 [ERROR] [sfbay.sunrisemovement.org] failed storage check: open /etc/caddy/storage/rw_test_511848594996758435: read-only file system - storage is probably misconfigured

May 07 10:32:42 brockovich caddy[366379]: 2020/05/07 10:32:42 http: TLS handshake error from [2601:42:0:6200:a427:f874:5070:1ae]:60785: remote error: tls: expired certificate
May 07 10:33:10 brockovich caddy[366379]: 2020/05/07 10:33:10 [INFO] [sfbay.sunrisemovement.org] Maintenance routine: certificate expires in -14h0m44.752754788s; queueing for renewal
May 07 10:33:10 brockovich caddy[366379]: 2020/05/07 10:33:10 [INFO] [www.sunrisemovement.dev] Maintenance routine: certificate expires in -13h30m44.753367867s; queueing for renewal
May 07 10:33:10 brockovich caddy[366379]: 2020/05/07 10:33:10 [INFO] [ri.sunrisemovement.dev] Maintenance routine: certificate expires in -12h20m44.753481437s; queueing for renewal
May 07 10:33:10 brockovich caddy[366379]: 2020/05/07 10:33:10 [INFO] [sunrisemovement.dev] Maintenance routine: certificate expires in -13h30m44.753575348s; queueing for renewal
May 07 10:33:10 brockovich caddy[366379]: 2020/05/07 10:33:10 [INFO] [philadelphia.sunrisemovement.dev] Maintenance routine: certificate expires in -11h40m44.753679177s; queueing for renewal
May 07 10:33:10 brockovich caddy[366379]: 2020/05/07 10:33:10 [INFO] [houston.sunrisemovement.dev] Maintenance routine: certificate expires in -11h40m44.753780147s; queueing for renewal
May 07 10:33:10 brockovich caddy[366379]: 2020/05/07 10:33:10 [INFO] [sfbay.sunrisemovement.org] Maintenance routine: attempting renewal with -14h0m44.753884827s remaining
May 07 10:33:10 brockovich caddy[366379]: 2020/05/07 10:33:10 [ERROR] [sfbay.sunrisemovement.org] failed storage check: open /etc/caddy/storage/rw_test_1147514851786515967: read-only file system - storage is probably misconfigured
May 07 10:33:10 brockovich caddy[366379]: 2020/05/07 10:33:10 [INFO] [www.sunrisemovement.dev] Maintenance routine: attempting renewal with -13h30m44.754167137s remaining
May 07 10:33:10 brockovich caddy[366379]: 2020/05/07 10:33:10 [ERROR] [www.sunrisemovement.dev] failed storage check: open /etc/caddy/storage/rw_test_6036242988766395678: read-only file system - storage is probably misconfigured
May 07 10:33:10 brockovich caddy[366379]: 2020/05/07 10:33:10 [INFO] [ri.sunrisemovement.dev] Maintenance routine: attempting renewal with -12h20m44.754380087s remaining
May 07 10:33:10 brockovich caddy[366379]: 2020/05/07 10:33:10 [ERROR] [ri.sunrisemovement.dev] failed storage check: open /etc/caddy/storage/rw_test_8962129509835205179: read-only file system - storage is probably misconfigured
May 07 10:33:10 brockovich caddy[366379]: 2020/05/07 10:33:10 [INFO] [sunrisemovement.dev] Maintenance routine: attempting renewal with -13h30m44.754623517s remaining
May 07 10:33:10 brockovich caddy[366379]: 2020/05/07 10:33:10 [ERROR] [sunrisemovement.dev] failed storage check: open /etc/caddy/storage/rw_test_7712519341641703623: read-only file system - storage is probably misconfigured
May 07 10:33:10 brockovich caddy[366379]: 2020/05/07 10:33:10 [INFO] [philadelphia.sunrisemovement.dev] Maintenance routine: attempting renewal with -11h40m44.754818026s remaining
May 07 10:33:10 brockovich caddy[366379]: 2020/05/07 10:33:10 [ERROR] [philadelphia.sunrisemovement.dev] failed storage check: open /etc/caddy/storage/rw_test_128955500673949390: read-only file system - storage is probably misconfigured
May 07 10:33:10 brockovich caddy[366379]: 2020/05/07 10:33:10 [INFO] [houston.sunrisemovement.dev] Maintenance routine: attempting renewal with -11h40m44.755071827s remaining
May 07 10:33:10 brockovich caddy[366379]: 2020/05/07 10:33:10 [ERROR] [houston.sunrisemovement.dev] failed storage check: open /etc/caddy/storage/rw_test_7285203397674450269: read-only file system - storage is probably misconfigured
May 07 10:33:11 brockovich caddy[366379]: 2020/05/07 10:33:11 http: TLS handshake error from [2601:42:0:6200:a427:f874:5070:1ae]:60789: remote error: tls: expired certificate
May 07 10:33:21 brockovich caddy[366379]: 2020/05/07 10:33:21 http: TLS handshake error from [2601:42:0:6200:a427:f874:5070:1ae]:60807: remote error: tls: expired certificate
May 07 10:40:07 brockovich caddy[366379]: 2020/05/07 10:40:07 http: TLS handshake error from 198.108.66.240:22688: no certificate available for '45.33.69.233'

I don’t understand why this is happening, here’s what the permissions on the relevant directories look like:

[nelson@brockovich storage]$ ls -la
total 28
drwxr-xr-x 7 http http 4096 May  6 10:52 .
drwxr-xr-x 3 http http 4096 Apr 29 08:32 ..
drwx------ 3 http http 4096 Apr  2 20:04 acme
drwx------ 4 http http 4096 Apr 10 16:40 certificates
drwx------ 2 http http 4096 May  6 10:52 locks
drwx------ 2 http http 4096 May  6 13:02 ocsp
drwx------ 3 http http 4096 Apr 10 16:40 pki

caddy is running as the http user according to top.

5. What I already tried:

6. Links to relevant resources:

skyfaller · May 7, 2020, 3:51pm

OK, now I have another suspicion. According to Automatic HTTPS — Caddy Documentation “The first time a root key is used, Caddy will try to install it into the system’s local trust store(s). If it does not have permission to do so, it will prompt for a password.” Since I was not actually logged into the server, I upgraded Caddy using Ansible, I don’t think ever saw this password prompt, so I guess the root key was never installed? Not sure what to do about that.

rmoriz · May 7, 2020, 3:58pm

I think ProtectSystem=full prevents writes to /etc: https://www.freedesktop.org/software/systemd/man/systemd.exec.html

skyfaller · May 7, 2020, 4:02pm

AHA! That is one thing that I changed when I updated the service file to the recommended service file. I guess maybe if I used the default storage location this wouldn’t be a problem, I’ll have to try that.

matt · May 7, 2020, 4:06pm

Correct, so, there’s the caddy trust command for unsupervised provisioning environments; since you need privileges to provision a machine, running caddy trust at that time will not prompt for a password, and you won’t need it after that.

Anyway, the storage check seems to have served you well. Caddy won’t try to get a certificate if it can’t successfully store it. We’ve seen this save the day a few times now!

skyfaller · May 7, 2020, 4:14pm

Is there any drawback to simply changing my storage location? The warning in the documentation scares me:

The data directory must not be treated as a cache. Its contents are not ephemeral or merely for the sake of performance. Caddy stores TLS certificates, private keys, OCSP staples, and other necessary information to the data directory. It should not be purged without understanding the implications.

If I attempt to move the directory in the config, won’t that be the same as purging the directory? Do I need to copy the contents to the new directory first, and are there any negative side effects of that?

matt · May 7, 2020, 4:23pm

Nope, just make sure Caddy can read it and write to it.

Yes, and probably not, no. Caddy doesn’t care where the files are, as long as it can read and write them.

skyfaller · May 8, 2020, 6:55pm

Sorry for straying slightly off-topic, but can you go into more detail about exactly what happens if you purge the data directory? I’m wondering in what circumstances I need to preserve the contents.

For example, if I move to a new virtual machine or web host, do I want to bring the contents of that data directory with me? Or does changing operating systems or IP addresses make it undesirable to attempt to use the same data directory contents?

matt · May 8, 2020, 6:57pm

The biggest example is TLS certificates; the data directory is where Caddy stores certificates, so if they aren’t there, Caddy will have to get new certificates from Let’s Encrypt (or whatever the issuer is). Doing that more than necessary can easily run you into rate limits.

skyfaller · May 9, 2020, 1:38am

I noticed the line in my log “http : user NOT in sudoers”. Would it be bad to let caddy’s user sudo? Is it preferable to use caddy trust? I used caddy trust manually and I still see Caddy making the same complaint in my logs.

Whitestrake · May 11, 2020, 12:11am

Principle of least privilege dictates you shouldn’t allow Caddy root access unless it’s absolutely required.

It’s up to you, though. I’ve had my fingers in the code and have a pretty good confidence that Caddy won’t pwn me, so I don’t have any concerns running it as root, which I do in a Docker container.

I believe that log line is probably less of a complaint and more of a note.

system · June 10, 2020, 12:11am

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.