Caddy won't reload, unknown field "issuers"

1. Caddy version (caddy version):

v2.3.0 h1:fnrqJLa3G5vfxcxmOH/+kJOcunPLhSBnjgIvjXV/QTA=

2. How I run Caddy:

a. System environment:

Arch Linux on a Linode VPS

systemd 247 (247.2-1-arch)
+PAM +AUDIT -SELINUX -IMA -APPARMOR +SMACK -SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +ZSTD +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=hybrid

b. Command:

/usr/bin/caddy run --environ --config /etc/caddy/Caddyfile

c. Service/unit/compose file:

# caddy.service
#
# For using Caddy with a config file.
#
# Make sure the ExecStart and ExecReload commands are correct
# for your installation.
#
# See https://caddyserver.com/docs/install for instructions.
#
# WARNING: This service does not use the --resume flag, so if you
# use the API to make changes, they will be overwritten by the
# Caddyfile next time the service is restarted. If you intend to
# use Caddy's API to configure it, add the --resume flag to the
# `caddy run` command or use the caddy-api.service file instead.

[Unit]
Description=Caddy webserver
Documentation=https://caddyserver.com/docs/
After=network.target network-online.target
Wants=network-online.target systemd-networkd-wait-online.service
StartLimitIntervalSec=14400
StartLimitBurst=10

[Service]
; User and group the process will run as.
User=caddy
Group=caddy
Environment=XDG_DATA_HOME=/var/lib
Environment=XDG_CONFIG_HOME=/var/lib/caddy/.config

; caddy command assumes the caddyfile adapter if filename starts with Caddyfile
ExecStart=/usr/bin/caddy run --environ --config /etc/caddy/Caddyfile
ExecReload=/usr/bin/caddy reload --config /etc/caddy/Caddyfile

# Do not allow the process to be restarted in a tight loop. If the
# process fails to start, something critical needs to be fixed.
Restart=on-abnormal

# Use graceful shutdown with a reasonable timeout
KillMode=mixed
KillSignal=SIGQUIT
TimeoutStopSec=5s

; Raise limit for file descriptors; see `man systemd.exec` for more limit settings.
LimitNOFILE=1048576
LimitNPROC=512

# Hardening options
PrivateTmp=true
PrivateDevices=true
ProtectHome=true
ProtectSystem=strict
ReadWritePaths=/var/lib/caddy /var/log/caddy
; Note that you may have to add capabilities required by any plugins in use.
CapabilityBoundingSet=CAP_NET_BIND_SERVICE
AmbientCapabilities=CAP_NET_BIND_SERVICE
NoNewPrivileges=true
ProtectKernelTunables=true
ProtectKernelModules=true
ProtectControlGroups=true
LockPersonality=true

[Install]
WantedBy=multi-user.target

d. My complete Caddyfile or JSON config:

# global options block: https://caddyserver.com/docs/caddyfile/options
{
	servers {
		protocol {
			experimental_http3
		}
	}
	email 	{{ admin_email }}
}

# reusable snippets: https://caddyserver.com/docs/caddyfile/concepts#snippets
(boilerplate) {
	encode gzip zstd
	file_server
}

# redirect no-www to www
(redir-to-www) {
	{args.0} {
		redir https://www.{args.0}{uri}
	}
}

# start site blocks

# test page
test.maximumethics.dev {
	root * /srv/maximumethics.dev/test/public/
	import boilerplate
	# https://caddyserver.com/docs/caddyfile/directives/templates
	# can't use default double curly braces b/c Ansible uses them
	templates {
		between (( ))
	}
}

# handcoded Rhode Island site

# Hugo sites
www.maximumethics.dev {
	root * /srv/maximumethics.dev/www/public
	encode gzip zstd
	file_server browse
}
import redir-to-www maximumethics.dev

3. The problem I’m having:

I’m trying to reload Caddy to take in the “new config”, but the only thing I changed in my Caddyfile was fixing the “experimental_http3” global option which was deprecated, and removing that entirely doesn’t change anything.

4. Error messages and/or full log output:

Here’s a relevant section of my systemd journal journalctl -u caddy:

Jan 12 18:40:20 america systemd[1]: Reloading Caddy webserver.
Jan 12 18:40:20 america caddy[1163664]: {"level":"info","ts":1610494820.195494,"msg":"using provided configuration","config_file":"/etc/caddy/Caddyfile","config_adapter":""}
Jan 12 18:40:20 america caddy[201731]: {"level":"info","ts":1610494820.1994202,"logger":"admin.api","msg":"received request","method":"POST","host":"localhost:2019","uri":"/load","remote_addr":"127.0.0.1:39972","headers":{"Accept-Encoding":["gzip"],"Content-Length":["1222"],"Content-Type":["application/json"],"Origin":["localhost:2019"],"User-Agent":["Go-http-client/1.1"]}}
Jan 12 18:40:20 america caddy[201731]: {"level":"info","ts":1610494820.2002597,"logger":"admin","msg":"admin endpoint started","address":"tcp/localhost:2019","enforce_origin":false,"origins":["localhost:2019","[::1]:2019","127.0.0.1:2019"]}
Jan 12 18:40:20 america caddy[201731]: {"level":"error","ts":1610494820.200391,"logger":"admin.api","msg":"request error","error":"loading config: loading new config: loading tls app module: decoding module config: tls: json: unknown field \"issuers\"","status_code":400}
Jan 12 18:40:20 america caddy[1163664]: reload: sending configuration to instance: caddy responded with error: HTTP 400: {"error":"loading config: loading new config: loading tls app module: decoding module config: tls: json: unknown field \"issuers\""}
Jan 12 18:40:20 america systemd[1]: caddy.service: Control process exited, code=exited, status=1/FAILURE
Jan 12 18:40:20 america systemd[1]: Reload failed for Caddy webserver.
Jan 12 18:40:20 america caddy[201731]: {"level":"info","ts":1610494820.7006083,"logger":"admin","msg":"stopped previous server"}

5. What I already tried:

I have no idea where to start, I searched my Caddyfile and other config files for the field “issuers” and had no joy. What is that field “issuers” and where is it coming from?

What is the last caddy version in your logs? (From the --environ flag)

I believe the last time I started Caddy instead of reloading it was using Caddy v2.2.1.

EDIT: Oh, is that the problem? That I updated the binary to 2.3.0, but never restarted the server, so it’s still using an older binary? I need to rethink my sysadmin practices.

Oct 14 15:22:05 america caddy[187616]: {"level":"info","ts":1602703325.0678995,"msg":"quitting process immediately","signal":"SIGQUIT"}
Oct 14 15:22:05 america systemd[1]: Stopping Caddy webserver...
Oct 14 15:22:05 america systemd[1]: caddy.service: Main process exited, code=exited, status=2/INVALIDARGUMENT
Oct 14 15:22:05 america systemd[1]: caddy.service: Failed with result 'exit-code'.
Oct 14 15:22:05 america systemd[1]: Stopped Caddy webserver.
Oct 14 15:22:05 america systemd[1]: Started Caddy webserver.
Oct 14 15:22:05 america caddy[201731]: caddy.HomeDir=/var/lib/caddy
Oct 14 15:22:05 america caddy[201731]: caddy.AppDataDir=/var/lib/caddy
Oct 14 15:22:05 america caddy[201731]: caddy.AppConfigDir=/var/lib/caddy/.config/caddy
Oct 14 15:22:05 america caddy[201731]: caddy.ConfigAutosavePath=/var/lib/caddy/.config/caddy/autosave.json
Oct 14 15:22:05 america caddy[201731]: caddy.Version=v2.2.1
Oct 14 15:22:05 america caddy[201731]: runtime.GOOS=linux
Oct 14 15:22:05 america caddy[201731]: runtime.GOARCH=amd64
Oct 14 15:22:05 america caddy[201731]: runtime.Compiler=gc
Oct 14 15:22:05 america caddy[201731]: runtime.NumCPU=1
Oct 14 15:22:05 america caddy[201731]: runtime.GOMAXPROCS=1
Oct 14 15:22:05 america caddy[201731]: runtime.Version=go1.15.2
Oct 14 15:22:05 america caddy[201731]: os.Getwd=/
Oct 14 15:22:05 america caddy[201731]: LANG=en_US.UTF-8
Oct 14 15:22:05 america caddy[201731]: PATH=/usr/local/sbin:/usr/local/bin:/usr/bin
Oct 14 15:22:05 america caddy[201731]: HOME=/var/lib/caddy
Oct 14 15:22:05 america caddy[201731]: LOGNAME=caddy
Oct 14 15:22:05 america caddy[201731]: USER=caddy
Oct 14 15:22:05 america caddy[201731]: INVOCATION_ID=0a90c0100896422499120b66e0696db1
Oct 14 15:22:05 america caddy[201731]: JOURNAL_STREAM=8:5209565
Oct 14 15:22:05 america caddy[201731]: XDG_DATA_HOME=/var/lib
Oct 14 15:22:05 america caddy[201731]: XDG_CONFIG_HOME=/var/lib/caddy/.config
Oct 14 15:22:05 america caddy[201731]: {"level":"info","ts":1602703325.1513174,"logger":"automigrate","msg":"beginning one-time data directory migration","old_dir":"/var/lib/caddy/.local/share/caddy","new_dir":"/var/lib/caddy","details":"https://github.com/caddyserver/caddy/issues/2955"}
Oct 14 15:22:05 america caddy[201731]: {"level":"error","ts":1602703325.151413,"logger":"automigrate","msg":"new data directory already exists; skipping auto-migration as conservative safety measure","old_dir":"/var/lib/caddy/.local/share/caddy","new_dir":"/var/lib/caddy","instructions":"https://github.com/caddyserver/caddy/issues/2955#issuecomment-570000333"}
Oct 14 15:22:05 america caddy[201731]: {"level":"info","ts":1602703325.151512,"msg":"using provided configuration","config_file":"/etc/caddy/Caddyfile","config_adapter":""}
Oct 14 15:22:05 america caddy[201731]: {"level":"info","ts":1602703325.1556077,"logger":"admin","msg":"admin endpoint started","address":"tcp/localhost:2019","enforce_origin":false,"origins":["localhost:2019","[::1]:2019","127.0.0.1:2019"]}
Oct 14 15:22:05 america caddy[201731]: {"level":"info","ts":1602703325.1640515,"logger":"http","msg":"server is listening only on the HTTPS port but has no TLS connection policies; adding one to enable TLS","server_name":"srv0","https_port":443}
Oct 14 15:22:05 america caddy[201731]: {"level":"info","ts":1602703325.1642013,"logger":"http","msg":"enabling automatic HTTP->HTTPS redirects","server_name":"srv0"}
Oct 14 15:22:05 america caddy[201731]: {"level":"info","ts":1602703325.1651764,"logger":"http","msg":"enabling experimental HTTP/3 listener","addr":":443"}
1 Like

Yeah just replacing the binary file on disk won’t stop and restart a running process… you should just need to do systemctl restart caddy.

So when you gave it the reload command, systemd spawned a temporary caddy reload which used the new binary to make the updated config, and sent it to the running caddy process, which was still the old version.

2 Likes

Yeah, I guess I’m an idiot, that was the problem, systemctl restart caddy seems to have brought it back up just fine. Sorry for my pandemic-addled brain. Thank you for explaining what happened!

2 Likes

That’s all of us, tbh.

2 Likes