1. The problem I’m having:
My Monitoring recently alerted me, that my certificates will expire in about 21 days. I thought caddy would handle it, but looked at the caddy logs to be sure and there were a bunch of errors about renewals failing with both let’s encrypt and zerossl.
2. Error messages and/or full log output:
It basically boils down to
Jan 09 14:45:00 proxy1 caddy[944950]: {"level":"error","ts":1704807900.171735,"logger":"tls.renew","msg":"could not get certificate from issuer","identifier":"<one of my domains>","issuer":"acme-v02.api.letsencrypt.org-directory","error":"HTTP 400 urn:ietf:params:acme:error:malformed - JWS verification error"}
3. Caddy version:
v2.7.6 h1:w0NymbG2m9PcvKWsrXO6EEkY9Ru4FJK8uQbYcev1p3A=
4. How I installed and ran Caddy:
a. System environment:
neofetch output
_,met$$$$$gg. root@proxy1
,g$$$$$$$$$$$$$$$P. -----------
,g$$P" """Y$$.". OS: Debian GNU/Linux 12 (bookworm) x86_64
,$$P' `$$$. Host: KVM/QEMU (Standard PC (i440FX + PIIX, 1996) pc-i440fx-8.1)
',$$P ,ggs. `$$b: Kernel: 6.1.0-17-amd64
`d$$' ,$P"' . $$$ Uptime: 7 days, 7 hours, 42 mins
$$P d$' , $$P Packages: 553 (dpkg)
$$: $$. - ,d$$' Shell: bash 5.2.15
$$; Y$b._ _,d$P' Resolution: 1280x800
Y$$. `.`"Y$$$$P"' CPU: QEMU Virtual version 2.5+ (1) @ 1.992GHz
`$$b "-.__ GPU: 00:02.0 Vendor 1234 Device 1111
`Y$$ Memory: 102MiB / 457MiB
`Y$$.
`$$b.
`Y$$b.
`"Y$b._
`"""
Basically a standard Debian Bookworm install. caddy was installed via the caddy apt repositories and is run via systemd
b. Command:
Started by systemd.
$ cat /proc/$(pidof caddy) | xargs -0 echo
/usr/bin/caddy run --environ --config /etc/caddy/Caddyfile
c. Service/unit/compose file:
The default one supplied by the apt package.
# caddy.service
#
# For using Caddy with a config file.
#
# Make sure the ExecStart and ExecReload commands are correct
# for your installation.
#
# See https://caddyserver.com/docs/install for instructions.
#
# WARNING: This service does not use the --resume flag, so if you
# use the API to make changes, they will be overwritten by the
# Caddyfile next time the service is restarted. If you intend to
# use Caddy's API to configure it, add the --resume flag to the
# `caddy run` command or use the caddy-api.service file instead.
[Unit]
Description=Caddy
Documentation=https://caddyserver.com/docs/
After=network.target network-online.target
Requires=network-online.target
[Service]
Type=notify
User=caddy
Group=caddy
ExecStart=/usr/bin/caddy run --environ --config /etc/caddy/Caddyfile
ExecReload=/usr/bin/caddy reload --config /etc/caddy/Caddyfile --force
TimeoutStopSec=5s
LimitNOFILE=1048576
LimitNPROC=512
PrivateTmp=true
ProtectSystem=full
AmbientCapabilities=CAP_NET_ADMIN CAP_NET_BIND_SERVICE
[Install]
WantedBy=multi-user.target
d. My complete Caddy config:
No, that is not the full config, but my Config contains a LOT of credentials. This shortened Config contains all TLS-Relevant changes I did. All Server blocks are
subdomain.domain.tld {
# A bunch of options, without any TLS options
}
{
email <my email>
on_demand_tls {
ask https://<one of my domains>/<one of my domains>
}
servers {
trusted_proxies static <a few subnets>
}
debug
}
localhost, 127.0.0.1, [::1] {
tls internal
respond "Hello World!"
}
https:// {
tls internal
respond "Hello Https!"
}
:443 {
tls internal
respond "Hello 443!"
}
I fail to see how the rest of the config would help diagnosing this issue, considering the very same config was used to get the certificates in the first place. I have a second machine with the exact same config, but without the issues. All my domain names are handled by two separate caddy instances.
