1. Output of caddy version:
v2.5.1 (Upgrade planned)
2. How I run Caddy:
Systemd as caddy user
a. System environment:
AWS EC2 Pool
Linux 5.13.0-1023-aws #25~20.04.1-Ubuntu SMP 2022 x86_64 x86_64 x86_64 GNU/Linux
b. Command:
service caddy start
c. Service/unit/compose file:
[Unit]
Description=Caddy HTTP/2 web server
Documentation=https://caddyserver.com/docs
After=network-online.target remote-fs.target nginx.service
Wants=network-online.target systemd-networkd-wait-online.service
Requires=nginx.service
BindsTo=nginx.service
[Service]
Type=notify
Restart=on-abort
; User and group the process will run as.
User=caddy
Group=caddy
; Letsencrypt-issued certificates will be written to this directory.
Environment=CADDYPATH=/etc/ssl/caddy/v2
; Always set "-root" to something safe in case it gets forgotten in the Caddyfile.
ExecStart=/usr/bin/caddy run --environ --config /etc/caddy/Caddyfile_v2 -watch
ExecReload=/usr/bin/caddy reload --config /etc/caddy/Caddyfile_v2 -watch
; Use graceful shutdown with a reasonable timeout
KillMode=mixed
KillSignal=SIGQUIT
TimeoutStopSec=5s
; Limit the number of file descriptors; see `man systemd.exec` for more limit settings.
LimitNOFILE=1048576
; Unmodified caddy is not expected to use more than that.
LimitNPROC=2048
; Use private /tmp and /var/tmp, which are discarded after caddy stops.
PrivateTmp=true
; Use a minimal /dev (May bring additional security if switched to 'true', but it may not work on Raspberry Pi's or other devices, so it has been disabled in this dist.)
PrivateDevices=false
; Hide /home, /root, and /run/user. Nobody will steal your SSH-keys.
ProtectHome=true
; Make /usr, /boot, /etc and possibly some more folders read-only.
ProtectSystem=full
; … except /etc/ssl/caddy, because we want Letsencrypt-certificates there.
; This merely retains r/w access rights, it does not add any new. Must still be writable on the host!
ReadWriteDirectories=/etc/ssl/caddy/v2
; The following additional security directives only work with systemd v229 or later.
; They further restrict privileges that can be gained by caddy. Uncomment if you like.
; Note that you may have to add capabilities required by any plugins in use.
;CapabilityBoundingSet=CAP_NET_BIND_SERVICE
AmbientCapabilities=CAP_NET_BIND_SERVICE
;NoNewPrivileges=true
[Install]
WantedBy=multi-user.target
d. My complete Caddy config:
# Global Options
# https://caddyserver.com/docs/caddyfile/options
{
admin off
grace_period 10m
#Order for rate limiting
order rate_limit before basicauth
# TLS Options
email ssl@example.com
on_demand_tls {
ask https://sites.example.com/allowed
}
#Set Certificate Location
storage file_system /etc/ssl/caddy/v2
}
:443 {
tls {
#ca https://acme-staging-v02.api.letsencrypt.org/directory
on_demand
}
#Set GeoBlocking
@geofilter {
maxmind_geolocation {
db_path /etc/caddy/GeoLite2-Country.mmdb
deny_countries RU RO VN
}
}
reverse_proxy @geofilter http://127.0.0.1:7777
#Set Rate Limiting
rate_limit {
zone dynamic {
key {remote_host}
events 60
window 120s
}
}
#CDN Setting
uri replace s3.amazonaws.com/example static.example.com
#Set Headers
header {
Strict-Transport-Security max-age=31536000;
X-Content-Type-Options nosniff
X-XSS-Protection "1; mode=block"
}
#Enable Compression
encode {
#gzip 9
}
#Define Logs
log {
output file /var/log/caddy/access.log {
roll_size 25
roll_keep 10
roll_keep_for 7d
}
format filter {
wrap console {
time_format rfc3339
time_key timestamp
}
fields {
request>headers>Accept delete
}
}
}
}
#Define Snippet for On-Demand TLS
(onDemand) {
tls {
on_demand
}
}
#Include Redirs
import redir2/*
import redir2MultiDomain/*
3. The problem I’m having:
All issued certs are from ZeroSSL, We have tried forcing a cert renewal by deleting old cert and OCSP files. While doing the reissue process, these errors were noticed in the logs
4. Error messages and/or full log output:
Sep 1 08:37:45 ip-172-31-6-120 caddy[42779]: {"level":"error","ts":1662021465.443806,"logger":"tls.obtain","msg":"could not get certificate from issuer","identifier":"www.example.com","issuer":"acme-v02.api.letsencrypt.org-directory","error":"HTTP 400 urn:ietf:params:acme:error:malformed - JWS verification error"}
# ll /etc/ssl/caddy/v2/certificates/
total 60
drwx------ 3 caddy caddy 6144 Apr 24 08:01 ./
drwxr-xr-x 6 caddy caddy 6144 Sep 12 11:04 ../
drwx------ 1072 caddy caddy 55296 Sep 12 09:14 acme.zerossl.com-v2-dv90/
5. What I already tried:
Forcing a renewal is the only thing we’ve tried this far
