Letsencrypt Certificate issuance issues

1. Output of caddy version:

v2.5.1 (Upgrade planned)

2. How I run Caddy:

Systemd as caddy user

a. System environment:

AWS EC2 Pool
Linux 5.13.0-1023-aws #25~20.04.1-Ubuntu SMP 2022 x86_64 x86_64 x86_64 GNU/Linux

b. Command:

service caddy start

c. Service/unit/compose file:

[Unit]
Description=Caddy HTTP/2 web server
Documentation=https://caddyserver.com/docs
After=network-online.target remote-fs.target nginx.service
Wants=network-online.target systemd-networkd-wait-online.service
Requires=nginx.service
BindsTo=nginx.service

[Service]
Type=notify
Restart=on-abort

; User and group the process will run as.
User=caddy
Group=caddy

; Letsencrypt-issued certificates will be written to this directory.
Environment=CADDYPATH=/etc/ssl/caddy/v2

; Always set "-root" to something safe in case it gets forgotten in the Caddyfile.
ExecStart=/usr/bin/caddy run --environ --config /etc/caddy/Caddyfile_v2 -watch
ExecReload=/usr/bin/caddy reload --config /etc/caddy/Caddyfile_v2 -watch

; Use graceful shutdown with a reasonable timeout
KillMode=mixed
KillSignal=SIGQUIT
TimeoutStopSec=5s

; Limit the number of file descriptors; see `man systemd.exec` for more limit settings.
LimitNOFILE=1048576
; Unmodified caddy is not expected to use more than that.
LimitNPROC=2048

; Use private /tmp and /var/tmp, which are discarded after caddy stops.
PrivateTmp=true
; Use a minimal /dev (May bring additional security if switched to 'true', but it may not work on Raspberry Pi's or other devices, so it has been disabled in this dist.)
PrivateDevices=false
; Hide /home, /root, and /run/user. Nobody will steal your SSH-keys.
ProtectHome=true
; Make /usr, /boot, /etc and possibly some more folders read-only.
ProtectSystem=full
; … except /etc/ssl/caddy, because we want Letsencrypt-certificates there.
;   This merely retains r/w access rights, it does not add any new. Must still be writable on the host!
ReadWriteDirectories=/etc/ssl/caddy/v2

; The following additional security directives only work with systemd v229 or later.
; They further restrict privileges that can be gained by caddy. Uncomment if you like.
; Note that you may have to add capabilities required by any plugins in use.
;CapabilityBoundingSet=CAP_NET_BIND_SERVICE
AmbientCapabilities=CAP_NET_BIND_SERVICE
;NoNewPrivileges=true

[Install]
WantedBy=multi-user.target

d. My complete Caddy config:

# Global Options
# https://caddyserver.com/docs/caddyfile/options
{
        admin off
        grace_period 10m

        #Order for rate limiting
        order rate_limit before basicauth

        # TLS Options
        email ssl@example.com
        on_demand_tls {
                ask https://sites.example.com/allowed
        }

        #Set Certificate Location
        storage file_system /etc/ssl/caddy/v2
}

:443 {
        tls {
                #ca https://acme-staging-v02.api.letsencrypt.org/directory
                on_demand
        }

        #Set GeoBlocking
        @geofilter {
                maxmind_geolocation {
                        db_path /etc/caddy/GeoLite2-Country.mmdb
                        deny_countries RU RO VN
                }
        }
        reverse_proxy @geofilter http://127.0.0.1:7777

        #Set Rate Limiting
        rate_limit {
                zone dynamic {
                        key {remote_host}
                        events 60
                        window 120s
                }
        }

        #CDN Setting
        uri replace s3.amazonaws.com/example static.example.com

        #Set Headers
        header {
                Strict-Transport-Security max-age=31536000;
                X-Content-Type-Options nosniff
                X-XSS-Protection "1; mode=block"
        }

        #Enable Compression
        encode {
                #gzip 9
        }

        #Define Logs
        log {
                output file /var/log/caddy/access.log {
                        roll_size 25
                        roll_keep 10
                        roll_keep_for 7d
                }
                format filter {
                        wrap console {
                                time_format rfc3339
                                time_key timestamp
                        }
                        fields {
                                request>headers>Accept delete
                        }
                }
        }
}

#Define Snippet for On-Demand TLS
(onDemand) {
        tls {
                on_demand
        }
}

#Include Redirs
import redir2/*
import redir2MultiDomain/*

3. The problem I’m having:

All issued certs are from ZeroSSL, We have tried forcing a cert renewal by deleting old cert and OCSP files. While doing the reissue process, these errors were noticed in the logs

4. Error messages and/or full log output:

Sep  1 08:37:45 ip-172-31-6-120 caddy[42779]: {"level":"error","ts":1662021465.443806,"logger":"tls.obtain","msg":"could not get certificate from issuer","identifier":"www.example.com","issuer":"acme-v02.api.letsencrypt.org-directory","error":"HTTP 400 urn:ietf:params:acme:error:malformed - JWS verification error"}
# ll /etc/ssl/caddy/v2/certificates/
total 60
drwx------    3 caddy caddy  6144 Apr 24 08:01 ./
drwxr-xr-x    6 caddy caddy  6144 Sep 12 11:04 ../
drwx------ 1072 caddy caddy 55296 Sep 12 09:14 acme.zerossl.com-v2-dv90/

5. What I already tried:

Forcing a renewal is the only thing we’ve tried this far

6. Links to relevant resources:

Interesting; how can we minimally reproduce this?

Also, what are the full logs, as requested by the help template, and not just one log entry in isolation?

Here is a longer snip of one of the domain renewals

Sep 12 09:17:26 ip-172-31-12-51 caddy[48505]: {"level":"info","ts":1662974246.3206336,"logger":"tls.on_demand","msg":"obtaining new certificate","server_name":"example.com"}
Sep 12 09:17:26 ip-172-31-12-51 caddy[48505]: {"level":"info","ts":1662974246.38222,"logger":"tls.obtain","msg":"acquiring lock","identifier":"example.com"}
Sep 12 09:17:26 ip-172-31-12-51 caddy[48505]: {"level":"info","ts":1662974246.4049256,"logger":"tls.obtain","msg":"lock acquired","identifier":"example.com"}
Sep 12 09:17:26 ip-172-31-12-51 caddy[48505]: {"level":"info","ts":1662974246.42483,"logger":"tls.issuance.acme","msg":"waiting on internal rate limiter","identifiers":["example.com"],"ca":"https://acme-v02.api.letsencrypt.org/directory","account":"ssl@example.com"}
Sep 12 09:17:26 ip-172-31-12-51 caddy[48505]: {"level":"info","ts":1662974246.4249146,"logger":"tls.issuance.acme","msg":"done waiting on internal rate limiter","identifiers":["example.com"],"ca":"https://acme-v02.api.letsencrypt.org/directory","account":"ssl@example.com"}
Sep 12 09:17:26 ip-172-31-12-51 caddy[48505]: {"level":"error","ts":1662974246.6479287,"logger":"tls.obtain","msg":"could not get certificate from issuer","identifier":"example.com","issuer":"acme-v02.api.letsencrypt.org-directory","error":"HTTP 400 urn:ietf:params:acme:error:malformed - JWS verification error"}
Sep 12 09:17:26 ip-172-31-12-51 caddy[48505]: {"level":"info","ts":1662974246.6680167,"logger":"tls.issuance.acme","msg":"waiting on internal rate limiter","identifiers":["example.com"],"ca":"https://acme.zerossl.com/v2/DV90","account":"ssl@example.com"}
Sep 12 09:17:26 ip-172-31-12-51 caddy[48505]: {"level":"info","ts":1662974246.6680648,"logger":"tls.issuance.acme","msg":"done waiting on internal rate limiter","identifiers":["example.com"],"ca":"https://acme.zerossl.com/v2/DV90","account":"ssl@example.com"}
Sep 12 09:17:43 ip-172-31-12-51 caddy[48505]: {"level":"info","ts":1662974263.413021,"logger":"tls.issuance.acme.acme_client","msg":"trying to solve challenge","identifier":"example.com","challenge_type":"http-01","ca":"https://acme.zerossl.com/v2/DV90"}
Sep 12 09:20:26 ip-172-31-12-51 caddy[48505]: {"level":"error","ts":1662974426.371843,"logger":"tls.issuance.acme.acme_client","msg":"deactivating authorization","identifier":"example.com","authz":"https://acme.zerossl.com/v2/DV90/authz/-Bo4CLVQtUfrfRCbuulVuw","error":"attempt 1: https://acme.zerossl.com/v2/DV90/authz/-Bo4CLVQtUfrfRCbuulVuw: context deadline exceeded"}
Sep 12 09:20:26 ip-172-31-12-51 caddy[48505]: {"level":"error","ts":1662974426.371862,"logger":"tls.obtain","msg":"could not get certificate from issuer","identifier":"example.com","issuer":"acme.zerossl.com-v2-DV90","error":"[example.com] solving challenges: [example.com] checking authorization status: attempt 1: https://acme.zerossl.com/v2/DV90/authz/-Bo4CLVQtUfrfRCbuulVuw: context deadline exceeded (order=https://acme.zerossl.com/v2/DV90/order/yzsm8ujlEbLFpuEZyxvVYw) (ca=https://acme.zerossl.com/v2/DV90)"}
Sep 12 09:20:26 ip-172-31-12-51 caddy[48505]: {"level":"error","ts":1662974426.371874,"logger":"tls.obtain","msg":"will retry","error":"[example.com] Obtain: [example.com] solving challenges: [example.com] checking authorization status: attempt 1: https://acme.zerossl.com/v2/DV90/authz/-Bo4CLVQtUfrfRCbuulVuw: context deadline exceeded (order=https://acme.zerossl.com/v2/DV90/order/yzsm8ujlEbLFpuEZyxvVYw) (ca=https://acme.zerossl.com/v2/DV90)","attempt":1,"retrying_in":60,"elapsed":179.966916899,"max_duration":2592000}
Sep 12 09:20:26 ip-172-31-12-51 caddy[48505]: {"level":"info","ts":1662974426.3718824,"logger":"tls.obtain","msg":"releasing lock","identifier":"example.com"}
Sep 12 09:59:24 ip-172-31-12-51 caddy[48505]: {"level":"info","ts":1662976764.384177,"logger":"tls.on_demand","msg":"obtaining new certificate","server_name":"example.com"}
Sep 12 09:59:24 ip-172-31-12-51 caddy[48505]: {"level":"info","ts":1662976764.4481459,"logger":"tls.obtain","msg":"acquiring lock","identifier":"example.com"}
Sep 12 09:59:24 ip-172-31-12-51 caddy[48505]: {"level":"info","ts":1662976764.478967,"logger":"tls.obtain","msg":"lock acquired","identifier":"example.com"}
Sep 12 09:59:24 ip-172-31-12-51 caddy[48505]: {"level":"info","ts":1662976764.50044,"logger":"tls.issuance.acme","msg":"waiting on internal rate limiter","identifiers":["example.com"],"ca":"https://acme-v02.api.letsencrypt.org/directory","account":"ssl@example.com"}
Sep 12 09:59:24 ip-172-31-12-51 caddy[48505]: {"level":"info","ts":1662976764.500488,"logger":"tls.issuance.acme","msg":"done waiting on internal rate limiter","identifiers":["example.com"],"ca":"https://acme-v02.api.letsencrypt.org/directory","account":"ssl@example.com"}
Sep 12 09:59:24 ip-172-31-12-51 caddy[48505]: {"level":"error","ts":1662976764.724304,"logger":"tls.obtain","msg":"could not get certificate from issuer","identifier":"example.com","issuer":"acme-v02.api.letsencrypt.org-directory","error":"HTTP 400 urn:ietf:params:acme:error:malformed - JWS verification error"}
Sep 12 09:59:24 ip-172-31-12-51 caddy[48505]: {"level":"info","ts":1662976764.7432709,"logger":"tls.issuance.acme","msg":"waiting on internal rate limiter","identifiers":["example.com"],"ca":"https://acme.zerossl.com/v2/DV90","account":"ssl@example.com"}
Sep 12 09:59:24 ip-172-31-12-51 caddy[48505]: {"level":"info","ts":1662976764.7433085,"logger":"tls.issuance.acme","msg":"done waiting on internal rate limiter","identifiers":["example.com"],"ca":"https://acme.zerossl.com/v2/DV90","account":"ssl@example.com"}
Sep 12 09:59:39 ip-172-31-12-51 caddy[48505]: {"level":"info","ts":1662976779.774856,"logger":"tls.issuance.acme.acme_client","msg":"trying to solve challenge","identifier":"example.com","challenge_type":"http-01","ca":"https://acme.zerossl.com/v2/DV90"}
Sep 12 10:02:24 ip-172-31-12-51 caddy[48505]: {"level":"error","ts":1662976944.4241793,"logger":"tls.issuance.acme.acme_client","msg":"deactivating authorization","identifier":"example.com","authz":"https://acme.zerossl.com/v2/DV90/authz/ObjFsBSAOc7fH__4IYip3A","error":"attempt 1: https://acme.zerossl.com/v2/DV90/authz/ObjFsBSAOc7fH__4IYip3A: context deadline exceeded"}
Sep 12 10:02:24 ip-172-31-12-51 caddy[48505]: {"level":"error","ts":1662976944.4242084,"logger":"tls.obtain","msg":"could not get certificate from issuer","identifier":"example.com","issuer":"acme.zerossl.com-v2-DV90","error":"[example.com] solving challenges: [example.com] context deadline exceeded (order=https://acme.zerossl.com/v2/DV90/order/5Sh7Y0zZ_yRqWSNuc-Dnkg) (ca=https://acme.zerossl.com/v2/DV90)"}
Sep 12 10:02:24 ip-172-31-12-51 caddy[48505]: {"level":"error","ts":1662976944.424223,"logger":"tls.obtain","msg":"will retry","error":"[example.com] Obtain: [example.com] solving challenges: [example.com] context deadline exceeded (order=https://acme.zerossl.com/v2/DV90/order/5Sh7Y0zZ_yRqWSNuc-Dnkg) (ca=https://acme.zerossl.com/v2/DV90)","attempt":1,"retrying_in":60,"elapsed":179.944944573,"max_duration":2592000}
Sep 12 10:02:24 ip-172-31-12-51 caddy[48505]: {"level":"info","ts":1662976944.424235,"logger":"tls.obtain","msg":"releasing lock","identifier":"example.com"}

I was testing some cert renewals, this is for a different domain

Sep 14 08:00:15 ip-172-31-1-231 caddy[23406]: {"level":"info","ts":1663142415.764753,"logger":"tls.issuance.acme","msg":"waiting on internal rate limiter","identifiers":["example.com"],"ca":"https://acme-v02.api.letsencrypt.org/directory","account":"ssl@example.com"}
Sep 14 08:00:15 ip-172-31-1-231 caddy[23406]: {"level":"info","ts":1663142415.7648256,"logger":"tls.issuance.acme","msg":"done waiting on internal rate limiter","identifiers":["example.com"],"ca":"https://acme-v02.api.letsencrypt.org/directory","account":"ssl@example.com"}
Sep 14 08:01:51 ip-172-31-1-231 caddy[23477]: {"level":"info","ts":1663142511.0376492,"logger":"tls.issuance.acme","msg":"waiting on internal rate limiter","identifiers":["example.com"],"ca":"https://acme-v02.api.letsencrypt.org/directory","account":"ssl@example.com"}
Sep 14 08:01:51 ip-172-31-1-231 caddy[23477]: {"level":"info","ts":1663142511.0379164,"logger":"tls.issuance.acme","msg":"done waiting on internal rate limiter","identifiers":["example.com"],"ca":"https://acme-v02.api.letsencrypt.org/directory","account":"ssl@example.com"}
Sep 14 08:01:51 ip-172-31-1-231 caddy[23477]: {"level":"error","ts":1663142511.1541908,"logger":"tls.obtain","msg":"could not get certificate from issuer","identifier":"example.com","issuer":"acme-v02.api.letsencrypt.org-directory","error":"HTTP 400 urn:ietf:params:acme:error:malformed - JWS verification error"}
Sep 14 08:29:30 ip-172-31-1-231 caddy[24341]: {"level":"info","ts":1663144170.7995408,"logger":"tls.issuance.acme","msg":"waiting on internal rate limiter","identifiers":["example.com"],"ca":"https://acme-v02.api.letsencrypt.org/directory","account":"ssl@example.com"}
Sep 14 08:29:30 ip-172-31-1-231 caddy[24341]: {"level":"info","ts":1663144170.7997956,"logger":"tls.issuance.acme","msg":"done waiting on internal rate limiter","identifiers":["example.com"],"ca":"https://acme-v02.api.letsencrypt.org/directory","account":"ssl@example.com"}
Sep 14 08:29:30 ip-172-31-1-231 caddy[24341]: {"level":"error","ts":1663144170.9080038,"logger":"tls.obtain","msg":"could not get certificate from issuer","identifier":"example.com","issuer":"acme-v02.api.letsencrypt.org-directory","error":"HTTP 400 urn:ietf:params:acme:error:malformed - JWS verification error"}
Sep 14 09:15:32 ip-172-31-1-231 caddy[24768]: {"level":"info","ts":1663146932.5741234,"logger":"tls.issuance.acme","msg":"waiting on internal rate limiter","identifiers":["example.com"],"ca":"https://acme-v02.api.letsencrypt.org/directory","account":"ssl@example.com"}
Sep 14 09:15:32 ip-172-31-1-231 caddy[24768]: {"level":"info","ts":1663146932.5743902,"logger":"tls.issuance.acme","msg":"done waiting on internal rate limiter","identifiers":["example.com"],"ca":"https://acme-v02.api.letsencrypt.org/directory","account":"ssl@example.com"}
Sep 14 09:15:32 ip-172-31-1-231 caddy[24768]: {"level":"error","ts":1663146932.8321526,"logger":"tls.obtain","msg":"could not get certificate from issuer","identifier":"example.com","issuer":"acme-v02.api.letsencrypt.org-directory","error":"HTTP 400 urn:ietf:params:acme:error:malformed - JWS verification error"}
Sep 14 09:35:46 ip-172-31-1-231 caddy[24768]: {"level":"info","ts":1663148146.9342084,"logger":"tls.issuance.acme","msg":"waiting on internal rate limiter","identifiers":["example.com"],"ca":"https://acme-v02.api.letsencrypt.org/directory","account":"ssl@example.com"}
Sep 14 09:35:46 ip-172-31-1-231 caddy[24768]: {"level":"info","ts":1663148146.9342778,"logger":"tls.issuance.acme","msg":"done waiting on internal rate limiter","identifiers":["example.com"],"ca":"https://acme-v02.api.letsencrypt.org/directory","account":"ssl@example.com"}
Sep 14 09:35:47 ip-172-31-1-231 caddy[24768]: {"level":"error","ts":1663148147.1409402,"logger":"tls.obtain","msg":"could not get certificate from issuer","identifier":"example.com","issuer":"acme-v02.api.letsencrypt.org-directory","error":"HTTP 400 urn:ietf:params:acme:error:malformed - JWS verification error"}

This might be fixed in the latest version. Your really should upgrade!

1 Like

I wonder, this seems to happen when the wrong account key is used. What if you try clearing out the acme account from the storage folder?

And yes upgrade if possible :slightly_smiling_face:

We have upgraded to 2.5.2 since I posted this originally and the updated post logs were generated after the upgrade.
I’ll check on the account key

Deleting the acme account seems to have done the trick, there are now a number of certificates from Letsencrypt, although many of these were/are valid/active certificates from ZeroSSL.

2 Likes

I’ll look into that. Thanks for trying it out and reporting back!