Dynu.net TLS challenge fails with JWS verification error

Help
1

1. The problem I’m having:

Unable to auto-retrieve TLS certificates and thus unable to serve content with TLS enabled. If I disable TLS explicitly (:80), the content is served just fine, so the router and dynu config themselves do not appear to the be the problem. It might howevert be a problem with the ACME challenge / dns provider plugin, maybe? I have a hard time understanding what exactly is going wrong from the error log alone.

This issue has persisted for several weeks now, I tried rebuilding caddy and minimizing the config, nothing changed or improved the issue.

2. Error messages and/or full log output:

Jan 31 19:49:51 Baikonur systemd[1]: Started Caddy web server.
░░ Subject: A start job for unit caddy.service has finished successfully
░░ Defined-By: systemd
░░ Support: https://lists.freedesktop.org/mailman/listinfo/systemd-devel
░░
░░ A start job for unit caddy.service has finished successfully.
░░
░░ The job identifier is 2591.
Jan 31 19:49:52 Baikonur caddy[96180]: {"level":"info","ts":1738349392.0109763,"logger":"tls","msg":"storage cleaning happened too recently; skipping for now","storage":"FileStorage:/var/lib/caddy","instance":"dcbf530e-0bdf-4a2f-9f5a-966731344eb3","try_again":1738435792.010972,"try_again_in":86399.999999018}
Jan 31 19:49:52 Baikonur caddy[96180]: {"level":"info","ts":1738349392.0111396,"logger":"tls","msg":"finished cleaning storage units"}
Jan 31 19:49:52 Baikonur caddy[96180]: {"level":"info","ts":1738349392.025736,"logger":"tls.obtain","msg":"acquiring lock","identifier":"senshi.dynu.net"}
Jan 31 19:49:52 Baikonur caddy[96180]: {"level":"info","ts":1738349392.049646,"logger":"tls.obtain","msg":"lock acquired","identifier":"senshi.dynu.net"}
Jan 31 19:49:52 Baikonur caddy[96180]: {"level":"info","ts":1738349392.0497854,"logger":"tls.obtain","msg":"obtaining certificate","identifier":"senshi.dynu.net"}
Jan 31 19:49:52 Baikonur caddy[96180]: {"level":"debug","ts":1738349392.0498283,"logger":"events","msg":"event","name":"cert_obtaining","id":"07d4bf4c-86db-458c-a76a-075d40577975","origin":"tls","data":{"identifier":"senshi.dynu.net"}}
Jan 31 19:49:52 Baikonur caddy[96180]: {"level":"debug","ts":1738349392.0500631,"logger":"tls","msg":"created CSR","identifiers":["senshi.dynu.net"],"san_dns_names":["senshi.dynu.net"],"san_emails":[],"common_name":"","extra_extensions":0}
Jan 31 19:49:52 Baikonur caddy[96180]: {"level":"debug","ts":1738349392.0508893,"logger":"tls.obtain","msg":"trying issuer 1/1","issuer":"acme-v02.api.letsencrypt.org-directory"}
Jan 31 19:49:52 Baikonur caddy[96180]: {"level":"debug","ts":1738349392.051555,"logger":"tls.issuance.acme","msg":"using existing ACME account because key found in storage associated with email","email":"caddy@zerossl.com","ca":"https://acme-v02.api.letsencrypt.org/directory"}
Jan 31 19:49:52 Baikonur caddy[96180]: {"level":"debug","ts":1738349392.051888,"logger":"tls.issuance.acme","msg":"using existing ACME account because key found in storage associated with email","email":"caddy@zerossl.com","ca":"https://acme-v02.api.letsencrypt.org/directory"}
Jan 31 19:49:52 Baikonur caddy[96180]: {"level":"info","ts":1738349392.0519385,"logger":"tls.issuance.acme","msg":"waiting on internal rate limiter","identifiers":["senshi.dynu.net"],"ca":"https://acme-v02.api.letsencrypt.org/directory","account":"caddy@zerossl.com"}
Jan 31 19:49:52 Baikonur caddy[96180]: {"level":"info","ts":1738349392.0519621,"logger":"tls.issuance.acme","msg":"done waiting on internal rate limiter","identifiers":["senshi.dynu.net"],"ca":"https://acme-v02.api.letsencrypt.org/directory","account":"caddy@zerossl.com"}
Jan 31 19:49:52 Baikonur caddy[96180]: {"level":"info","ts":1738349392.051994,"logger":"tls.issuance.acme","msg":"using ACME account","account_id":"https://acme-v02.api.letsencrypt.org/acme/acct/729702067","account_contact":["mailto:caddy@zerossl.com"]}
Jan 31 19:49:52 Baikonur caddy[96180]: {"level":"debug","ts":1738349392.5568404,"msg":"http request","method":"GET","url":"https://acme-v02.api.letsencrypt.org/directory","headers":{"User-Agent":["Caddy/2.9.1 CertMagic acmez (linux; amd64)"]},"response_headers":{"Cache-Control":["public, max-age=0, no-cache"],"Content-Length":["828"],"Content-Type":["application/json"],"Date":["Fri, 31 Jan 2025 18:49:52 GMT"],"Server":["nginx"],"Strict-Transport-Security":["max-age=604800"],"X-Frame-Options":["DENY"]},"status_code":200}
Jan 31 19:49:52 Baikonur caddy[96180]: {"level":"debug","ts":1738349392.5570908,"msg":"creating order","account":"https://acme-v02.api.letsencrypt.org/acme/acct/729702067","identifiers":["senshi.dynu.net"]}
Jan 31 19:49:52 Baikonur caddy[96180]: {"level":"debug","ts":1738349392.702651,"msg":"http request","method":"HEAD","url":"https://acme-v02.api.letsencrypt.org/acme/new-nonce","headers":{"User-Agent":["Caddy/2.9.1 CertMagic acmez (linux; amd64)"]},"response_headers":{"Cache-Control":["public, max-age=0, no-cache"],"Date":["Fri, 31 Jan 2025 18:49:52 GMT"],"Link":["<https://acme-v02.api.letsencrypt.org/directory>;rel=\"index\""],"Replay-Nonce":["hXNDuA8iB_9t_EPbqHZsLqE07vRFcDc-NLRXKPjs_TTsqUBatIA"],"Server":["nginx"],"Strict-Transport-Security":["max-age=604800"],"X-Frame-Options":["DENY"]},"status_code":200}
Jan 31 19:49:52 Baikonur caddy[96180]: {"level":"debug","ts":1738349392.849179,"msg":"http request","method":"POST","url":"https://acme-v02.api.letsencrypt.org/acme/new-order","headers":{"Content-Type":["application/jose+json"],"User-Agent":["Caddy/2.9.1 CertMagic acmez (linux; amd64)"]},"response_headers":{"Boulder-Requester":["729702067"],"Cache-Control":["public, max-age=0, no-cache"],"Content-Length":["107"],"Content-Type":["application/problem+json"],"Date":["Fri, 31 Jan 2025 18:49:52 GMT"],"Link":["<https://acme-v02.api.letsencrypt.org/directory>;rel=\"index\""],"Replay-Nonce":["hXNDuA8iO1nFgRtxeuRdtz0dYjS-A9cPgw4MkcgcJk03tldoNEM"],"Server":["nginx"]},"status_code":400}
Jan 31 19:49:52 Baikonur caddy[96180]: {"level":"error","ts":1738349392.8493168,"logger":"tls.obtain","msg":"could not get certificate from issuer","identifier":"senshi.dynu.net","issuer":"acme-v02.api.letsencrypt.org-directory","error":"HTTP 400 urn:ietf:params:acme:error:malformed - JWS verification error"}
Jan 31 19:49:52 Baikonur caddy[96180]: {"level":"debug","ts":1738349392.8493607,"logger":"events","msg":"event","name":"cert_failed","id":"0189102a-de80-4366-8e42-313da2b6956f","origin":"tls","data":{"error":{},"identifier":"senshi.dynu.net","issuers":["acme-v02.api.letsencrypt.org-directory"],"renewal":false}}
Jan 31 19:49:52 Baikonur caddy[96180]: {"level":"error","ts":1738349392.8494127,"logger":"tls.obtain","msg":"will retry","error":"[senshi.dynu.net] Obtain: [senshi.dynu.net] creating new order: attempt 1: https://acme-v02.api.letsencrypt.org/acme/new-order: HTTP 400 urn:ietf:params:acme:error:malformed - JWS verification error (ca=https://acme-v02.api.letsencrypt.org/directory)","attempt":1,"retrying_in":60,"elapsed":0.799747113,"max_duration":2592000}
Jan 31 19:50:52 Baikonur caddy[96180]: {"level":"info","ts":1738349452.8506606,"logger":"tls.obtain","msg":"obtaining certificate","identifier":"senshi.dynu.net"}
Jan 31 19:50:52 Baikonur caddy[96180]: {"level":"debug","ts":1738349452.8507266,"logger":"events","msg":"event","name":"cert_obtaining","id":"26094dae-0879-46bd-b07f-a302f36bcc9d","origin":"tls","data":{"identifier":"senshi.dynu.net"}}
Jan 31 19:50:52 Baikonur caddy[96180]: {"level":"debug","ts":1738349452.8509262,"logger":"tls","msg":"created CSR","identifiers":["senshi.dynu.net"],"san_dns_names":["senshi.dynu.net"],"san_emails":[],"common_name":"","extra_extensions":0}
Jan 31 19:50:52 Baikonur caddy[96180]: {"level":"debug","ts":1738349452.8517165,"logger":"tls.obtain","msg":"trying issuer 1/1","issuer":"acme-v02.api.letsencrypt.org-directory"}
Jan 31 19:50:52 Baikonur caddy[96180]: {"level":"debug","ts":1738349452.8521183,"logger":"tls.issuance.acme","msg":"using existing ACME account because key found in storage associated with email","email":"caddy@zerossl.com","ca":"https://acme-staging-v02.api.letsencrypt.org/directory"}
Jan 31 19:50:52 Baikonur caddy[96180]: {"level":"info","ts":1738349452.8521516,"logger":"tls.issuance.acme","msg":"using ACME account","account_id":"https://acme-staging-v02.api.letsencrypt.org/acme/acct/64387964","account_contact":["mailto:caddy@zerossl.com"]}
Jan 31 19:50:53 Baikonur caddy[96180]: {"level":"debug","ts":1738349453.517121,"msg":"http request","method":"GET","url":"https://acme-staging-v02.api.letsencrypt.org/directory","headers":{"User-Agent":["Caddy/2.9.1 CertMagic acmez (linux; amd64)"]},"response_headers":{"Cache-Control":["public, max-age=0, no-cache"],"Content-Length":["974"],"Content-Type":["application/json"],"Date":["Fri, 31 Jan 2025 18:50:53 GMT"],"Server":["nginx"],"Strict-Transport-Security":["max-age=604800"],"X-Frame-Options":["DENY"]},"status_code":200}
Jan 31 19:50:53 Baikonur caddy[96180]: {"level":"debug","ts":1738349453.517288,"msg":"creating order","account":"https://acme-staging-v02.api.letsencrypt.org/acme/acct/64387964","identifiers":["senshi.dynu.net"]}
Jan 31 19:50:53 Baikonur caddy[96180]: {"level":"debug","ts":1738349453.6688187,"msg":"http request","method":"HEAD","url":"https://acme-staging-v02.api.letsencrypt.org/acme/new-nonce","headers":{"User-Agent":["Caddy/2.9.1 CertMagic acmez (linux; amd64)"]},"response_headers":{"Cache-Control":["public, max-age=0, no-cache"],"Date":["Fri, 31 Jan 2025 18:50:53 GMT"],"Link":["<https://acme-staging-v02.api.letsencrypt.org/directory>;rel=\"index\""],"Replay-Nonce":["xMYr_xatcWGGVRDr_Cs9Xi2YFZoH_HYbGp9to0HJ6oZBkhRJxUo"],"Server":["nginx"],"Strict-Transport-Security":["max-age=604800"],"X-Frame-Options":["DENY"]},"status_code":200}
Jan 31 19:50:53 Baikonur caddy[96180]: {"level":"debug","ts":1738349453.8520005,"msg":"http request","method":"POST","url":"https://acme-staging-v02.api.letsencrypt.org/acme/new-order","headers":{"Content-Type":["application/jose+json"],"User-Agent":["Caddy/2.9.1 CertMagic acmez (linux; amd64)"]},"response_headers":{"Boulder-Requester":["64387964"],"Cache-Control":["public, max-age=0, no-cache"],"Content-Length":["357"],"Content-Type":["application/json"],"Date":["Fri, 31 Jan 2025 18:50:53 GMT"],"Link":["<https://acme-staging-v02.api.letsencrypt.org/directory>;rel=\"index\""],"Location":["https://acme-staging-v02.api.letsencrypt.org/acme/order/64387964/22331289564"],"Replay-Nonce":["xMYr_xatSKEWbHrfH_jUilkITVWgeIRteOZg48CD3VdyK0onJiw"],"Server":["nginx"],"Strict-Transport-Security":["max-age=604800"],"X-Frame-Options":["DENY"]},"status_code":201}
Jan 31 19:50:54 Baikonur caddy[96180]: {"level":"debug","ts":1738349454.0107203,"msg":"http request","method":"POST","url":"https://acme-staging-v02.api.letsencrypt.org/acme/authz/64387964/15681669274","headers":{"Content-Type":["application/jose+json"],"User-Agent":["Caddy/2.9.1 CertMagic acmez (linux; amd64)"]},"response_headers":{"Boulder-Requester":["64387964"],"Cache-Control":["public, max-age=0, no-cache"],"Content-Length":["666"],"Content-Type":["application/json"],"Date":["Fri, 31 Jan 2025 18:50:53 GMT"],"Link":["<https://acme-staging-v02.api.letsencrypt.org/directory>;rel=\"index\""],"Replay-Nonce":["N9Ep_yiTZ_XvgJ-OzqwT6fsQid24NOjS7leT3k7XTzHSIJR0eT8"],"Server":["nginx"],"Strict-Transport-Security":["max-age=604800"],"X-Frame-Options":["DENY"]},"status_code":200}
Jan 31 19:50:54 Baikonur caddy[96180]: {"level":"debug","ts":1738349454.0110302,"msg":"skipping challenge initiation because authorization is not pending","identifier":"senshi.dynu.net","authz_status":"valid"}
Jan 31 19:50:54 Baikonur caddy[96180]: {"level":"info","ts":1738349454.0110567,"msg":"authorization finalized","identifier":"senshi.dynu.net","authz_status":"valid"}
Jan 31 19:50:54 Baikonur caddy[96180]: {"level":"info","ts":1738349454.0110786,"msg":"validations succeeded; finalizing order","order":"https://acme-staging-v02.api.letsencrypt.org/acme/order/64387964/22331289564"}
Jan 31 19:50:54 Baikonur caddy[96180]: {"level":"debug","ts":1738349454.659099,"msg":"http request","method":"POST","url":"https://acme-staging-v02.api.letsencrypt.org/acme/finalize/64387964/22331289564","headers":{"Content-Type":["application/jose+json"],"User-Agent":["Caddy/2.9.1 CertMagic acmez (linux; amd64)"]},"response_headers":{"Boulder-Requester":["64387964"],"Cache-Control":["public, max-age=0, no-cache"],"Content-Length":["362"],"Content-Type":["application/json"],"Date":["Fri, 31 Jan 2025 18:50:54 GMT"],"Link":["<https://acme-staging-v02.api.letsencrypt.org/directory>;rel=\"index\""],"Location":["https://acme-staging-v02.api.letsencrypt.org/acme/order/64387964/22331289564"],"Replay-Nonce":["N9Ep_yiTeu9ixVmylQTJCH4tXnJkTkfrYlIkHVnvDMOSmgezRHY"],"Retry-After":["3"],"Server":["nginx"],"Strict-Transport-Security":["max-age=604800"],"X-Frame-Options":["DENY"]},"status_code":200}
Jan 31 19:50:57 Baikonur caddy[96180]: {"level":"debug","ts":1738349457.8207188,"msg":"http request","method":"POST","url":"https://acme-staging-v02.api.letsencrypt.org/acme/order/64387964/22331289564","headers":{"Content-Type":["application/jose+json"],"User-Agent":["Caddy/2.9.1 CertMagic acmez (linux; amd64)"]},"response_headers":{"Cache-Control":["public, max-age=0, no-cache"],"Content-Length":["469"],"Content-Type":["application/json"],"Date":["Fri, 31 Jan 2025 18:50:57 GMT"],"Link":["<https://acme-staging-v02.api.letsencrypt.org/directory>;rel=\"index\""],"Replay-Nonce":["xMYr_xatubHzFbpPwniY8S9kAVfnwuWxS-5i7Fwhi5vCue9K2RU"],"Server":["nginx"],"Strict-Transport-Security":["max-age=604800"],"X-Frame-Options":["DENY"]},"status_code":200}
Jan 31 19:50:57 Baikonur caddy[96180]: {"level":"debug","ts":1738349457.9806201,"msg":"http request","method":"POST","url":"https://acme-staging-v02.api.letsencrypt.org/acme/cert/2b1fa79e5e5c0ecb26f07f51c6625a7399de","headers":{"Content-Type":["application/jose+json"],"User-Agent":["Caddy/2.9.1 CertMagic acmez (linux; amd64)"]},"response_headers":{"Cache-Control":["public, max-age=0, no-cache"],"Content-Length":["2978"],"Content-Type":["application/pem-certificate-chain"],"Date":["Fri, 31 Jan 2025 18:50:57 GMT"],"Link":["<https://acme-staging-v02.api.letsencrypt.org/directory>;rel=\"index\"","<https://acme-staging-v02.api.letsencrypt.org/acme/cert/2b1fa79e5e5c0ecb26f07f51c6625a7399de/1>;rel=\"alternate\""],"Replay-Nonce":["N9Ep_yiT-jijGjywegQVEaFR_UmHwwHH7bguzppKAt25hUfmQZk"],"Server":["nginx"],"Strict-Transport-Security":["max-age=604800"],"X-Frame-Options":["DENY"]},"status_code":200}
Jan 31 19:50:57 Baikonur caddy[96180]: {"level":"debug","ts":1738349457.9808307,"msg":"getting renewal info","names":["senshi.dynu.net"]}
Jan 31 19:50:58 Baikonur caddy[96180]: {"level":"debug","ts":1738349458.138653,"msg":"http request","method":"GET","url":"https://acme-staging-v02.api.letsencrypt.org/draft-ietf-acme-ari-03/renewalInfo/oXQaBm1Qt4YtSizBfrSNiElszRY.Kx-nnl5cDssm8H9RxmJac5ne","headers":{"User-Agent":["Caddy/2.9.1 CertMagic acmez (linux; amd64)"]},"response_headers":{"Cache-Control":["public, max-age=0, no-cache"],"Content-Length":["101"],"Content-Type":["application/json"],"Date":["Fri, 31 Jan 2025 18:50:58 GMT"],"Link":["<https://acme-staging-v02.api.letsencrypt.org/directory>;rel=\"index\""],"Retry-After":["21600"],"Server":["nginx"],"Strict-Transport-Security":["max-age=604800"],"X-Frame-Options":["DENY"]},"status_code":200}
Jan 31 19:50:58 Baikonur caddy[96180]: {"level":"info","ts":1738349458.1388505,"msg":"got renewal info","names":["senshi.dynu.net"],"window_start":1743444713,"window_end":1743617513,"selected_time":1743522089,"recheck_after":1738371058.138839,"explanation_url":""}
Jan 31 19:50:58 Baikonur caddy[96180]: {"level":"debug","ts":1738349458.298328,"msg":"http request","method":"POST","url":"https://acme-staging-v02.api.letsencrypt.org/acme/cert/2b1fa79e5e5c0ecb26f07f51c6625a7399de/1","headers":{"Content-Type":["application/jose+json"],"User-Agent":["Caddy/2.9.1 CertMagic acmez (linux; amd64)"]},"response_headers":{"Cache-Control":["public, max-age=0, no-cache"],"Content-Length":["2421"],"Content-Type":["application/pem-certificate-chain"],"Date":["Fri, 31 Jan 2025 18:50:58 GMT"],"Link":["<https://acme-staging-v02.api.letsencrypt.org/directory>;rel=\"index\"","<https://acme-staging-v02.api.letsencrypt.org/acme/cert/2b1fa79e5e5c0ecb26f07f51c6625a7399de/0>;rel=\"alternate\""],"Replay-Nonce":["xMYr_xatUJNLuWqrLPqYqCLZq5oghQ13RiN0CsJfV8803bP5wt8"],"Server":["nginx"],"Strict-Transport-Security":["max-age=604800"],"X-Frame-Options":["DENY"]},"status_code":200}
Jan 31 19:50:58 Baikonur caddy[96180]: {"level":"debug","ts":1738349458.298471,"msg":"getting renewal info","names":["senshi.dynu.net"]}
Jan 31 19:50:58 Baikonur caddy[96180]: {"level":"debug","ts":1738349458.4535458,"msg":"http request","method":"GET","url":"https://acme-staging-v02.api.letsencrypt.org/draft-ietf-acme-ari-03/renewalInfo/oXQaBm1Qt4YtSizBfrSNiElszRY.Kx-nnl5cDssm8H9RxmJac5ne","headers":{"User-Agent":["Caddy/2.9.1 CertMagic acmez (linux; amd64)"]},"response_headers":{"Cache-Control":["public, max-age=0, no-cache"],"Content-Length":["101"],"Content-Type":["application/json"],"Date":["Fri, 31 Jan 2025 18:50:58 GMT"],"Link":["<https://acme-staging-v02.api.letsencrypt.org/directory>;rel=\"index\""],"Retry-After":["21600"],"Server":["nginx"],"Strict-Transport-Security":["max-age=604800"],"X-Frame-Options":["DENY"]},"status_code":200}
Jan 31 19:50:58 Baikonur caddy[96180]: {"level":"info","ts":1738349458.4536326,"msg":"got renewal info","names":["senshi.dynu.net"],"window_start":1743444713,"window_end":1743617513,"selected_time":1743537363,"recheck_after":1738371058.4536235,"explanation_url":""}
Jan 31 19:50:58 Baikonur caddy[96180]: {"level":"info","ts":1738349458.4536903,"msg":"successfully downloaded available certificate chains","count":2,"first_url":"https://acme-staging-v02.api.letsencrypt.org/acme/cert/2b1fa79e5e5c0ecb26f07f51c6625a7399de"}
Jan 31 19:50:58 Baikonur caddy[96180]: {"level":"debug","ts":1738349458.4537091,"logger":"tls.issuance.acme","msg":"selected certificate chain","url":"https://acme-staging-v02.api.letsencrypt.org/acme/cert/2b1fa79e5e5c0ecb26f07f51c6625a7399de"}
Jan 31 19:50:58 Baikonur caddy[96180]: {"level":"debug","ts":1738349458.4541473,"logger":"tls.issuance.acme","msg":"using existing ACME account because key found in storage associated with email","email":"caddy@zerossl.com","ca":"https://acme-v02.api.letsencrypt.org/directory"}
Jan 31 19:50:58 Baikonur caddy[96180]: {"level":"info","ts":1738349458.4541707,"logger":"tls.issuance.acme","msg":"waiting on internal rate limiter","identifiers":["senshi.dynu.net"],"ca":"https://acme-v02.api.letsencrypt.org/directory","account":"caddy@zerossl.com"}
Jan 31 19:50:58 Baikonur caddy[96180]: {"level":"info","ts":1738349458.454194,"logger":"tls.issuance.acme","msg":"done waiting on internal rate limiter","identifiers":["senshi.dynu.net"],"ca":"https://acme-v02.api.letsencrypt.org/directory","account":"caddy@zerossl.com"}
Jan 31 19:50:58 Baikonur caddy[96180]: {"level":"info","ts":1738349458.4542248,"logger":"tls.issuance.acme","msg":"using ACME account","account_id":"https://acme-v02.api.letsencrypt.org/acme/acct/729702067","account_contact":["mailto:caddy@zerossl.com"]}
Jan 31 19:50:58 Baikonur caddy[96180]: {"level":"debug","ts":1738349458.4542491,"msg":"creating order","account":"https://acme-v02.api.letsencrypt.org/acme/acct/729702067","identifiers":["senshi.dynu.net"]}
Jan 31 19:50:58 Baikonur caddy[96180]: {"level":"debug","ts":1738349458.601714,"msg":"http request","method":"HEAD","url":"https://acme-v02.api.letsencrypt.org/acme/new-nonce","headers":{"User-Agent":["Caddy/2.9.1 CertMagic acmez (linux; amd64)"]},"response_headers":{"Cache-Control":["public, max-age=0, no-cache"],"Date":["Fri, 31 Jan 2025 18:50:58 GMT"],"Link":["<https://acme-v02.api.letsencrypt.org/directory>;rel=\"index\""],"Replay-Nonce":["aTttXYyy6hGDXqrD7nTwtwH4ObzlDgRxDFNrj_0X5rtzpc-HCtg"],"Server":["nginx"],"Strict-Transport-Security":["max-age=604800"],"X-Frame-Options":["DENY"]},"status_code":200}
Jan 31 19:50:58 Baikonur caddy[96180]: {"level":"debug","ts":1738349458.7497706,"msg":"http request","method":"POST","url":"https://acme-v02.api.letsencrypt.org/acme/new-order","headers":{"Content-Type":["application/jose+json"],"User-Agent":["Caddy/2.9.1 CertMagic acmez (linux; amd64)"]},"response_headers":{"Boulder-Requester":["729702067"],"Cache-Control":["public, max-age=0, no-cache"],"Content-Length":["107"],"Content-Type":["application/problem+json"],"Date":["Fri, 31 Jan 2025 18:50:58 GMT"],"Link":["<https://acme-v02.api.letsencrypt.org/directory>;rel=\"index\""],"Replay-Nonce":["hXNDuA8i9eERESdZ6P6U9hRI5vFQsaNao79G-nYgMY2Q-uMd7oQ"],"Server":["nginx"]},"status_code":400}
Jan 31 19:50:58 Baikonur caddy[96180]: {"level":"error","ts":1738349458.749921,"logger":"tls.obtain","msg":"could not get certificate from issuer","identifier":"senshi.dynu.net","issuer":"acme-v02.api.letsencrypt.org-directory","error":"HTTP 400 urn:ietf:params:acme:error:malformed - JWS verification error"}
Jan 31 19:50:58 Baikonur caddy[96180]: {"level":"debug","ts":1738349458.7499635,"logger":"events","msg":"event","name":"cert_failed","id":"5eb9a41e-0e12-4929-86cd-a937fbea510f","origin":"tls","data":{"error":{"Err":{}},"identifier":"senshi.dynu.net","issuers":["acme-v02.api.letsencrypt.org-directory"],"renewal":false}}
Jan 31 19:50:58 Baikonur caddy[96180]: {"level":"info","ts":1738349458.7500281,"logger":"tls.obtain","msg":"releasing lock","identifier":"senshi.dynu.net"}
Jan 31 19:50:58 Baikonur caddy[96180]: {"level":"error","ts":1738349458.7502027,"logger":"tls","msg":"job failed","error":"senshi.dynu.net: obtaining certificate: [senshi.dynu.net] Obtain: [senshi.dynu.net] creating new order: attempt 1: https://acme-v02.api.letsencrypt.org/acme/new-order: HTTP 400 urn:ietf:params:acme:error:malformed - JWS verification error (ca=https://acme-v02.api.letsencrypt.org/directory)"}

3. Caddy version:

v2.9.1 h1:OEYiZ7DbCzAWVb6TNEkjRcSCRGHVoZsJinoDR/n9oaY=

4. How I installed and ran Caddy:

Downloaded custom binary for linux-amd64 including the dynu dns provider plugin.

a. System environment:

Archlinux, bare metal homeserver

b. Command:

c. Service/unit/compose file:

Unchanged from default:

# caddy.service
#
# For using Caddy with a config file.
#
# Make sure the ExecStart and ExecReload commands are correct
# for your installation.
#
# See https://caddyserver.com/docs/install for instructions.
#
# WARNING: This service does not use the --resume flag, so if you
# use the API to make changes, they will be overwritten by the
# Caddyfile next time the service is restarted. If you intend to
# use Caddy's API to configure it, add the --resume flag to the
# `caddy run` command or use the caddy-api.service file instead.

[Unit]
Description=Caddy web server
Documentation=https://caddyserver.com/docs/
After=network-online.target
Wants=network-online.target
StartLimitIntervalSec=14400
StartLimitBurst=10

[Service]
Type=notify
User=caddy
Group=caddy
Environment=XDG_DATA_HOME=/var/lib
Environment=XDG_CONFIG_HOME=/etc
ExecStartPre=/usr/bin/caddy validate --config /etc/caddy/Caddyfile
ExecStart=/usr/bin/caddy run --environ --config /etc/caddy/Caddyfile
ExecReload=/usr/bin/caddy reload --config /etc/caddy/Caddyfile --force
ExecStopPost=/usr/bin/rm -f /run/caddy/admin.socket

# Do not allow the process to be restarted in a tight loop. If the
# process fails to start, something critical needs to be fixed.
Restart=on-abnormal

# Use graceful shutdown with a reasonable timeout
TimeoutStopSec=5s

LimitNOFILE=1048576
LimitNPROC=512

# Hardening options
AmbientCapabilities=CAP_NET_BIND_SERVICE
CapabilityBoundingSet=CAP_NET_BIND_SERVICE
DevicePolicy=closed
LockPersonality=true
MemoryAccounting=true
MemoryDenyWriteExecute=true
NoNewPrivileges=true
PrivateDevices=true
PrivateTmp=true
ProcSubset=pid
ProtectClock=true
ProtectControlGroups=true
ProtectHome=true
ProtectHostname=true
ProtectKernelLogs=true
ProtectKernelModules=true
ProtectKernelTunables=true
ProtectProc=invisible
ProtectSystem=strict
RemoveIPC=true
ReadWritePaths=/var/lib/caddy /var/log/caddy /run/caddy
RestrictNamespaces=true
RestrictRealtime=true
RestrictSUIDSGID=true

d. My complete Caddy config:

{
  admin "unix//run/caddy/admin.socket"

  acme_dns dynu {
    api_token <token>
    own_domain senshi.dynu.net
  }
}

# Import additional caddy config files in /etc/caddy/conf.d/
import /etc/caddy/conf.d/fileserver

conf.d/fileserver :

senshi.dynu.net {
  root * /srv/ftp
  file_server browse
  basic_auth * {
    <name> <pwd>
  }
}

5. Links to relevant resources:

2

This error usually happens when the key file is corrupted for some reason. We’ve implemented additional controls to avoid corruption, so one way to fix it now is to delete caddy data directory so it’ll generate new keys and certificates.

2 Likes
3

That did fix it, indeed. I’ve been using this server for years now, so good chance this happened during one of the recent upgrades of caddy or any other Arch package or somesuch. But that’s just guessing.

Thank you very much for the quick fix, though!

4

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.