ACME DNS challenge fails after CNAME/NS internal redirection

Help
1

(Sorry for the repost, realized I had a credential in my previous one, so I deleted it until I could revoke that credential)

1. The problem I’m having:

I’ve been using GitHub - caddy-dns/google-domains: Support for ACME DNS challenge through Google Domains to get wildcard DNS certificates for *.schafers.me, where I have schafers.me registered on Google Domains, but it recently started failing.

A week ago, I could go to https://paperless.schafers.me no problem, but when I went there today it told me that the cert had expired. To isolate the issue, I rebuilt my Caddy setup from scratch (i.e. removed the old Docker volumes, to ensure it wasn’t a caching issue), and it does appear that I cannot get new certs right now.

I’m hoping this doesn’t have to do with the Google Domains to Squarespace cutover, though it might. Thanks for the help, and if any other information would be helpful let me know.

Note that you’ll likely (ideally) see no public A/CNAME DNS records for schafers.me — this is expected, as I’m using pihole internally to resolve those domains.

2. Error messages and/or full log output:

{"level":"info","ts":1687454455.242228,"msg":"using provided configuration","config_file":"/etc/caddy/Caddyfile","config_adapter":"caddyfile"}
{"level":"info","ts":1687454455.2446353,"logger":"admin","msg":"admin endpoint started","address":"localhost:2019","enforce_origin":false,"origins":["//[::1]:2019","//127.0.0.1:2019","//localhost:2019"]}
{"level":"info","ts":1687454455.245281,"logger":"http","msg":"server is listening only on the HTTPS port but has no TLS connection policies; adding one to enable TLS","server_name":"srv0","https_port":443}
{"level":"info","ts":1687454455.2452981,"logger":"http","msg":"enabling automatic HTTP->HTTPS redirects","server_name":"srv0"}
{"level":"info","ts":1687454455.2455208,"logger":"tls.cache.maintenance","msg":"started background certificate maintenance","cache":"0xc0006d85b0"}
{"level":"info","ts":1687454455.2466936,"logger":"http","msg":"enabling HTTP/3 listener","addr":":443"}
{"level":"info","ts":1687454455.2467017,"logger":"tls","msg":"cleaning storage unit","description":"FileStorage:/data/caddy"}
{"level":"info","ts":1687454455.2467341,"logger":"tls","msg":"finished cleaning storage units"}
{"level":"info","ts":1687454455.246744,"msg":"failed to sufficiently increase receive buffer size (was: 208 kiB, wanted: 2048 kiB, got: 416 kiB). See https://github.com/quic-go/quic-go/wiki/UDP-Receive-Buffer-Size for details."}
{"level":"debug","ts":1687454455.2467835,"logger":"http","msg":"starting server loop","address":"[::]:443","tls":true,"http3":true}
{"level":"info","ts":1687454455.2467916,"logger":"http.log","msg":"server running","name":"srv0","protocols":["h1","h2","h3"]}
{"level":"debug","ts":1687454455.2468104,"logger":"http","msg":"starting server loop","address":"[::]:80","tls":false,"http3":false}
{"level":"info","ts":1687454455.246814,"logger":"http.log","msg":"server running","name":"remaining_auto_https_redirects","protocols":["h1","h2","h3"]}
{"level":"info","ts":1687454455.2468162,"logger":"http","msg":"enabling automatic TLS certificate management","domains":["*.schafers.me","schafers.me"]}
{"level":"info","ts":1687454455.246975,"msg":"autosaved config (load with --resume flag)","file":"/config/caddy/autosave.json"}
{"level":"info","ts":1687454455.2469802,"msg":"serving initial configuration"}
{"level":"info","ts":1687454455.2471132,"logger":"tls.obtain","msg":"acquiring lock","identifier":"*.schafers.me"}
{"level":"info","ts":1687454455.2471733,"logger":"tls.obtain","msg":"acquiring lock","identifier":"schafers.me"}
{"level":"info","ts":1687454455.24842,"logger":"tls.obtain","msg":"lock acquired","identifier":"*.schafers.me"}
{"level":"info","ts":1687454455.2485268,"logger":"tls.obtain","msg":"obtaining certificate","identifier":"*.schafers.me"}
{"level":"debug","ts":1687454455.2485578,"logger":"events","msg":"event","name":"cert_obtaining","id":"23736de5-9969-42d1-9fd3-75bfd032b542","origin":"tls","data":{"identifier":"*.schafers.me"}}
{"level":"debug","ts":1687454455.2487595,"logger":"tls.obtain","msg":"trying issuer 1/2","issuer":"acme-v02.api.letsencrypt.org-directory"}
{"level":"info","ts":1687454455.2493732,"logger":"tls.obtain","msg":"lock acquired","identifier":"schafers.me"}
{"level":"info","ts":1687454455.2494652,"logger":"tls.obtain","msg":"obtaining certificate","identifier":"schafers.me"}
{"level":"debug","ts":1687454455.2494965,"logger":"events","msg":"event","name":"cert_obtaining","id":"0762322d-63d4-4f28-bcc0-7a881f461123","origin":"tls","data":{"identifier":"schafers.me"}}
{"level":"debug","ts":1687454455.2496784,"logger":"tls.obtain","msg":"trying issuer 1/2","issuer":"acme-v02.api.letsencrypt.org-directory"}
{"level":"debug","ts":1687454455.357049,"logger":"http.acme_client","msg":"http request","method":"GET","url":"https://acme-v02.api.letsencrypt.org/directory","headers":{"User-Agent":["Caddy/2.6.4 CertMagic acmez (linux; amd64)"]},"response_headers":{"Cache-Control":["public, max-age=0, no-cache"],"Content-Length":["752"],"Content-Type":["application/json"],"Date":["Thu, 22 Jun 2023 17:20:55 GMT"],"Server":["nginx"],"Strict-Transport-Security":["max-age=604800"],"X-Frame-Options":["DENY"]},"status_code":200}
{"level":"debug","ts":1687454455.3844032,"logger":"http.acme_client","msg":"http request","method":"HEAD","url":"https://acme-v02.api.letsencrypt.org/acme/new-nonce","headers":{"User-Agent":["Caddy/2.6.4 CertMagic acmez (linux; amd64)"]},"response_headers":{"Cache-Control":["public, max-age=0, no-cache"],"Date":["Thu, 22 Jun 2023 17:20:55 GMT"],"Link":["<https://acme-v02.api.letsencrypt.org/directory>;rel=\"index\""],"Replay-Nonce":["2712eqsmX0Au3ARLWyJ1PWjRIapIDNUTLyl84aJhiR47qgs"],"Server":["nginx"],"Strict-Transport-Security":["max-age=604800"],"X-Frame-Options":["DENY"]},"status_code":200}
{"level":"debug","ts":1687454455.3844898,"logger":"http.acme_client","msg":"http request","method":"HEAD","url":"https://acme-v02.api.letsencrypt.org/acme/new-nonce","headers":{"User-Agent":["Caddy/2.6.4 CertMagic acmez (linux; amd64)"]},"response_headers":{"Cache-Control":["public, max-age=0, no-cache"],"Date":["Thu, 22 Jun 2023 17:20:55 GMT"],"Link":["<https://acme-v02.api.letsencrypt.org/directory>;rel=\"index\""],"Replay-Nonce":["2712mNoQoqwLAOUIrncFFkNQnf34HJBDblg1-6IL7hvfJIU"],"Server":["nginx"],"Strict-Transport-Security":["max-age=604800"],"X-Frame-Options":["DENY"]},"status_code":200}
{"level":"debug","ts":1687454455.4233956,"logger":"http.acme_client","msg":"http request","method":"POST","url":"https://acme-v02.api.letsencrypt.org/acme/new-acct","headers":{"Content-Type":["application/jose+json"],"User-Agent":["Caddy/2.6.4 CertMagic acmez (linux; amd64)"]},"response_headers":{"Boulder-Requester":["1169582307"],"Cache-Control":["public, max-age=0, no-cache"],"Content-Length":["266"],"Content-Type":["application/json"],"Date":["Thu, 22 Jun 2023 17:20:55 GMT"],"Link":["<https://acme-v02.api.letsencrypt.org/directory>;rel=\"index\"","<https://letsencrypt.org/documents/LE-SA-v1.3-September-21-2022.pdf>;rel=\"terms-of-service\""],"Location":["https://acme-v02.api.letsencrypt.org/acme/acct/1169582307"],"Replay-Nonce":["2712tJXfWhsn6qiyVQxuAcSiCH8MnBVVViACzzIJETLK_fY"],"Server":["nginx"],"Strict-Transport-Security":["max-age=604800"],"X-Frame-Options":["DENY"]},"status_code":201}
{"level":"info","ts":1687454455.4237008,"logger":"http","msg":"waiting on internal rate limiter","identifiers":["*.schafers.me"],"ca":"https://acme-v02.api.letsencrypt.org/directory","account":""}
{"level":"info","ts":1687454455.4237075,"logger":"http","msg":"done waiting on internal rate limiter","identifiers":["*.schafers.me"],"ca":"https://acme-v02.api.letsencrypt.org/directory","account":""}
{"level":"debug","ts":1687454455.4451993,"logger":"http.acme_client","msg":"http request","method":"POST","url":"https://acme-v02.api.letsencrypt.org/acme/new-acct","headers":{"Content-Type":["application/jose+json"],"User-Agent":["Caddy/2.6.4 CertMagic acmez (linux; amd64)"]},"response_headers":{"Boulder-Requester":["1169582317"],"Cache-Control":["public, max-age=0, no-cache"],"Content-Length":["266"],"Content-Type":["application/json"],"Date":["Thu, 22 Jun 2023 17:20:55 GMT"],"Link":["<https://acme-v02.api.letsencrypt.org/directory>;rel=\"index\"","<https://letsencrypt.org/documents/LE-SA-v1.3-September-21-2022.pdf>;rel=\"terms-of-service\""],"Location":["https://acme-v02.api.letsencrypt.org/acme/acct/1169582317"],"Replay-Nonce":["2712VWd1CfSLZmYWa8xHILo8CIqUBtEUjkHeoJA_YMlamao"],"Server":["nginx"],"Strict-Transport-Security":["max-age=604800"],"X-Frame-Options":["DENY"]},"status_code":201}
{"level":"info","ts":1687454455.4454038,"logger":"http","msg":"waiting on internal rate limiter","identifiers":["schafers.me"],"ca":"https://acme-v02.api.letsencrypt.org/directory","account":""}
{"level":"info","ts":1687454455.445412,"logger":"http","msg":"done waiting on internal rate limiter","identifiers":["schafers.me"],"ca":"https://acme-v02.api.letsencrypt.org/directory","account":""}
{"level":"debug","ts":1687454455.4719536,"logger":"http.acme_client","msg":"http request","method":"POST","url":"https://acme-v02.api.letsencrypt.org/acme/new-order","headers":{"Content-Type":["application/jose+json"],"User-Agent":["Caddy/2.6.4 CertMagic acmez (linux; amd64)"]},"response_headers":{"Boulder-Requester":["1169582307"],"Cache-Control":["public, max-age=0, no-cache"],"Content-Length":["339"],"Content-Type":["application/json"],"Date":["Thu, 22 Jun 2023 17:20:55 GMT"],"Link":["<https://acme-v02.api.letsencrypt.org/directory>;rel=\"index\""],"Location":["https://acme-v02.api.letsencrypt.org/acme/order/1169582307/190260098567"],"Replay-Nonce":["2712XnX3maKbbWwge999YZVajAkfwrDjXm2c_eL1FQUju30"],"Server":["nginx"],"Strict-Transport-Security":["max-age=604800"],"X-Frame-Options":["DENY"]},"status_code":201}
{"level":"debug","ts":1687454455.5029151,"logger":"http.acme_client","msg":"http request","method":"POST","url":"https://acme-v02.api.letsencrypt.org/acme/authz-v3/239094609787","headers":{"Content-Type":["application/jose+json"],"User-Agent":["Caddy/2.6.4 CertMagic acmez (linux; amd64)"]},"response_headers":{"Boulder-Requester":["1169582307"],"Cache-Control":["public, max-age=0, no-cache"],"Content-Length":["385"],"Content-Type":["application/json"],"Date":["Thu, 22 Jun 2023 17:20:55 GMT"],"Link":["<https://acme-v02.api.letsencrypt.org/directory>;rel=\"index\""],"Replay-Nonce":["4397kKqgptAErsAbzwRlU-nNF8i_NI8cCxSjbAl_ovygPNs"],"Server":["nginx"],"Strict-Transport-Security":["max-age=604800"],"X-Frame-Options":["DENY"]},"status_code":200}
{"level":"debug","ts":1687454455.5029342,"logger":"http.acme_client","msg":"http request","method":"POST","url":"https://acme-v02.api.letsencrypt.org/acme/new-order","headers":{"Content-Type":["application/jose+json"],"User-Agent":["Caddy/2.6.4 CertMagic acmez (linux; amd64)"]},"response_headers":{"Boulder-Requester":["1169582317"],"Cache-Control":["public, max-age=0, no-cache"],"Content-Length":["337"],"Content-Type":["application/json"],"Date":["Thu, 22 Jun 2023 17:20:55 GMT"],"Link":["<https://acme-v02.api.letsencrypt.org/directory>;rel=\"index\""],"Location":["https://acme-v02.api.letsencrypt.org/acme/order/1169582317/190260098637"],"Replay-Nonce":["43977k9H7XhAB0pcFPXRnzHlFoxX1POp15IlG61d_O4PvlE"],"Server":["nginx"],"Strict-Transport-Security":["max-age=604800"],"X-Frame-Options":["DENY"]},"status_code":201}
{"level":"info","ts":1687454455.5030966,"logger":"http.acme_client","msg":"trying to solve challenge","identifier":"*.schafers.me","challenge_type":"dns-01","ca":"https://acme-v02.api.letsencrypt.org/directory"}
{"level":"debug","ts":1687454455.5311184,"logger":"http.acme_client","msg":"http request","method":"POST","url":"https://acme-v02.api.letsencrypt.org/acme/authz-v3/239094609837","headers":{"Content-Type":["application/jose+json"],"User-Agent":["Caddy/2.6.4 CertMagic acmez (linux; amd64)"]},"response_headers":{"Boulder-Requester":["1169582317"],"Cache-Control":["public, max-age=0, no-cache"],"Content-Length":["795"],"Content-Type":["application/json"],"Date":["Thu, 22 Jun 2023 17:20:55 GMT"],"Link":["<https://acme-v02.api.letsencrypt.org/directory>;rel=\"index\""],"Replay-Nonce":["439784xAwxNXQ8PwkgVlNLiMchaCj22LJTx3GqODRAMByZ8"],"Server":["nginx"],"Strict-Transport-Security":["max-age=604800"],"X-Frame-Options":["DENY"]},"status_code":200}
{"level":"info","ts":1687454455.531231,"logger":"http.acme_client","msg":"trying to solve challenge","identifier":"schafers.me","challenge_type":"dns-01","ca":"https://acme-v02.api.letsencrypt.org/directory"}
{"level":"error","ts":1687454456.1553552,"logger":"http.acme_client","msg":"cleaning up solver","identifier":"schafers.me","challenge_type":"dns-01","error":"no memory of presenting a DNS record for \"_acme-challenge.schafers.me\" (usually OK if presenting also failed)"}
{"level":"error","ts":1687454456.1574724,"logger":"http.acme_client","msg":"cleaning up solver","identifier":"*.schafers.me","challenge_type":"dns-01","error":"no memory of presenting a DNS record for \"_acme-challenge.schafers.me\" (usually OK if presenting also failed)"}
{"level":"debug","ts":1687454456.1895,"logger":"http.acme_client","msg":"http request","method":"POST","url":"https://acme-v02.api.letsencrypt.org/acme/authz-v3/239094609787","headers":{"Content-Type":["application/jose+json"],"User-Agent":["Caddy/2.6.4 CertMagic acmez (linux; amd64)"]},"response_headers":{"Boulder-Requester":["1169582307"],"Cache-Control":["public, max-age=0, no-cache"],"Content-Length":["389"],"Content-Type":["application/json"],"Date":["Thu, 22 Jun 2023 17:20:56 GMT"],"Link":["<https://acme-v02.api.letsencrypt.org/directory>;rel=\"index\""],"Replay-Nonce":["2712gNacfhOlVn_Pbdg7K36QeUBE75_oHtPIVraVRWjVRXY"],"Server":["nginx"],"Strict-Transport-Security":["max-age=604800"],"X-Frame-Options":["DENY"]},"status_code":200}
{"level":"error","ts":1687454456.1896667,"logger":"tls.obtain","msg":"could not get certificate from issuer","identifier":"*.schafers.me","issuer":"acme-v02.api.letsencrypt.org-directory","error":"[*.schafers.me] solving challenges: presenting for challenge: adding temporary record for zone \"me.\": HTTP 400: Request contains an invalid argument. (order=https://acme-v02.api.letsencrypt.org/acme/order/1169582307/190260098567) (ca=https://acme-v02.api.letsencrypt.org/directory)"}
{"level":"debug","ts":1687454456.189719,"logger":"tls.obtain","msg":"trying issuer 2/2","issuer":"acme.zerossl.com-v2-DV90"}
{"level":"warn","ts":1687454456.189936,"logger":"tls.issuance.zerossl","msg":"missing email address for ZeroSSL; it is strongly recommended to set one for next time"}
{"level":"debug","ts":1687454456.1989698,"logger":"http.acme_client","msg":"http request","method":"POST","url":"https://acme-v02.api.letsencrypt.org/acme/authz-v3/239094609837","headers":{"Content-Type":["application/jose+json"],"User-Agent":["Caddy/2.6.4 CertMagic acmez (linux; amd64)"]},"response_headers":{"Boulder-Requester":["1169582317"],"Cache-Control":["public, max-age=0, no-cache"],"Content-Length":["799"],"Content-Type":["application/json"],"Date":["Thu, 22 Jun 2023 17:20:56 GMT"],"Link":["<https://acme-v02.api.letsencrypt.org/directory>;rel=\"index\""],"Replay-Nonce":["2712Uid1YaXdkZuQ2jyECN92W6ukRfP3Q7Ox3NRoq34_5WE"],"Server":["nginx"],"Strict-Transport-Security":["max-age=604800"],"X-Frame-Options":["DENY"]},"status_code":200}
{"level":"error","ts":1687454456.199133,"logger":"tls.obtain","msg":"could not get certificate from issuer","identifier":"schafers.me","issuer":"acme-v02.api.letsencrypt.org-directory","error":"[schafers.me] solving challenges: presenting for challenge: adding temporary record for zone \"me.\": HTTP 400: Request contains an invalid argument. (order=https://acme-v02.api.letsencrypt.org/acme/order/1169582317/190260098637) (ca=https://acme-v02.api.letsencrypt.org/directory)"}
{"level":"debug","ts":1687454456.199166,"logger":"tls.obtain","msg":"trying issuer 2/2","issuer":"acme.zerossl.com-v2-DV90"}
{"level":"warn","ts":1687454456.1993742,"logger":"tls.issuance.zerossl","msg":"missing email address for ZeroSSL; it is strongly recommended to set one for next time"}
{"level":"info","ts":1687454457.1126952,"logger":"tls.issuance.zerossl","msg":"generated EAB credentials","key_id":"Xla00VZjdiCJOY22SOwPhQ"}
{"level":"info","ts":1687454457.1246517,"logger":"tls.issuance.zerossl","msg":"generated EAB credentials","key_id":"hVYHJvkO1nHtyG_N42aMNA"}
{"level":"debug","ts":1687454457.4531426,"logger":"http.acme_client","msg":"http request","method":"GET","url":"https://acme.zerossl.com/v2/DV90","headers":{"User-Agent":["Caddy/2.6.4 CertMagic acmez (linux; amd64)"]},"response_headers":{"Access-Control-Allow-Origin":["*"],"Content-Length":["645"],"Content-Type":["application/json"],"Date":["Thu, 22 Jun 2023 17:20:57 GMT"],"Server":["nginx"],"Strict-Transport-Security":["max-age=15724800; includeSubDomains"]},"status_code":200}
{"level":"debug","ts":1687454457.9070299,"logger":"http.acme_client","msg":"http request","method":"HEAD","url":"https://acme.zerossl.com/v2/DV90/newNonce","headers":{"User-Agent":["Caddy/2.6.4 CertMagic acmez (linux; amd64)"]},"response_headers":{"Access-Control-Allow-Origin":["*"],"Cache-Control":["max-age=0, no-cache, no-store"],"Content-Type":["application/octet-stream"],"Date":["Thu, 22 Jun 2023 17:20:57 GMT"],"Link":["<https://acme.zerossl.com/v2/DV90>;rel=\"index\""],"Replay-Nonce":["numEhcqbSqrItRJcSg_6MZouwIU4sssj9hgyyltJ2yg"],"Server":["nginx"],"Strict-Transport-Security":["max-age=15724800; includeSubDomains"]},"status_code":200}
{"level":"debug","ts":1687454457.909323,"logger":"http.acme_client","msg":"http request","method":"HEAD","url":"https://acme.zerossl.com/v2/DV90/newNonce","headers":{"User-Agent":["Caddy/2.6.4 CertMagic acmez (linux; amd64)"]},"response_headers":{"Access-Control-Allow-Origin":["*"],"Cache-Control":["max-age=0, no-cache, no-store"],"Content-Type":["application/octet-stream"],"Date":["Thu, 22 Jun 2023 17:20:57 GMT"],"Link":["<https://acme.zerossl.com/v2/DV90>;rel=\"index\""],"Replay-Nonce":["usfeT85-0BQx9YXZTJm6dUvAF01oml_Gul3042K_-dA"],"Server":["nginx"],"Strict-Transport-Security":["max-age=15724800; includeSubDomains"]},"status_code":200}
{"level":"debug","ts":1687454458.3673923,"logger":"http.acme_client","msg":"http request","method":"POST","url":"https://acme.zerossl.com/v2/DV90/newAccount","headers":{"Content-Type":["application/jose+json"],"User-Agent":["Caddy/2.6.4 CertMagic acmez (linux; amd64)"]},"response_headers":{"Access-Control-Allow-Origin":["*"],"Cache-Control":["max-age=0, no-cache, no-store"],"Content-Length":["579"],"Content-Type":["application/json"],"Date":["Thu, 22 Jun 2023 17:20:58 GMT"],"Link":["<https://acme.zerossl.com/v2/DV90>;rel=\"index\""],"Location":["https://acme.zerossl.com/v2/DV90/account/Xla00VZjdiCJOY22SOwPhQ"],"Replay-Nonce":["4LQSbJUmW706beISqug4CKwf1aPgscbyxSMNOfoPcr0"],"Server":["nginx"],"Strict-Transport-Security":["max-age=15724800; includeSubDomains"]},"status_code":201}
{"level":"info","ts":1687454458.3679845,"logger":"http","msg":"waiting on internal rate limiter","identifiers":["*.schafers.me"],"ca":"https://acme.zerossl.com/v2/DV90","account":""}
{"level":"info","ts":1687454458.3679996,"logger":"http","msg":"done waiting on internal rate limiter","identifiers":["*.schafers.me"],"ca":"https://acme.zerossl.com/v2/DV90","account":""}
{"level":"debug","ts":1687454458.3821716,"logger":"http.acme_client","msg":"http request","method":"POST","url":"https://acme.zerossl.com/v2/DV90/newAccount","headers":{"Content-Type":["application/jose+json"],"User-Agent":["Caddy/2.6.4 CertMagic acmez (linux; amd64)"]},"response_headers":{"Access-Control-Allow-Origin":["*"],"Cache-Control":["max-age=0, no-cache, no-store"],"Content-Length":["579"],"Content-Type":["application/json"],"Date":["Thu, 22 Jun 2023 17:20:58 GMT"],"Link":["<https://acme.zerossl.com/v2/DV90>;rel=\"index\""],"Location":["https://acme.zerossl.com/v2/DV90/account/hVYHJvkO1nHtyG_N42aMNA"],"Replay-Nonce":["VqOQHxVyw-ozTFGUzfQunSviEakLCaDg2x7bIGl9y7U"],"Server":["nginx"],"Strict-Transport-Security":["max-age=15724800; includeSubDomains"]},"status_code":201}
{"level":"info","ts":1687454458.3826718,"logger":"http","msg":"waiting on internal rate limiter","identifiers":["schafers.me"],"ca":"https://acme.zerossl.com/v2/DV90","account":""}
{"level":"info","ts":1687454458.3826919,"logger":"http","msg":"done waiting on internal rate limiter","identifiers":["schafers.me"],"ca":"https://acme.zerossl.com/v2/DV90","account":""}
{"level":"debug","ts":1687454458.8625748,"logger":"http.acme_client","msg":"http request","method":"POST","url":"https://acme.zerossl.com/v2/DV90/newOrder","headers":{"Content-Type":["application/jose+json"],"User-Agent":["Caddy/2.6.4 CertMagic acmez (linux; amd64)"]},"response_headers":{"Access-Control-Allow-Origin":["*"],"Cache-Control":["max-age=0, no-cache, no-store"],"Content-Length":["275"],"Content-Type":["application/json"],"Date":["Thu, 22 Jun 2023 17:20:58 GMT"],"Location":["https://acme.zerossl.com/v2/DV90/order/BwwXNToiSefLyUq3-fV9_g"],"Replay-Nonce":["jVNSdWNPzulOHJQOfSU_hpA2kJvYs9YDi0bULPB2k9M"],"Server":["nginx"],"Strict-Transport-Security":["max-age=15724800; includeSubDomains"]},"status_code":201}
{"level":"debug","ts":1687454458.87125,"logger":"http.acme_client","msg":"http request","method":"POST","url":"https://acme.zerossl.com/v2/DV90/newOrder","headers":{"Content-Type":["application/jose+json"],"User-Agent":["Caddy/2.6.4 CertMagic acmez (linux; amd64)"]},"response_headers":{"Access-Control-Allow-Origin":["*"],"Cache-Control":["max-age=0, no-cache, no-store"],"Content-Length":["273"],"Content-Type":["application/json"],"Date":["Thu, 22 Jun 2023 17:20:58 GMT"],"Location":["https://acme.zerossl.com/v2/DV90/order/rc8HjPxWrcaDTKIRZYK1Vw"],"Replay-Nonce":["Ult-Bh-DJyPouQH5si6pTclWccJAKdkHEuAwNzf-Y_M"],"Server":["nginx"],"Strict-Transport-Security":["max-age=15724800; includeSubDomains"]},"status_code":201}
{"level":"debug","ts":1687454459.3396475,"logger":"http.acme_client","msg":"http request","method":"POST","url":"https://acme.zerossl.com/v2/DV90/authz/Aza_xRdp39Y_82pg1n4NkA","headers":{"Content-Type":["application/jose+json"],"User-Agent":["Caddy/2.6.4 CertMagic acmez (linux; amd64)"]},"response_headers":{"Access-Control-Allow-Origin":["*"],"Cache-Control":["max-age=0, no-cache, no-store"],"Content-Length":["295"],"Content-Type":["application/json"],"Date":["Thu, 22 Jun 2023 17:20:59 GMT"],"Link":["<https://acme.zerossl.com/v2/DV90>;rel=\"index\""],"Replay-Nonce":["z_xF-yKJMX1yasr320c2Yzef-4S3DAzBr3wjf605vbY"],"Retry-After":["5"],"Server":["nginx"],"Strict-Transport-Security":["max-age=15724800; includeSubDomains"]},"status_code":200}
{"level":"info","ts":1687454459.339872,"logger":"http.acme_client","msg":"trying to solve challenge","identifier":"*.schafers.me","challenge_type":"dns-01","ca":"https://acme.zerossl.com/v2/DV90"}
{"level":"debug","ts":1687454459.3892882,"logger":"http.acme_client","msg":"http request","method":"POST","url":"https://acme.zerossl.com/v2/DV90/authz/tCITk-ruNXEiDCIC7iQOFQ","headers":{"Content-Type":["application/jose+json"],"User-Agent":["Caddy/2.6.4 CertMagic acmez (linux; amd64)"]},"response_headers":{"Access-Control-Allow-Origin":["*"],"Cache-Control":["max-age=0, no-cache, no-store"],"Content-Length":["441"],"Content-Type":["application/json"],"Date":["Thu, 22 Jun 2023 17:20:59 GMT"],"Link":["<https://acme.zerossl.com/v2/DV90>;rel=\"index\""],"Replay-Nonce":["X1sFSY3NAV5SgQM0Rleit6IARnXYEndh70O7yTlN4Uw"],"Retry-After":["5"],"Server":["nginx"],"Strict-Transport-Security":["max-age=15724800; includeSubDomains"]},"status_code":200}
{"level":"info","ts":1687454459.3894763,"logger":"http.acme_client","msg":"trying to solve challenge","identifier":"schafers.me","challenge_type":"dns-01","ca":"https://acme.zerossl.com/v2/DV90"}
{"level":"error","ts":1687454459.4265914,"logger":"http.acme_client","msg":"cleaning up solver","identifier":"*.schafers.me","challenge_type":"dns-01","error":"no memory of presenting a DNS record for \"_acme-challenge.schafers.me\" (usually OK if presenting also failed)"}
{"level":"error","ts":1687454459.4657571,"logger":"http.acme_client","msg":"cleaning up solver","identifier":"schafers.me","challenge_type":"dns-01","error":"no memory of presenting a DNS record for \"_acme-challenge.schafers.me\" (usually OK if presenting also failed)"}
{"level":"debug","ts":1687454459.9523506,"logger":"http.acme_client","msg":"http request","method":"POST","url":"https://acme.zerossl.com/v2/DV90/authz/Aza_xRdp39Y_82pg1n4NkA","headers":{"Content-Type":["application/jose+json"],"User-Agent":["Caddy/2.6.4 CertMagic acmez (linux; amd64)"]},"response_headers":{"Access-Control-Allow-Origin":["*"],"Cache-Control":["max-age=0, no-cache, no-store"],"Content-Length":["139"],"Content-Type":["application/json"],"Date":["Thu, 22 Jun 2023 17:20:59 GMT"],"Link":["<https://acme.zerossl.com/v2/DV90>;rel=\"index\""],"Replay-Nonce":["yqpsLJZPalhLrWZqOGzz5Rf-9Uo4YR9Uz9_Ey96MA1c"],"Retry-After":["86400"],"Server":["nginx"],"Strict-Transport-Security":["max-age=15724800; includeSubDomains"]},"status_code":200}
{"level":"error","ts":1687454459.9524999,"logger":"tls.obtain","msg":"could not get certificate from issuer","identifier":"*.schafers.me","issuer":"acme.zerossl.com-v2-DV90","error":"[*.schafers.me] solving challenges: presenting for challenge: adding temporary record for zone \"me.\": HTTP 400: Request contains an invalid argument. (order=https://acme.zerossl.com/v2/DV90/order/BwwXNToiSefLyUq3-fV9_g) (ca=https://acme.zerossl.com/v2/DV90)"}
{"level":"debug","ts":1687454459.9525383,"logger":"events","msg":"event","name":"cert_failed","id":"969d8a66-d1a1-4e99-a970-43828e001a0c","origin":"tls","data":{"error":{},"identifier":"*.schafers.me","issuers":["acme-v02.api.letsencrypt.org-directory","acme.zerossl.com-v2-DV90"],"renewal":false}}
{"level":"error","ts":1687454459.9526136,"logger":"tls.obtain","msg":"will retry","error":"[*.schafers.me] Obtain: [*.schafers.me] solving challenges: presenting for challenge: adding temporary record for zone \"me.\": HTTP 400: Request contains an invalid argument. (order=https://acme.zerossl.com/v2/DV90/order/BwwXNToiSefLyUq3-fV9_g) (ca=https://acme.zerossl.com/v2/DV90)","attempt":1,"retrying_in":60,"elapsed":4.704178995,"max_duration":2592000}
{"level":"debug","ts":1687454459.966599,"logger":"http.acme_client","msg":"http request","method":"POST","url":"https://acme.zerossl.com/v2/DV90/authz/tCITk-ruNXEiDCIC7iQOFQ","headers":{"Content-Type":["application/jose+json"],"User-Agent":["Caddy/2.6.4 CertMagic acmez (linux; amd64)"]},"response_headers":{"Access-Control-Allow-Origin":["*"],"Cache-Control":["max-age=0, no-cache, no-store"],"Content-Length":["123"],"Content-Type":["application/json"],"Date":["Thu, 22 Jun 2023 17:20:59 GMT"],"Link":["<https://acme.zerossl.com/v2/DV90>;rel=\"index\""],"Replay-Nonce":["oo0mpANZ-67vBTwVeVOIC7GKGmZeH9xijX1S9sSpyig"],"Retry-After":["86400"],"Server":["nginx"],"Strict-Transport-Security":["max-age=15724800; includeSubDomains"]},"status_code":200}
{"level":"error","ts":1687454459.9667583,"logger":"tls.obtain","msg":"could not get certificate from issuer","identifier":"schafers.me","issuer":"acme.zerossl.com-v2-DV90","error":"[schafers.me] solving challenges: presenting for challenge: adding temporary record for zone \"me.\": HTTP 400: Request contains an invalid argument. (order=https://acme.zerossl.com/v2/DV90/order/rc8HjPxWrcaDTKIRZYK1Vw) (ca=https://acme.zerossl.com/v2/DV90)"}
{"level":"debug","ts":1687454459.9668007,"logger":"events","msg":"event","name":"cert_failed","id":"47230678-7be7-4d1f-80f2-5c415f5fb31b","origin":"tls","data":{"error":{},"identifier":"schafers.me","issuers":["acme-v02.api.letsencrypt.org-directory","acme.zerossl.com-v2-DV90"],"renewal":false}}
{"level":"error","ts":1687454459.9668322,"logger":"tls.obtain","msg":"will retry","error":"[schafers.me] Obtain: [schafers.me] solving challenges: presenting for challenge: adding temporary record for zone \"me.\": HTTP 400: Request contains an invalid argument. (order=https://acme.zerossl.com/v2/DV90/order/rc8HjPxWrcaDTKIRZYK1Vw) (ca=https://acme.zerossl.com/v2/DV90)","attempt":1,"retrying_in":60,"elapsed":4.717446732,"max_duration":2592000}
{"level":"debug","ts":1687454500.2165458,"logger":"events","msg":"event","name":"tls_get_certificate","id":"b46af989-6c8c-4227-8611-0e643c101137","origin":"tls","data":{"client_hello":{"CipherSuites":[4865,4867,4866,49195,49199,52393,52392,49196,49200,49162,49161,49171,49172,156,157,47,53],"ServerName":"paperless.schafers.me","SupportedCurves":[29,23,24,25,256,257],"SupportedPoints":"AA==","SignatureSchemes":[1027,1283,1539,2052,2053,2054,1025,1281,1537,515,513],"SupportedProtos":["h2","http/1.1"],"SupportedVersions":[772,771],"Conn":{}}}}
{"level":"debug","ts":1687454500.2167127,"logger":"tls.handshake","msg":"no matching certificates and no custom selection logic","identifier":"paperless.schafers.me"}
{"level":"debug","ts":1687454500.2167208,"logger":"tls.handshake","msg":"no matching certificates and no custom selection logic","identifier":"*.schafers.me"}
{"level":"debug","ts":1687454500.2167258,"logger":"tls.handshake","msg":"no matching certificates and no custom selection logic","identifier":"*.*.me"}
{"level":"debug","ts":1687454500.2167304,"logger":"tls.handshake","msg":"no matching certificates and no custom selection logic","identifier":"*.*.*"}
{"level":"debug","ts":1687454500.2167377,"logger":"tls.handshake","msg":"all external certificate managers yielded no certificates and no errors","remote_ip":"192.168.20.247","remote_port":"49547","sni":"paperless.schafers.me"}
{"level":"debug","ts":1687454500.2167447,"logger":"tls.handshake","msg":"no certificate matching TLS ClientHello","remote_ip":"192.168.20.247","remote_port":"49547","server_name":"paperless.schafers.me","remote":"192.168.20.247:49547","identifier":"paperless.schafers.me","cipher_suites":[4865,4867,4866,49195,49199,52393,52392,49196,49200,49162,49161,49171,49172,156,157,47,53],"cert_cache_fill":0,"load_if_necessary":true,"obtain_if_necessary":true,"on_demand":false}
{"level":"debug","ts":1687454500.216809,"logger":"http.stdlib","msg":"http: TLS handshake error from 192.168.20.247:49547: no certificate available for 'paperless.schafers.me'"}

3. Caddy version:

$ docker exec caddy caddy version
v2.6.4 h1:2hwYqiRwk1tf3VruhMpLcYTg+11fCdr8S3jhNAdnPy8=

4. How I installed and ran Caddy:

Dockerfile:

FROM caddy:builder AS builder

RUN xcaddy build \
    --with github.com/caddy-dns/google-domains

FROM caddy

COPY --from=builder /usr/bin/caddy /usr/bin/caddy

a. System environment:

Docker on Ubuntu Server

b. Command:

Within Ansible

- name: Create the Caddy custom build
  block:
    - name: Build the new Docker image
      become: true
      community.docker.docker_image:
        build:
          path: /etc/caddy/
        name: "caddy-custom"
        source: build
- name: Start Caddy
  community.docker.docker_container:
    name: caddy
    image: caddy-custom
    restart_policy: unless-stopped
    restart: "{{ caddyfile.changed }}"
    state: started
    network_mode: caddy-network
    ports:
      - 80:80
      - 443:443
    networks:
      - name: paperless-network
    volumes:
      - /etc/caddy/Caddyfile:/etc/caddy/Caddyfile
      - caddy_data:/data
      - caddy_config:/config

d. My complete Caddy config:

{
	debug
}

schafers.me {
	tls {
		dns google_domains [REDACTED CREDENTIAL]
	}

	redir https://www.{host}{uri}
}

*.schafers.me {
	tls {
		dns google_domains [REDACTED CREDENTIAL]
	}

	@paperless host paperless.schafers.me
	handle @paperless {
		reverse_proxy paperless-ng:8000
	}

	# Fallback for otherwise unhandled domains
	handle {
		abort
	}
}
2

Note that I’ve tried doing:

	tls {
		dns google_domains [CREDENTIAL]
		resolvers 8.8.8.8:53
	}

To hopefully make sure that my pihole redirection of schafers.me to my local webserver isn’t the issue… but I do still wonder if that’s getting in the way somehow.

3

FYI, you will need to rotate your API key, since I would assume some bot will have gotten it.

Are you sure you have an authoritative NS record for your domain?

What if you try without the wildcard… wonder if it’s something funky with Google Domains (other than that it’s going away soon).

1 Like
4

Yep, I deleted the post to give myself 10 minutes of security-by-obscurity while I revoked all of my API keys, minted a new one, and set myself up with that before reposting. Cheap lesson in key rotation, all things considered.

Ahh, this is looking suspicious.

[11:51:12] dschafer@dan-mbp14-2023:~$ dig schafers.me ns

; <<>> DiG 9.10.6 <<>> schafers.me ns
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 34051
;; flags: qr aa rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 1232
;; QUESTION SECTION:
;schafers.me.			IN	NS

;; ANSWER SECTION:
schafers.me.		0	IN	CNAME	cactusranch.webi.schafers.me.

;; Query time: 6 msec
;; SERVER: 192.168.0.192#53(192.168.0.192)
;; WHEN: Thu Jun 22 11:51:22 PDT 2023
;; MSG SIZE  rcvd: 82

[11:51:22] dschafer@dan-mbp14-2023:~$ dig @8.8.8.8 schafers.me ns

; <<>> DiG 9.10.6 <<>> @8.8.8.8 schafers.me ns
; (1 server found)
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 34602
;; flags: qr rd ra ad; QUERY: 1, ANSWER: 4, AUTHORITY: 0, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 512
;; QUESTION SECTION:
;schafers.me.			IN	NS

;; ANSWER SECTION:
schafers.me.		21600	IN	NS	ns-cloud-a1.googledomains.com.
schafers.me.		21600	IN	NS	ns-cloud-a4.googledomains.com.
schafers.me.		21600	IN	NS	ns-cloud-a3.googledomains.com.
schafers.me.		21600	IN	NS	ns-cloud-a2.googledomains.com.

;; Query time: 126 msec
;; SERVER: 8.8.8.8#53(8.8.8.8)
;; WHEN: Thu Jun 22 11:51:31 PDT 2023
;; MSG SIZE  rcvd: 161

I have a CNAME in pihole right now:

cname=schafers.me,cactusranch.webi.schafers.me

which was designed to say “hey, when I go to schafers.me in my browser, go to cactusranch.webi.schafers.me instead” (and then I have an A record in pihole pointing cactusranch.webi.schafers.me at my reverse proxy).

When I remove that CNAME, it works fine. So things are working once more. Thank you!

I’ve updated the title of this post so that future searchers might find this — it had nothing to do with Google Domains after all, it was the CNAME/NS internal stuff that was the issue.

–

Just to further my understanding — is there a way to force Caddy to use 8.8.8.8 for the NS record lookup (instead of the system’s DNS resolver). Given that I’m doing internal shenanigans, that seems like it will be more resilient going forward anyway.

1 Like
5

Add a resolvers subdirective:

Glad you figured it out!!

6

Sounds like the approach I had in ACME DNS challenge fails after CNAME/NS internal redirection - #2 by dschafer was correct; I’ll give it another shot. I wonder if I ran into some DNS caching issues somehow when I tried that last time.

EDIT: Yep, I tried this again and it seems to work!

Thanks for the help!

2 Likes
7

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.