Verify DNS provider configuration is correct - dns-01 challenge fails - cloudflare

1. Caddy version (caddy version):

v2.4.5

2. How I run Caddy:

a. System environment:

debian buster LXC, proxmox - only runningcaddy and ddclient
I use pihole for dns, with a split dns (so i can access site/services via url internally.
I locked access down to only cloudflare ips
router redirects all port 53 requests to pihole
xcaddy build --with github.com/caddy-dns/cloudflare

b. Command:

systemctl start caddy

c. Service/unit/compose file:

d. My complete Caddyfile or JSON config:

zanj.cc {
        tls {
                dns cloudflare API_TOKEN
        }
}

3. The problem I’m having:

config broken - dns-01 challenge times out. It worked for ~6 months - I’m not sure what changed.

4. Error messages and/or full log output:

Sep 06 19:39:11 : {"level":"info","ts":1630957151.8386655,"logger":"tls.obtain","msg":"acquiring lock","identifier":"zanj.cc"}
Sep 06 19:39:11 : {"level":"info","ts":1630957151.8422577,"logger":"tls.obtain","msg":"lock acquired","identifier":"zanj.cc"}
Sep 06 19:39:11 : {"level":"info","ts":1630957151.8442597,"logger":"tls.issuance.acme","msg":"waiting on internal rate limiter","identifiers":["zanj.cc"],"ca":"https://acme-v02.api.letsencrypt.org/directory","account":""}
Sep 06 19:39:11 : {"level":"info","ts":1630957151.8443556,"logger":"tls.issuance.acme","msg":"done waiting on internal rate limiter","identifiers":["zanj.cc"],"ca":"https://acme-v02.api.letsencrypt.org/directory","account":""}
Sep 06 19:39:13 : {"level":"info","ts":1630957153.6146562,"logger":"tls.issuance.acme.acme_client","msg":"trying to solve challenge","identifier":"zanj.cc","challenge_type":"dns-01","ca":"https://acme-v02.api.letsencrypt.org/directory"}
Sep 06 19:41:17 : {"level":"error","ts":1630957277.3974266,"logger":"tls.obtain","msg":"could not get certificate from issuer","identifier":"zanj.cc","issuer":"acme-v02.api.letsencrypt.org-directory","error":"[zanj.cc] solving challenges: waiting for solver certmagic.solverWrapper to be ready: timed out waiting for record to fully propagate; verify DNS provider configuration is correct - last error: <nil> (order=https://acme-v02.api.letsencrypt.org/acme/order/190167180/22511250220) (ca=https://acme-v02.api.letsencrypt.org/directory)"}

5. What I already tried:

This did all work fine previously but to troubleshoot my network, in case I’ve changed something since setting up this server half a year ago, I turned off all network stuff: set dhcp server’s dns to 1.1.1.1 (to bypass local dns server = pihole) & turned off all extra firewall rules.

tried the recent github.com/caddy-dns/cloudflare issues, including adding resolvers 1.1.1.1 to Caddyfile.

Even stopped the lxc and spun up a new one, reinstalled caddy via xcaddy build --with github.com/caddy-dns/cloudflare - produces the same DNS error when running.

The TXT records do appear in cloudflare DNS - so the api key is working (though I also re-rolled this to check)

I’m at a loss of what else to try now, I see there are several previous issues here with same error messages but nothing has helped really… appreciate any help!

Try turning on debug mode to see if it shows more detail. Add this at the top of your Caddyfile:

{
	debug
}

It’s possible that split DNS making Caddy not see the TXT record after pushing the update. In that case setting resolvers 1.1.1.1 should fix that.

But :man_shrugging: I don’t use Cloudflare so I’m not sure what else to suggest.

Thanks francis

here’s with debug on:

Sep 06 21:03:11 : {"level":"info","ts":1630962191.2186203,"logger":"tls.obtain","msg":"acquiring lock","identifier":"zanj.cc"}
Sep 06 21:03:11 : {"level":"info","ts":1630962191.2223136,"logger":"tls.obtain","msg":"lock acquired","identifier":"zanj.cc"}
Sep 06 21:03:11 : {"level":"debug","ts":1630962191.222582,"logger":"tls.obtain","msg":"trying issuer 1/2","issuer":"acme-v02.api.letsencrypt.org-directory"}
Sep 06 21:03:11 : {"level":"info","ts":1630962191.2228062,"logger":"tls.issuance.acme","msg":"waiting on internal rate limiter","identifiers":["zanj.cc"],"ca":"https://acme-v02.api.letsencrypt.org/directory","account":""}
Sep 06 21:03:11 : {"level":"info","ts":1630962191.2228127,"logger":"tls.issuance.acme","msg":"done waiting on internal rate limiter","identifiers":["zanj.cc"],"ca":"https://acme-v02.api.letsencrypt.org/directory","account":""}
Sep 06 21:03:11 : {"level":"debug","ts":1630962191.2268453,"logger":"tls.obtain","msg":"trying issuer 1/2","issuer":"acme-v02.api.letsencrypt.org-directory"}
Sep 06 21:03:11 : {"level":"debug","ts":1630962191.7923746,"logger":"tls.issuance.acme.acme_client","msg":"http request","method":"GET","url":"https://acme-v02.api.letsencrypt.org/directory","headers":{"User-Agent":["Caddy/2.4.5 CertMagic acmez (linux; amd64)"]},"response_headers":{"Cache-Control":["public, max-age=0, no-cache"],"Content-Length":["658"],"Content-Type":["application/json"],"Date":["Mon, 06 Sep 2021 21:03:11 GMT"],"Server":["nginx"],"Strict-Transport-Security":["max-age=604800"],"X-Frame-Options":["DENY"]},"status_code":200}
Sep 06 21:03:11 : {"level":"debug","ts":1630962191.9205813,"logger":"tls.issuance.acme.acme_client","msg":"http request","method":"HEAD","url":"https://acme-v02.api.letsencrypt.org/acme/new-nonce","headers":{"User-Agent":["Caddy/2.4.5 CertMagic acmez (linux; amd64)"]},"response_headers":{"Cache-Control":["public, max-age=0, no-cache"],"Date":["Mon, 06 Sep 2021 21:03:11 GMT"],"Link":["<https://acme-v02.api.letsencrypt.org/directory>;rel=\"index\""],"Replay-Nonce":["0101t3P0Lxe9RlInQcoq_Rdoh_4cwV6-wpbneVcgB4Hm9nI"],"Server":["nginx"],"Strict-Transport-Security":["max-age=604800"],"X-Frame-Options":["DENY"]},"status_code":200}
Sep 06 21:03:12 : {"level":"debug","ts":1630962192.346478,"logger":"tls.issuance.acme.acme_client","msg":"http request","method":"HEAD","url":"https://acme-v02.api.letsencrypt.org/acme/new-nonce","headers":{"User-Agent":["Caddy/2.4.5 CertMagic acmez (linux; amd64)"]},"response_headers":{"Cache-Control":["public, max-age=0, no-cache"],"Date":["Mon, 06 Sep 2021 21:03:12 GMT"],"Link":["<https://acme-v02.api.letsencrypt.org/directory>;rel=\"index\""],"Replay-Nonce":["0102zeUcI7fg_riXzEFsl5dMGTch2Pj14fLPQbUvT7pkASE"],"Server":["nginx"],"Strict-Transport-Security":["max-age=604800"],"X-Frame-Options":["DENY"]},"status_code":200}
Sep 06 21:03:12 : {"level":"debug","ts":1630962192.3818724,"logger":"tls.issuance.acme.acme_client","msg":"http request","method":"POST","url":"https://acme-v02.api.letsencrypt.org/acme/new-order","headers":{"Content-Type":["application/jose+json"],"User-Agent":["Caddy/2.4.5 CertMagic acmez (linux; amd64)"]},"response_headers":{"Boulder-Requester":["190167180"],"Cache-Control":["public, max-age=0, no-cache"],"Content-Length":["330"],"Content-Type":["application/json"],"Date":["Mon, 06 Sep 2021 21:03:12 GMT"],"Link":["<https://acme-v02.api.letsencrypt.org/directory>;rel=\"index\""],"Location":["https://acme-v02.api.letsencrypt.org/acme/order/190167180/22524946870"],"Replay-Nonce":["0101fnQAppMrIYRgNnnxdBLylLdQsYOe24m67FKt0ATcwvo"],"Server":["nginx"],"Strict-Transport-Security":["max-age=604800"],"X-Frame-Options":["DENY"]},"status_code":201}
Sep 06 21:03:12 : {"level":"debug","ts":1630962192.5943043,"logger":"tls.issuance.acme.acme_client","msg":"http request","method":"POST","url":"https://acme-v02.api.letsencrypt.org/acme/authz-v3/28612803450","headers":{"Content-Type":["application/jose+json"],"User-Agent":["Caddy/2.4.5 CertMagic acmez (linux; amd64)"]},"response_headers":{"Boulder-Requester":["190167180"],"Cache-Control":["public, max-age=0, no-cache"],"Content-Length":["788"],"Content-Type":["application/json"],"Date":["Mon, 06 Sep 2021 21:03:12 GMT"],"Link":["<https://acme-v02.api.letsencrypt.org/directory>;rel=\"index\""],"Replay-Nonce":["0102OE7ppKx8T5vv2H83VXWJAHzTOvLm6naAjmdzmgMR6DA"],"Server":["nginx"],"Strict-Transport-Security":["max-age=604800"],"X-Frame-Options":["DENY"]},"status_code":200}
Sep 06 21:03:12 : {"level":"info","ts":1630962192.5946596,"logger":"tls.issuance.acme.acme_client","msg":"trying to solve challenge","identifier":"zanj.cc","challenge_type":"dns-01","ca":"https://acme-v02.api.letsencrypt.org/directory"}
Sep 06 21:03:12 : {"level":"debug","ts":1630962192.7772567,"logger":"tls.issuance.acme.acme_client","msg":"http request","method":"POST","url":"https://acme-v02.api.letsencrypt.org/acme/new-order","headers":{"Content-Type":["application/jose+json"],"User-Agent":["Caddy/2.4.5 CertMagic acmez (linux; amd64)"]},"response_headers":{"Boulder-Requester":["190167180"],"Cache-Control":["public, max-age=0, no-cache"],"Content-Length":["334"],"Content-Type":["application/json"],"Date":["Mon, 06 Sep 2021 21:03:12 GMT"],"Link":["<https://acme-v02.api.letsencrypt.org/directory>;rel=\"index\""],"Location":["https://acme-v02.api.letsencrypt.org/acme/order/190167180/22524948310"],"Replay-Nonce":["01017QlI5cvrjIwB8By6ZQa0gKV6KZe50qaSavjg_lQJ8aA"],"Server":["nginx"],"Strict-Transport-Security":["max-age=604800"],"X-Frame-Options":["DENY"]},"status_code":201}
Sep 06 21:03:12 : {"level":"debug","ts":1630962192.968534,"logger":"tls.issuance.acme.acme_client","msg":"http request","method":"POST","url":"https://acme-v02.api.letsencrypt.org/acme/authz-v3/28612805220","headers":{"Content-Type":["application/jose+json"],"User-Agent":["Caddy/2.4.5 CertMagic acmez (linux; amd64)"]},"response_headers":{"Boulder-Requester":["190167180"],"Cache-Control":["public, max-age=0, no-cache"],"Content-Length":["792"],"Content-Type":["application/json"],"Date":["Mon, 06 Sep 2021 21:03:12 GMT"],"Link":["<https://acme-v02.api.letsencrypt.org/directory>;rel=\"index\""],"Replay-Nonce":["0101e_tKwwjxDg3bat550b_ukaJlv7yssVFMfxgPi6cG_50"],"Server":["nginx"],"Strict-Transport-Security":["max-age=604800"],"X-Frame-Options":["DENY"]},"status_code":200}
Sep 06 21:05:16 : {"level":"debug","ts":1630962316.2646008,"logger":"tls.issuance.acme.acme_client","msg":"http request","method":"POST","url":"https://acme-v02.api.letsencrypt.org/acme/authz-v3/28612805220","headers":{"Content-Type":["application/jose+json"],"User-Agent":["Caddy/2.4.5 CertMagic acmez (linux; amd64)"]},"response_headers":{"Boulder-Requester":["190167180"],"Cache-Control":["public, max-age=0, no-cache"],"Content-Length":["796"],"Content-Type":["application/json"],"Date":["Mon, 06 Sep 2021 21:05:16 GMT"],"Link":["<https://acme-v02.api.letsencrypt.org/directory>;rel=\"index\""],"Replay-Nonce":["0101eaJ7dxUlfk31OFaX1ISz-NwAXiYveKynyt7TkCEyW1s"],"Server":["nginx"],"Strict-Transport-Security":["max-age=604800"],"X-Frame-Options":["DENY"]},"status_code":200}
Sep 06 21:05:16 : {"level":"debug","ts":1630962316.2652605,"logger":"tls.obtain","msg":"trying issuer 2/2","issuer":"acme.zerossl.com-v2-DV90"}
Sep 06 21:05:16 : {"level":"warn","ts":1630962316.2655997,"logger":"tls.issuance.zerossl","msg":"missing email address for ZeroSSL; it is strongly recommended to set one for next time"}
Sep 06 21:05:16 : {"level":"debug","ts":1630962316.4006872,"logger":"tls.issuance.acme.acme_client","msg":"http request","method":"POST","url":"https://acme-v02.api.letsencrypt.org/acme/authz-v3/28612803450","headers":{"Content-Type":["application/jose+json"],"User-Agent":["Caddy/2.4.5 CertMagic acmez (linux; amd64)"]},"response_headers":{"Boulder-Requester":["190167180"],"Cache-Control":["public, max-age=0, no-cache"],"Content-Length":["792"],"Content-Type":["application/json"],"Date":["Mon, 06 Sep 2021 21:05:16 GMT"],"Link":["<https://acme-v02.api.letsencrypt.org/directory>;rel=\"index\""],"Replay-Nonce":["0102ldmrP9GHYKl0HazpIVx7D70prm81sysPkcNKDWXNXmE"],"Server":["nginx"],"Strict-Transport-Security":["max-age=604800"],"X-Frame-Options":["DENY"]},"status_code":200}
Sep 06 21:05:16 : {"level":"error","ts":1630962316.4009762,"logger":"tls.obtain","msg":"could not get certificate from issuer","identifier":"zanj.cc","issuer":"acme-v02.api.letsencrypt.org-directory","error":"[zanj.cc] solving challenges: waiting for solver certmagic.solverWrapper to be ready: timed out waiting for record to fully propagate; verify DNS provider configuration is correct - last error: <nil> (order=https://acme-v02.api.letsencrypt.org/acme/order/190167180/22524946870) (ca=https://acme-v02.api.letsencrypt.org/directory)"}
Sep 06 21:05:16 : {"level":"debug","ts":1630962316.4011366,"logger":"tls.obtain","msg":"trying issuer 2/2","issuer":"acme.zerossl.com-v2-DV90"}
Sep 06 21:05:16 : {"level":"warn","ts":1630962316.401295,"logger":"tls.issuance.zerossl","msg":"missing email address for ZeroSSL; it is strongly recommended to set one for next time"}
Sep 06 21:05:16 : {"level":"info","ts":1630962316.821989,"logger":"tls.issuance.zerossl","msg":"generated EAB credentials","key_id":"TwzBMObpLUMV2XdnYCSVdw"}
Sep 06 21:05:16 : {"level":"info","ts":1630962316.8238626,"logger":"tls.issuance.zerossl","msg":"generated EAB credentials","key_id":"H-7vFrAuKc95kWHSjlUGIw"}
Sep 06 21:05:17 : {"level":"debug","ts":1630962317.3142114,"logger":"tls.issuance.acme.acme_client","msg":"http request","method":"GET","url":"https://acme.zerossl.com/v2/DV90","headers":{"User-Agent":["Caddy/2.4.5 CertMagic acmez (linux; amd64)"]},"response_headers":{"Access-Control-Allow-Origin":["*"],"Cache-Control":["max-age=-1"],"Content-Length":["645"],"Content-Type":["application/json"],"Date":["Mon, 06 Sep 2021 21:05:17 GMT"],"Server":["nginx"],"Strict-Transport-Security":["max-age=15552000"]},"status_code":200}
Sep 06 21:05:17 : {"level":"debug","ts":1630962317.668621,"logger":"tls.issuance.acme.acme_client","msg":"http request","method":"HEAD","url":"https://acme.zerossl.com/v2/DV90/newNonce","headers":{"User-Agent":["Caddy/2.4.5 CertMagic acmez (linux; amd64)"]},"response_headers":{"Access-Control-Allow-Origin":["*"],"Cache-Control":["max-age=-1"],"Content-Type":["application/octet-stream"],"Date":["Mon, 06 Sep 2021 21:05:17 GMT"],"Link":["<https://acme.zerossl.com/v2/DV90>;rel=\"index\""],"Replay-Nonce":["AwWVFXBynLMYMAZlx7v7K8yzdMuj82juDq1CjJrr3qk"],"Server":["nginx"],"Strict-Transport-Security":["max-age=15552000"]},"status_code":200}
Sep 06 21:05:17 : {"level":"debug","ts":1630962317.758527,"logger":"tls.issuance.acme.acme_client","msg":"http request","method":"HEAD","url":"https://acme.zerossl.com/v2/DV90/newNonce","headers":{"User-Agent":["Caddy/2.4.5 CertMagic acmez (linux; amd64)"]},"response_headers":{"Access-Control-Allow-Origin":["*"],"Cache-Control":["max-age=-1"],"Content-Type":["application/octet-stream"],"Date":["Mon, 06 Sep 2021 21:05:17 GMT"],"Link":["<https://acme.zerossl.com/v2/DV90>;rel=\"index\""],"Replay-Nonce":["3i1v8vH8BIo8aeI4FvreNXlhACDQl5E8uyPE9ViKBz0"],"Server":["nginx"],"Strict-Transport-Security":["max-age=15552000"]},"status_code":200}
Sep 06 21:05:18 : {"level":"debug","ts":1630962318.0557957,"logger":"tls.issuance.acme.acme_client","msg":"http request","method":"POST","url":"https://acme.zerossl.com/v2/DV90/newAccount","headers":{"Content-Type":["application/jose+json"],"User-Agent":["Caddy/2.4.5 CertMagic acmez (linux; amd64)"]},"response_headers":{"Access-Control-Allow-Origin":["*"],"Cache-Control":["max-age=0, no-cache, no-store","max-age=-1"],"Content-Length":["579"],"Content-Type":["application/json"],"Date":["Mon, 06 Sep 2021 21:05:18 GMT"],"Location":["https://acme.zerossl.com/v2/DV90/account/TwzBMObpLUMV2XdnYCSVdw"],"Replay-Nonce":["eVH4lswK5F8hSv-LLDx9zowht4L4Pi_6MpuJh2NccJo"],"Server":["nginx"],"Status":[""],"Strict-Transport-Security":["max-age=15552000"]},"status_code":201}
Sep 06 21:05:18 : {"level":"debug","ts":1630962318.1879897,"logger":"tls.issuance.acme.acme_client","msg":"http request","method":"POST","url":"https://acme.zerossl.com/v2/DV90/newAccount","headers":{"Content-Type":["application/jose+json"],"User-Agent":["Caddy/2.4.5 CertMagic acmez (linux; amd64)"]},"response_headers":{"Access-Control-Allow-Origin":["*"],"Cache-Control":["max-age=0, no-cache, no-store","max-age=-1"],"Content-Length":["579"],"Content-Type":["application/json"],"Date":["Mon, 06 Sep 2021 21:05:18 GMT"],"Location":["https://acme.zerossl.com/v2/DV90/account/H-7vFrAuKc95kWHSjlUGIw"],"Replay-Nonce":["T2QRTjZcR1NXUkyXfeV-cDhi07u-QXr46_1GZsV9zVw"],"Server":["nginx"],"Status":[""],"Strict-Transport-Security":["max-age=15552000"]},"status_code":201}
Sep 06 21:05:18 : {"level":"info","ts":1630962318.1884897,"logger":"tls.issuance.acme","msg":"waiting on internal rate limiter","identifiers":["zanj.cc"],"ca":"https://acme.zerossl.com/v2/DV90","account":""}
Sep 06 21:05:18 : {"level":"info","ts":1630962318.1886828,"logger":"tls.issuance.acme","msg":"done waiting on internal rate limiter","identifiers":["zanj.cc"],"ca":"https://acme.zerossl.com/v2/DV90","account":""}
Sep 06 21:05:18 : {"level":"debug","ts":1630962318.451321,"logger":"tls.issuance.acme.acme_client","msg":"http request","method":"POST","url":"https://acme.zerossl.com/v2/DV90/newOrder","headers":{"Content-Type":["application/jose+json"],"User-Agent":["Caddy/2.4.5 CertMagic acmez (linux; amd64)"]},"response_headers":{"Access-Control-Allow-Origin":["*"],"Cache-Control":["max-age=0, no-cache, no-store","max-age=-1"],"Content-Length":["273"],"Content-Type":["application/json"],"Date":["Mon, 06 Sep 2021 21:05:18 GMT"],"Location":["https://acme.zerossl.com/v2/DV90/order/oFEqdzKp7aH_99JM3RZh1w"],"Replay-Nonce":["kB__iGipiUFWBaDM2o8q_DmkqxzOfbI5v7qYJzoKPiw"],"Server":["nginx"],"Status":[""],"Strict-Transport-Security":["max-age=15552000"]},"status_code":201}
Sep 06 21:05:18 : {"level":"debug","ts":1630962318.6361442,"logger":"tls.issuance.acme.acme_client","msg":"http request","method":"POST","url":"https://acme.zerossl.com/v2/DV90/newOrder","headers":{"Content-Type":["application/jose+json"],"User-Agent":["Caddy/2.4.5 CertMagic acmez (linux; amd64)"]},"response_headers":{"Access-Control-Allow-Origin":["*"],"Cache-Control":["max-age=0, no-cache, no-store","max-age=-1"],"Content-Length":["269"],"Content-Type":["application/json"],"Date":["Mon, 06 Sep 2021 21:05:18 GMT"],"Location":["https://acme.zerossl.com/v2/DV90/order/COmGbgu4jxYq1romFsnqCg"],"Replay-Nonce":["VGCPsrLJYtDnjvCvJvZOqD2wOuamp3QKZSd4DdXL6Lw"],"Server":["nginx"],"Status":[""],"Strict-Transport-Security":["max-age=15552000"]},"status_code":201}
Sep 06 21:05:18 : {"level":"debug","ts":1630962318.8162355,"logger":"tls.issuance.acme.acme_client","msg":"http request","method":"POST","url":"https://acme.zerossl.com/v2/DV90/authz/9z1CngFW1TkV_3Nctn76oQ","headers":{"Content-Type":["application/jose+json"],"User-Agent":["Caddy/2.4.5 CertMagic acmez (linux; amd64)"]},"response_headers":{"Access-Control-Allow-Origin":["*"],"Cache-Control":["max-age=-1"],"Content-Length":["441"],"Content-Type":["application/json"],"Date":["Mon, 06 Sep 2021 21:05:18 GMT"],"Link":["<https://acme.zerossl.com/v2/DV90>;rel=\"index\""],"Replay-Nonce":["goW_dI_VfddHoW_jYHU2FYqN4OMs1nddPVLoW1e7tFU"],"Retry-After":["5"],"Server":["nginx"],"Strict-Transport-Security":["max-age=15552000"]},"status_code":200}
Sep 06 21:05:19 : {"level":"debug","ts":1630962319.0101697,"logger":"tls.issuance.acme.acme_client","msg":"http request","method":"POST","url":"https://acme.zerossl.com/v2/DV90/authz/841dL8yV6Vxzir4LcYIaNA","headers":{"Content-Type":["application/jose+json"],"User-Agent":["Caddy/2.4.5 CertMagic acmez (linux; amd64)"]},"response_headers":{"Access-Control-Allow-Origin":["*"],"Cache-Control":["max-age=-1"],"Content-Length":["437"],"Content-Type":["application/json"],"Date":["Mon, 06 Sep 2021 21:05:18 GMT"],"Link":["<https://acme.zerossl.com/v2/DV90>;rel=\"index\""],"Replay-Nonce":["ZwUd8Gf6y9FAAjXmuek5iFA9oCBRkAWG8mZ0oGs8Ez0"],"Retry-After":["5"],"Server":["nginx"],"Strict-Transport-Security":["max-age=15552000"]},"status_code":200}
Sep 06 21:05:19 : {"level":"info","ts":1630962319.010456,"logger":"tls.issuance.acme.acme_client","msg":"trying to solve challenge","identifier":"zanj.cc","challenge_type":"dns-01","ca":"https://acme.zerossl.com/v2/DV90"}

The split dns is all off now, I checked it is in fact using 1.1.1.1, and it seems to be (via dig). My router doesn’t have any dns cache and the new lxc will definitely not have one - i made it since disabling/byassing pihole (when i set the dhcp server’s to offer 1.1.1.1 to clients). I did try the resolvers but that didn’t change anything.

Thanks though, if the debug log doesn’t help, I’m hoping Matt comes along - I’ve read he uses cloudflare…

So I bodged this (not a solution!) - I disabled proxy on cloudflare, ie it just handles dns, then removed the tls option from the config, and domain/subdomain got certificates through http-01. Turned proxy back on in cloudflare and now it’s all fine again … until that is I need to renew the certificate in 90 days!

I’d still appreciate any input on where the dns-01 challenge might have been falling down…

I have still not made any progress (got this first site working fine as above - through turning off cloudflare proxy and dumping the tls section from the caddyfile… I double checked and the challenge which did work was http-01, which I think is the default first one to try in a default address line?

I wonder if someone might be able to suggest where the dns-01 challenge falls down? I want to make sure my local network isn’t causing problems, as even after i think I made everything normal/drop firewall rules/split dns etc, it still seems like it could be my lan at fault as I’ve seen plenty of people use /caddy-dns/cloudflare without dramas…

  1. In my lan, do I need to forward both port 80 and 443 to the host running caddy?
  2. Does this error: timed out waiting for record to fully propagate; verify DNS provider configuration is correct mean theres an issue with my caddy host’s networking/dns/other?
  3. or might it be something to do with cloudflare setup - perhaps there’s some setting in the cf firewall/ssl/other settings i’ve changed and since overlooked?
  4. Does the the challenge come back to my caddy server at all, or is it more: once caddy has set the TXT record _acme-challenge, then does letsencrypt/zerossl communicate direct with cloudflare?

As must be clear, I don’t really understand how this all works, and I’m not really making any headway with the docs/similar forum issues, or elsewhere! The few issues which seem similar are often closed unfinished, hopefully I can get to the bottom of this and perhaps a future searcher might find use!? Hope my questions aren’t too stupid!

Even though this is working with the original domain, I have subdomains I want to also use for this and would love to have the cloudflare proxy working! Thanks.

To solve both HTTP and TLS-ALPN challenges, it’s strongly recommended, yes.

It’s not necessary if using the DNS though, because ACME CAs don’t need to reach your server directly.

Caddy has a repeated loop where it tries to query for the DNS TXT record to see if the call it made to Cloudflare’s API had the intended effect. Once it sees the TXT record, then it will move onto the next step.

But what happens for you is Caddy times out because it wasn’t able to find the TXT record during that loop, and gives up after a while (I think by default, the timeout is 2 minutes).

If you’re saying that you do see the TXT record on your domain within those two minutes, then it means that something in your setup is probably preventing Caddy from using public DNS, and instead reporting maybe some too-aggressively-cached DNS records or your local DNS override.

That’s what the resolvers config is supposed to work around, but :man_shrugging:

ACME CAs will do their own DNS queries to verify the challenge in the TXT record, but only once Caddy tells them “ok I did it, should be good to go”, but it doesn’t reach that step because Caddy isn’t able to verify for itself that it worked.

The ACME CAs don’t really “know” about cloudflare, they just do DNS queries. Only Caddy needs to “know” about cloudflare, because you need to use their API to allow Caddy to set the TXT record during automation.

2 Likes

Thank you so much for going into all this detail Francis, really helps me with the next troubleshooting steps - maybe not what to do but at least where to look!!

I’ve gone back to check the timing of the TXT record appearing on the cloudflare dashboard… It showed within about 20 seconds, so that step should be ok…

OK this all seems to come back to my network DNS, which I have possibly changed since setting up caddy months ago so that would make sense.

I don’t have a very-simple network at home, but it’s not that complicated! I do have a couple of pihole DNS servers, and until I got this issue last week I did have a NAT rule to redirect all :53 requests to these. I have :853 DoT blocked (caddy doesnt use dns over tls does it???). But I’ve since disabled the NAT rule, and I think validated that the caddy server can indeed access 1.1.1.1:53 :

  • I’ve checked https://1.1.1.1/help from another machine in the same vlan - and it suggests it IS accessible - so I don’t think there will be any lingering firewall rules blocking/redirecting to pihole.

  • I also checked dig on the caddy server CT:

; <<>> DiG 9.11.5-P4-5.1+deb10u5-Debian <<>> @1.1.1.1 caddy.community
; (1 server found)
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 5355
;; flags: qr rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 1232
;; QUESTION SECTION:
;caddy.community.               IN      A

;; ANSWER SECTION:
caddy.community.        300     IN      A       159.89.152.193

;; Query time: 75 msec
;; SERVER: 1.1.1.1#53(1.1.1.1)
;; WHEN: Fri Sep 10 08:27:42 UTC 2021
;; MSG SIZE  rcvd: 60

Seems OK!?

Next I’ve tried clearing the server’s dns cache with this systemd-resolve --flush-caches, I did just now after the TXT record appeared and… I think… before the end of the 2 minute window! No dice. Not sure if caddy uses the host cache or not, or whether clearing it here makes any difference - clearly not!

Last gasp, am I staring right at the issue in an obvious place? My last idea would be to test caddy on a pi directly plugged into my modem skipping the router and network all together but that seems a bit drastic, and doesnt really fix the network problems.

EDIT:
This is all with resolvers 1.1.1.1 included now.

Not directly, I think. It’ll use whatever DNS servers are configured on your system by default, using Go’s stdlib and the http://github.com/miekg/dns library. The queries happen here:

I’m not sure what else to suggest :frowning_face:

1 Like