Verify DNS provider configuration is correct - dns-01 challenge fails - cloudflare

shar · September 6, 2021, 8:07pm

1. Caddy version (`caddy version`):

v2.4.5

2. How I run Caddy:

a. System environment:

debian buster LXC, proxmox - only runningcaddy and ddclient
I use pihole for dns, with a split dns (so i can access site/services via url internally.
I locked access down to only cloudflare ips
router redirects all port 53 requests to pihole
xcaddy build --with github.com/caddy-dns/cloudflare

b. Command:

systemctl start caddy

c. Service/unit/compose file:

d. My complete Caddyfile or JSON config:

zanj.cc {
        tls {
                dns cloudflare API_TOKEN
        }
}

3. The problem I’m having:

config broken - dns-01 challenge times out. It worked for ~6 months - I’m not sure what changed.

4. Error messages and/or full log output:

Sep 06 19:39:11 : {"level":"info","ts":1630957151.8386655,"logger":"tls.obtain","msg":"acquiring lock","identifier":"zanj.cc"}
Sep 06 19:39:11 : {"level":"info","ts":1630957151.8422577,"logger":"tls.obtain","msg":"lock acquired","identifier":"zanj.cc"}
Sep 06 19:39:11 : {"level":"info","ts":1630957151.8442597,"logger":"tls.issuance.acme","msg":"waiting on internal rate limiter","identifiers":["zanj.cc"],"ca":"https://acme-v02.api.letsencrypt.org/directory","account":""}
Sep 06 19:39:11 : {"level":"info","ts":1630957151.8443556,"logger":"tls.issuance.acme","msg":"done waiting on internal rate limiter","identifiers":["zanj.cc"],"ca":"https://acme-v02.api.letsencrypt.org/directory","account":""}
Sep 06 19:39:13 : {"level":"info","ts":1630957153.6146562,"logger":"tls.issuance.acme.acme_client","msg":"trying to solve challenge","identifier":"zanj.cc","challenge_type":"dns-01","ca":"https://acme-v02.api.letsencrypt.org/directory"}
Sep 06 19:41:17 : {"level":"error","ts":1630957277.3974266,"logger":"tls.obtain","msg":"could not get certificate from issuer","identifier":"zanj.cc","issuer":"acme-v02.api.letsencrypt.org-directory","error":"[zanj.cc] solving challenges: waiting for solver certmagic.solverWrapper to be ready: timed out waiting for record to fully propagate; verify DNS provider configuration is correct - last error: <nil> (order=https://acme-v02.api.letsencrypt.org/acme/order/190167180/22511250220) (ca=https://acme-v02.api.letsencrypt.org/directory)"}

5. What I already tried:

This did all work fine previously but to troubleshoot my network, in case I’ve changed something since setting up this server half a year ago, I turned off all network stuff: set dhcp server’s dns to 1.1.1.1 (to bypass local dns server = pihole) & turned off all extra firewall rules.

tried the recent github.com/caddy-dns/cloudflare issues, including adding resolvers 1.1.1.1 to Caddyfile.

Even stopped the lxc and spun up a new one, reinstalled caddy via xcaddy build --with github.com/caddy-dns/cloudflare - produces the same DNS error when running.

The TXT records do appear in cloudflare DNS - so the api key is working (though I also re-rolled this to check)

I’m at a loss of what else to try now, I see there are several previous issues here with same error messages but nothing has helped really… appreciate any help!

francislavoie · September 6, 2021, 9:01pm

Try turning on debug mode to see if it shows more detail. Add this at the top of your Caddyfile:

{
	debug
}

It’s possible that split DNS making Caddy not see the TXT record after pushing the update. In that case setting resolvers 1.1.1.1 should fix that.

But I don’t use Cloudflare so I’m not sure what else to suggest.

shar · September 6, 2021, 9:11pm

Thanks francis

here’s with debug on:

Sep 06 21:03:11 : {"level":"info","ts":1630962191.2186203,"logger":"tls.obtain","msg":"acquiring lock","identifier":"zanj.cc"}
Sep 06 21:03:11 : {"level":"info","ts":1630962191.2223136,"logger":"tls.obtain","msg":"lock acquired","identifier":"zanj.cc"}
Sep 06 21:03:11 : {"level":"debug","ts":1630962191.222582,"logger":"tls.obtain","msg":"trying issuer 1/2","issuer":"acme-v02.api.letsencrypt.org-directory"}
Sep 06 21:03:11 : {"level":"info","ts":1630962191.2228062,"logger":"tls.issuance.acme","msg":"waiting on internal rate limiter","identifiers":["zanj.cc"],"ca":"https://acme-v02.api.letsencrypt.org/directory","account":""}
Sep 06 21:03:11 : {"level":"info","ts":1630962191.2228127,"logger":"tls.issuance.acme","msg":"done waiting on internal rate limiter","identifiers":["zanj.cc"],"ca":"https://acme-v02.api.letsencrypt.org/directory","account":""}
Sep 06 21:03:11 : {"level":"debug","ts":1630962191.2268453,"logger":"tls.obtain","msg":"trying issuer 1/2","issuer":"acme-v02.api.letsencrypt.org-directory"}
Sep 06 21:03:11 : {"level":"debug","ts":1630962191.7923746,"logger":"tls.issuance.acme.acme_client","msg":"http request","method":"GET","url":"https://acme-v02.api.letsencrypt.org/directory","headers":{"User-Agent":["Caddy/2.4.5 CertMagic acmez (linux; amd64)"]},"response_headers":{"Cache-Control":["public, max-age=0, no-cache"],"Content-Length":["658"],"Content-Type":["application/json"],"Date":["Mon, 06 Sep 2021 21:03:11 GMT"],"Server":["nginx"],"Strict-Transport-Security":["max-age=604800"],"X-Frame-Options":["DENY"]},"status_code":200}
Sep 06 21:03:11 : {"level":"debug","ts":1630962191.9205813,"logger":"tls.issuance.acme.acme_client","msg":"http request","method":"HEAD","url":"https://acme-v02.api.letsencrypt.org/acme/new-nonce","headers":{"User-Agent":["Caddy/2.4.5 CertMagic acmez (linux; amd64)"]},"response_headers":{"Cache-Control":["public, max-age=0, no-cache"],"Date":["Mon, 06 Sep 2021 21:03:11 GMT"],"Link":["<https://acme-v02.api.letsencrypt.org/directory>;rel=\"index\""],"Replay-Nonce":["0101t3P0Lxe9RlInQcoq_Rdoh_4cwV6-wpbneVcgB4Hm9nI"],"Server":["nginx"],"Strict-Transport-Security":["max-age=604800"],"X-Frame-Options":["DENY"]},"status_code":200}
Sep 06 21:03:12 : {"level":"debug","ts":1630962192.346478,"logger":"tls.issuance.acme.acme_client","msg":"http request","method":"HEAD","url":"https://acme-v02.api.letsencrypt.org/acme/new-nonce","headers":{"User-Agent":["Caddy/2.4.5 CertMagic acmez (linux; amd64)"]},"response_headers":{"Cache-Control":["public, max-age=0, no-cache"],"Date":["Mon, 06 Sep 2021 21:03:12 GMT"],"Link":["<https://acme-v02.api.letsencrypt.org/directory>;rel=\"index\""],"Replay-Nonce":["0102zeUcI7fg_riXzEFsl5dMGTch2Pj14fLPQbUvT7pkASE"],"Server":["nginx"],"Strict-Transport-Security":["max-age=604800"],"X-Frame-Options":["DENY"]},"status_code":200}
Sep 06 21:03:12 : {"level":"debug","ts":1630962192.3818724,"logger":"tls.issuance.acme.acme_client","msg":"http request","method":"POST","url":"https://acme-v02.api.letsencrypt.org/acme/new-order","headers":{"Content-Type":["application/jose+json"],"User-Agent":["Caddy/2.4.5 CertMagic acmez (linux; amd64)"]},"response_headers":{"Boulder-Requester":["190167180"],"Cache-Control":["public, max-age=0, no-cache"],"Content-Length":["330"],"Content-Type":["application/json"],"Date":["Mon, 06 Sep 2021 21:03:12 GMT"],"Link":["<https://acme-v02.api.letsencrypt.org/directory>;rel=\"index\""],"Location":["https://acme-v02.api.letsencrypt.org/acme/order/190167180/22524946870"],"Replay-Nonce":["0101fnQAppMrIYRgNnnxdBLylLdQsYOe24m67FKt0ATcwvo"],"Server":["nginx"],"Strict-Transport-Security":["max-age=604800"],"X-Frame-Options":["DENY"]},"status_code":201}
Sep 06 21:03:12 : {"level":"debug","ts":1630962192.5943043,"logger":"tls.issuance.acme.acme_client","msg":"http request","method":"POST","url":"https://acme-v02.api.letsencrypt.org/acme/authz-v3/28612803450","headers":{"Content-Type":["application/jose+json"],"User-Agent":["Caddy/2.4.5 CertMagic acmez (linux; amd64)"]},"response_headers":{"Boulder-Requester":["190167180"],"Cache-Control":["public, max-age=0, no-cache"],"Content-Length":["788"],"Content-Type":["application/json"],"Date":["Mon, 06 Sep 2021 21:03:12 GMT"],"Link":["<https://acme-v02.api.letsencrypt.org/directory>;rel=\"index\""],"Replay-Nonce":["0102OE7ppKx8T5vv2H83VXWJAHzTOvLm6naAjmdzmgMR6DA"],"Server":["nginx"],"Strict-Transport-Security":["max-age=604800"],"X-Frame-Options":["DENY"]},"status_code":200}
Sep 06 21:03:12 : {"level":"info","ts":1630962192.5946596,"logger":"tls.issuance.acme.acme_client","msg":"trying to solve challenge","identifier":"zanj.cc","challenge_type":"dns-01","ca":"https://acme-v02.api.letsencrypt.org/directory"}
Sep 06 21:03:12 : {"level":"debug","ts":1630962192.7772567,"logger":"tls.issuance.acme.acme_client","msg":"http request","method":"POST","url":"https://acme-v02.api.letsencrypt.org/acme/new-order","headers":{"Content-Type":["application/jose+json"],"User-Agent":["Caddy/2.4.5 CertMagic acmez (linux; amd64)"]},"response_headers":{"Boulder-Requester":["190167180"],"Cache-Control":["public, max-age=0, no-cache"],"Content-Length":["334"],"Content-Type":["application/json"],"Date":["Mon, 06 Sep 2021 21:03:12 GMT"],"Link":["<https://acme-v02.api.letsencrypt.org/directory>;rel=\"index\""],"Location":["https://acme-v02.api.letsencrypt.org/acme/order/190167180/22524948310"],"Replay-Nonce":["01017QlI5cvrjIwB8By6ZQa0gKV6KZe50qaSavjg_lQJ8aA"],"Server":["nginx"],"Strict-Transport-Security":["max-age=604800"],"X-Frame-Options":["DENY"]},"status_code":201}
Sep 06 21:03:12 : {"level":"debug","ts":1630962192.968534,"logger":"tls.issuance.acme.acme_client","msg":"http request","method":"POST","url":"https://acme-v02.api.letsencrypt.org/acme/authz-v3/28612805220","headers":{"Content-Type":["application/jose+json"],"User-Agent":["Caddy/2.4.5 CertMagic acmez (linux; amd64)"]},"response_headers":{"Boulder-Requester":["190167180"],"Cache-Control":["public, max-age=0, no-cache"],"Content-Length":["792"],"Content-Type":["application/json"],"Date":["Mon, 06 Sep 2021 21:03:12 GMT"],"Link":["<https://acme-v02.api.letsencrypt.org/directory>;rel=\"index\""],"Replay-Nonce":["0101e_tKwwjxDg3bat550b_ukaJlv7yssVFMfxgPi6cG_50"],"Server":["nginx"],"Strict-Transport-Security":["max-age=604800"],"X-Frame-Options":["DENY"]},"status_code":200}
Sep 06 21:05:16 : {"level":"debug","ts":1630962316.2646008,"logger":"tls.issuance.acme.acme_client","msg":"http request","method":"POST","url":"https://acme-v02.api.letsencrypt.org/acme/authz-v3/28612805220","headers":{"Content-Type":["application/jose+json"],"User-Agent":["Caddy/2.4.5 CertMagic acmez (linux; amd64)"]},"response_headers":{"Boulder-Requester":["190167180"],"Cache-Control":["public, max-age=0, no-cache"],"Content-Length":["796"],"Content-Type":["application/json"],"Date":["Mon, 06 Sep 2021 21:05:16 GMT"],"Link":["<https://acme-v02.api.letsencrypt.org/directory>;rel=\"index\""],"Replay-Nonce":["0101eaJ7dxUlfk31OFaX1ISz-NwAXiYveKynyt7TkCEyW1s"],"Server":["nginx"],"Strict-Transport-Security":["max-age=604800"],"X-Frame-Options":["DENY"]},"status_code":200}
Sep 06 21:05:16 : {"level":"debug","ts":1630962316.2652605,"logger":"tls.obtain","msg":"trying issuer 2/2","issuer":"acme.zerossl.com-v2-DV90"}
Sep 06 21:05:16 : {"level":"warn","ts":1630962316.2655997,"logger":"tls.issuance.zerossl","msg":"missing email address for ZeroSSL; it is strongly recommended to set one for next time"}
Sep 06 21:05:16 : {"level":"debug","ts":1630962316.4006872,"logger":"tls.issuance.acme.acme_client","msg":"http request","method":"POST","url":"https://acme-v02.api.letsencrypt.org/acme/authz-v3/28612803450","headers":{"Content-Type":["application/jose+json"],"User-Agent":["Caddy/2.4.5 CertMagic acmez (linux; amd64)"]},"response_headers":{"Boulder-Requester":["190167180"],"Cache-Control":["public, max-age=0, no-cache"],"Content-Length":["792"],"Content-Type":["application/json"],"Date":["Mon, 06 Sep 2021 21:05:16 GMT"],"Link":["<https://acme-v02.api.letsencrypt.org/directory>;rel=\"index\""],"Replay-Nonce":["0102ldmrP9GHYKl0HazpIVx7D70prm81sysPkcNKDWXNXmE"],"Server":["nginx"],"Strict-Transport-Security":["max-age=604800"],"X-Frame-Options":["DENY"]},"status_code":200}
Sep 06 21:05:16 : {"level":"error","ts":1630962316.4009762,"logger":"tls.obtain","msg":"could not get certificate from issuer","identifier":"zanj.cc","issuer":"acme-v02.api.letsencrypt.org-directory","error":"[zanj.cc] solving challenges: waiting for solver certmagic.solverWrapper to be ready: timed out waiting for record to fully propagate; verify DNS provider configuration is correct - last error: <nil> (order=https://acme-v02.api.letsencrypt.org/acme/order/190167180/22524946870) (ca=https://acme-v02.api.letsencrypt.org/directory)"}
Sep 06 21:05:16 : {"level":"debug","ts":1630962316.4011366,"logger":"tls.obtain","msg":"trying issuer 2/2","issuer":"acme.zerossl.com-v2-DV90"}
Sep 06 21:05:16 : {"level":"warn","ts":1630962316.401295,"logger":"tls.issuance.zerossl","msg":"missing email address for ZeroSSL; it is strongly recommended to set one for next time"}
Sep 06 21:05:16 : {"level":"info","ts":1630962316.821989,"logger":"tls.issuance.zerossl","msg":"generated EAB credentials","key_id":"TwzBMObpLUMV2XdnYCSVdw"}
Sep 06 21:05:16 : {"level":"info","ts":1630962316.8238626,"logger":"tls.issuance.zerossl","msg":"generated EAB credentials","key_id":"H-7vFrAuKc95kWHSjlUGIw"}
Sep 06 21:05:17 : {"level":"debug","ts":1630962317.3142114,"logger":"tls.issuance.acme.acme_client","msg":"http request","method":"GET","url":"https://acme.zerossl.com/v2/DV90","headers":{"User-Agent":["Caddy/2.4.5 CertMagic acmez (linux; amd64)"]},"response_headers":{"Access-Control-Allow-Origin":["*"],"Cache-Control":["max-age=-1"],"Content-Length":["645"],"Content-Type":["application/json"],"Date":["Mon, 06 Sep 2021 21:05:17 GMT"],"Server":["nginx"],"Strict-Transport-Security":["max-age=15552000"]},"status_code":200}
Sep 06 21:05:17 : {"level":"debug","ts":1630962317.668621,"logger":"tls.issuance.acme.acme_client","msg":"http request","method":"HEAD","url":"https://acme.zerossl.com/v2/DV90/newNonce","headers":{"User-Agent":["Caddy/2.4.5 CertMagic acmez (linux; amd64)"]},"response_headers":{"Access-Control-Allow-Origin":["*"],"Cache-Control":["max-age=-1"],"Content-Type":["application/octet-stream"],"Date":["Mon, 06 Sep 2021 21:05:17 GMT"],"Link":["<https://acme.zerossl.com/v2/DV90>;rel=\"index\""],"Replay-Nonce":["AwWVFXBynLMYMAZlx7v7K8yzdMuj82juDq1CjJrr3qk"],"Server":["nginx"],"Strict-Transport-Security":["max-age=15552000"]},"status_code":200}
Sep 06 21:05:17 : {"level":"debug","ts":1630962317.758527,"logger":"tls.issuance.acme.acme_client","msg":"http request","method":"HEAD","url":"https://acme.zerossl.com/v2/DV90/newNonce","headers":{"User-Agent":["Caddy/2.4.5 CertMagic acmez (linux; amd64)"]},"response_headers":{"Access-Control-Allow-Origin":["*"],"Cache-Control":["max-age=-1"],"Content-Type":["application/octet-stream"],"Date":["Mon, 06 Sep 2021 21:05:17 GMT"],"Link":["<https://acme.zerossl.com/v2/DV90>;rel=\"index\""],"Replay-Nonce":["3i1v8vH8BIo8aeI4FvreNXlhACDQl5E8uyPE9ViKBz0"],"Server":["nginx"],"Strict-Transport-Security":["max-age=15552000"]},"status_code":200}
Sep 06 21:05:18 : {"level":"debug","ts":1630962318.0557957,"logger":"tls.issuance.acme.acme_client","msg":"http request","method":"POST","url":"https://acme.zerossl.com/v2/DV90/newAccount","headers":{"Content-Type":["application/jose+json"],"User-Agent":["Caddy/2.4.5 CertMagic acmez (linux; amd64)"]},"response_headers":{"Access-Control-Allow-Origin":["*"],"Cache-Control":["max-age=0, no-cache, no-store","max-age=-1"],"Content-Length":["579"],"Content-Type":["application/json"],"Date":["Mon, 06 Sep 2021 21:05:18 GMT"],"Location":["https://acme.zerossl.com/v2/DV90/account/TwzBMObpLUMV2XdnYCSVdw"],"Replay-Nonce":["eVH4lswK5F8hSv-LLDx9zowht4L4Pi_6MpuJh2NccJo"],"Server":["nginx"],"Status":[""],"Strict-Transport-Security":["max-age=15552000"]},"status_code":201}
Sep 06 21:05:18 : {"level":"debug","ts":1630962318.1879897,"logger":"tls.issuance.acme.acme_client","msg":"http request","method":"POST","url":"https://acme.zerossl.com/v2/DV90/newAccount","headers":{"Content-Type":["application/jose+json"],"User-Agent":["Caddy/2.4.5 CertMagic acmez (linux; amd64)"]},"response_headers":{"Access-Control-Allow-Origin":["*"],"Cache-Control":["max-age=0, no-cache, no-store","max-age=-1"],"Content-Length":["579"],"Content-Type":["application/json"],"Date":["Mon, 06 Sep 2021 21:05:18 GMT"],"Location":["https://acme.zerossl.com/v2/DV90/account/H-7vFrAuKc95kWHSjlUGIw"],"Replay-Nonce":["T2QRTjZcR1NXUkyXfeV-cDhi07u-QXr46_1GZsV9zVw"],"Server":["nginx"],"Status":[""],"Strict-Transport-Security":["max-age=15552000"]},"status_code":201}
Sep 06 21:05:18 : {"level":"info","ts":1630962318.1884897,"logger":"tls.issuance.acme","msg":"waiting on internal rate limiter","identifiers":["zanj.cc"],"ca":"https://acme.zerossl.com/v2/DV90","account":""}
Sep 06 21:05:18 : {"level":"info","ts":1630962318.1886828,"logger":"tls.issuance.acme","msg":"done waiting on internal rate limiter","identifiers":["zanj.cc"],"ca":"https://acme.zerossl.com/v2/DV90","account":""}
Sep 06 21:05:18 : {"level":"debug","ts":1630962318.451321,"logger":"tls.issuance.acme.acme_client","msg":"http request","method":"POST","url":"https://acme.zerossl.com/v2/DV90/newOrder","headers":{"Content-Type":["application/jose+json"],"User-Agent":["Caddy/2.4.5 CertMagic acmez (linux; amd64)"]},"response_headers":{"Access-Control-Allow-Origin":["*"],"Cache-Control":["max-age=0, no-cache, no-store","max-age=-1"],"Content-Length":["273"],"Content-Type":["application/json"],"Date":["Mon, 06 Sep 2021 21:05:18 GMT"],"Location":["https://acme.zerossl.com/v2/DV90/order/oFEqdzKp7aH_99JM3RZh1w"],"Replay-Nonce":["kB__iGipiUFWBaDM2o8q_DmkqxzOfbI5v7qYJzoKPiw"],"Server":["nginx"],"Status":[""],"Strict-Transport-Security":["max-age=15552000"]},"status_code":201}
Sep 06 21:05:18 : {"level":"debug","ts":1630962318.6361442,"logger":"tls.issuance.acme.acme_client","msg":"http request","method":"POST","url":"https://acme.zerossl.com/v2/DV90/newOrder","headers":{"Content-Type":["application/jose+json"],"User-Agent":["Caddy/2.4.5 CertMagic acmez (linux; amd64)"]},"response_headers":{"Access-Control-Allow-Origin":["*"],"Cache-Control":["max-age=0, no-cache, no-store","max-age=-1"],"Content-Length":["269"],"Content-Type":["application/json"],"Date":["Mon, 06 Sep 2021 21:05:18 GMT"],"Location":["https://acme.zerossl.com/v2/DV90/order/COmGbgu4jxYq1romFsnqCg"],"Replay-Nonce":["VGCPsrLJYtDnjvCvJvZOqD2wOuamp3QKZSd4DdXL6Lw"],"Server":["nginx"],"Status":[""],"Strict-Transport-Security":["max-age=15552000"]},"status_code":201}
Sep 06 21:05:18 : {"level":"debug","ts":1630962318.8162355,"logger":"tls.issuance.acme.acme_client","msg":"http request","method":"POST","url":"https://acme.zerossl.com/v2/DV90/authz/9z1CngFW1TkV_3Nctn76oQ","headers":{"Content-Type":["application/jose+json"],"User-Agent":["Caddy/2.4.5 CertMagic acmez (linux; amd64)"]},"response_headers":{"Access-Control-Allow-Origin":["*"],"Cache-Control":["max-age=-1"],"Content-Length":["441"],"Content-Type":["application/json"],"Date":["Mon, 06 Sep 2021 21:05:18 GMT"],"Link":["<https://acme.zerossl.com/v2/DV90>;rel=\"index\""],"Replay-Nonce":["goW_dI_VfddHoW_jYHU2FYqN4OMs1nddPVLoW1e7tFU"],"Retry-After":["5"],"Server":["nginx"],"Strict-Transport-Security":["max-age=15552000"]},"status_code":200}
Sep 06 21:05:19 : {"level":"debug","ts":1630962319.0101697,"logger":"tls.issuance.acme.acme_client","msg":"http request","method":"POST","url":"https://acme.zerossl.com/v2/DV90/authz/841dL8yV6Vxzir4LcYIaNA","headers":{"Content-Type":["application/jose+json"],"User-Agent":["Caddy/2.4.5 CertMagic acmez (linux; amd64)"]},"response_headers":{"Access-Control-Allow-Origin":["*"],"Cache-Control":["max-age=-1"],"Content-Length":["437"],"Content-Type":["application/json"],"Date":["Mon, 06 Sep 2021 21:05:18 GMT"],"Link":["<https://acme.zerossl.com/v2/DV90>;rel=\"index\""],"Replay-Nonce":["ZwUd8Gf6y9FAAjXmuek5iFA9oCBRkAWG8mZ0oGs8Ez0"],"Retry-After":["5"],"Server":["nginx"],"Strict-Transport-Security":["max-age=15552000"]},"status_code":200}
Sep 06 21:05:19 : {"level":"info","ts":1630962319.010456,"logger":"tls.issuance.acme.acme_client","msg":"trying to solve challenge","identifier":"zanj.cc","challenge_type":"dns-01","ca":"https://acme.zerossl.com/v2/DV90"}

The split dns is all off now, I checked it is in fact using 1.1.1.1, and it seems to be (via dig). My router doesn’t have any dns cache and the new lxc will definitely not have one - i made it since disabling/byassing pihole (when i set the dhcp server’s to offer 1.1.1.1 to clients). I did try the resolvers but that didn’t change anything.

Thanks though, if the debug log doesn’t help, I’m hoping Matt comes along - I’ve read he uses cloudflare…

shar · September 7, 2021, 11:22am

So I bodged this (not a solution!) - I disabled proxy on cloudflare, ie it just handles dns, then removed the tls option from the config, and domain/subdomain got certificates through http-01. Turned proxy back on in cloudflare and now it’s all fine again … until that is I need to renew the certificate in 90 days!

I’d still appreciate any input on where the dns-01 challenge might have been falling down…

shar · September 8, 2021, 2:20pm

I have still not made any progress (got this first site working fine as above - through turning off cloudflare proxy and dumping the tls section from the caddyfile… I double checked and the challenge which did work was http-01, which I think is the default first one to try in a default address line?

I wonder if someone might be able to suggest where the dns-01 challenge falls down? I want to make sure my local network isn’t causing problems, as even after i think I made everything normal/drop firewall rules/split dns etc, it still seems like it could be my lan at fault as I’ve seen plenty of people use /caddy-dns/cloudflare without dramas…

In my lan, do I need to forward both port 80 and 443 to the host running caddy?
Does this error: timed out waiting for record to fully propagate; verify DNS provider configuration is correct mean theres an issue with my caddy host’s networking/dns/other?
or might it be something to do with cloudflare setup - perhaps there’s some setting in the cf firewall/ssl/other settings i’ve changed and since overlooked?
Does the the challenge come back to my caddy server at all, or is it more: once caddy has set the TXT record _acme-challenge, then does letsencrypt/zerossl communicate direct with cloudflare?

As must be clear, I don’t really understand how this all works, and I’m not really making any headway with the docs/similar forum issues, or elsewhere! The few issues which seem similar are often closed unfinished, hopefully I can get to the bottom of this and perhaps a future searcher might find use!? Hope my questions aren’t too stupid!

Even though this is working with the original domain, I have subdomains I want to also use for this and would love to have the cloudflare proxy working! Thanks.

francislavoie · September 8, 2021, 4:55pm

To solve both HTTP and TLS-ALPN challenges, it’s strongly recommended, yes.

It’s not necessary if using the DNS though, because ACME CAs don’t need to reach your server directly.

Caddy has a repeated loop where it tries to query for the DNS TXT record to see if the call it made to Cloudflare’s API had the intended effect. Once it sees the TXT record, then it will move onto the next step.

But what happens for you is Caddy times out because it wasn’t able to find the TXT record during that loop, and gives up after a while (I think by default, the timeout is 2 minutes).

If you’re saying that you do see the TXT record on your domain within those two minutes, then it means that something in your setup is probably preventing Caddy from using public DNS, and instead reporting maybe some too-aggressively-cached DNS records or your local DNS override.

That’s what the resolvers config is supposed to work around, but

ACME CAs will do their own DNS queries to verify the challenge in the TXT record, but only once Caddy tells them “ok I did it, should be good to go”, but it doesn’t reach that step because Caddy isn’t able to verify for itself that it worked.

The ACME CAs don’t really “know” about cloudflare, they just do DNS queries. Only Caddy needs to “know” about cloudflare, because you need to use their API to allow Caddy to set the TXT record during automation.

shar · September 10, 2021, 8:45am

Thank you so much for going into all this detail Francis, really helps me with the next troubleshooting steps - maybe not what to do but at least where to look!!

I’ve gone back to check the timing of the TXT record appearing on the cloudflare dashboard… It showed within about 20 seconds, so that step should be ok…

OK this all seems to come back to my network DNS, which I have possibly changed since setting up caddy months ago so that would make sense.

I don’t have a very-simple network at home, but it’s not that complicated! I do have a couple of pihole DNS servers, and until I got this issue last week I did have a NAT rule to redirect all :53 requests to these. I have :853 DoT blocked (caddy doesnt use dns over tls does it???). But I’ve since disabled the NAT rule, and I think validated that the caddy server can indeed access 1.1.1.1:53 :

I’ve checked https://1.1.1.1/help from another machine in the same vlan - and it suggests it IS accessible - so I don’t think there will be any lingering firewall rules blocking/redirecting to pihole.
I also checked dig on the caddy server CT:

; <<>> DiG 9.11.5-P4-5.1+deb10u5-Debian <<>> @1.1.1.1 caddy.community
; (1 server found)
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 5355
;; flags: qr rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 1232
;; QUESTION SECTION:
;caddy.community.               IN      A

;; ANSWER SECTION:
caddy.community.        300     IN      A       159.89.152.193

;; Query time: 75 msec
;; SERVER: 1.1.1.1#53(1.1.1.1)
;; WHEN: Fri Sep 10 08:27:42 UTC 2021
;; MSG SIZE  rcvd: 60

Seems OK!?

Next I’ve tried clearing the server’s dns cache with this systemd-resolve --flush-caches, I did just now after the TXT record appeared and… I think… before the end of the 2 minute window! No dice. Not sure if caddy uses the host cache or not, or whether clearing it here makes any difference - clearly not!

Last gasp, am I staring right at the issue in an obvious place? My last idea would be to test caddy on a pi directly plugged into my modem skipping the router and network all together but that seems a bit drastic, and doesnt really fix the network problems.

EDIT:
This is all with resolvers 1.1.1.1 included now.

francislavoie · September 10, 2021, 10:14am

Not directly, I think. It’ll use whatever DNS servers are configured on your system by default, using Go’s stdlib and the GitHub - miekg/dns: DNS library in Go library. The queries happen here:

github.com

caddyserver/certmagic/blob/b74a9ce8088fde76facb32d545d2494f52b227fe/dnsutil.go#L143


      
          func createDNSMsg(fqdn string, rtype uint16, recursive bool) *dns.Msg {
          	m := new(dns.Msg)
          	m.SetQuestion(fqdn, rtype)
          	m.SetEdns0(4096, false)
          	if !recursive {
          		m.RecursionDesired = false
          	}
          	return m
          }
          
          func sendDNSQuery(m *dns.Msg, ns string) (*dns.Msg, error) {
          	udp := &dns.Client{Net: "udp", Timeout: dnsTimeout}
          	in, _, err := udp.Exchange(m, ns)
          	// two kinds of errors we can handle by retrying with TCP:
          	// truncation and timeout; see https://github.com/caddyserver/caddy/issues/3639
          	truncated := in != nil && in.Truncated
          	timeoutErr := err != nil && strings.Contains(err.Error(), "timeout")
          	if truncated || timeoutErr {
          		tcp := &dns.Client{Net: "tcp", Timeout: dnsTimeout}
          		in, _, err = tcp.Exchange(m, ns)
          	}

I’m not sure what else to suggest

system · October 6, 2021, 8:07pm

This topic was automatically closed after 30 days. New replies are no longer allowed.

shar · February 14, 2022, 4:22pm

1. Caddy version (`caddy version`):

2.4.5

2. How I run Caddy:

a. System environment:

debian 10 LXC on proxmox
only services running are caddy and ddclient

b. Command:

systemctl start caddy

d. My complete Caddyfile or JSON config:

zanj.cc {
        tls {
                dns cloudflare API_TOKEN
        }
}

3. The problem I’m having:

This is the same problem I’ve written about previously at this thread which is now locked - I’ve put up with it by turning off the Cloudflare dns proxy and removing the tls line from my Caddyfile, renewing certs and then turning the cloudflare proxy back on. ie not automatic - I’ve come back to do this manually a few times since originally posting this issue - finally finding some free time to (hopefully) find a solution!

I’ve tried this with a container, starting from a fresh caddy & cloudflare dns installer, with the container resolv.conf giving 1.1.1.1 and 1.0.0.1 as dns servers, verified by $ dig caddy.community returning 1.1.1.1#53 as expected (I had a redirect to only allow the pihole servers access to 53, but have turned this off for now).

I still get the following errors in next section.

4. Error messages and/or full log output:

Feb 14 15:58:33 caddy caddy[12711]: {"level":"info","ts":1644854313.6137033,"logger":"tls.issuance.acme.acme_client","msg":"trying to solve challenge","identifier":"zanj.cc","challenge_type":"dns-01","ca":"https://acme-staging-v02.api.letsencrypt.org/directory"}
Feb 14 16:00:37 caddy caddy[12711]: {"level":"error","ts":1644854437.4932487,"logger":"tls.obtain","msg":"could not get certificate from issuer","identifier":"zanj.cc","issuer":"acme-v02.api.letsencrypt.org-directory","error":"[zanj.cc] solving challenges: waiting for solver certmagic.solverWrapper to be ready: timed out waiting for record to fully propagate; verify DNS provider configuration is correct - last error: <nil> (order=https://acme-staging-v02.api.letsencrypt.org/acme/order/25750648/1787790298) (ca=https://acme-staging-v02.api.letsencrypt.org/directory)"}

5. What I already tried:

Verified dig caddy.community returns 1.1.1.1#53 as expected
Created new lxc and installed caddy & cloudflare dns challenger as per the install instructions
Watched the cloudflare DNS dashboard after starting caddy (systemctl restart caddy), waited until the log shows trying to solve challenge - and within ~15 seconds a TXT record is added: _acme-challenge and contents LONG_STRING_OF_TEXT
As per previous thread - I’m not sure what is next to try?!

6. Links to relevant resources:

My previous thread https://caddy.community/t/verify-dns-provider-configuration-is-correct-dns-01-challenge-fails-cloudflare/13568)

francislavoie · February 14, 2022, 8:30pm

I reopened this topic and moved your new reply back into here.

I’m not sure what the problem is though hopefully someone else can help

shar · February 14, 2022, 9:14pm

Thanks! I hope so too!

Mohammed90 · February 14, 2022, 10:00pm

Nothing seems wrong from here. It isn’t clear why Caddy can’t see the TXT record. Can we experiment to see how long it might take for Caddy to actually see the record? Set propagation_timeout to something larger than 2 minutes. 10 minutes? 20? Feel free to stop experimenting at something reasonable.

shar · February 15, 2022, 10:23am

Thanks Mohammed… I just tried with a 10 and 30 min timeout. No dice.

Stopped caddy, then deleted the txt dns record on cloudflare.
Restarted caddy and waited for the log to show a challenge was sent.
I then did this:

$ dig @ray.ns.cloudflare.com _acme-challenge.zanj.cc txt

which gave response:

; <<>> DiG 9.11.5-P4-5.1+deb10u6-Debian <<>> @ray.ns.cloudflare.com _acme-challenge.zanj.cc txt
; (6 servers found)
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 63156
;; flags: qr rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 1232
;; QUESTION SECTION:
;_acme-challenge.zanj.cc. IN      TXT

;; ANSWER SECTION:
_acme-challenge.zanj.cc. 290 IN   TXT     "Aw6w3LTKnSbZi2DCpKxdihiN3DVoysXpVC-BM8MQRTo"

;; Query time: 19 msec
;; SERVER: 108.162.193.138#53(108.162.193.138)
;; WHEN: Tue Feb 15 09:58:46 UTC 2022
;; MSG SIZE  rcvd: 114

The answer matches the new record I can see on the cloudflare dns dashboard. Which, if I’m not misunderstanding something(?), re-enforces my view that the container/my network doesn’t have anything cacheing or otherwise preventing caddy from seeing the txt record. No idea what else it could be!

Mohammed90 · February 15, 2022, 11:40am

Odd. Can you rebuild Caddy but with CGO enabled? Use this command:

CGO_ENABLED=1 xcaddy build --with github.com/caddy-dns/cloudflare

shar · February 15, 2022, 1:14pm

Thanks, I don’t understand what CGO is but more than happy to try other things… I have done that - rebuilt and it appears to have worked, version is updated to 2.4.6 from previous 2.4.5.

Nothing new in the logs after starting caddy though, it still looks to be failing to see the record, timeout 10m, dig gives the correct reply as before.

Mohammed90 · February 15, 2022, 5:28pm

Go netstack can be backed either by libc or pure Go. The default mode when building caddy using xcaddy is to check if CGO_ENABLED is explicitly enabled, otherwise it opts for pure-Go build, which means Go uses the pure Go netstack. I was trying to see if the C/libc backed netstack of Go works differently.

Bear with me as we’re developing the hypotheses and test them out Do you happen to have SELinux enabled?

shar · February 15, 2022, 8:40pm

Thanks Mohammed, really kind of you to explain and to help!

This lxc does not have selinux.

shar · February 17, 2022, 11:07am

I’ve tried to do a bit more research but not sure if I’m on the right track… can’t see I’ve made any progress tbh!

Does Caddy by default use nameservers from /etc/resolv.conf? I’ve made these 1.1.1.1 and 1.0.0.1. Regardless, I have also set resolvers 1.1.1.1 in the caddyfile but wanted to make sure I had both options covered.

I can rundig @ray.ns.cloudflare.com _acme-challenge.zanj.cc txt and get a TXT response which matches what I can see in the cloudflare dashboard. Does this mean that a) the cloudflare dns plugin is doing its thing correctly, and b) the container has correct outbound firewall permissions? Does this eliminate, correspondingly a) the cloudflare API stuff, and b) my lan/proxmox setup from being at fault in preventing caddy from reading the txt file on cloudflare?

Yet everything I’ve read suggests that the error i see is usually caused by local network issues!

Could it be a permission thing on the local machine - caddy doesn’t need superuser to run whatever it does equivalent to dig ?

Verify DNS provider configuration is correct - dns-01 challenge fails - cloudflare

1. Caddy version (caddy version):

2. How I run Caddy:

a. System environment:

b. Command:

c. Service/unit/compose file:

d. My complete Caddyfile or JSON config:

3. The problem I’m having:

4. Error messages and/or full log output:

5. What I already tried:

1. Caddy version (caddy version):

2. How I run Caddy:

a. System environment:

b. Command:

d. My complete Caddyfile or JSON config:

3. The problem I’m having:

4. Error messages and/or full log output:

5. What I already tried:

6. Links to relevant resources:

1. Caddy version (`caddy version`):

1. Caddy version (`caddy version`):