[SOLVED] Caddy and Let's Encrypt CloudFlare DNS challenge not working

1. The problem I’m having:

I cannot obtain a TLS certificate via Let’s Encrypt using CloudFlare DNS challenge.
I’ve verified that caddy can successfully create the ACME TXT record on CloudFlare.
However, caddy does not seem to be able to confirm that the record is created. I tried to configure my Caddyfile with propagation_timeout -1 in the hope that it would not check if the record was created, but this does not seem to work and the challenge cannot complete.

2. Error messages and/or full log output:

 ./caddy_darwin_arm64_custom run
2024/05/06 14:42:43.970 INFO    using adjacent Caddyfile
2024/05/06 14:42:43.972 INFO    admin   admin endpoint started  {"address": "localhost:2019", "enforce_origin": false, "origins": ["//[::1]:2019", "//127.0.0.1:2019", "//localhost:2019"]}
2024/05/06 14:42:43.972 INFO    tls.cache.maintenance   started background certificate maintenance      {"cache": "0x140006a3d00"}
2024/05/06 14:42:43.972 INFO    http.auto_https server is listening only on the HTTPS port but has no TLS connection policies; adding one to enable TLS {"server_name": "srv0", "https_port": 443}
2024/05/06 14:42:43.972 INFO    http.auto_https enabling automatic HTTP->HTTPS redirects        {"server_name": "srv0"}
2024/05/06 14:42:43.972 INFO    http    enabling HTTP/3 listener        {"addr": ":443"}
2024/05/06 14:42:43.973 INFO    http.log        server running  {"name": "srv0", "protocols": ["h1", "h2", "h3"]}
2024/05/06 14:42:43.973 INFO    http.log        server running  {"name": "remaining_auto_https_redirects", "protocols": ["h1", "h2", "h3"]}
2024/05/06 14:42:43.973 INFO    http    enabling automatic TLS certificate management   {"domains": ["vault2.rhaidiz.net"]}
2024/05/06 14:42:43.973 INFO    autosaved config (load with --resume flag)      {"file": "/Users/federicodemeo/Library/Application Support/Caddy/autosave.json"}
2024/05/06 14:42:43.973 INFO    serving initial configuration
2024/05/06 14:42:43.974 INFO    tls.obtain      acquiring lock  {"identifier": "vault2.rhaidiz.net"}
2024/05/06 14:42:43.979 WARN    tls     storage cleaning happened too recently; skipping for now        {"storage": "FileStorage:/Users/federicodemeo/Library/Application Support/Caddy", "instance": "9ae59042-52e9-4d4e-b09d-79bae33db1a6", "try_again": "2024/05/07 14:42:43.979", "try_again_in": 86399.999999542}
2024/05/06 14:42:43.979 INFO    tls     finished cleaning storage units
2024/05/06 14:42:43.983 INFO    tls.obtain      lock acquired   {"identifier": "vault2.rhaidiz.net"}
2024/05/06 14:42:43.983 INFO    tls.obtain      obtaining certificate   {"identifier": "vault2.rhaidiz.net"}
2024/05/06 14:42:43.985 INFO    tls.issuance.acme       waiting on internal rate limiter        {"identifiers": ["vault2.rhaidiz.net"], "ca": "https://acme-v02.api.letsencrypt.org/directory", "account": ""}
2024/05/06 14:42:43.986 INFO    tls.issuance.acme       done waiting on internal rate limiter   {"identifiers": ["vault2.rhaidiz.net"], "ca": "https://acme-v02.api.letsencrypt.org/directory", "account": ""}
2024/05/06 14:42:46.283 INFO    tls.issuance.acme.acme_client   trying to solve challenge       {"identifier": "vault2.rhaidiz.net", "challenge_type": "dns-01", "ca": "https://acme-v02.api.letsencrypt.org/directory"}
2024/05/06 14:42:49.313 ERROR   tls.issuance.acme.acme_client   challenge failed        {"identifier": "vault2.rhaidiz.net", "challenge_type": "dns-01", "problem": {"type": "urn:ietf:params:acme:error:dns", "title": "", "detail": "DNS problem: NXDOMAIN looking up TXT for _acme-challenge.vault2.rhaidiz.net - check that a DNS record exists for this domain", "instance": "", "subproblems": []}}
2024/05/06 14:42:49.313 ERROR   tls.issuance.acme.acme_client   validating authorization        {"identifier": "vault2.rhaidiz.net", "problem": {"type": "urn:ietf:params:acme:error:dns", "title": "", "detail": "DNS problem: NXDOMAIN looking up TXT for _acme-challenge.vault2.rhaidiz.net - check that a DNS record exists for this domain", "instance": "", "subproblems": []}, "order": "https://acme-v02.api.letsencrypt.org/acme/order/1711691297/267097986617", "attempt": 1, "max_attempts": 3}
2024/05/06 14:42:49.313 ERROR   tls.obtain      could not get certificate from issuer   {"identifier": "vault2.rhaidiz.net", "issuer": "acme-v02.api.letsencrypt.org-directory", "error": "HTTP 400 urn:ietf:params:acme:error:dns - DNS problem: NXDOMAIN looking up TXT for _acme-challenge.vault2.rhaidiz.net - check that a DNS record exists for this domain"}
2024/05/06 14:42:49.317 INFO    tls.issuance.zerossl    waiting on internal rate limiter        {"identifiers": ["vault2.rhaidiz.net"], "ca": "https://acme.zerossl.com/v2/DV90", "account": "caddy@zerossl.com"}
2024/05/06 14:42:49.317 INFO    tls.issuance.zerossl    done waiting on internal rate limiter   {"identifiers": ["vault2.rhaidiz.net"], "ca": "https://acme.zerossl.com/v2/DV90", "account": "caddy@zerossl.com"}
2024/05/06 14:42:50.785 INFO    tls.issuance.zerossl.acme_client        trying to solve challenge       {"identifier": "vault2.rhaidiz.net", "challenge_type": "dns-01", "ca": "https://acme.zerossl.com/v2/DV90"}

3. Caddy version:

v2.7.6 h1:w0NymbG2m9PcvKWsrXO6EEkY9Ru4FJK8uQbYcev1p3A=

4. How I installed and ran Caddy:

I went to Download Caddy, searched for cloudflare-dns, selected it, made sure Apple Silicon was selected and downloaded.
I run caddy like this ./caddy_darwin_arm64_custom run.

a. System environment:

MacOS 14.4.1.

b. Command:

 ./caddy_darwin_arm64_custom run

c. Service/unit/compose file:

N/A

d. My complete Caddy config:

vault2.rhaidiz.net {
	respond "Welcome to TLS"
	tls {
		dns cloudflare REDACTED
		propagation_timeout -1
		resolvers 8.8.8.8
	}
}

5. Links to relevant resources:

N/A

Hello @rhaidiz,

I see troubles with the DNS and the NAME SERVERS;
that would be a problem for Let’s Encrypt even with the DNS-01 challenge

Edit: may bad; I had a type missed the ending t on the TLD. Sorry! :frowning:

And from around the world all get “Not found”
here Permanent link to this check report for vault2.rhaidiz.ne

Using the SOA’s

$ nslookup -q=ns vault2.rhaidiz.net brady.ns.cloudflare.com.
Server:         brady.ns.cloudflare.com.
Address:        172.64.35.215#53

*** Can't find vault2.rhaidiz.net: No answer

Just using the my local name resolver

$ nslookup -q=ns vault2.rhaidiz.net
Server:         127.0.0.53
Address:        127.0.0.53#53

Non-authoritative answer:
*** Can't find vault2.rhaidiz.net: No answer

Authoritative answers can be found from:
rhaidiz.net
        origin = brady.ns.cloudflare.com
        mail addr = dns.cloudflare.com
        serial = 2340494265
        refresh = 10000
        retry = 2400
        expire = 604800
        minimum = 1800

Edit: and I kind of find that hidden in your post too.
(I think those line get a bit long, and hard to find the correct detail if one is not sure what they are looking for).

Uhm, I’m not sure I follow, I might be missing something.
The DNS record for vault2.rhaidiz.net is configured and I can see it also from around the world.

As I’m not using different namersers vault2.rhaidiz.net I don’t understand the need of checking that record.

The error I receive in the logs suggests that Caddy cannot find the TXT record for _acme-challenge.vault2.rhaidiz.net, even tough I can see it in the CloudFlare dashboard.

The resolvers option with the Google DNS (yes, I’ve tried also with 1.1.1.1) should also help in allowing Caddy to see the record, but it isn’t.
Assuming there is no issue in the implementation of the DNS challenge, it would look very much like a networking issue. However, I tried certbot and it can complete the DNS challenge easily and without problems.
There must be something in the network stack of Caddy that is not working differently.

1 Like

@rhaidiz yes you are correct; I made a typo, sorry! :frowning:
I’m looking further right now.

Here is a list of issued certificates crt.sh | vault2.rhaidiz.net, the latest being 2024-05-06. So it looks like you were successful in getting a certificate.

@rhaidiz,

Using the online tool https://unboundtest.com/ yields
https://unboundtest.com/m/TXT/_acme-challenge.vault2.rhaidiz.net/FDJV3BLR

Query results for TXT _acme-challenge.vault2.rhaidiz.net

Response:
;; opcode: QUERY, status: NOERROR, id: 30039
;; flags: qr rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version 0; flags: do; udp: 512

;; QUESTION SECTION:
;_acme-challenge.vault2.rhaidiz.net.	IN	 TXT

;; ANSWER SECTION:
_acme-challenge.vault2.rhaidiz.net.	0	IN	TXT	"oWdOmG_Luck5hX2pJFMw0AZ6wmGRi3HmtNTnwj165aE"

----- Unbound logs -----
May 06 16:18:28 unbound1.19[2025593:0] debug: creating udp6 socket ::1 1053
May 06 16:18:28 unbound1.19[2025593:0] debug: creating tcp6 socket ::1 1053

This seems to show a DNS TXT record is present
;; ANSWER SECTION:
_acme-challenge.vault2.rhaidiz.net. 0 IN TXT “oWdOmG_Luck5hX2pJFMw0AZ6wmGRi3HmtNTnwj165aE”

Yes, as I mentioned I tried certbot and with that I could get a certificate.
I’m also now noticing something interesting, I fired up Wireshark and noticed that the DNS request is made for the following domain.
_acme-challenge.vault2.rhaidiz.net.RhAidIz.net: type TXT, class IN.

What am I missing? Why is it _acme-challenge.vault2.rhaidiz.net.RhAidIz.net and not _acme-challenge.vault2.rhaidiz.net ??

1 Like

Hmm.

The UPPER and lower case is part of the validation (randomizing the domain name’s case, since it it case insensitive, is to help improve validation).
Edit: also the link here if one read all the output will see Upper and lower case mix usage in the domain name.

But the issue that the rhaidiz.net is being appended to the whole domain name is a bit confusing presently.

Still investigating

I get this with nslookup, which seem correct.

$ nslookup -q=txt _acme-challenge.vault2.rhaidiz.net brady.ns.cloudflare.com.
Server:         brady.ns.cloudflare.com.
Address:        172.64.35.215#53

_acme-challenge.vault2.rhaidiz.net      text = "oWdOmG_Luck5hX2pJFMw0AZ6wmGRi3HmtNTnwj165aE"

@rhaidiz side note:

You are not at the limit, I presently use “2 of 5 weekly certificates”.

Testing and debugging are best done using the Staging Environment as the Rate Limits are much higher.

And here I see a second TXT Record
https://unboundtest.com/m/TXT/_acme-challenge.vault2.rhaidiz.net/5VYFF5WM

Query results for TXT _acme-challenge.vault2.rhaidiz.net

Response:
;; opcode: QUERY, status: NOERROR, id: 17823
;; flags: qr rd ra; QUERY: 1, ANSWER: 2, AUTHORITY: 0, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version 0; flags: do; udp: 512

;; QUESTION SECTION:
;_acme-challenge.vault2.rhaidiz.net.	IN	 TXT

;; ANSWER SECTION:
_acme-challenge.vault2.rhaidiz.net.	0	IN	TXT	"uAI5x7kl2jufQGGd5ZfvHSpUiopiok_gn4iQaXRHXtA"
_acme-challenge.vault2.rhaidiz.net.	0	IN	TXT	"J1WUvcdUbnq9t2tZRXXI_cLQTrld_obp_tnuEpRym_0"

----- Unbound logs -----
May 06 16:44:22 unbound1.19[2025688:0] debug: creating udp6 socket ::1 1053
May 06 16:44:22 unbound1.19[2025688:0] debug: creating tcp6 socket ::1 1053

I am not finding anything presently @rhaidiz.

Kindly wait for more knowledgeable Caddy community volunteers to assist.

3 Likes

Thanks for the heads up.

I’ve changed the config as follows

vault2.rhaidiz.net {
	respond "Welcome to TLS"
	tls {
	issuer acme {
	    dir https://acme-staging-v02.api.letsencrypt.org/directory
		dns cloudflare REDACTED
		propagation_timeout 20m
		resolvers 8.8.8.8
		}
	}
}

I am now seeing via Wireshark that it tries the correct TXT record and could successfully complete the challenge.
I really have no idea what is happening, I’m quite confused :thinking: !

1 Like

Yeah, when Cloudflare is involve I often get lost (that is not meant to be taken as a negative about Cloudflare, just my own limitations).

And now again I get this

18:49:22.256431 IP 192.168.33.99.36856 > 8.8.8.8.53: 1748+ [1au] TXT? _acme-challenge.vault2.rhaidiz.net.RHAIDiz.net. (75)
18:49:22.266564 IP 8.8.8.8.53 > 192.168.33.99.36856: 1748 NXDomain 0/1/1 (163)

what is going on :see_no_evil: !

1 Like

Kindly wait for more knowledgeable Caddy community volunteers to assist. :slight_smile:

This is interesting; I see

$ nslookup -q=txt _acme-challenge.vault2.rhaidiz.net brady.ns.cloudflare.com.
Server:         brady.ns.cloudflare.com.
Address:        162.159.44.215#53

_acme-challenge.vault2.rhaidiz.net      text = "sHBkgmBq7GwPs1eawvsWCCcqOMHp9J3DQ-M-_Z_m7_k"

Yet this online test does not see any TXT

1 Like

I am running some tests and currently have removed the record. It might come up and down as I’m trying things out.

1 Like

Update on this. I’ve been suspecting this for a while, but has I had a pihole setup I didn’t really get too deep into tinkering with other configuration, but essentially my ISP is doing some DNS interference on my network. At first I thought they where doing this on their side, turns out they are doing this directly into the router they provided me with. Thankfully this idiotic behavior can be disabled and now Caddy can solve the challenge with absolutely no issue.
I’m still a little puzzled as to why Caddy was appending the domain two times when it had difficulty in finding the TXT record, which also did not always happened. Must be some specifics on DNS that I’m not familiar with.

Thank you @Bruce5051 for your prompt support, much appreciate it :pray: .

3 Likes