Recent Issues with Cloudflare DNS Challenges

1. Caddy version (caddy version):

v2.4.0-beta.1 h1:Ed/tIaN3p6z8M3pEiXWJL/T8JmCqV62FrSJCHKquW/I=

2. How I run Caddy:

Via Docker (Deployed from Portainer, if it matters): slothcroissant/caddy-cloudflaredns - Docker Hub

Dockerfile:

version: "2"

services:
  app:
    image: slothcroissant/caddy-cloudflaredns
    container_name: caddy
    environment:
      - CLOUDFLARE_API_TOKEN=12345abcdef
      - ACME_AGREE=true
      - CLOUDFLARE_EMAIL=me@email.com
    volumes:
      - /mnt/zfs/docker/caddy/config:/config
      - /mnt/zfs/docker/caddy/data:/data
      - /mnt/zfs/docker/caddy/Caddyfile:/etc/caddy/Caddyfile
    restart: always
    ports:
      - 80:80
      - 443:443
    networks:
      - mybridge

networks:
  mybridge:
    external:
      name: mybridge

System environment:

Ubuntu 20.04 Server & Docker version 20.10.2, build 2291f61

My complete Caddyfile or JSON config:

{
	acme_dns cloudflare {env.CLOUDFLARE_API_TOKEN}
	email me@email.com
}

(snippet) {
	header {
		Strict-Transport-Security "max-age=31536000; includeSubdomains"
		X-Real-IP {http.request.remote}
		Host {host}
		X-XSS-Protection "1; mode=block"
		X-Frame-Options "DENY"
		-server
	}
	tls {
		resolvers 1.1.1.1,1.0.0.1,8.8.8.8,8.8.4.4
	}
	log {
		output file /data/logs/caddy.log {
			roll_size 20MiB
			roll_keep 5
		}
	}
}

dns03.ryanb.tv {
	reverse_proxy dns03.lan:80
	import snippet
}

3. The problem I’m having:

First off, this has been working great since the Caddy V1 days. No issues with Cloudflare in years. In fact, I moved from my own wildcard cert (provided free by my work for lab use) because I enjoy how easy Caddy is with LetsEncrypt and its hands-off renewals.

The gist here is that some of my sites are coming due for their cert renewals, and the cert renewal is failing as the API calls to Cloudflare don’t seem to be properly plumbing the TXT challenge records. I’m using one site here as an example, as the logs, issues and details of each are exactly the same.

4. Error messages and/or full log output:

{"level":"info","ts":1615905294.5129788,"logger":"tls.cache.maintenance","msg":"certificate expires soon; queuing for renewal","identifiers":["dns03.ryanb.tv"],"remaining":1901721.487022019}
{"level":"info","ts":1615905305.2825034,"logger":"tls.renew","msg":"renewing certificate","identifier":"dns03.ryanb.tv","remaining":1901710.717501787}
{"level":"info","ts":1615905305.6328773,"logger":"tls.issuance.acme.acme_client","msg":"trying to solve challenge","identifier":"dns03.ryanb.tv","challenge_type":"dns-01","ca":"https://acme-staging-v02.api.letsencrypt.org/directory"}
{"level":"error","ts":1615905306.7431986,"logger":"tls.issuance.acme.acme_client","msg":"challenge failed","identifier":"dns03.ryanb.tv","challenge_type":"dns-01","status_code":400,"problem_type":"urn:ietf:params:acme:error:dns","error":"DNS problem: NXDOMAIN looking up TXT for _acme-challenge.dns03.ryanb.tv - check that a DNS record exists for this domain"}
{"level":"error","ts":1615905306.7432632,"logger":"tls.issuance.acme.acme_client","msg":"validating authorization","identifier":"dns03.ryanb.tv","error":"authorization failed: HTTP 400 urn:ietf:params:acme:error:dns - DNS problem: NXDOMAIN looking up TXT for _acme-challenge.dns03.ryanb.tv - check that a DNS record exists for this domain","order":"https://acme-staging-v02.api.letsencrypt.org/acme/order/13551060/10605424","attempt":1,"max_attempts":3}
{"level":"error","ts":1615905308.0974076,"logger":"tls.renew","msg":"will retry","error":"[dns03.ryanb.tv] Renew: [dns03.ryanb.tv] solving challenges: dns03.ryanb.tv: no solvers available for remaining challenges (configured=[dns-01] offered=[http-01 dns-01 tls-alpn-01] remaining=[http-01 tls-alpn-01]) (order=https://acme-staging-v02.api.letsencrypt.org/acme/order/13551060/10605455) (ca=https://acme-staging-v02.api.letsencrypt.org/directory)","attempt":5,"retrying_in":600,"elapsed":613.557235658,"max_duration":2592000}

5. What I already tried:

  1. Reviewed a similar issue: Cloudflare DNS: Propagation fails due NXDOMAIN - #16 by matt, but this was on an old version of Caddy, and the fix was to upgrade to the newest version at the time (Aug 2020). So just in case, I pushed updates to my Docker build (Updated Dockerfile to use xcaddy · SlothCroissant/caddy-cloudflaredns@f4df659 · GitHub I needed an excuse to implement xcaddy from caddy-builder anyways), and rebuilt the container with the new version.
  2. Rolled my Cloudflare API token (which has been working for literal years up till now), in case there was some issue there
  3. Looked for some way to debug ACME challenges deeper, but the log directive doesn’t seem to work for me (I need to make a dedicated post around my logging strategy, it’s been weird"
  4. Re-org’d my Caddyfile to take advantage of the new global directives, in case some syntax issue was causing my problem.

You don’t need this anymore btw, this is a leftover from Caddy v1. ACME terms agreement is automatic by simply using Caddy.

I don’t think this is supposed to be a comma separated list, it should be space separated.


I think you might be running into this problem, where you end up with a bad build of Caddy:

Try building like this:

RUN xcaddy build v2.3.0 \
    --with github.com/caddy-dns/cloudflare@eda8e5aa22232e9c279b0df7531f20c331b331c6

This resolved it for me. So the issue was that Cloudflare plugin had a mismatched version from caddy itself? Forgive me I know nearly nothing of Go :slight_smile:

/srv # caddy version
v2.3.0 h1:fnrqJLa3G5vfxcxmOH/+kJOcunPLhSBnjgIvjXV/QTA=

What’s the long-term solution to get back to a more updated, generalized (using latest, etc) build? I don’t want to be stuck on 2.3 forever, or this specific Cloudflare build - I did that once and ended up causing all sorts of issues a year later when some things changed and broke, so I like to stay up-to-date nowadays.

Thanks!

@SlothCroissant What are in the debug logs? (Enable debug mode in your Caddyfile.) That will show all the ACME stuff.

FWIW, the Cloudflare plugin works 100% for my use case, so I’m not sure what the problem is.

@matt happy to rebuild my container images on the latest again (maybe I’ll put it on a dev tag or whatever for testing) and test further.

Can you confirm for me that this is all I need to add to logging?

(snippet) {
        header {
                Strict-Transport-Security "max-age=31536000; includeSubdomains"
                X-Real-IP {http.request.remote}
                Host {host}
                X-XSS-Protection "1; mode=block"
                X-Frame-Options "DENY"
                -server
        }
        tls me@email.com {
                dns cloudflare {env.CLOUDFLARE_API_TOKEN}
        }
        log {
                output file /data/logs/caddy.log {
                        roll_size 20MiB
                        roll_keep 5
                }
                level debug   <---------
        }
}

I think I’m going to make a post at some point to clean up my logging a bit better. I’d love to see application logging posted to docker log and a certain file, HTTP access logs to a second file, and HTTP error logs to yet another file, but for now it’s just whatever I have above :smiley:

Adding debug to the global options block should suffice.

2 Likes

Thanks; that enables HTTP access logs only (which don’t have debug-level emissions, currently) – so do what Basil suggested which is to put this at the top of your Caddyfile (in the global options block):

{
    debug
}
1 Like

Sounds good - building a dev tag image to test, will report back once done. Thanks!

1 Like

@matt here’s the info:

Using dev image tag: Docker Hub. Dockerfile: caddy-cloudflaredns/Dockerfile at dev · SlothCroissant/caddy-cloudflaredns · GitHub

Version:

v2.4.0-beta.1 h1:Ed/tIaN3p6z8M3pEiXWJL/T8JmCqV62FrSJCHKquW/I=

Caddyfile (for testing):

{
        debug
}

(snippet) {
        header {
                Strict-Transport-Security "max-age=31536000; includeSubdomains"
                X-XSS-Protection "1; mode=block"
                X-Frame-Options "DENY"
                -server
        }
        tls {
                issuer acme {
                        email me@email.com
                        dns cloudflare {env.CLOUDFLARE_API_TOKEN}
                        resolvers 1.1.1.1 1.0.0.1 8.8.8.8 8.8.4.4
                }
        }
        log {
                output file /data/logs/caddy.log {
                        roll_size 20MiB
                        roll_keep 5
                }
        }
}

headerstest.ryanb.tv {
        reverse_proxy docker01.lan:8097
        import snippet
}
{"level":"info","ts":1615921927.9838674,"msg":"using provided configuration","config_file":"/etc/caddy/Caddyfile","config_adapter":"caddyfile"}
{"level":"info","ts":1615921927.9865224,"logger":"admin","msg":"admin endpoint started","address":"tcp/localhost:2019","enforce_origin":false,"origins":["localhost:2019","[::1]:2019","127.0.0.1:2019"]}
{"level":"info","ts":1615921927.9868786,"logger":"tls.cache.maintenance","msg":"started background certificate maintenance","cache":"0xc0003b0770"}
{"level":"info","ts":1615921927.98701,"logger":"http","msg":"server is listening only on the HTTPS port but has no TLS connection policies; adding one to enable TLS","server_name":"srv0","https_port":443}
{"level":"info","ts":1615921927.987035,"logger":"http","msg":"enabling automatic HTTP->HTTPS redirects","server_name":"srv0"}
{"level":"debug","ts":1615921927.9875472,"logger":"http","msg":"starting server loop","address":"[::]:443","http3":false,"tls":true}
{"level":"debug","ts":1615921927.9876034,"logger":"http","msg":"starting server loop","address":"[::]:80","http3":false,"tls":false}
{"level":"info","ts":1615921927.9876156,"logger":"http","msg":"enabling automatic TLS certificate management","domains":["headerstest.ryanb.tv"]}
{"level":"info","ts":1615921927.987919,"msg":"autosaved config","file":"/config/caddy/autosave.json"}
{"level":"info","ts":1615921927.9879391,"msg":"serving initial configuration"}
{"level":"info","ts":1615921927.988374,"logger":"tls.obtain","msg":"acquiring lock","identifier":"headerstest.ryanb.tv"}
{"level":"info","ts":1615921927.989246,"logger":"tls.obtain","msg":"lock acquired","identifier":"headerstest.ryanb.tv"}
{"level":"info","ts":1615921928.0164888,"logger":"tls.issuance.acme","msg":"waiting on internal rate limiter","identifiers":["headerstest.ryanb.tv"]}
{"level":"info","ts":1615921928.0165884,"logger":"tls.issuance.acme","msg":"done waiting on internal rate limiter","identifiers":["headerstest.ryanb.tv"]}
{"level":"info","ts":1615921928.046092,"logger":"tls","msg":"cleaned up storage units"}
{"level":"debug","ts":1615921928.2167864,"logger":"tls.issuance.acme.acme_client","msg":"http request","method":"GET","url":"https://acme-v02.api.letsencrypt.org/directory","headers":{"User-Agent":["Caddy/2.4.0-beta.1 CertMagic acmez (linux; amd64)"]},"status_code":200,"response_headers":{"Cache-Control":["public, max-age=0, no-cache"],"Content-Length":["658"],"Content-Type":["application/json"],"Date":["Tue, 16 Mar 2021 19:12:08 GMT"],"Server":["nginx"],"Strict-Transport-Security":["max-age=604800"],"X-Frame-Options":["DENY"]}}
{"level":"debug","ts":1615921928.2664735,"logger":"tls.issuance.acme.acme_client","msg":"http request","method":"HEAD","url":"https://acme-v02.api.letsencrypt.org/acme/new-nonce","headers":{"User-Agent":["Caddy/2.4.0-beta.1 CertMagic acmez (linux; amd64)"]},"status_code":200,"response_headers":{"Cache-Control":["public, max-age=0, no-cache"],"Date":["Tue, 16 Mar 2021 19:12:08 GMT"],"Link":["<https://acme-v02.api.letsencrypt.org/directory>;rel=\"index\""],"Replay-Nonce":["01040EL5yOBsE8diixWdvaP5vLWzXIhtSZgLk8kpvQnd6qk"],"Server":["nginx"],"Strict-Transport-Security":["max-age=604800"],"X-Frame-Options":["DENY"]}}
{"level":"debug","ts":1615921928.6143923,"logger":"tls.issuance.acme.acme_client","msg":"http request","method":"POST","url":"https://acme-v02.api.letsencrypt.org/acme/new-order","headers":{"Content-Type":["application/jose+json"],"User-Agent":["Caddy/2.4.0-beta.1 CertMagic acmez (linux; amd64)"]},"status_code":201,"response_headers":{"Boulder-Requester":["85557177"],"Cache-Control":["public, max-age=0, no-cache"],"Content-Length":["341"],"Content-Type":["application/json"],"Date":["Tue, 16 Mar 2021 19:12:08 GMT"],"Link":["<https://acme-v02.api.letsencrypt.org/directory>;rel=\"index\""],"Location":["https://acme-v02.api.letsencrypt.org/acme/order/85557177/8488708367"],"Replay-Nonce":["0103pOi5dyBIQJ1suyGyTQVGOmNGj0lrDbnxOb0-OtcF2XE"],"Server":["nginx"],"Strict-Transport-Security":["max-age=604800"],"X-Frame-Options":["DENY"]}}
{"level":"debug","ts":1615921928.7030046,"logger":"tls.issuance.acme.acme_client","msg":"http request","method":"POST","url":"https://acme-v02.api.letsencrypt.org/acme/authz-v3/11613087122","headers":{"Content-Type":["application/jose+json"],"User-Agent":["Caddy/2.4.0-beta.1 CertMagic acmez (linux; amd64)"]},"status_code":200,"response_headers":{"Boulder-Requester":["85557177"],"Cache-Control":["public, max-age=0, no-cache"],"Content-Length":["801"],"Content-Type":["application/json"],"Date":["Tue, 16 Mar 2021 19:12:08 GMT"],"Link":["<https://acme-v02.api.letsencrypt.org/directory>;rel=\"index\""],"Replay-Nonce":["0103QEbaqvUVSB_7HP8q24ibfzE3yatv_Ced5pTluQXs0j4"],"Server":["nginx"],"Strict-Transport-Security":["max-age=604800"],"X-Frame-Options":["DENY"]}}
{"level":"debug","ts":1615921928.7031846,"logger":"tls.issuance.acme.acme_client","msg":"no solver configured","challenge_type":"tls-alpn-01"}
{"level":"info","ts":1615921928.7032025,"logger":"tls.issuance.acme.acme_client","msg":"trying to solve challenge","identifier":"headerstest.ryanb.tv","challenge_type":"dns-01","ca":"https://acme-v02.api.letsencrypt.org/directory"}
{"level":"debug","ts":1615921929.966319,"logger":"tls.issuance.acme.acme_client","msg":"http request","method":"POST","url":"https://acme-v02.api.letsencrypt.org/acme/chall-v3/11613087122/3FRdzA","headers":{"Content-Type":["application/jose+json"],"User-Agent":["Caddy/2.4.0-beta.1 CertMagic acmez (linux; amd64)"]},"status_code":200,"response_headers":{"Boulder-Requester":["85557177"],"Cache-Control":["public, max-age=0, no-cache"],"Content-Length":["185"],"Content-Type":["application/json"],"Date":["Tue, 16 Mar 2021 19:12:09 GMT"],"Link":["<https://acme-v02.api.letsencrypt.org/directory>;rel=\"index\"","<https://acme-v02.api.letsencrypt.org/acme/authz-v3/11613087122>;rel=\"up\""],"Location":["https://acme-v02.api.letsencrypt.org/acme/chall-v3/11613087122/3FRdzA"],"Replay-Nonce":["0104GhMma8Tc-qiXoH7i3HOAbrZ4GV2jSygIT0dxFHs1VT0"],"Server":["nginx"],"Strict-Transport-Security":["max-age=604800"],"X-Frame-Options":["DENY"]}}
{"level":"debug","ts":1615921929.9664564,"logger":"tls.issuance.acme.acme_client","msg":"challenge accepted","identifier":"headerstest.ryanb.tv","challenge_type":"dns-01"}
{"level":"debug","ts":1615921930.302825,"logger":"tls.issuance.acme.acme_client","msg":"http request","method":"POST","url":"https://acme-v02.api.letsencrypt.org/acme/authz-v3/11613087122","headers":{"Content-Type":["application/jose+json"],"User-Agent":["Caddy/2.4.0-beta.1 CertMagic acmez (linux; amd64)"]},"status_code":200,"response_headers":{"Boulder-Requester":["85557177"],"Cache-Control":["public, max-age=0, no-cache"],"Content-Length":["619"],"Content-Type":["application/json"],"Date":["Tue, 16 Mar 2021 19:12:10 GMT"],"Link":["<https://acme-v02.api.letsencrypt.org/directory>;rel=\"index\""],"Replay-Nonce":["0103VBq-s54AMDfDqf2DUao25VdpaoEvZCvx2VQahvmB_Xo"],"Server":["nginx"],"Strict-Transport-Security":["max-age=604800"],"X-Frame-Options":["DENY"]}}
{"level":"error","ts":1615921930.4536796,"logger":"tls.issuance.acme.acme_client","msg":"challenge failed","identifier":"headerstest.ryanb.tv","challenge_type":"dns-01","status_code":400,"problem_type":"urn:ietf:params:acme:error:dns","error":"DNS problem: NXDOMAIN looking up TXT for _acme-challenge.headerstest.ryanb.tv - check that a DNS record exists for this domain"}
{"level":"error","ts":1615921930.4537308,"logger":"tls.issuance.acme.acme_client","msg":"validating authorization","identifier":"headerstest.ryanb.tv","error":"authorization failed: HTTP 400 urn:ietf:params:acme:error:dns - DNS problem: NXDOMAIN looking up TXT for _acme-challenge.headerstest.ryanb.tv - check that a DNS record exists for this domain","order":"https://acme-v02.api.letsencrypt.org/acme/order/85557177/8488708367","attempt":1,"max_attempts":3}
{"level":"debug","ts":1615921931.6880887,"logger":"tls.issuance.acme.acme_client","msg":"http request","method":"POST","url":"https://acme-v02.api.letsencrypt.org/acme/new-order","headers":{"Content-Type":["application/jose+json"],"User-Agent":["Caddy/2.4.0-beta.1 CertMagic acmez (linux; amd64)"]},"status_code":201,"response_headers":{"Boulder-Requester":["85557177"],"Cache-Control":["public, max-age=0, no-cache"],"Content-Length":["341"],"Content-Type":["application/json"],"Date":["Tue, 16 Mar 2021 19:12:11 GMT"],"Link":["<https://acme-v02.api.letsencrypt.org/directory>;rel=\"index\""],"Location":["https://acme-v02.api.letsencrypt.org/acme/order/85557177/8488709287"],"Replay-Nonce":["0104T7f6lxeigzP_ctrs1WSWHnA8W0IdzfF9VZ7jv0KQyaY"],"Server":["nginx"],"Strict-Transport-Security":["max-age=604800"],"X-Frame-Options":["DENY"]}}
{"level":"debug","ts":1615921931.7820208,"logger":"tls.issuance.acme.acme_client","msg":"http request","method":"POST","url":"https://acme-v02.api.letsencrypt.org/acme/authz-v3/11613088243","headers":{"Content-Type":["application/jose+json"],"User-Agent":["Caddy/2.4.0-beta.1 CertMagic acmez (linux; amd64)"]},"status_code":200,"response_headers":{"Boulder-Requester":["85557177"],"Cache-Control":["public, max-age=0, no-cache"],"Content-Length":["801"],"Content-Type":["application/json"],"Date":["Tue, 16 Mar 2021 19:12:11 GMT"],"Link":["<https://acme-v02.api.letsencrypt.org/directory>;rel=\"index\""],"Replay-Nonce":["0104lP-57vptS9FcDokwX6l74UToLcpzUS-UKH9jz8fggfc"],"Server":["nginx"],"Strict-Transport-Security":["max-age=604800"],"X-Frame-Options":["DENY"]}}
{"level":"debug","ts":1615921931.7822125,"logger":"tls.issuance.acme.acme_client","msg":"no solver configured","challenge_type":"tls-alpn-01"}
{"level":"debug","ts":1615921931.7822394,"logger":"tls.issuance.acme.acme_client","msg":"no solver configured","challenge_type":"http-01"}
{"level":"debug","ts":1615921931.8904192,"logger":"tls.issuance.acme.acme_client","msg":"http request","method":"POST","url":"https://acme-v02.api.letsencrypt.org/acme/authz-v3/11613088243","headers":{"Content-Type":["application/jose+json"],"User-Agent":["Caddy/2.4.0-beta.1 CertMagic acmez (linux; amd64)"]},"status_code":200,"response_headers":{"Boulder-Requester":["85557177"],"Cache-Control":["public, max-age=0, no-cache"],"Content-Length":["805"],"Content-Type":["application/json"],"Date":["Tue, 16 Mar 2021 19:12:11 GMT"],"Link":["<https://acme-v02.api.letsencrypt.org/directory>;rel=\"index\""],"Replay-Nonce":["01036uAbQLOpuq8Wrz6zr8VoMggxw1LXBQ8VzdOcTW6DuXQ"],"Server":["nginx"],"Strict-Transport-Security":["max-age=604800"],"X-Frame-Options":["DENY"]}}
{"level":"error","ts":1615921931.8905716,"logger":"tls.obtain","msg":"will retry","error":"[headerstest.ryanb.tv] Obtain: [headerstest.ryanb.tv] solving challenges: headerstest.ryanb.tv: no solvers available for remaining challenges (configured=[dns-01] offered=[http-01 dns-01 tls-alpn-01] remaining=[http-01 tls-alpn-01]) (order=https://acme-v02.api.letsencrypt.org/acme/order/85557177/8488709287) (ca=https://acme-v02.api.letsencrypt.org/directory)","attempt":1,"retrying_in":60,"elapsed":3.901303165,"max_duration":2592000}

I don’t see anything interesting in here around Cloudflare API calls to update the DNS TXT records, etc. I feel like that’s what’s failing - I never do see those show up in my Cloudflare dashboard (though I could just be missing them if they’re only there for a few seconds before getting cleaned up).

@matt thoughts on what I’m seeing above? Is there something unique about my caddy build process that might be causing issues?

The workaround works fine, but I just don’t want to be stuck on 2.3 forever.

Thanks!

Sorry, will get back to this in a bit (taking a few days off).

Yesterday when I was poking around on the Caddy repo, I noticed that the commit that actually made Caddy use the latest changes for libdns happened after v2.4.0-beta.1 was released, which I think is why things got all messed up.

@SlothCroissant, could you try this?

RUN xcaddy build 427bbe99d076a76766266d1386ce16a38ce56fa6 \
    --with github.com/caddy-dns/cloudflare

The difference here is that this commit instead uses a later commit from Caddy instead of using the earlier commit from the cloudflare plugin. I think it might work, especially since Matt said he doesn’t have the same problems (I assume he’s using a later build with these changes)

1 Like

@francislavoie popped that into the dev branch and built the image based off of it. looks to work great (I can provide logs if needed).

Is the “main” version (without putting a commit ID) not as up to date?

Thanks! If you saw the issuance happen in the logs and it worked then that sounds good to me! :+1: Appreciate the help!

Basically, v2.4.0-beta.1 which is what the cloudflare plugin marks as its requirement, actually doesn’t have the change that that version bump (from v2.3.0 to v2.4.0-beta.1 on the cloudflare plugin) was intended to “solve”. There’s no second beta yet, so there’s no tagged version that exists that has the correct code in the main Caddy repo at this time, so right now the only options are either to downgrade the cloudflare plugin to the version that required v2.3.0, or upgrade the Caddy version to a later commit that does have the change required to be compatible with the code changes in the cloudflare plugin’s latest version.

I hope that doesn’t add to the confusion, I’m not totally sure how else to explain it :sweat_smile:

2 Likes

Thanks @francislavoie for your assistance here.

I’m happy to run on this commit for some time till the next “full” release of 2.4.x, which in theory would have the proper fixes applied. Once that happens, I’ll test again with the “latest” code, instead of a specific commit, and report back if I have any issues.

Thanks again!

This issue was plaguing me for the past week or so until I found this post. Thanks for the info.

v2.4.0-beta.2 has just been released, so you may use that now, with the latest version of the cloudflare plugin.

1 Like

This topic was automatically closed after 30 days. New replies are no longer allowed.