Cloudflare DNS Challenge Fails with 401 Status

1. Caddy version (caddy version):

v2.1.1 h1:X9k1+ehZPYYrSqBvf/ocUgdLSRIuiNiMo7CvyGUQKeA=

2. How I run Caddy:

a. System environment:

Raspberry Pi OS May 27th version
systemd

b. Command:

to run:

sudo systemctl start caddy

to build, I used xcaddy:

GOOS=linux GOARCH=arm GOARM=7 xcaddy build --with github.com/caddy-dns/cloudflare

As you might infer, I cross-compiled caddy on a regular x86 linux machine to run on a ARMv7 Raspberry Pi 4

c. Service/unit/compose file:

systemd

caddy.service

[Unit]
Description=Caddy
Documentation=https://caddyserver.com/docs/
After=network.target

[Service]
User=caddy
Group=caddy
ExecStart=/usr/bin/caddy run --environ --config /etc/caddy/Caddyfile
ExecReload=/usr/bin/caddy reload --config /etc/caddy/Caddyfile
TimeoutStopSec=5s
LimitNOFILE=1048576
LimitNPROC=512
PrivateTmp=true
ProtectSystem=full
AmbientCapabilities=CAP_NET_BIND_SERVICE

[Install]
WantedBy=multi-user.target

d. My complete Caddyfile or JSON config:

nizar.cf:1996 {
  encode gzip

### important part starts here
  tls {
    dns cloudflare API_TOKEN_PLAINTEXT_REDACTED
  }
###important part ends here

  redir /cloud /cloud/
  redir /cloud/.well-known/caldav /cloud/remote.php/dav 301
  redir /cloud/.well-known/carddav /cloud/remote.php/dav 301
  route /cloud/* {
    root * /var/www
    php_fastcgi unix//run/php/php7.3-fpm.sock
    file_server
  }

  redir /bitw /bitwarden/
  redir /bitw/ /bitwarden/
  redir /bitwarden /bitwarden/
  reverse_proxy /bitwarden/* localhost:3401

  redir /rss /miniflux/
  redir /rss/ /miniflux/
  redir /miniflux /miniflux/
  reverse_proxy /miniflux/* unix//run/miniflux/miniflux.sock
}

3. The problem I’m having:

The DNS challenge for TLS seems to be failing with cloudflare with a 401 error. I cannot access my server using the domain name.

4. Error messages and/or full log output:

Jul 26 03:55:20 raspberrypi caddy[30881]: 2020/07/26 03:55:20 [INFO] [nizar.cf] acme: Preparing to solve DNS-01
Jul 26 03:55:21 raspberrypi caddy[30881]: 2020/07/26 03:55:21 [INFO] [nizar.cf] acme: Cleaning DNS-01 challenge
Jul 26 03:55:21 raspberrypi caddy[30881]: 2020/07/26 03:55:21 [WARN] [nizar.cf] acme: cleaning up failed: no memory of presenting a DNS record for nizar.cf
Jul 26 03:55:21 raspberrypi caddy[30881]: 2020/07/26 03:55:21 [INFO] Deactivating auth: https://acme-staging-v02.api.letsencrypt.org/acme/authz-v3/82793320
Jul 26 03:55:21 raspberrypi caddy[30881]: 2020/07/26 03:55:21 [ERROR] error: one or more domains had a problem:
Jul 26 03:55:21 raspberrypi caddy[30881]: [nizar.cf] [nizar.cf] acme: error presenting token: got error status: HTTP 401: []
Jul 26 03:55:21 raspberrypi caddy[30881]:  (challenge=dns-01 remaining=[])
Jul 26 03:55:23 raspberrypi caddy[30881]: 2020/07/26 03:55:23 [ERROR] attempt 5: [nizar.cf] Obtain: [nizar.cf] error: one or more domains had a problem:
Jul 26 03:55:23 raspberrypi caddy[30881]: [nizar.cf] [nizar.cf] acme: error presenting token: got error status: HTTP 401: []
Jul 26 03:55:23 raspberrypi caddy[30881]:  - retrying in 10m0s (10m21.77428467s/720h0m0s elapsed)...

5. What I already tried:

I tried changing the API token, expanding its permissions beyond whats required. I tried using a wrong token to ensure that I do indeed get a different error, so that is not exactly the issue. I also ensured the token was correct by running the curl command cloudflare provides, and it returned successfully.

I saw that others have had similar issues to me [1][2]

In both cases, the issue seemed to be that the users were using the API key instead of token. I made sure that I am in fact using an API token (which requires permissions set) and not the global keys.

6. Links to relevant resources:

Others who had the same issue
[1] V2 Caddyfile problem with Cloudflare plugin - #7 by francislavoie
[2] Compiled Caddy 2.0.0 with this module but TLS handshake error during challenge · Issue #1 · caddy-dns/cloudflare · GitHub

Please help me fix this, I would really appreciate it!

I am planning on opening a thread in the cloudflare support forums about this, since it seems more likely to be a cloudflare issue and not a Caddy one. Just so I can isolate the issue from Caddy and make it more easily reproducible, would it be possible for someone to please help me write a curl equivalent to the API request to cloudflare made by the cloudflare plugin for Caddy? I tried to figure it out myself, but I am somewhat new to coding, and have only coded with Javascript. I was not able to understand it well.

It seems that this is where the request happens:

so something like:

curl -H "Authorization: Bearer INSERT_TOKEN_HERE" https://api.cloudflare.com/client/v4

I would really appreciate the help!

We’ve got const baseURL = "https://api.cloudflare.com/client/v4" and qs is just a generated URL query string using url.Values. The zone name is the domain name.

Basically, the URL should be: https://api.cloudflare.com/client/v4/zones?name=nizar.cf

1 Like

Not sure if this is exactly how Caddy does it, but here is what I tried:

command:

curl -X GET "https://api.cloudflare.com/client/v4/zones?name=nizar.cf" -H "Authorization: Bearer TOKEN_HERE" -H "Content-Type: application/json"

then, with the zone ID:

curl -X GET "https://api.cloudflare.com/client/v4/zones/56cae7d1410a21bfac59350ddefd39f6/" -H "Authorization: Bearer Bearer TOKEN_HERE" -H "Content-Type: application/json"

then, with the following:

curl -X GET "https://api.cloudflare.com/client/v4/zones/56cae7d1410a21bfac59350ddefd39f6/dns_records?name=nizar.cf" -H "Authorization: Bearer TOKEN_HERE" -H "Content-Type: application/json"

All 3 returned successfully. I pasted the results below respectively. I can’t seem to be able to isolate the issue from caddy. It looks like I am at a dead end :disappointed:

results:

{
    "result": [
        {
            "id": "56cae7d1410a21bfac59350ddefd39f6",
            "name": "nizar.cf",
            "status": "active",
            "paused": false,
            "type": "full",
            "development_mode": 0,
            "name_servers": [
                "brenda.ns.cloudflare.com",
                "burt.ns.cloudflare.com"
            ],
            "original_name_servers": [
                "ns04.freenom.com",
                "ns03.freenom.com",
                "ns02.freenom.com",
                "ns01.freenom.com"
            ],
            "original_registrar": null,
            "original_dnshost": null,
            "modified_on": "2020-07-25T18:43:19.773670Z",
            "created_on": "2020-07-25T02:54:21.532575Z",
            "activated_on": "2020-07-25T18:43:19.773670Z",
            "meta": {
                "step": 2,
                "wildcard_proxiable": false,
                "custom_certificate_quota": 0,
                "page_rule_quota": 3,
                "phishing_detected": false,
                "multiple_railguns_allowed": false
            },
            "owner": {
                "id": "f8104333b4912e8bead1828af9961921",
                "type": "user",
                "email": "REDACTED_EMAIL"
            },
            "account": {
                "id": "f19b44ea0009041582a875fb236a5ee2",
                "name": "REDACTED_EMAIL"
            },
            "permissions": [
                "#access:edit",
                "#access:read",
                "#analytics:read",
                "#app:edit",
                "#auditlogs:read",
                "#billing:edit",
                "#billing:read",
                "#cache_purge:edit",
                "#dns_records:edit",
                "#dns_records:read",
                "#lb:edit",
                "#lb:read",
                "#legal:edit",
                "#legal:read",
                "#logs:edit",
                "#logs:read",
                "#member:edit",
                "#member:read",
                "#organization:edit",
                "#organization:read",
                "#ssl:edit",
                "#ssl:read",
                "#stream:edit",
                "#stream:read",
                "#subscription:edit",
                "#subscription:read",
                "#teams:edit",
                "#teams:read",
                "#teams:report",
                "#waf:edit",
                "#waf:read",
                "#webhooks:edit",
                "#webhooks:read",
                "#worker:edit",
                "#worker:read",
                "#zone:edit",
                "#zone:read",
                "#zone_settings:edit",
                "#zone_settings:read"
            ],
            "plan": {
                "id": "0feeeeeeeeeeeeeeeeeeeeeeeeeeeeee",
                "name": "Free Website",
                "price": 0,
                "currency": "USD",
                "frequency": "",
                "is_subscribed": false,
                "can_subscribe": false,
                "legacy_id": "free",
                "legacy_discount": false,
                "externally_managed": false
            }
        }
    ],
    "result_info": {
        "page": 1,
        "per_page": 20,
        "total_pages": 1,
        "count": 1,
        "total_count": 1
    },
    "success": true,
    "errors": [],
    "messages": []
}

Then this:

{
    "result": {
        "id": "56cae7d1410a21bfac59350ddefd39f6",
        "name": "nizar.cf",
        "status": "active",
        "paused": false,
        "type": "full",
        "development_mode": 0,
        "name_servers": [
            "brenda.ns.cloudflare.com",
            "burt.ns.cloudflare.com"
        ],
        "original_name_servers": [
            "ns04.freenom.com",
            "ns03.freenom.com",
            "ns02.freenom.com",
            "ns01.freenom.com"
        ],
        "original_registrar": null,
        "original_dnshost": null,
        "modified_on": "2020-07-25T18:43:19.773670Z",
        "created_on": "2020-07-25T02:54:21.532575Z",
        "activated_on": "2020-07-25T18:43:19.773670Z",
        "meta": {
            "step": 2,
            "wildcard_proxiable": false,
            "custom_certificate_quota": 0,
            "page_rule_quota": 3,
            "phishing_detected": false,
            "multiple_railguns_allowed": false
        },
        "owner": {
            "id": "f8104333b4912e8bead1828af9961921",
            "type": "user",
            "email": "REDACTED_EMAIL"
        },
        "account": {
            "id": "f19b44ea0009041582a875fb236a5ee2",
            "name": "REDACTED_EMAIL"
        },
        "permissions": [
            "#access:edit",
            "#access:read",
            "#analytics:read",
            "#app:edit",
            "#auditlogs:read",
            "#billing:edit",
            "#billing:read",
            "#cache_purge:edit",
            "#dns_records:edit",
            "#dns_records:read",
            "#lb:edit",
            "#lb:read",
            "#legal:edit",
            "#legal:read",
            "#logs:edit",
            "#logs:read",
            "#member:edit",
            "#member:read",
            "#organization:edit",
            "#organization:read",
            "#ssl:edit",
            "#ssl:read",
            "#stream:edit",
            "#stream:read",
            "#subscription:edit",
            "#subscription:read",
            "#teams:edit",
            "#teams:read",
            "#teams:report",
            "#waf:edit",
            "#waf:read",
            "#webhooks:edit",
            "#webhooks:read",
            "#worker:edit",
            "#worker:read",
            "#zone:edit",
            "#zone:read",
            "#zone_settings:edit",
            "#zone_settings:read"
        ],
        "plan": {
            "id": "0feeeeeeeeeeeeeeeeeeeeeeeeeeeeee",
            "name": "Free Website",
            "price": 0,
            "currency": "USD",
            "frequency": "",
            "is_subscribed": false,
            "can_subscribe": false,
            "legacy_id": "free",
            "legacy_discount": false,
            "externally_managed": false
        }
    },
    "success": true,
    "errors": [],
    "messages": []
}

and last, this:

{
    "result": [
        {
            "id": "926dd0ee7fe95c70c4e9bfca7b0c41d4",
            "zone_id": "56cae7d1410a21bfac59350ddefd39f6",
            "zone_name": "nizar.cf",
            "name": "nizar.cf",
            "type": "A",
            "content": "100.36.30.171",
            "proxiable": true,
            "proxied": true,
            "ttl": 1,
            "locked": false,
            "meta": {
                "auto_added": false,
                "managed_by_apps": false,
                "managed_by_argo_tunnel": false,
                "source": "primary"
            },
            "created_on": "2020-07-28T00:13:42.493708Z",
            "modified_on": "2020-07-28T00:13:42.493708Z"
        }
    ],
    "success": true,
    "errors": [],
    "messages": [],
    "result_info": {
        "page": 1,
        "per_page": 20,
        "count": 1,
        "total_count": 1,
        "total_pages": 1
    }
}

Are you sure Caddy is configured with the same (correct) API token and not API key?

1 Like

Hi Matt! Unfortunately yes. I made sure that it was the token and not the key. and to double check, I tried generating new keys, but that did not work either.

One reason I am sure the token is correct is that, when I do try to intentionally change token to make it wrong, I get a different error. I pasted it below:

[nizar.cf] [nizar.cf] acme: error presenting token: got error status: HTTP 403: [{Code:9109 Message:Invalid access token}]

I also tried giving the token more permissions than recommended (not pictured), but that did not work either.

1 Like

I see, and you’re getting HTTP 401.

If you can get Caddy to print the request (URL, headers, body if any) to the console and try again, that would be most helpful at this point (I’m not sure what else is wrong).

1 Like

sorry to ask, but how exactly can I get Caddy to do that? is there a certain directive I can add to Caddyfile to make it output that? I can see the log directive but I thought that was for access logs only.

No, you’ll have to add log.Println() lines to the code (or we need to add debug logging, which I don’t think we do currently; it’s all still pretty new) and recompile.

I apologize for the delayed response. It took me a while to understand the Golang ecosystem with its dependencies before I could figure out how to make Go take my edited dependency instead. Finally got it working now! Here is the printed requests:

&{GET https://api.cloudflare.com/client/v4/zones?name=nizar.cf. HTTP/1.1 1 1 map[Authorization:[Bearer REDACTED_TOKEN]] <nil> <nil> 0 [] false api.cloudflare.com map[] map[] <nil> map[]   <nil> <nil> <nil> 0x283a03c}
&{POST https://api.cloudflare.com/client/v4/zones/56cae7d1410a21bfac59350ddefd39f6/dns_records HTTP/1.1 1 1 map[Authorization:[Bearer REDACTED_API_TOKEN] Content-Type:[application/json]] {0x2d07180} 0x2d84f0 178 [] false api.cloudflare.com map[] map[] <nil> map[]   <nil> <nil> <nil> 0x283a03c}

The error I spoke about above printed after the second request (POST request).

It looks like the log did not print out absolutely everything and left out some of the nested fields. Should I print those too? How can I access them? Currently, I am just doing log.Println() under doApiRequest function in client.go

Great job figuring that out!

You tried the GET request with curl, but the POST request is the one that is failing. Have you tried doing the POST request with curl too?

(You can print more about the request using fmt.Printf("%#v", req) – for pointers though, you’ll need to dereference them: fmt.Printf("%#v %#v", req, req.FieldName))

1 Like

I have uncovered the issue. Upon sending the POST request with curl, I got a more descriptive error message:

"error": "You cannot use this API for domains with a .cf, .ga, .gq, .ml, or .tk TLD (top-level domain). To configure the DNS settings for this domain, use the Cloudflare Dashboard."

Looks like I don’t have many options here, but the issue is clearly not a Caddy issue. I apologize for the trouble, and thank you for all the help!

2 Likes

Thanks for investigating! What was the full output from that curl POST request? libdns should have been able to report the error message in the first place:

But as you saw it was blank. It looks like Cloudflare was sending a malformed or undocumented response body structure and I’d like to know what it is.

this was all I received in that JSON:

A post was split to a new topic: Using Cloudflare certificates

Thanks

Hm, indeed, that is not the same structure you get as their documentation would have you think:

So… that’s annoying. They’re violating their API contract. And that’s why there was no actual error message up front, which is why this was hard to debug. I hope they’ll fix that.

1 Like

This topic was automatically closed after 30 days. New replies are no longer allowed.