Caddy v2 - Cloudflare HTTP 403, requires permission to list zones

1. Caddy version (caddy version):

lee@compute01:~/docker/caddy_broken_test$ docker run --rm --name=caddy_test -it caddy_test caddy version
v2.1.1 => /src/caddy

2. How I run Caddy:

I’m just trying to get a wildcard cert working with Caddy using cloudflare, my domain is leeroy.xyz. I have followed this and given zone:read, and dns:edit for all zones and all accounts.

To elaborate a bit more on how I’ve implemented, I’m using docker on ubuntu, and am using a dockerfile to build a docker image which combines the official caddy docker image, and runs caddy-builder to replace the caddy binary so it includes dns cloudflare functionality.

^^ If there’s a better way by the way, I’m all ears. It took me a while to get this far so I assume I’m not doing it the easiest way.

Image builds, docker runs, it tries to list zones to start doing DNS-01 business, and fails due to permissions listing zones. I have since implemented it using certbot to grab certificates, and statically linked certificate files, but this is not desirable.

I would like to help find this problem and fix it.

a. System environment:

Docker 19.03.11 & Ubuntu 20.04, just trying to run manually, systemd/cron not involved.

lee@compute01:~/docker/caddy_broken_test$ docker --version
Docker version 19.03.11, build 42e35e61f3

lee@compute01:~/docker/caddy_broken_test$ lsb_release -d
Description: Ubuntu 20.04 LTS

b. Command:

docker commands

docker build -t caddy_test .

docker run --rm --name=caddy_test \
-it -p 8080:80 -p 8443:443 \
-v "$(pwd)/Caddyfile":/etc/caddy/Caddyfile \
-v "$(pwd)/caddy_data":/data \
-e CLOUDFLARE_API_TOKEN='redacted' \
caddy_test

c. Service/unit/compose file:

Dockerfile:

FROM caddy:builder AS builder

RUN caddy-builder \
    github.com/caddy-dns/cloudflare

FROM caddy:alpine

COPY --from=builder /usr/bin/caddy /usr/bin/caddy

d. My complete Caddyfile or JSON config:

Caddyfile

{
    acme_ca "https://acme-staging-v02.api.letsencrypt.org/directory"
}

(wildcard_cert) {
    tls {
        dns cloudflare {env.CLOUDFLARE_API_TOKEN}
    }
}


*.leeroy.xyz {
    import wildcard_cert
    respond "Hello, world!"
}

3. The problem I’m having:

I’m just trying to get a wildcard cert working with Caddy using cloudflare, and it fails when it tries to list zones doing DNS-01 business. I have followed this and given zone:read, and dns:edit for all zones and all accounts.

Image builds, docker runs, it tries to list zones to start doing DNS-01 business, and fails due to permissions listing zones.

4. Error messages and/or full log output:

lee@compute01:~/docker/caddy_broken_test$ docker run --rm --name=caddy_test -it -v "$(pwd)/Caddyfile":/etc/caddy/Caddyfile -v "$(pwd)/caddy_data":/data -e CLOUDFLARE_API_TOKEN='REDACTED' caddy_test
2020/07/16 11:48:50.966	INFO	using provided configuration	{"config_file": "/etc/caddy/Caddyfile", "config_adapter": "caddyfile"}
2020/07/16 11:48:50.969	INFO	admin	admin endpoint started	{"address": "tcp/localhost:2019", "enforce_origin": false, "origins": ["[::1]:2019", "127.0.0.1:2019", "localhost:2019"]}
2020/07/16 11:48:50.969	INFO	http	server is listening only on the HTTPS port but has no TLS connection policies; adding one to enable TLS	{"server_name": "srv0", "https_port": 443}
2020/07/16 11:48:50.969	INFO	http	enabling automatic HTTP->HTTPS redirects	{"server_name": "srv0"}
2020/07/16 11:48:50 [INFO][cache:0xc0002ef260] Started certificate maintenance routine
2020/07/16 11:48:50.971	INFO	tls	cleaned up storage units
2020/07/16 11:48:50.971	INFO	http	enabling automatic TLS certificate management	{"domains": ["*.leeroy.xyz"]}
2020/07/16 11:48:50.972	INFO	autosaved config	{"file": "/config/caddy/autosave.json"}
2020/07/16 11:48:50.972	INFO	serving initial configuration
2020/07/16 11:48:50 [INFO][*.leeroy.xyz] Obtain certificate; acquiring lock...
2020/07/16 11:48:50 [INFO][*.leeroy.xyz] Obtain: Lock acquired; proceeding...
2020/07/16 11:48:52 [INFO][*.leeroy.xyz] Waiting on rate limiter...
2020/07/16 11:48:52 [INFO][*.leeroy.xyz] Done waiting
2020/07/16 11:48:52 [INFO] [*.leeroy.xyz] acme: Obtaining bundled SAN certificate given a CSR
2020/07/16 11:48:53 [INFO] [*.leeroy.xyz] AuthURL: https://acme-staging-v02.api.letsencrypt.org/acme/authz-v3/78193488
2020/07/16 11:48:53 [INFO] [*.leeroy.xyz] acme: use dns-01 solver
2020/07/16 11:48:53 [INFO] [*.leeroy.xyz] acme: Preparing to solve DNS-01
2020/07/16 11:48:54 [INFO] [*.leeroy.xyz] acme: Cleaning DNS-01 challenge
2020/07/16 11:48:54 [WARN] [*.leeroy.xyz] acme: cleaning up failed: no memory of presenting a DNS record for leeroy.xyz 
2020/07/16 11:48:54 [INFO] Deactivating auth: https://acme-staging-v02.api.letsencrypt.org/acme/authz-v3/78193488
2020/07/16 11:48:55 [ERROR] error: one or more domains had a problem:
[*.leeroy.xyz] [*.leeroy.xyz] acme: error presenting token: got error status: HTTP 403: [{Code:0 Message:Actor 'com.cloudflare.api.token.REDACTED' requires permission 'com.cloudflare.api.account.zone.list' to list zones}]
 (challenge=dns-01 remaining=[])
2020/07/16 11:48:57 [ERROR] attempt 1: [*.leeroy.xyz] Obtain: [*.leeroy.xyz] error: one or more domains had a problem:
[*.leeroy.xyz] [*.leeroy.xyz] acme: error presenting token: got error status: HTTP 403: [{Code:0 Message:Actor 'com.cloudflare.api.token.REDACTED' requires permission 'com.cloudflare.api.account.zone.list' to list zones}]
 - retrying in 1m0s (6.237607754s/720h0m0s elapsed)...
^C2020/07/16 11:48:58.526	INFO	shutting down	{"signal": "SIGINT"}
2020/07/16 11:48:58 [INFO][cache:0xc0002ef260] Stopped certificate maintenance routine
2020/07/16 11:48:58 [INFO][*.leeroy.xyz] Obtain: Releasing lock
2020/07/16 11:48:58.527	INFO	admin	stopped previous server
2020/07/16 11:48:58.527	INFO	shutdown done	{"signal": "SIGINT"}

5. What I already tried:

I have tried simplifying my config as much as humanly possible to demonstrate the problem and make it easy to reproduce.

I repeated the process on macOS, same behaviour.

I have tumbled and also tried recreating the API token, although I can prove the token is ok:

lee@compute01:~/docker/caddy_broken_test$ curl -sX GET "https://api.cloudflare.com/client/v4/zones?match=all"     -H "Authorization: Bearer REDACTED"     -H "Content-Type:application/json"| jq '.result[] | {id: .id, name: .name}'
{
  "id": "235ccec4ab265bc87340273aa5d7da46",
  "name": "leeroy.xyz"
}

6. Links to relevant resources:

Specify Zone ID · Issue #2 · caddy-dns/cloudflare · GitHub ← I believe this would sort my issues, but I am not a programmer. I’ve stared at Go, and I’m just not capable of contributing.

[DNS mode] Cloudflare New API Tokens · Issue #2398 · acmesh-official/acme.sh · GitHub ← some robust conversation in the acme.sh issues section about the same thing

Please help!

Howdy @leeramsay, welcome to the Caddy community.

This really seems like a Cloudflare API issue. Their dashboard is telling you one thing (you’ve got permissions), your CLI response is fine, but to Caddy, they say you don’t have permissions.

At best, the error they’re giving Caddy is wildly misleading/incorrect and there’s something else going on.

Have you opened a support ticket with Cloudflare to see what they say?

1 Like

I’m guessing you’re using the Cloudflare proxy. It might be easier to use a Cloudflare issued origin cert, as you won’t have to mess around with their API or compile a custom build of Caddy.

@Whitestrake
Hi there, first off, thanks for responding and trying to help! Respectfully I disagree with root cause being cloudflare specifically with my issue.

I may not have explained myself clearly, but I worked around caddy not working by implementing certbot to provide certs for caddy, certbot is working fine using the same token and domain.

I also have acme.sh running on a different host on my same network, and that too is working fine (albeit, with acme.sh, they allow you to specicy your zone ID, which allowed it to work). I am also able to list zones, list dns entries, create dns entries, and delete them using curl with the same API token.

I don’t think you can take from above that cloudflare is at fault, and also, if I did reach out to cloudflare support, the most I think I could expect is that they would assist in making sure curl reflects the permissions granted by the admin console, and if so ‘well the rest is on you’, which leads us back here…

@samjmckenzie

I am not using cloudflare proxy, it’s an internal to my network service I’m attempting to use DNS-01 as a method to get certificates

1 Like

I did find the root cause for my problem.

Internal to my network, I have a zone in my DNS server for leeroy.xyz, so that I can run a wildcard entry for *.leeroy.xyz → my docker host, so that I don’t need to create a new DNS entry every time I run another service.

Having an internal zone, which is what caddy is using for DNS resolution, breaks DNS validation. I know that’s obvious in hindsight, but I was thrown off by:

1 - Certbot & acme.sh OK
2 - The cryptic error message that caddy returns

I removed the zone from my DNS server and just specified specific dns overrides for specific domain names, this is now allowing Caddy to request wildcard certs using my domain.

Thanks for everyones help, but if I may offer a suggestion, perhaps look into the error handling within the dns-cloudflare plugin? I’m happy to work with any developer who wants to take a look at this, I can restore my DNS configuration back to the state where it was breaking Caddy, and run as many tests/dev versions of code as is needed.

I would like to contribute to making Caddy the most reliable and robust service it can be, it would be better if the error messages it throws are accurate, I definitely do not have an issue with cloudflare and listing DNS/zone permissions, yet that’s what I was being told the problem was!

3 Likes

@leeramsay could you open up an issue on the cloudflare plugin repo to track the issue?

Glad you figured out the problem!

I just spotted this issue, does it seem relevant to your situation?

Maybe, but the error Caddy was returning was simply the error that Cloudflare was giving Caddy:

Caddy would have to do some serious leg work to infer the root cause here.

I’m still not even sure how having split DNS would cause this issue. Basically, Caddy is just trying to connect to CF’s API and set a text record. Having your own DNS resolution for the domain you’re setting a record for in the CF API shouldn’t have any bearing on the response from CF.

With split DNS issues I’d expect to see instead symptoms we’ve seen other users report in the past: Caddy waits a long time for DNS propagation and eventually times out without completing the DNS challenge. In those cases we’ve spotted the problem pretty quick.

1 Like

This topic was automatically closed after 30 days. New replies are no longer allowed.