Caddy Cloudflare DNS Challenge Not Working within Docker

PREFACE: I have my own custom caddy build with xcaddy with the cloudflare DNS module installed on my server as a service and starts and runs fine and gets my certificates from the DNS challenge from my CF account just fine with my credentials. Recently, I have been wanting to run caddy in a docker container instead, but I am not able to receive my cert due to the DNS challenge failing and I am not sure why. I am using the same API token from my CF account that I use for my caddy binary on my server and works fine, but trying it within docker gives me a weird issue for some reason. I have listed below what I am doing for my docker build and what the error is.

1. Caddy version (caddy version): Caddy v2.5.0

2. How I run Caddy:

Created my custom docker image using a dockerfile with:

FROM caddy:2.5.0-builder AS builder

RUN xcaddy build --with github.com/caddy-dns/cloudflare --with github.com/greenpau/caddy-trace

FROM caddy:2.5.0

COPY --from=builder /usr/bin/caddy /usr/bin/caddy

Image builds fine and I am able to use it on my server no problem. Everything starts.

I build on my laptop for my pi, (armv7 userland)

docker buildx build --platform linux/arm/v7 -f Dockerfile . -t azoller12/caddy

a. System environment:

Here is my system:

OS: Raspbian GNU/Linux 11 (bullseye) aarch64
Host: Raspberry Pi 4 Model B Rev 1.4
Kernel: 5.15.33-v8+
CPU: BCM2835 (4) @ 1.800GHz
Memory: 795MiB / 1892MiB

docker compose file, compose.yml:

networks:
  default:
    name: caddy_net

services:
  caddy:
    container_name: caddy
    image: azoller12/custom-caddy:2.5.0
    restart: unless-stopped
    ports:
      - 80:80
      - 443:443
    volumes:
      - /home/alexz/docker/caddy/Caddyfile:/etc/caddy/Caddyfile
      - /home/alexz/docker/caddy/site:/srv
      - /home/alexz/docker/caddy/caddy_data:/data
      - /home/alexz/docker/caddy/caddy_config:/config

All of my volumes have proper permissions for the root user. The logs don’t indicate any permission issues.

Caddyfile snippet of TLS challenge:

*.azollerhome.xyz {

        header {
        # disable FLoC tracking
        Permissions-Policy interest-cohort=()

        # enable HSTS
        Strict-Transport-Security max-age=31536000

        # disable clients from sniffing the media type
        X-Content-Type-Options nosniff

        # clickjacking protection
        X-Frame-Options DENY

        }


        tls {
                protocols tls1.3
                dns cloudflare custom_api_token
        }

        @home host home.azollerhome.xyz
        handle @home {
                    reverse_proxy localhost:81
        }

        ......

Error Message/Logs (Docker logs)

This error message continually loops.

2022-05-01T08:04:24.585226785Z {"level":"error","ts":1651392264.5849605,"logger":"tls.obtain","msg":"could not get certificate from issuer","identifier":"*.azollerhome.xyz","issuer":"acme-v02.api.letsencrypt.org-directory","error":"[*.azollerhome.xyz] solving challenges: presenting for challenge: adding temporary record for zone xyz.: got error status: HTTP 403: [{Code:9109 Message:Invalid access token}] (order=https://acme-v02.api.letsencrypt.org/acme/order/519669747/84827465237) (ca=https://acme-v02.api.letsencrypt.org/directory)"}
2022-05-01T08:04:24.586578341Z {"level":"info","ts":1651392264.5864255,"logger":"tls.issuance.acme","msg":"waiting on internal rate limiter","identifiers":["*.azollerhome.xyz"],"ca":"https://acme.zerossl.com/v2/DV90","account":"alexzoller1@protonmail.com"}
2022-05-01T08:04:24.586637210Z {"level":"info","ts":1651392264.586479,"logger":"tls.issuance.acme","msg":"done waiting on internal rate limiter","identifiers":["*.azollerhome.xyz"],"ca":"https://acme.zerossl.com/v2/DV90","account":"alexzoller1@protonmail.com"}
^[[<0;42;16m2022-05-01T08:05:06.972162480Z {"level":"info","ts":1651392306.9718812,"logger":"tls.issuance.acme.acme_client","msg":"trying to solve challenge","identifier":"*.azollerhome.xyz","challenge_type":"dns-01","ca":"https://acme.zerossl.com/v2/DV90"}
2022-05-01T08:05:08.259174582Z {"level":"error","ts":1651392308.2571886,"logger":"tls.issuance.acme.acme_client","msg":"cleaning up solver","identifier":"*.azollerhome.xyz","challenge_type":"dns-01","error":"no memory of presenting a DNS record for azollerhome.xyz (probably OK if presenting failed)"}
2022-05-01T08:05:21.396422100Z {"level":"error","ts":1651392321.3961182,"logger":"tls.obtain","msg":"could not get certificate from issuer","identifier":"*.azollerhome.xyz","issuer":"acme.zerossl.com-v2-DV90","error":"[*.azollerhome.xyz] solving challenges: presenting for challenge: adding temporary record for zone xyz.: got error status: HTTP 403: [{Code:9109 Message:Invalid access token}] (order=https://acme.zerossl.com/v2/DV90/order/sl8lhveAS3-T2EXxAmUMrw) (ca=https://acme.zerossl.com/v2/DV90)"}
2022-05-01T08:05:21.396534839Z {"level":"error","ts":1651392321.3962412,"logger":"tls.obtain","msg":"will retry","error":"[*.azollerhome.xyz] Obtain: [*.azollerhome.xyz] solving challenges: presenting for challenge: adding temporary record for zone xyz.: got error status: HTTP 403: [{Code:9109 Message:Invalid access token}] (order=https://acme.zerossl.com/v2/DV90/order/sl8lhveAS3-T2EXxAmUMrw) (ca=https://acme.zerossl.com/v2/DV90)","attempt":1,"retrying_in":60,"elapsed":58.005679197,"max_duration":2592000}

Things I have tried

  1. I have a CAA rule to only allow certs from letsencrypt.org. I even turned this off but still did not help.
  2. I have created new API keys twice to no avail. Using the same API key that I use for my binary caddy service on my host didn’t work even when it works if I do not use docker. The API token has Zone.DNS Edit and Zone.Zone Read rights.
  3. I have also just used the global “Zone API” token for my account to use and did not work either.

I am just confused to why it says "Invalid access token" when that same token works with my Caddyfile outside of docker. I am not sure what is happening here.

When I run docker exec caddy caddy list-modules it shows the cloudflare dns module was successfully built:

.........
tls.issuance.acme
tls.issuance.internal
tls.issuance.zerossl
tls.stek.distributed
tls.stek.standard

  Standard modules: 95

dns.providers.cloudflare
http.handlers.trace

  Non-standard modules: 2

  Unknown modules: 0

That reads to me like it’s trying to set a record on the entire .xyz zone and not just on your domain… or something like that.

Try configuring resolvers in your Caddy config for tls, set it to 1.1.1.1 and see if that helps. My hunch is that certmagic’s logic for finding the authoritative zone is failing because it ends up using a bad DNS resolver that can’t determine it correctly.

@matt maybe you can confirm how that libdns zone thing works again? Is it possible that’s the issue?

Thanks for the reply!

Using resolvers 1.1.1.1 seemed to do the trick…

I am just curious as to why I didn’t need to explicitly set the resolver when running caddy as a bare service on my server versus running it in docker. I am running dnsmasq on my server using it as my DNS server, maybe that is the issue. I may need to configure dnsmasq to properly listen on the docker interface? Anyways, I think this can be closed now. I think this would make the most sense due to the fact that I needed to explicitly set the DNS resolvers like you suggested.

1 Like

Docker does some DNS magic. More than I know how to explain here, but you can do some research into it.

I know that it copies the resolv.conf from the host machine into the container, and changes one of the lines to point it to using Docker’s built-in DNS resolver (which is used to resolve container names to container IPs). I couldn’t say what exactly happens with the DNS queries though.