DNS-01 Challenge Failing for DigitalOcean

1. Caddy version (caddy version):

2.3.0

2. How I run Caddy:

It is based on a custom image. Here it is the Dockerfile:


FROM caddy:2.3.0-builder AS builder

RUN xcaddy build \
    --with github.com/caddy-dns/digitalocean

FROM caddy:2.3.0

COPY --from=builder /usr/bin/caddy /usr/bin/caddy

a. System environment:

OS: Ubuntu 20.04
Docker:

Client:
 Version:           19.03.8
 API version:       1.40
 Go version:        go1.13.8
 Git commit:        afacb8b7f0
 Built:             Fri Dec 18 12:15:19 2020
 OS/Arch:           linux/amd64
 Experimental:      false

Server:
 Engine:
  Version:          19.03.8
  API version:      1.40 (minimum version 1.12)
  Go version:       go1.13.8
  Git commit:       afacb8b7f0
  Built:            Fri Dec  4 23:02:49 2020
  OS/Arch:          linux/amd64
  Experimental:     false
 containerd:
  Version:          1.3.3-0ubuntu2.3
  GitCommit:        
 runc:
  Version:          spec: 1.0.1-dev
  GitCommit:        
 docker-init:
  Version:          0.18.0
  GitCommit: 

b. Command:

docker-compose up

c. Service/unit/compose file:

docker-compose.yml:

version: "3.5"

networks:
    web:
        driver: bridge
    backend:
        driver: bridge
        internal: true

services:
    caddy:
        container_name: caddy
        image: "custom-caddy:2.3.0"
        init: true
        ports:
            - "80:80"
            - "443:443"
        volumes:
            - ./Caddyfile:/etc/caddy/Caddyfile
            - ./data:/server/caddy
            - ../frontend/dist:/dist
        networks:
            - web
            - backend
        depends_on:
            - api
        environment:
            - DIGITAL_OCEAN_TOKEN=${DIGITAL_OCEAN_TOKEN}
...

d. My complete Caddyfile or JSON config:

I tried to reduce the configuration to the minimum necessary to reproduce the problem:

*.pronus.io {
    tls {
        dns digitalocean {env.DIGITAL_OCEAN_TOKEN}
    }
}

3. The problem I’m having:

DNS challenge does not work.

4. Error messages and/or full log output:

caddy    | {"level":"info","ts":1617667674.743623,"msg":"using provided configuration","config_file":"/etc/caddy/Caddyfile","config_adapter":"caddyfile"}
caddy    | {"level":"info","ts":1617667674.7496188,"logger":"admin","msg":"admin endpoint started","address":"tcp/localhost:2019","enforce_origin":false,"origins":["localhost:2019","[::1]:2019","127.0.0.1:2019"]}
caddy    | {"level":"info","ts":1617667674.7512286,"logger":"http","msg":"server is listening only on the HTTPS port but has no TLS connection policies; adding one to enable TLS","server_name":"srv0","https_port":443}
caddy    | {"level":"info","ts":1617667674.7517993,"logger":"http","msg":"enabling automatic HTTP->HTTPS redirects","server_name":"srv0"}
caddy    | {"level":"info","ts":1617667674.7544327,"logger":"http","msg":"enabling automatic TLS certificate management","domains":["*.pronus.io"]}
caddy    | {"level":"info","ts":1617667674.7549808,"msg":"autosaved config","file":"/config/caddy/autosave.json"}
caddy    | {"level":"info","ts":1617667674.7550008,"msg":"serving initial configuration"}
caddy    | {"level":"info","ts":1617667674.7576463,"logger":"tls.obtain","msg":"acquiring lock","identifier":"*.pronus.io"}
caddy    | {"level":"info","ts":1617667674.7579458,"logger":"tls.obtain","msg":"lock acquired","identifier":"*.pronus.io"}
caddy    | {"level":"info","ts":1617667674.7643893,"logger":"tls.cache.maintenance","msg":"started background certificate maintenance","cache":"0xc0001af030"}
caddy    | {"level":"info","ts":1617667674.770398,"logger":"tls","msg":"cleaned up storage units"}
caddy    | {"level":"info","ts":1617667674.7963264,"logger":"tls.issuance.acme","msg":"waiting on internal rate limiter","identifiers":["*.pronus.io"]}
caddy    | {"level":"info","ts":1617667674.79732,"logger":"tls.issuance.acme","msg":"done waiting on internal rate limiter","identifiers":["*.pronus.io"]}
caddy    | {"level":"info","ts":1617667675.6327436,"logger":"tls.issuance.acme.acme_client","msg":"trying to solve challenge","identifier":"*.pronus.io","challenge_type":"dns-01","ca":"https://acme-v02.api.letsencrypt.org/directory"}
caddy    | {"level":"error","ts":1617667797.0061057,"logger":"tls.obtain","msg":"will retry","error":"[*.pronus.io] Obtain: [*.pronus.io] solving challenges: waiting for solver *certmagic.DNS01Solver to be ready: timed out waiting for record to fully propagate; verify DNS provider configuration is correct - last error: <nil> (order=https://acme-v02.api.letsencrypt.org/acme/order/118079098/8888535988) (ca=https://acme-v02.api.letsencrypt.org/directory)","attempt":1,"retrying_in":60,"elapsed":122.24814519,"max_duration":2592000}
caddy    | {"level":"info","ts":1617667857.4311733,"logger":"tls.issuance.acme.acme_client","msg":"trying to solve challenge","identifier":"*.pronus.io","challenge_type":"dns-01","ca":"https://acme-staging-v02.api.letsencrypt.org/directory"}
caddy    | {"level":"error","ts":1617667979.83649,"logger":"tls.obtain","msg":"will retry","error":"[*.pronus.io] Obtain: [*.pronus.io] solving challenges: waiting for solver *certmagic.DNS01Solver to be ready: timed out waiting for record to fully propagate; verify DNS provider configuration is correct - last error: <nil> (order=https://acme-staging-v02.api.letsencrypt.org/acme/order/18930537/25886206) (ca=https://acme-staging-v02.api.letsencrypt.org/directory)","attempt":2,"retrying_in":120,"elapsed":305.078530849,"max_duration":2592000}

5. What I already tried:

I tried to change the domain from codelab.pronus.io to *.pronus.io. None worked. I have no ideas about how to solve this. Any help?

6. Links to relevant resources:

This is how the DNS records are configured:


DNS records
Type 	Hostname   Value TTL (seconds) 	
TXT 	_acme-challenge.codelab.pronus.io.pronus.io Fp5zURDQniTb1sUAXgjNeHWWaxNRAeRD4MzzKWPkR4w 1800
A 	pronus.io directs to 159.65.236.186 3600
A 	*.pronus.io directs to 159.65.236.186 3600

This doesn’t look right, the zone is doubled up here (pronus.io appears twice).

This might be a bug with the digitalocean DNS plugin. Might be best to ask on that repo for help.

You’re not persisting the /data volume in the Caddy container. You definitely should do that, because it holds long-term data like your certificates, keys, and some other identifiers for Caddy to properly function. See the note in the docs on Docker Hub

Thanks, Francis, I’ll ask on that repo for help.

On the second point, I’d rather keep the data directory on the host so I don’t lose it when the container is destroyed.

It seems related to Challenge never solved: waiting for solver *certmagic.DNS01Solver to be ready · Issue #8 · caddy-dns/digitalocean · GitHub

Yes, that’s what I mean. And to do that, you need to use something like ./data:/data. The path /server/caddy is not used by Caddy inside the container. The official docker image stores the data in /data, not /server/caddy.

Thanks for the heads up!