Rootless Podman ACME/DNS Challenge Problem

1. The problem I’m having:

I am attempting to use the DNS-01 challenge with the Cloudflare plugin on an Arch Linux host (rootless Podman). My domain uses the .top TLD, which apparently can have slow propagation as of recently.

Despite configuring propagation_delay 600s and propagation_timeout -1 in both global options and site blocks, Caddy consistently aborts the challenge at exactly 128–132 seconds with a timeout error. It appears the certmagic.solverWrapper is enforcing a hard-coded ~2-minute timeout and ignoring the Caddyfile directives intended to extend or disable that check.

I can see the _acme-challenge TXT records successfully created in my Cloudflare dashboard, but Caddy seems to give up before the TLD nameservers sync.

I feel like I’ve tried various things, but nothing seems to be correcting the problem. I unfortunately don’t remember all that I’ve attempted by this point.

2. Error messages and/or full log output:

{"level":"error","ts":1774293174.5387354,"logger":"tls.obtain","msg":"could not get certificate from issuer","identifier":"adguard.famdam.top","issuer":"acme-v02.api.letsencrypt.org-directory","error":"[adguard.famdam.top] solving challenges: waiting for solver certmagic.solverWrapper to be ready: timed out waiting for record to fully propagate; verify DNS provider configuration is correct - last error: <nil> (order=https://acme-v02.api.letsencrypt.org/acme/order/2237117955/493550461651) (ca=https://acme-v02.api.letsencrypt.org/directory)"}
{"level":"error","ts":1774293174.5388987,"logger":"tls.obtain","msg":"will retry","error":"[adguard.famdam.top] Obtain: [adguard.famdam.top] solving challenges: waiting for solver certmagic.solverWrapper to be ready: timed out waiting for record to fully propagate; verify DNS provider configuration is correct - last error: <nil> (order=https://acme-v02.api.letsencrypt.org/acme/order/2237117955/493550461651) (ca=https://acme-v02.api.letsencrypt.org/directory)","attempt":1,"retrying_in":60,"elapsed":131.589497927,"max_duration":2592000}

3. Caddy version:

v2.10.2 h1:g/gTYjGMD0dec+UgMw8SnfmJ3I9+M2TdvoRL/Ovu6U8=

4. How I installed and ran Caddy:

a. System environment:

  • OS: Arch Linux (Kernel: linux-hardened)
  • Deployment: Rootless Podman with Caddy using systemd socket for ports 80 TCP and 443 TCP/UDP
  • Hardware: Intel NUC with OPNsense, AdGuard Home, and Unbound | Dell Optiplex running Caddy and other containerized services

b. Command:

Podman Containerfile:

FROM caddy:2.10.2-builder AS builder

RUN xcaddy build \
    --with github.com/caddy-dns/cloudflare \
    --with github.com/WeidiDeng/caddy-cloudflare-ip \
    --with github.com/mholt/caddy-dynamicdns \
    --with github.com/hslatman/caddy-crowdsec-bouncer

FROM caddy:2.10.2

COPY --from=builder /usr/bin/caddy /usr/bin/caddy

Then create the image:

podman build -t caddy-custom ~/.local/share/containers/storage/caddy/

c. Service/unit/compose file:

caddy.container file:

[Unit]
AssertPathExists=%h/.local/share/containers/storage/caddy/Caddyfile

[Container]
ContainerName=caddy
Image=localhost/caddy-custom
Exec=/usr/bin/caddy run --config /etc/caddy/Caddyfile
Environment=EMAIL=redacted
Environment=LOG_FILE=/data/access.log
Secret=CLOUDFLARE_API_TOKEN,type=env,target=CF_API_TOKEN
Secret=NAMECHEAP_API_USER,type=env,target=NAMECHEAP_API_USER
Secret=CROWDSEC_API_KEY,type=env,target=CROWDSEC_API_KEY
Volume=%h/.local/share/containers/storage/caddy/Caddyfile:/etc/caddy/Caddyfile
Volume=%h/.local/share/containers/storage/caddy/caddy-config:/config
Volume=%h/.local/share/containers/storage/caddy/caddy-data:/data
Volume=%h/.local/share/containers/storage/caddy/log.d:/data/log.d
Volume=/srv/www:/srv/www:ro
Notify=true
Memory=256m

HealthCmd=caddy validate --config /etc/caddy/Caddyfile || exit 1
HealthInterval=30s
HealthRetries=3

PodmanArgs=--dns=1.1.1.1
AddHost=host.containers.internal:host-gateway

Network=crowdsec.network
Network=vaultwarden.network
Network=immich.network
Network=radicale.network
Network=atlas-cmms.network
Network=ntfy.network
Network=mealie.network

[Install]
WantedBy=default.target

[Service]
ExecReload=/usr/bin/podman exec caddy /usr/bin/caddy reload --config /etc/caddy/Caddyfile --force

d. My complete Caddy config:

{
        order crowdsec first
        crowdsec {
                api_url http://crowdsec:8080
                api_key {env.CROWDSEC_API_KEY}
        }
        dns cloudflare {env.CF_API_TOKEN}
        acme_dns cloudflare {env.CF_API_TOKEN} {
                propagation_delay 600s
                propagation_timeout -1
                resolvers 1.1.1.1 8.8.8.8
        }
        dynamic_dns {
                provider cloudflare {env.CF_API_TOKEN}
                domains {
                        famdam.top @
                }
                versions ipv4
        }
        servers {
                trusted_proxies cloudflare {
                        interval 12h
                        timeout 15s
                }
        }
        pki {
                ca local {
                        name "Homelab"
                }
        }
}

(cloudflare_dns) {
        tls {
                dns cloudflare {env.CF_API_TOKEN}
                propagation_delay 600s
                propagation_timeout -1
                resolvers 1.1.1.1 8.8.8.8
        }
}

famdam.top {
        bind fd/3 {
                protocols h1
        }
        bind fd/4 {
                protocols h1 h2
        }
        bind fdgram/5 {
                protocols h3
        }

        log {
                output file /data/log.d/famdam.log {
                        roll_local_time
                        mode 644
                        roll_keep_for 48h
                }
        }

        root * /srv/www

        encode zstd gzip

        file_server {
                browse
        }
}

adguard.famdam.top {
        import cloudflare_dns
        crowdsec
        bind fd/3 {
                protocols h1
        }
        bind fd/4 {
                protocols h1 h2
        }
        bind fdgram/5 {
                protocols h3
        }

        log {
                output file /data/log.d/adguard.log {
                        roll_local_time
                        mode 644
                        roll_keep_for 48h
                }
        }
       reverse_proxy host.containers.internal:8081 {
                header_up Host {upstream_hostport}
                header_up X-Real-IP {remote_host}
                header_up X-Forwarded-For {remote_host}
                header_up X-Forwarded-Proto {scheme}
        }
        encode zstd gzip
}

# SAMPLE SITE BLOCK THAT REFLECTS VIRTUALLY ALL MY OTHER SITE BLOCKS WITH MINOR EDITS
service.famdam.top {
        import cloudflare_dns
        crowdsec
        bind fd/3 {
                protocols h1
        }
        bind fd/4 {
                protocols h1 h2
        }
        bind fdgram/5 {
                protocols h3
        }

        log {
                output file /data/log.d/service.log {
                        roll_local_time
                        mode 644
                        roll_keep_for 48h
                }
        }
       reverse_proxy service:80
}

5. Links to relevant resources:

I guess I had an incorrectly configured firewall rule in OPNsense. It made ALL traffic use 192.168.1.1 as the DNS, which also included the Caddy server in the container. I reconfigured it and it now propagates correctly. The errors are misleading in that context, but not a huge deal. Figure I’ll at least keep this issue up on the forums in case someone else ever deals with it.

1 Like