Acme DNS-01 Challenge Failing

1. The problem I’m having:

I am trying to use Caddy to do a DNS-01 challenge such that I can have certificates not just for my exposed domains, but also for domains for my internal network. However, I am unable to make it work and I spent approx. 10-20h trying every iteration I could find.

I tried things such as:

  1. Adding resolvers 1.1.1.1 8.8.8.8, with one or more DNS servers
  2. Adding various combinations of propagation_delay and propagation_timeout
  3. Various combinations of the Caddyfile config, such as having only the p3x-001 subdomain listed, having only the wildcard domain listed, having the wildcard domain point to the internal ip…
  4. Using debugging tools such as trying the nslookup -type=TXT _acme-challenge.chaos-gang.click command to make sure the config is correct in namecheap. It seems like it is.

Additionally, I tried to use ngnix proxy manager, to check whether my configurations work in terms of setting up my domain and so on, and indeed, I could make it work with ngnix proxy manager. So my conclusion is that the issue must be somewhere in Caddy or its acmedns plugin.

The “furthest” I could get is that caddy attempts a challenge as in below error output and fails. There was one initial instance where the challenge succeeded for the domain, but the domain didn’t really work, instead it spammed redirects until the browser makes it stop. Restarting Caddy caused it to get into the present state and the challenge never passed again.

If relevant, here are my namecheap domain configs:


where 10.183.4.20 is caddy’s ip. Used the exact same config for ngnix (and as I mentioned, it worked fine).

2. Error messages and/or full log output:

I suppose this is the most relevant log output:

Mar 09 13:00:56 reverse-proxy caddy[4332]: {"level":"info","ts":1741525256.0745935,"msg":"trying to solve challenge","identifier":"p3x-001.chaos-gang.click","challenge_type":"dns-01","ca":"https://acme-staging-v02.api.letsencrypt.org/directory"}
Mar 09 13:02:57 reverse-proxy caddy[4332]: {"level":"error","ts":1741525377.4201367,"logger":"tls.obtain","msg":"could not get certificate from issuer","identifier":"p3x-001.chaos-gang.click","issuer":"acme-v02.api.letsencrypt.org-directory","error":"[p3x-001.chaos-gang.click] solving challenges: waiting for solver certmagic.solverWrapper to be ready: timed out waiting for record to fully propagate; verify DNS provider configuration is correct - last error: <nil> (order=https://acme-staging-v02.api.letsencrypt.org/acme/order/188646604/23133217974) (ca=https://acme-staging-v02.api.letsencrypt.org/directory)"}

3. Caddy version:

v2.9.1 h1:OEYiZ7DbCzAWVb6TNEkjRcSCRGHVoZsJinoDR/n9oaY=

4. How I installed and ran Caddy:

a. System environment:

Ubuntu 24.04.2 LTS (GNU/Linux 6.8.12-8-pve x86_64)
Running under an LXC in proxmox

b. Command:

For building:

xcaddy build --with github.com/caddy-dns/acmedns

For starting:

systemd start caddy

c. Service/unit/compose file:

[Unit]
Description=Caddy Web Server
Documentation=https://caddyserver.com/docs/
After=network.target network-online.target
Requires=network-online.target

[Service]
Type=notify
ExecStart=/usr/bin/caddy run --environ --config /etc/caddy/Caddyfile
ExecReload=/usr/bin/caddy reload --config /etc/caddy/Caddyfile
TimeoutStopSec=5s
LimitNOFILE=1048576
LimitNPROC=512
PrivateTmp=true
ProtectSystem=full
AmbientCapabilities=CAP_NET_BIND_SERVICE

[Install]
WantedBy=multi-user.target

d. My complete Caddy config:

{
    admin :2019
    email myemail@gmail.com
}

chaos-gang.click *.chaos-gang.click {
	tls {
		dns acmedns /etc/caddy/acmedns.json
	}
}

p3x-001.chaos-gang.click {
    tls {
		dns acmedns /etc/caddy/acmedns.json
	}
    reverse_proxy 10.183.3.10:8006
}

e. My acmedns.json config

{
    "username": "username",
    "password": "password",
    "fulldomain": "http://b20457bf-b2c0-4ddf-baae-a0b6cce3e8ff.auth.acme-dns.io",
    "subdomain": "b20457bf-b2c0-4ddf-baae-a0b6cce3e8ff",
    "server_url": "https://auth.acme-dns.io",
    "allowfrom": []
}

Well it turns out that spending whatever I spent so far + 1h was the key… The problem was that I was “unlucky” by having my first pick of the target service (at 10.183.3.10) be proxmox. Essentially, proxmox does some of its own stuff when it comes to http → https redirects and that is what caused the infinite loops on a valid configuration. As far as I found out, there are two ways to solve this (I modified the format of the caddyfile config a bit because I like the internal routes to be grouped like this):

a. Trust proxmox’s certificate in caddy’s host:

For the certificate:

  1. On your caddy node vim /usr/local/share/ca-certificates/proxmox-ca.crt and paste in what you got from cat /etc/pve/pve-root-ca.pem from your proxmox node.
  2. Do update-ca-certificates on caddy’s node
  3. Restart caddy

Modify Caddyfile:

*.chaos-gang.click {
    tls {
        dns acmedns /etc/caddy/acmedns.json
    }

    @p3x-001 host p3x-001.chaos-gang.click
    handle @p3x-001 {
        reverse_proxy https://10.183.3.10:8006 {
            header_up Host {host}
            header_up X-Forwarded-Proto {scheme}
            header_up X-Forwarded-For {remote}
            header_up X-Real-IP {remote}
        }
    }

    @internalIP not remote_ip 10.183.0.0/16
    handle @internalIP {
        respond "403 Forbidden" 403
    }

    respond "403 Forbidden" 403
}
b. Use tls_insecure_skip_verify

In my case, all of the traffic in this thread (apart from dns lookup) happens on the internal network, so (as far as I am aware) skipping the verification so proxmox stops redirecting should be fine.

*.chaos-gang.click {
    tls {
        dns acmedns /etc/caddy/acmedns.json
    }

    @p3x-001 host p3x-001.chaos-gang.click
    handle @p3x-001 {
        reverse_proxy https://10.183.3.10:8006 {
            transport http {
                tls_insecure_skip_verify
            }
            header_up Host {host}
            header_up X-Forwarded-Proto {scheme}
            header_up X-Forwarded-For {remote}
            header_up X-Real-IP {remote}
        }
    }
}

As far as my understanding goes, the benefit of option a) is that your TLS setup is verified also from caddy → internal service (proxmox). While option b) allows for some mitm attacks and whatnot (though again, this is on the local network).