DNS-01 challenge tries to modify tld instead of domain

wigster · March 25, 2025, 9:01am

1. The problem I’m having:

I am using caddy as reverse proxy and trying to obtain ssl certificates for my internal domain names using DNS-01 challenge. The domain provider is Porkbun.com and the domain is in the .eu TLD. I build the caddy docker image myself with the Porkbun module included.

This worked for the first few domains and then stopped working. The certificates I obtained for the first domains are fine. Any new domain I add fails in the manner I describe.

One complication is that I have two separate caddy instances which run on two separate machines and proxy the services on those machines only. Thus what happened was that

caddy machine 1 – got all the certificates
caddy machine 2 – failed in this way
caddy machine 1+2 – now fail in this way for new certificates.

I show the logs from machine 2, where caddy proxies pihole. It is the same on machine1.

Just to check, when I try to add a record using the API directly from the machine, things are fine:

curl -X POST "https://api.porkbun.com/api/json/v3/dns/create/tumy.eu" -H "Content-Type: application/json" -d '{"apikey":"'"$PORKBUN_API_KEY"'", "secretapikey": "'"$PORKBUN_API_SECRET_KEY"'", "name": "_acme-chall
enge.t$umy.eu", "type": "TXT", "content": "abc123", "ttl": 600}'

{"status":"SUCCESS","id":462431394}

2. Error messages and/or full log output:

{"level":"info","ts":1742891333.6318808,"msg":"using config from file","file":"/etc/caddy/Caddyfile"}
{"level":"info","ts":1742891333.6526072,"msg":"adapted config to JSON","adapter":"caddyfile"}
{"level":"info","ts":1742891333.6663747,"logger":"admin","msg":"admin endpoint started","address":"localhost:2019","enforce_origin":false,"origins":["//[::1]:2019","//127.0.0.1:2019","//localhost:2019"]}
{"level":"info","ts":1742891333.6687977,"logger":"tls.cache.maintenance","msg":"started background certificate maintenance","cache":"0x3a901e0"}
{"level":"info","ts":1742891333.6701553,"logger":"http.auto_https","msg":"server is listening only on the HTTPS port but has no TLS connection policies; adding one to enable TLS","server_name":"srv0","https_port":443}
{"level":"info","ts":1742891333.6703072,"logger":"http.auto_https","msg":"enabling automatic HTTP->HTTPS redirects","server_name":"srv0"}
{"level":"info","ts":1742891333.6760488,"logger":"http","msg":"enabling HTTP/3 listener","addr":":443"}
{"level":"info","ts":1742891333.6775963,"msg":"failed to sufficiently increase receive buffer size (was: 176 kiB, wanted: 7168 kiB, got: 2048 kiB). See https://github.com/quic-go/quic-go/wiki/UDP-Buffer-Sizes for details."}
{"level":"info","ts":1742891333.6787794,"logger":"http.log","msg":"server running","name":"srv0","protocols":["h1","h2","h3"]}
{"level":"warn","ts":1742891333.6794913,"logger":"http","msg":"HTTP/2 skipped because it requires TLS","network":"tcp","addr":":80"}
{"level":"warn","ts":1742891333.6796112,"logger":"http","msg":"HTTP/3 skipped because it requires TLS","network":"tcp","addr":":80"}
{"level":"info","ts":1742891333.67967,"logger":"http.log","msg":"server running","name":"remaining_auto_https_redirects","protocols":["h1","h2","h3"]}
{"level":"info","ts":1742891333.6797242,"logger":"http","msg":"enabling automatic TLS certificate management","domains":["pihole2.tumy.eu"]}
{"level":"info","ts":1742891333.6829388,"msg":"autosaved config (load with --resume flag)","file":"/config/caddy/autosave.json"}
{"level":"info","ts":1742891333.6830797,"msg":"serving initial configuration"}
{"level":"info","ts":1742891333.7011476,"logger":"tls","msg":"cleaning storage unit","storage":"FileStorage:/data/caddy"}
{"level":"info","ts":1742891333.7032113,"logger":"tls.obtain","msg":"acquiring lock","identifier":"pihole2.tumy.eu"}
{"level":"info","ts":1742891333.7120397,"logger":"tls","msg":"finished cleaning storage units"}
{"level":"info","ts":1742891333.7141957,"logger":"tls.obtain","msg":"lock acquired","identifier":"pihole2.tumy.eu"}
{"level":"info","ts":1742891333.7149687,"logger":"tls.obtain","msg":"obtaining certificate","identifier":"pihole2.tumy.eu"}
{"level":"info","ts":1742891333.8340821,"logger":"tls.issuance.acme","msg":"waiting on internal rate limiter","identifiers":["pihole2.tumy.eu"],"ca":"https://acme-v02.api.letsencrypt.org/directory","account":""}
{"level":"info","ts":1742891333.8345752,"logger":"tls.issuance.acme","msg":"done waiting on internal rate limiter","identifiers":["pihole2.tumy.eu"],"ca":"https://acme-v02.api.letsencrypt.org/directory","account":""}
{"level":"info","ts":1742891333.8348353,"logger":"tls.issuance.acme","msg":"using ACME account","account_id":"https://acme-v02.api.letsencrypt.org/acme/acct/2298935526","account_contact":[]}
{"level":"info","ts":1742891335.0322537,"msg":"trying to solve challenge","identifier":"pihole2.tumy.eu","challenge_type":"dns-01","ca":"https://acme-v02.api.letsencrypt.org/directory"}
{"level":"error","ts":1742891336.2789683,"msg":"cleaning up solver","identifier":"pihole2.tumy.eu","challenge_type":"dns-01","error":"no memory of presenting a DNS record for \"_acme-challenge.pihole2.tumy.eu\" (usually OK if presenting also failed)","stacktrace":"github.com/mholt/acmez/v3.(*Client).solveChallenges.func1\n\tgithub.com/mholt/acmez/v3@v3.0.0/client.go:318\ngithub.com/mholt/acmez/v3.(*Client).solveChallenges\n\tgithub.com/mholt/acmez/v3@v3.0.0/client.go:363\ngithub.com/mholt/acmez/v3.(*Client).ObtainCertificate\n\tgithub.com/mholt/acmez/v3@v3.0.0/client.go:136\ngithub.com/caddyserver/certmagic.(*ACMEIssuer).doIssue\n\tgithub.com/caddyserver/certmagic@v0.21.6/acmeissuer.go:477\ngithub.com/caddyserver/certmagic.(*ACMEIssuer).Issue\n\tgithub.com/caddyserver/certmagic@v0.21.6/acmeissuer.go:371\ngithub.com/caddyserver/caddy/v2/modules/caddytls.(*ACMEIssuer).Issue\n\tgithub.com/caddyserver/caddy/v2@v2.9.1/modules/caddytls/acmeissuer.go:249\ngithub.com/caddyserver/certmagic.(*Config).obtainCert.func2\n\tgithub.com/caddyserver/certmagic@v0.21.6/config.go:626\ngithub.com/caddyserver/certmagic.doWithRetry\n\tgithub.com/caddyserver/certmagic@v0.21.6/async.go:104\ngithub.com/caddyserver/certmagic.(*Config).obtainCert\n\tgithub.com/caddyserver/certmagic@v0.21.6/config.go:700\ngithub.com/caddyserver/certmagic.(*Config).ObtainCertAsync\n\tgithub.com/caddyserver/certmagic@v0.21.6/config.go:505\ngithub.com/caddyserver/certmagic.(*Config).manageOne.func1\n\tgithub.com/caddyserver/certmagic@v0.21.6/config.go:415\ngithub.com/caddyserver/certmagic.(*jobManager).worker\n\tgithub.com/caddyserver/certmagic@v0.21.6/async.go:73"}
{"level":"error","ts":1742891336.4930828,"logger":"tls.obtain","msg":"could not get certificate from issuer","identifier":"pihole2.tumy.eu","issuer":"acme-v02.api.letsencrypt.org-directory","error":"[pihole2.tumy.eu] solving challenges: presenting for challenge: adding temporary record for zone \"eu.\": Invalid http response status, {\"status\":\"ERROR\",\"message\":\"Invalid domain.\"} (order=https://acme-v02.api.letsencrypt.org/acme/order/2298935526/367186829556) (ca=https://acme-v02.api.letsencrypt.org/directory)"}
{"level":"error","ts":1742891336.494296,"logger":"tls.obtain","msg":"will retry","error":"[pihole2.tumy.eu] Obtain: [pihole2.tumy.eu] solving challenges: presenting for challenge: adding temporary record for zone \"eu.\": Invalid http response status, {\"status\":\"ERROR\",\"message\":\"Invalid domain.\"} (order=https://acme-v02.api.letsencrypt.org/acme/order/2298935526/367186829556) (ca=https://acme-v02.api.letsencrypt.org/directory)","attempt":1,"retrying_in":60,"elapsed":2.779980867,"max_duration":2592000}

3. Caddy version:

caddy version

v2.9.1 h1:OEYiZ7DbCzAWVb6TNEkjRcSCRGHVoZsJinoDR/n9oaY=

From caddy list-modules:

  Standard modules: 124

dns.providers.porkbun

  Non-standard modules: 1

  Unknown modules: 0

4. How I installed and ran Caddy:

a. System environment:

machine2 — RPi2B, Debian Bookworm, armv7l
machien1 — RPi3A, Debian Bookworm, armv8

Linux fiddlestix2 6.6.74+rpt-rpi-v7 #1 SMP Raspbian 1:6.6.74-1+rpt1 (2025-01-27) armv7l

Client: Docker Engine - Community
 Version:           28.0.1
 API version:       1.48
 Go version:        go1.23.6
 Git commit:        068a01e
 Built:             Wed Feb 26 10:41:11 2025
 OS/Arch:           linux/arm
 Context:           default

Server: Docker Engine - Community
 Engine:
  Version:          28.0.1
  API version:      1.48 (minimum version 1.24)
  Go version:       go1.23.6
  Git commit:       bbd0a17
  Built:            Wed Feb 26 10:41:11 2025
  OS/Arch:          linux/arm
  Experimental:     false
 containerd:
  Version:          1.7.25
  GitCommit:        bcc810d6b9066471b0b6fa75f557a15a1cbf31bb
 runc:
  Version:          1.2.4
  GitCommit:        v1.2.4-0-g6c52b3f
 docker-init:
  Version:          0.19.0
  GitCommit:        de40ad0

b. Command:

Runs in docker container built on the machine with porkbun module included. See below.

PASTE OVER THIS, BETWEEN THE ``` LINES.
Please use the preview pane to ensure it looks nice.

c. Service/unit/compose file:

Docker compose yaml

  caddy:
    container_name: caddy
    build:
      context: /home/wigster/docker/caddy
      dockerfile: Dockerfile-caddy
    restart: unless-stopped
    ports:
      - "80:80"
      - "443:443"
      - "443:443/udp"
    environment:
      - TZ=${TZ}
      - PORKBUN_API_KEY=${PORKBUN_API_KEY}
      - PORKBUN_API_SECRET_KEY=${PORKBUN_API_SECRET_KEY}
    volumes:
      - /srv/docker/caddy/Caddyfile:/etc/caddy/Caddyfile
      - caddy_data:/data
      - caddy_config:/config

volumes:
  caddy_data:
    external: true
  caddy_config:

Dockerfile to build the caddy image

FROM caddy:builder AS builder
RUN xcaddy build --with github.com/caddy-dns/porkbun
FROM caddy:latest
COPY --from=builder /usr/bin/caddy /usr/bin/caddy

d. My complete Caddy config:

(get_cert) {
        tls {
                dns porkbun {
                        api_key {$PORKBUN_API_KEY}
                        api_secret_key {$PORKBUN_API_SECRET_KEY}
                }
        }
}
# Porkbun domain
pihole2.tumy.eu {
        import get_cert
        rewrite / /admin
        reverse_proxy 192.168.0.3:8080
}

5. Links to relevant resources:

hmoffatt · March 25, 2025, 6:16pm

Do you have any stale records in the domain that might be causing this, like a CNAME?

wigster · March 25, 2025, 8:04pm

This is a completely new domain I’ve just bought and there is nothing else there, beyond pointing to my external static IP.

However – I have just discovered that I cannot actually nslookup tumy.eu from inside my network. Apparently when I set this as my pi-hole domain name, it is treated as local and does not resolve externally. This then appears to block the dns-01 challenge from completing.

I’ve just changed it to local.tumy.eu inside pihole, but this is clearly not correct.

I want pihole to treat *.tumy.eu as my local search domain
I want caddy to obtain the certificates from subdomains of tumy.eu using dns-01 challenge.

What should be the correct setting?

I have pihole v6.

hmoffatt · March 25, 2025, 9:52pm

You’d have to ask a pihole forum about that.

TheRettom · March 26, 2025, 1:07am

If you are trying to access a service locally when it has a public IP for the A/AAAA record, then it will fail to resolve. You need to specify in your router or local DNS that local.tumy.eu has an internal IP address, not a public.

Best way to do it is to assign the IP of the machine hosting it to that domain.

victor · March 26, 2025, 1:18am

I had this exact issue a while back. My domain I was issuing was also my local search domain.

Try changing your local domain to edge.domain.tld on your pihole.

wigster · March 26, 2025, 4:11am

Is edge here a special keyword, or in any choice ok? I don’t see any documentation that it would be special. If not:

I am not trying to access an internal service from outside. I just mean that if I follow the standard recommended setup

I get a domain domain.tld
I set this up as the local search domain in my DHCP server, i.e. pihole
domain.tld is now not accessible on the local network because pihole treats it as local
caddy cannot complete the dns-01 challenge

I can work around this by adding a prefix for my local search domain like suggested-- edge.domain.tld

caddy can obtain certificates for subdomains of domain.tld, i.e. service.domain.tld
my DHCP servers issues internal fqdns for my local machines which look like server.edge.domain.tld

This seems to spoil the point of having everything on the network at a single subdomain level, whether servers or services.

victor · March 26, 2025, 5:09am

Edge is just a random keyword.

I know what you’re saying, but that’s the only way I was able to get it to work.

TheRettom · March 26, 2025, 3:43pm

Any will work. edge in this context is just a subdomain for domain.tld. The same function but a different name of pihole.domain.tld.

I’m aware, but the fact that tumy.eu is a hostname for the router’s external IP means that it will not properly resolve when attempting to access it from the same local network tumy.eu is hosted from. You need to specify in a local DNS, such as a hostname setting in your router, that it has an internal IP. It will only resolve as a local IP address within that network.

Otherwise, if you do not want this service publicly accessible, change the IP from your public IP to the local IP.

It’s not really a workaround, it’s giving Pihole its own subdomain of tumy.eu. Caddy can obtain wildcard (*) certificates for that domain, or you can specify each subdomain in the Caddyfile.

I’m not really sure if having a fqdns prefix would fix the problem I specified above. I haven’t tried it, and I haven’t researched it. I would believe it wouldn’t, since it is usually tied to the router, and it still wouldn’t know which specific internal IP is the server. Or maybe I’m misunderstanding the concept.

I’m not really sure what you mean by this. Each service on the server needs its own domain or subfolder, such as pihole.tumy.eu or tumy.eu/pihole. You can’t have multiple services accessible from subdomain.domain.tld alone. All your router needs to know is if a service is internal or external and what its IP is so it can resolve properly. The server handling the request (the server being Caddy on your machine) will then reverse proxy Pihole so you can access Pihole with HTTPS.