DNS challenge for internal sites not working

1. The problem I’m having:

I try to use Caddy with the DNS Challenge and the cloudflare module to automatically issue certificates for sites that are only internally accessible so there are no public DNS records for the subdomain i try to add (immich.voege.dev), only the main domain (voege.dev).
The DNS Challenge fails, and after a bit of googling and reading other threads here, it seems to be a DNS problem. I found multiple solutions to use a external DNS resolver with the “resolvers” directive, which does not seem to work for me.
Im guessing it’s still a DNS problem, but I’m not sure what do next.

I also tried to run Caddy in a LXC on proxmox with the same error, so it probably isn’t an issue with the installation itself?

2. Error messages and/or full log output:

2025-01-26T15:02:32	Error	caddy	 "error","ts":"2025-01-26T14:02:32Z","logger":"tls","msg":"job failed","error":"immich.voege.dev: obtaining certificate: context canceled"}
2025-01-26T15:02:32	Error	caddy	 "error","ts":"2025-01-26T14:02:32Z","logger":"tls.obtain","msg":"unable to unlock","identifier":"immich.voege.dev","lock_key":"issue_cert_immich.voege.dev","error":"remove /var/db/caddy/data/caddy/locks/issue_cert_immich.voege.dev.lock: no such file or directory"}
2025-01-26T15:02:32	Warning	caddy	 "warn","ts":"2025-01-26T14:02:32Z","logger":"admin.api","msg":"exiting; byeee!! 👋"}
2025-01-26T15:01:50	Error	caddy	 "error","ts":"2025-01-26T14:01:50Z","logger":"tls.obtain","msg":"will retry","error":"[immich.voege.dev] Obtain: [immich.voege.dev] solving challenges: presenting for challenge: adding temporary record for zone \"voege.dev.\": got error status: HTTP 403: [{Code:10000 Message:Authentication error ErrorChain:[]}] (order=https://acme-v02.api.letsencrypt.org/acme/order/2184904935/348228649835) (ca=https://acme-v02.api.letsencrypt.org/directory)","attempt":1,"retrying_in":60,"elapsed":4.500923974,"max_duration":2592000}
2025-01-26T15:01:50	Error	caddy	 "error","ts":"2025-01-26T14:01:50Z","logger":"tls.obtain","msg":"could not get certificate from issuer","identifier":"immich.voege.dev","issuer":"acme-v02.api.letsencrypt.org-directory","error":"[immich.voege.dev] solving challenges: presenting for challenge: adding temporary record for zone \"voege.dev.\": got error status: HTTP 403: [{Code:10000 Message:Authentication error ErrorChain:[]}] (order=https://acme-v02.api.letsencrypt.org/acme/order/2184904935/348228649835) (ca=https://acme-v02.api.letsencrypt.org/directory)"}
2025-01-26T15:01:49	Error	caddy	 "error","ts":"2025-01-26T14:01:49Z","msg":"cleaning up solver","identifier":"immich.voege.dev","challenge_type":"dns-01","error":"no memory of presenting a DNS record for \"_acme-challenge.immich.voege.dev\" (usually OK if presenting also failed)","stacktrace":"github.com/mholt/acmez/v3.(*Client).solveChallenges.func1\n\tgithub.com/mholt/acmez/v3@v3.0.0/client.go:318\ngithub.com/mholt/acmez/v3.(*Client).solveChallenges\n\tgithub.com/mholt/acmez/v3@v3.0.0/client.go:363\ngithub.com/mholt/acmez/v3.(*Client).ObtainCertificate\n\tgithub.com/mholt/acmez/v3@v3.0.0/client.go:136\ngithub.com/caddyserver/certmagic.(*ACMEIssuer).doIssue\n\tgithub.com/caddyserver/certmagic@v0.21.6/acmeissuer.go:477\ngithub.com/caddyserver/certmagic.(*ACMEIssuer).Issue\n\tgithub.com/caddyserver/certmagic@v0.21.6/acmeissuer.go:371\ngithub.com/caddyserver/caddy/v2/modules/caddytls.(*ACMEIssuer).Issue\n\tgithub.com/caddyserver/caddy/v2@v2.9.1/modules/caddytls/acmeissuer.go:249\ngithub.com/caddyserver/certmagic.(*Config).obtainCert.func2\n\tgithub.com/caddyserver/certmagic@v0.21.6/config.go:626\ngithub.com/caddyserver/certmagic.doWithRetry\n\tgithub.com/caddyserver/certmagic@v0.21.6/async.go:104\ngithub.com/caddyserver/certmagic.(*Config).obtainCert\n\tgithub.com/caddyserver/certmagic@v0.21.6/config.go:700\ngithub.com/caddyserver/certmagic.(*Config).ObtainCertAsync\n\tgithub.com/caddyserver/certmagic@v0.21.6/config.go:505\ngithub.com/caddyserver/certmagic.(*Config).manageOne.func1\n\tgithub.com/caddyserver/certmagic@v0.21.6/config.go:415\ngithub.com/caddyserver/certmagic.(*jobManager).worker\n\tgithub.com/caddyserver/certmagic@v0.21.6/async.go:73"}
2025-01-26T15:01:45	Warning	caddy	 "warn","ts":"2025-01-26T14:01:45Z","logger":"http","msg":"HTTP/3 skipped because it requires TLS","network":"tcp","addr":":80"}
2025-01-26T15:01:45	Warning	caddy	 "warn","ts":"2025-01-26T14:01:45Z","logger":"http","msg":"HTTP/2 skipped because it requires TLS","network":"tcp","addr":":80"}

3. Caddy version:

v2.9.1 (with the OPNsense Plugin)

4. How I installed and ran Caddy:

Installed on OPNsense using the caddy plugin.

a. System environment:

OPNsense 24.7.12-amd64
FreeBSD 14.1-RELEASE-p6

b. Command:

Not really a command, as I use the OPNsense plugin.

d. My complete Caddy config:

{
  "apps": {
    "http": {
      "grace_period": 10000000000,
      "servers": {
        "srv0": {
          "listen": [
            ":443"
          ],
          "protocols": [
            "h1",
            "h2",
            "h3"
          ],
          "routes": [
            {
              "handle": [
                {
                  "handler": "subroute",
                  "routes": [
                    {
                      "handle": [
                        {
                          "handler": "subroute",
                          "routes": [
                            {
                              "handle": [
                                {
                                  "handler": "reverse_proxy",
                                  "upstreams": [
                                    {
                                      "dial": "10.1.1.14:2283"
                                    }
                                  ]
                                }
                              ]
                            }
                          ]
                        }
                      ]
                    }
                  ]
                }
              ],
              "match": [
                {
                  "host": [
                    "immich.voege.dev"
                  ]
                }
              ],
              "terminal": true
            }
          ]
        }
      }
    },
    "tls": {
      "automation": {
        "policies": [
          {
            "issuers": [
              {
                "challenges": {
                  "dns": {
                    "provider": {
                      "api_token": "API_KEY_HERE",
                      "name": "cloudflare"
                    },
                    "resolvers": [
                      "1.1.1.1"
                    ]
                  }
                },
                "email": "REDACTED",
                "module": "acme"
              }
            ],
            "subjects": [
              "immich.voege.dev"
            ]
          }
        ]
      }
    }
  },
  "logging": {
    "logs": {
      "default": {
        "encoder": {
          "format": "json",
          "time_format": "rfc3339"
        },
        "writer": {
          "address": "unixgram//var/run/caddy/log.sock",
          "output": "net"
        }
      }
    }
  }
}

5. Links to relevant resources:

Fixed. After looking at the logs again, it obviously was a permission error with the API Key, recreated it and now it’s working.

What permission error specifically? I am also having this issue, and have tried recreating my key as well as ensuring it has the right permissions.

Thanks

I am not 100% sure, but the I thin the Token Ibcretead for Caddy dis not have the Zone-Zone-Read and Zone-DNS-Edit permissions in cloudflare as outlined here: Modules - Caddy Documentation