Hetzner Wildcard ACME Challenge Failure

1. Output of caddy version:

v2.5.2

2. How I run Caddy:

I’m trying to setup wildcard Let’s Encrypt certs with reverse proxy. I am using Hetzner as a DNS provider so I replaced the binary in /usr/bin/caddy and /usr/share/caddy with the binary containing the Hetzner DNS plugin.

a. System environment:

Ubuntu 22.04 containers on Proxmox.
192.168.42.6 is my service
192.168.42.8 is Caddy

b. Command:

caddy run

c. Service/unit/compose file:

Not using Docker/unit/compose

d. My complete Caddy config:

*.example.com {
        tls {
                dns hetzner <myAPIKey>
        }
        @test host test.example.com
        handle @test {
                reverse_proxy 192.168.42.6:8000
        }
}

3. The problem I’m having:

The ACME challenge is not working (see error message below). On my Hetzner DNS panel I see the TXT record appear for about 5 seconds and disappear when I run caddy, so my API access seems to be working. My guess is that the error is due to a firewall setting because I see “connection refused” in the error log with a randomized port. Can I specify what port for the Let’s Encrypt ACME challenge to use so I change my firewall settings?

4. Error messages and/or full log output:

tls.obtain      could not get certificate from issuer   {"identifier": "*.example.com", "issuer": "acme-v02.api.letsencrypt.org-directory", "error": "[*.example.com] solving challenges: waiting for solver certmagic.solverWrapper to be ready: checking DNS propagation of _acme-challenge.example.com: read udp 192.168.42.8:51285->88.198.229.192:53: read: connection refused

5. What I already tried:

Since I’m following the vaultwarden guide but using a different DNS provider and a different service, I’ve put an A name record of test.example.com with my Caddy private IP address.

6. Links to relevant resources:

Looks like your DNS resolver is misbehaving, or refusing connections from Caddy. You’ll need to figure out what’s going on.

Is that IP address of the DNS server of your ISP?

Hi Francis,

Thanks for the extremely fast reply! I completely forgot to add that I’m using unbound on my OpenWRT router. I’ve set it so I don’t use any upstream DNS providers like Cloudflare, Quad9, etc and allowed private rebinds.

It looks like 88.198.229.192 is Hetzner’s name server. See their documentation. Is it some configuration I need to do on their panel?

I couldn’t say. I don’t use Hetzner.

Worth saying though, what Caddy is doing is trying to check if the DNS change it had the DNS plugin made actually took effect, by querying the DNS, before actually asking the ACME issuer to continue. This is an optional step that Caddy does, but it can be turned off. Unfortunately though, it’s a bit goofy to configure for it to work for both the default issuers (Let’s Encrypt and ZeroSSL) since we didn’t set up a shortcut for it at the top level of tls config:

tls {
	issuer acme {
		dns hetzner <myAPIKey>
		propagation_timeout -1
	}
	issuer zerossl {
		dns hetzner <myAPIKey>
		propagation_timeout -1
	}
}

Alternatively you could configure a different DNS resolver for Caddy to use during DNS issuance operations, like this:

tls {
	dns hetzner <myAPIKey>
	resolvers 1.1.1.1
}
1 Like

Thanks again Francis for the response!

So I tried your first solution:

*.example.com {
        tls {
                issuer acme {
                        dns hetzner <myAPIKey>
                        propagation_timeout -1
                }
                issuer zerossl {
                        dns hetzner <myAPIKey>
                        propagation_timeout -1
                }
        }
        @test host test.example.com
        handle @test {
                reverse_proxy 192.168.42.6:8000
        }
}

The error message changed to this.

ERROR   tls.obtain      could not get certificate from issuer   {"identifier": "*.example.com", "issuer": "acme-v02.api.letsencrypt.org-directory", "error": "HTTP 403 urn:ietf:params:acme:error:unauthorized - During secondary validation: Incorrect TXT record \"7qlg6xRi8mYYyajmLY6nqmQDtdkPJ8zPWAO7x-46ulY\" found at _acme-challenge.example.com"}

When I open the DNS records on my panel, I don’t see the 7qlg6xRi8mYYyajmLY6nqmQDtdkPJ8zPWAO7x-46ulY\ TXT record. The panel shows a different TXT record Caddy creates using the API.

Additionally when I rerun Caddy, the log seems to indicate that it still tries to validate the same 7qlg6xRi8mYYyajmLY6nqmQDtdkPJ8zPWAO7x-46ulY\ TXT record. On rerun, the API plugin creates a fresh TXT record in the Hetzner panel for the ACME challenge but Caddy still tries to validate the old record. Thus, the challenge fails.

The second solution probably will not work for me since I block any requests to upstream resolvers on my network.

Try clearing out Caddy’s storage then restart it, to start from a clean slate. You might want to clear out the DNS TXT records by hand as well.

If you aren’t already (since you didn’t say you are), you should run Caddy as a systemd service, for reliability.

Caddy’s storage location depends on how you run it, since it uses the current user’s environment to determine the storage location.

I ran whereis caddy and got caddy: /usr/bin/caddy /etc/caddy /usr/share/caddy
Do you mean I should clear my etc folder?

EDIT: I found it in ./local/share/caddy. See docs

I’ve also cleared out relevant DNS TXT records manually but the API seems to delete the TXT records automatically…

For running Caddy as a systemd service, do you mean caddy start or sudo systemctl enable caddy

Thank you!

Cleared out my storage and these are my error logs now. The DNS TXT records just don’t seem to match up with Caddy’s log. I’m not sure what causing the mismatch in TXT records between the API and Caddy.

2022/08/19 18:08:14.383 ERROR   tls.issuance.acme.acme_client   challenge failed        {"identifier": "*.example.com", "challenge_type": "dns-01", "problem": {"type": "urn:ietf:params:acme:error:unauthorized", "title": "", "detail": "Incorrect TXT record \"iU5gwgV6B1TtLvzG9Hyu9aXXyVYIrlKwOfWhCC3rG3A\" (and 1 more) found at _acme-challenge.example.com", "instance": "", "subproblems": []}}
2022/08/19 18:08:14.383 ERROR   tls.issuance.acme.acme_client   validating authorization        {"identifier": "*.example.com", "problem": {"type": "urn:ietf:params:acme:error:unauthorized", "title": "", "detail": "Incorrect TXT record \"iU5gwgV6B1TtLvzG9Hyu9aXXyVYIrlKwOfWhCC3rG3A\" (and 1 more) found at _acme-challenge.example.com", "instance": "", "subproblems": []}, "order": "https://acme-staging-v02.api.letsencrypt.org/acme/order/65120214/3717862254", "attempt": 1, "max_attempts": 3}
2022/08/19 18:08:14.383 ERROR   tls.obtain      could not get certificate from issuer   {"identifier": "*.example.com", "issuer": "acme-staging-v02.api.letsencrypt.org-directory", "error": "HTTP 403 urn:ietf:params:acme:error:unauthorized - Incorrect TXT record \"iU5gwgV6B1TtLvzG9Hyu9aXXyVYIrlKwOfWhCC3rG3A\" (and 1 more) found at _acme-challenge.example.com"}

With systemctl, yes.

caddy start is not reliable, it just starts an instance of Caddy in the background as the current user, but it will not get restarted if you reboot your machine etc.

See the docs for running Caddy as a service:

Keep in mind that the storage location when running as a service is different, because it runs as the caddy user, whose HOME is /var/lib/caddy

So I did more digging. I’m not sure if this is a plugin issue or Caddy itself but I get the error

2022/08/20 16:14:19.743 ERROR   tls.obtain      will retry      {"error": "[*.example.com] Obtain: [*.example.com] solving challenge: *.example.com: [*.example.com] authorization failed: HTTP 403 urn:ietf:params:acme:error:unauthorized - Incorrect TXT record \"mJ6vwt-p6AOpWOaTIt88T0839wsf0EDcXb_wkG4wWC0\" (and 1 more) found at _acme-challenge.example.com

But the output of dig txt _acme-challenge.example.com shows the correct txt records.

_acme-challenge.example.com. 300 IN	TXT	"4rK0B5oehzCJ-1mnb-VCVFSNB_dEox0vb4VQBNYWxm0"
_acme-challenge.example.com. 300 IN	TXT	"mJ6vwt-p6AOpWOaTIt88T0839wsf0EDcXb_wkG4wWC0"

So it looks like my configuration is correct and TXT is on my DNS records? But the challenge just fails for no reason.

Tried the DuckDNS, Cloudflare, and another third party plugin for my registrar.
The only thing that worked was DuckDNS plugin which is not ideal since I want to use a custom domain.

Cloudflare, Hetzner, and my registrar’s plugin all have the same issue with correct TXT records shown using dig txt _acme-challenge.example.com but caddy complains the TXT record doesn’t match.

You still can; use the dns_challenge_override_domain feature to delegate the challenge to your duckdns domain, and use a CNAME from your custom domain to your duckdns domain.

That way, the plugin will write the TXT record to duckdns, but ACME issuers will follow the CNAME from _acme-challenge.example.com to _acme-challenge.your.duckdns.org to find the challenge value.

example.com {
	tls {
		dns duckdns <myAPIKey>
		dns_challenge_override_domain your.duckdns.org
	}
}

Either way, I have no idea why Caddy would be having a mismatch between the challenges. Are you absolutely sure you cleared the correct storage location? That’s really the only explanation IMO, is that you didn’t actually clear the right one.

1 Like

Thanks Francis! . As for clearing the storage location, I cleared $HOME/.local/share/caddy when I ran caddy run. I also setup caddy as a service via systemd and got the same error of incorrect TXT record after clearing /var/lib/caddy/.local/share/caddy.

I really appreciate the workaround and will use it. However, I still want to resolve the root cause of this issue.

In the debug message, I see this message:

2022/08/21 20:15:48.295 DEBUG tls.issuance.acme.acme_client http request {"method": "POST", "url": "https://acme-staging-v02.api.letsencrypt.org/acme/new-order", "headers": {"Content-Type":["application/jose+json"],"User-Agent":["Caddy/2.5.2 CertMagic acmez (linux; amd64)"]}, "response_headers": {"Boulder-Requester":["65315204"],"Cache-Control":["public, max-age=0, no-cache"],"Content-Length":["345"],"Content-Type":["application/json"],"Date":["Sun, 21 Aug 2022 20:15:48 GMT"],"Link":["<https://acme-staging-v02.api.letsencrypt.org/directory>;rel=\"index\""],"Location":["https://acme-staging-v02.api.letsencrypt.org/acme/order/65315204/3746448364"],"Replay-Nonce":["0002S5y3NKlJUQPA59Ld1uYHovMVDDGDJGV-RvdM-h3IDeE"],"Server":["nginx"],"Strict-Transport-Security":["max-age=604800"],"X-Frame-Options":["DENY"]}, "status_code": 201}

When I open the acme order link in the debug (https://acme-staging-v02.api.letsencrypt.org/acme/order/65315204/3746448364), the JSON says

{
  "type": "urn:ietf:params:acme:error:malformed",
  "detail": "Method not allowed",
  "status": 405
}

Is there anything I can do to debug this error?

That’s working correctly, that’s not an error in the logs.

That order URL is meant to be used by Caddy with the HTTP POST method to provide the certificate signing request. It doesn’t work with GET requests, there’s nothing to see there.

Hi Francis. I tried the DuckDNS workaround but no luck. I set a CNAME for *.example.com to my.duckdns.org and made sure that the records propagated. I still have the same Incorrect TXT Error.

Error:

tls.obtain      could not get certificate from issuer   {"identifier": "*.example.com", "issuer": "acme-staging-v02.api.letsencrypt.org-directory", "error": "HTTP 403 urn:ietf:params:acme:error:unauthorized - Incorrect TXT record \"VSL-iTIw0Nhqifrxaz3LW_JNWIOHCArpfQFgfVDVnj8\" (and 1 more) found at _acme-challenge.example.com"}

Caddyfile:

*.example.com {
        tls {
                issuer zerossl {
                        dns duckdns <API_KEY>
                        dns_challenge_override_domain my.duckdns.org
                        propagation_timeout -1
                }
                issuer acme {
                        dns duckdns <API_KEY>
                        dns_challenge_override_domain my.duckdns.org
                        propagation_timeout -1
                }
        }
        @test host https://test.example.com

        handle @test {
               reverse_proxy 192.168.42.6:8000
        }
}

However, like I said before DuckDNS by itself works without any issue. So I’ve confirmed that Caddy and the DuckDNS plugin works without any issue.

Caddyfile:

my.duckdns.org {
         tls {
                 issuer acme {
                         dns duckdns <API_KEY>
                         propagation_timeout -1
                 }
                 issuer zerossl {
                         dns duckdns <API_KEY>
                         propagation_timeout -1
                 }
         }
         reverse_proxy 192.168.42.6:8000
}

Did you set up a CNAME record on your example.com domain from _acme.challenge.example.com to _acme-challenge.my.duckdns.org?

The host matcher doesn’t take a scheme, it only takes the hostname. Remove https:// there.

This really would be easier to help you debug if you didn’t redact your domain. Right now I just have to make guesses. I could play around with dig to find out what I can see.

1 Like

I didn’t before but I tried now. Still same error even after propagation.
Sure. I can give the full unredacted Caddyfile (other than my key).

{
        acme_ca https://acme-staging-v02.api.letsencrypt.org/directory
}
*.seh.app {
        tls {
                issuer zerossl {
                        dns duckdns <api_key>
                        dns_challenge_override_domain sehapp.duckdns.org
                        propagation_timeout -1
                }
                issuer acme {
                        dns duckdns <api_key>
                        dns_challenge_override_domain sehapp.duckdns.org
                        propagation_timeout -1
                }
        }
        @st host st.seh.app

        handle @st {
                reverse_proxy 192.168.42.6:8384
        }
}

Running dig _acme-challenge.seh.app, I’m not seeing the CNAME.

1 Like

So sorry about that! I had a typo in the DNS records. It should be updated now.
I’m still getting Incorrect TXT record after clearing caddy storage and running dig to make sure the DNS records are updated.

1 Like

Yeah that looks fine now:

$ dig TXT _acme-challenge.seh.app

;; ANSWER SECTION:
_acme-challenge.seh.app. 276	IN	CNAME	_acme-challenge.sehapp.duckdns.org.
_acme-challenge.sehapp.duckdns.org. 60 IN TXT	"QCFSTOvKqdrp6g1231kVDYcVVTQBEEh-tA_pXQ8RTZM"
1 Like