I’m trying to setup wildcard Let’s Encrypt certs with reverse proxy. I am using Hetzner as a DNS provider so I replaced the binary in /usr/bin/caddy and /usr/share/caddy with the binary containing the Hetzner DNS plugin.
a. System environment:
Ubuntu 22.04 containers on Proxmox.
192.168.42.6 is my service
192.168.42.8 is Caddy
The ACME challenge is not working (see error message below). On my Hetzner DNS panel I see the TXT record appear for about 5 seconds and disappear when I run caddy, so my API access seems to be working. My guess is that the error is due to a firewall setting because I see “connection refused” in the error log with a randomized port. Can I specify what port for the Let’s Encrypt ACME challenge to use so I change my firewall settings?
4. Error messages and/or full log output:
tls.obtain could not get certificate from issuer {"identifier": "*.example.com", "issuer": "acme-v02.api.letsencrypt.org-directory", "error": "[*.example.com] solving challenges: waiting for solver certmagic.solverWrapper to be ready: checking DNS propagation of _acme-challenge.example.com: read udp 192.168.42.8:51285->88.198.229.192:53: read: connection refused
5. What I already tried:
Since I’m following the vaultwarden guide but using a different DNS provider and a different service, I’ve put an A name record of test.example.com with my Caddy private IP address.
Thanks for the extremely fast reply! I completely forgot to add that I’m using unbound on my OpenWRT router. I’ve set it so I don’t use any upstream DNS providers like Cloudflare, Quad9, etc and allowed private rebinds.
It looks like 88.198.229.192 is Hetzner’s name server. See their documentation. Is it some configuration I need to do on their panel?
Worth saying though, what Caddy is doing is trying to check if the DNS change it had the DNS plugin made actually took effect, by querying the DNS, before actually asking the ACME issuer to continue. This is an optional step that Caddy does, but it can be turned off. Unfortunately though, it’s a bit goofy to configure for it to work for both the default issuers (Let’s Encrypt and ZeroSSL) since we didn’t set up a shortcut for it at the top level of tls config:
ERROR tls.obtain could not get certificate from issuer {"identifier": "*.example.com", "issuer": "acme-v02.api.letsencrypt.org-directory", "error": "HTTP 403 urn:ietf:params:acme:error:unauthorized - During secondary validation: Incorrect TXT record \"7qlg6xRi8mYYyajmLY6nqmQDtdkPJ8zPWAO7x-46ulY\" found at _acme-challenge.example.com"}
When I open the DNS records on my panel, I don’t see the 7qlg6xRi8mYYyajmLY6nqmQDtdkPJ8zPWAO7x-46ulY\ TXT record. The panel shows a different TXT record Caddy creates using the API.
Additionally when I rerun Caddy, the log seems to indicate that it still tries to validate the same 7qlg6xRi8mYYyajmLY6nqmQDtdkPJ8zPWAO7x-46ulY\ TXT record. On rerun, the API plugin creates a fresh TXT record in the Hetzner panel for the ACME challenge but Caddy still tries to validate the old record. Thus, the challenge fails.
The second solution probably will not work for me since I block any requests to upstream resolvers on my network.
Cleared out my storage and these are my error logs now. The DNS TXT records just don’t seem to match up with Caddy’s log. I’m not sure what causing the mismatch in TXT records between the API and Caddy.
2022/08/19 18:08:14.383 ERROR tls.issuance.acme.acme_client challenge failed {"identifier": "*.example.com", "challenge_type": "dns-01", "problem": {"type": "urn:ietf:params:acme:error:unauthorized", "title": "", "detail": "Incorrect TXT record \"iU5gwgV6B1TtLvzG9Hyu9aXXyVYIrlKwOfWhCC3rG3A\" (and 1 more) found at _acme-challenge.example.com", "instance": "", "subproblems": []}}
2022/08/19 18:08:14.383 ERROR tls.issuance.acme.acme_client validating authorization {"identifier": "*.example.com", "problem": {"type": "urn:ietf:params:acme:error:unauthorized", "title": "", "detail": "Incorrect TXT record \"iU5gwgV6B1TtLvzG9Hyu9aXXyVYIrlKwOfWhCC3rG3A\" (and 1 more) found at _acme-challenge.example.com", "instance": "", "subproblems": []}, "order": "https://acme-staging-v02.api.letsencrypt.org/acme/order/65120214/3717862254", "attempt": 1, "max_attempts": 3}
2022/08/19 18:08:14.383 ERROR tls.obtain could not get certificate from issuer {"identifier": "*.example.com", "issuer": "acme-staging-v02.api.letsencrypt.org-directory", "error": "HTTP 403 urn:ietf:params:acme:error:unauthorized - Incorrect TXT record \"iU5gwgV6B1TtLvzG9Hyu9aXXyVYIrlKwOfWhCC3rG3A\" (and 1 more) found at _acme-challenge.example.com"}
caddy start is not reliable, it just starts an instance of Caddy in the background as the current user, but it will not get restarted if you reboot your machine etc.
See the docs for running Caddy as a service:
Keep in mind that the storage location when running as a service is different, because it runs as the caddy user, whose HOME is /var/lib/caddy
So I did more digging. I’m not sure if this is a plugin issue or Caddy itself but I get the error
2022/08/20 16:14:19.743 ERROR tls.obtain will retry {"error": "[*.example.com] Obtain: [*.example.com] solving challenge: *.example.com: [*.example.com] authorization failed: HTTP 403 urn:ietf:params:acme:error:unauthorized - Incorrect TXT record \"mJ6vwt-p6AOpWOaTIt88T0839wsf0EDcXb_wkG4wWC0\" (and 1 more) found at _acme-challenge.example.com
But the output of dig txt _acme-challenge.example.com shows the correct txt records.
_acme-challenge.example.com. 300 IN TXT "4rK0B5oehzCJ-1mnb-VCVFSNB_dEox0vb4VQBNYWxm0"
_acme-challenge.example.com. 300 IN TXT "mJ6vwt-p6AOpWOaTIt88T0839wsf0EDcXb_wkG4wWC0"
So it looks like my configuration is correct and TXT is on my DNS records? But the challenge just fails for no reason.
Tried the DuckDNS, Cloudflare, and another third party plugin for my registrar.
The only thing that worked was DuckDNS plugin which is not ideal since I want to use a custom domain.
Cloudflare, Hetzner, and my registrar’s plugin all have the same issue with correct TXT records shown using dig txt _acme-challenge.example.com but caddy complains the TXT record doesn’t match.
You still can; use the dns_challenge_override_domain feature to delegate the challenge to your duckdns domain, and use a CNAME from your custom domain to your duckdns domain.
That way, the plugin will write the TXT record to duckdns, but ACME issuers will follow the CNAME from _acme-challenge.example.com to _acme-challenge.your.duckdns.org to find the challenge value.
Either way, I have no idea why Caddy would be having a mismatch between the challenges. Are you absolutely sure you cleared the correct storage location? That’s really the only explanation IMO, is that you didn’t actually clear the right one.
Thanks Francis! . As for clearing the storage location, I cleared $HOME/.local/share/caddy when I ran caddy run. I also setup caddy as a service via systemd and got the same error of incorrect TXT record after clearing /var/lib/caddy/.local/share/caddy.
I really appreciate the workaround and will use it. However, I still want to resolve the root cause of this issue.
That’s working correctly, that’s not an error in the logs.
That order URL is meant to be used by Caddy with the HTTP POST method to provide the certificate signing request. It doesn’t work with GET requests, there’s nothing to see there.
Hi Francis. I tried the DuckDNS workaround but no luck. I set a CNAME for *.example.com to my.duckdns.org and made sure that the records propagated. I still have the same Incorrect TXT Error.
Error:
tls.obtain could not get certificate from issuer {"identifier": "*.example.com", "issuer": "acme-staging-v02.api.letsencrypt.org-directory", "error": "HTTP 403 urn:ietf:params:acme:error:unauthorized - Incorrect TXT record \"VSL-iTIw0Nhqifrxaz3LW_JNWIOHCArpfQFgfVDVnj8\" (and 1 more) found at _acme-challenge.example.com"}
Did you set up a CNAME record on your example.com domain from _acme.challenge.example.com to _acme-challenge.my.duckdns.org?
The host matcher doesn’t take a scheme, it only takes the hostname. Remove https:// there.
This really would be easier to help you debug if you didn’t redact your domain. Right now I just have to make guesses. I could play around with dig to find out what I can see.
So sorry about that! I had a typo in the DNS records. It should be updated now.
I’m still getting Incorrect TXT record after clearing caddy storage and running dig to make sure the DNS records are updated.