1. Caddy version (caddy version):
v2.4.3 h1:Y1FaV2N4WO3rBqxSYA8UZsZTQdN+PwcoOcAiZTM8C0I=
2. How I run Caddy:
a. System environment:
Running in a Debian Buster based devcontainer in docker on a windows 10 machine.
dig -v
DiG 9.11.5-P4-5.1+deb10u5-Debian
go version
go version go1.16 linux/amd64
uname -r
Linux 13090f1c5e4a 5.10.16.3-microsoft-standard-WSL2 #1 SMP Fri Apr 2 22:23:49 UTC 2021 x86_64 GNU/Linux
b. Command:
xcaddy build \
--with github.com/caddy-dns/loopia
./caddy run -watch
c. Service/unit/compose file:
Not used
d. My complete Caddyfile or JSON config:
{
email "${MY_EMAIL}"
debug
}
(loopia) {
tls {
issuer acme {
propagation_timeout "10m"
resolvers "ns1.loopia.se"
dns loopia {
username "{$LOOPIA_USER}@loopiaapi"
password "{$LOOPIA_PASSWORD}"
}
}
}
}
wms.lcl.kapi.se {
root var/app-wms
try_files {path} /index.html
file_server *
import loopia
}
3. The problem I’m having:
When caddy is trying to get a certificate for the domain it fails. When checking dns traffic with tcpdump -i eth0 I see that the queries for _acme-challenge.wms.lcl.kapi.se. get a NXDomain response but when I do the same check using dig @ns1.loopia.se -t TXT _acme-challenge.wms.lcl.kapi.se. I get the expected response.
To me, it looks like certmagic does something strange when checking DNS. Cant see much but dig uses DNS-Cookies and Caddy doesn’t.
4. Error messages and/or full log output:
2021/08/12 08:36:06.620 INFO tls.obtain lock acquired {"identifier": "wms.lcl.kapi.se"}
2021/08/12 08:36:06.625 DEBUG tls.obtain trying issuer 1/1 {"issuer": "acme-v02.api.letsencrypt.org-directory"}
2021/08/12 08:36:06.625 INFO tls.issuance.acme waiting on internal rate limiter {"identifiers": ["wms.lcl.kapi.se"], "ca": "https://acme-v02.api.letsencrypt.org/directory", "account": "peter.magnusson@rikstvatt.se"}
2021/08/12 08:36:06.625 INFO tls.issuance.acme done waiting on internal rate limiter {"identifiers": ["wms.lcl.kapi.se"], "ca": "https://acme-v02.api.letsencrypt.org/directory", "account": "peter.magnusson@rikstvatt.se"}
2021/08/12 08:36:07.434 DEBUG tls.issuance.acme.acme_client http request {"method": "HEAD", "url": "https://acme-v02.api.letsencrypt.org/acme/new-nonce", "headers": {"User-Agent":["Caddy/2.4.3 CertMagic acmez (linux; amd64)"]}, "status_code": 200, "response_headers": {"Cache-Control":["public, max-age=0, no-cache"],"Date":["Thu, 12 Aug 2021 08:36:07 GMT"],"Link":["<https://acme-v02.api.letsencrypt.org/directory>;rel=\"index\""],"Replay-Nonce":["0101Lfo-wY58nlMQ7QmYre9TfKCoQyeaDvCmdifuOKz0uko"],"Server":["nginx"],"Strict-Transport-Security":["max-age=604800"],"X-Frame-Options":["DENY"]}}
2021/08/12 08:36:07.778 DEBUG tls.issuance.acme.acme_client http request {"method": "POST", "url": "https://acme-v02.api.letsencrypt.org/acme/new-order", "headers": {"Content-Type":["application/jose+json"],"User-Agent":["Caddy/2.4.3 CertMagic acmez (linux; amd64)"]}, "status_code": 201, "response_headers": {"Boulder-Requester":["157425801"],"Cache-Control":["public, max-age=0, no-cache"],"Content-Length":["338"],"Content-Type":["application/json"],"Date":["Thu, 12 Aug 2021 08:36:07 GMT"],"Link":["<https://acme-v02.api.letsencrypt.org/directory>;rel=\"index\""],"Location":["https://acme-v02.api.letsencrypt.org/acme/order/157425801/16328392990"],"Replay-Nonce":["0101Y_F2tB-2IKvlBktrxzJ3wXV62JWDsIASqrTGMn13Bi8"],"Server":["nginx"],"Strict-Transport-Security":["max-age=604800"],"X-Frame-Options":["DENY"]}}
2021/08/12 08:36:07.948 DEBUG tls.issuance.acme.acme_client http request {"method": "POST", "url": "https://acme-v02.api.letsencrypt.org/acme/authz-v3/21152454710", "headers": {"Content-Type":["application/jose+json"],"User-Agent":["Caddy/2.4.3 CertMagic acmez (linux; amd64)"]}, "status_code": 200, "response_headers": {"Boulder-Requester":["157425801"],"Cache-Control":["public, max-age=0, no-cache"],"Content-Length":["796"],"Content-Type":["application/json"],"Date":["Thu, 12 Aug 2021 08:36:07 GMT"],"Link":["<https://acme-v02.api.letsencrypt.org/directory>;rel=\"index\""],"Replay-Nonce":["0102iRnrgWod2Y0-AhLZN-WoIBvyUDgH787F56qplPl-Nxg"],"Server":["nginx"],"Strict-Transport-Security":["max-age=604800"],"X-Frame-Options":["DENY"]}}
2021/08/12 08:36:07.949 DEBUG tls.issuance.acme.acme_client no solver configured {"challenge_type": "tls-alpn-01"}
2021/08/12 08:36:07.949 INFO tls.issuance.acme.acme_client trying to solve challenge {"identifier": "wms.lcl.kapi.se", "challenge_type": "dns-01", "ca": "https://acme-v02.api.letsencrypt.org/directory"}
2021/08/12 08:46:14.410 DEBUG tls.issuance.acme.acme_client http request {"method": "POST", "url": "https://acme-v02.api.letsencrypt.org/acme/authz-v3/21152454710", "headers": {"Content-Type":["application/jose+json"],"User-Agent":["Caddy/2.4.3 CertMagic acmez (linux; amd64)"]}, "status_code": 400, "response_headers": {"Boulder-Requester":["157425801"],"Cache-Control":["public, max-age=0, no-cache"],"Content-Length":["173"],"Content-Type":["application/problem+json"],"Date":["Thu, 12 Aug 2021 08:46:14 GMT"],"Link":["<https://acme-v02.api.letsencrypt.org/directory>;rel=\"index\""],"Replay-Nonce":["0001_2clx8FIQqRp1RgvdkpJuk7zZarysZPCQ8hF5L2hssM"],"Server":["nginx"]}}
2021/08/12 08:46:14.410 DEBUG tls.issuance.acme.acme_client server rejected our nonce; retrying {"detail": "JWS has an invalid anti-replay nonce: \"0102iRnrgWod2Y0-AhLZN-WoIBvyUDgH787F56qplPl-Nxg\"", "error": "HTTP 400 urn:ietf:params:acme:error:badNonce - JWS has an invalid anti-replay nonce: \"0102iRnrgWod2Y0-AhLZN-WoIBvyUDgH787F56qplPl-Nxg\""}
2021/08/12 08:46:14.938 DEBUG tls.issuance.acme.acme_client http request {"method": "POST", "url": "https://acme-v02.api.letsencrypt.org/acme/authz-v3/21152454710", "headers": {"Content-Type":["application/jose+json"],"User-Agent":["Caddy/2.4.3 CertMagic acmez (linux; amd64)"]}, "status_code": 200, "response_headers": {"Boulder-Requester":["157425801"],"Cache-Control":["public, max-age=0, no-cache"],"Content-Length":["800"],"Content-Type":["application/json"],"Date":["Thu, 12 Aug 2021 08:46:14 GMT"],"Link":["<https://acme-v02.api.letsencrypt.org/directory>;rel=\"index\""],"Replay-Nonce":["0002MWbHdKHXT7duV8hwujVJ3kVnIB_9BBQycKlZlugVmfU"],"Server":["nginx"],"Strict-Transport-Security":["max-age=604800"],"X-Frame-Options":["DENY"]}}
2021/08/12 08:46:14.939 ERROR tls.obtain could not get certificate from issuer {"identifier": "wms.lcl.kapi.se", "issuer": "acme-v02.api.letsencrypt.org-directory", "error": "[wms.lcl.kapi.se] solving challenges: waiting for solver certmagic.solverWrapper to be ready: timed out waiting for record to fully propagate; verify DNS provider configuration is correct - last error: <nil> (order=https://acme-v02.api.letsencrypt.org/acme/order/157425801/16328392990) (ca=https://acme-v02.api.letsencrypt.org/directory)"}
2021/08/12 08:46:14.939 ERROR tls.obtain will retry {"error": "[wms.lcl.kapi.se] Obtain: [wms.lcl.kapi.se] solving challenges: waiting for solver certmagic.solverWrapper to be ready: timed out waiting for record to fully propagate; verify DNS provider configuration is correct - last error: <nil> (order=https://acme-v02.api.letsencrypt.org/acme/order/157425801/16328392990) (ca=https://acme-v02.api.letsencrypt.org/directory)", "attempt": 1, "retrying_in": 60, "elapsed": 608.318517, "max_duration": 2592000}
dig @ns1.loopia.se -t TXT _acme-challenge.wms.lcl.kapi.se. +short
"KB_VOSn6vQEgd6J17xOo6kZFijQ0lwlcgNI5fy7u66A"
tcpdump -i eth0
08:39:49.846232 IP 13090f1c5e4a.59928 > 192.168.65.5.domain: 42898+ AAAA? ns1.loopia.se. (31)
08:39:49.846500 IP 13090f1c5e4a.57847 > 192.168.65.5.domain: 65078+ A? ns1.loopia.se. (31)
08:39:49.851407 IP 192.168.65.5.domain > 13090f1c5e4a.57847: 65078 1/0/0 A 93.188.0.20 (47)
08:39:49.851763 IP 192.168.65.5.domain > 13090f1c5e4a.59928: 42898 1/0/0 AAAA 2a02:250:ffff::20 (59)
08:39:49.852770 IP 13090f1c5e4a.46521 > 93.188.0.20.domain: 50461+ [1au] NS? kapi.se. (36)
08:39:49.868921 IP 93.188.0.20.domain > 13090f1c5e4a.46521: 50461*- 2/0/1 NS ns1.loopia.se., NS ns2.loopia.se. (79)
08:39:49.871513 IP 13090f1c5e4a.44159 > 192.168.65.5.domain: 7998+ A? ns1.loopia.se. (31)
08:39:49.871797 IP 13090f1c5e4a.44762 > 192.168.65.5.domain: 1691+ AAAA? ns1.loopia.se. (31)
08:39:49.876270 IP 192.168.65.5.domain > 13090f1c5e4a.44159: 7998 1/0/0 A 93.188.0.20 (47)
08:39:49.877260 IP 192.168.65.5.domain > 13090f1c5e4a.44762: 1691 1/0/0 AAAA 2a02:250:ffff::20 (59)
08:39:49.879320 IP 13090f1c5e4a.39624 > 93.188.0.20.domain: 15126 [1au] TXT? _acme-challenge.wms.lcl.kapi.se. (60)
08:39:49.896179 IP 93.188.0.20.domain > 13090f1c5e4a.39624: 15126 NXDomain*- 0/1/1 (116)
5. What I already tried:
- changing resolvers to just about everything I can think of
- changing propagation timeout in various steps from 2m to 15m
- logging DNS traffic with tcp dump
- testing DNS lookup using
dig @ns1.loopia.se -t TXT _acme-challenge.wms.lcl.kapi.se. +shortat the same time as caddy gets NXDomain - validated DNS propagation using https://dnschecker.org
6. Links to relevant resources:
- pcap with 2 different DNS queries, first from caddy, second from dig. https://gist.github.com/kmpm/9ec2930e71f7fcfc53da705a986fe50d/raw/64ddea2058514e72b0dd2c4436d65b6f85b15828/one-of-each.pcap