Caddy fails to retrieve aws challenge txt record value

Hello Everybody!
I’d like to ask for help regarding setting up caddy as a docker container to supply an endpoint with a valid letsencrypt certificate as a reverse proxy.

First I build a caddy 2.0.0 container with the aws route53 provider.
This is my Caddyfile:

{
  debug
  email karstensiemer@live.de
}
unifi.karstensiemer.de {
  reverse_proxy https://192.168.1.1:8443 {
       transport http {
           tls
           tls_insecure_skip_verify
       }
  }
  tls {
    dns route53 {
      max_retries 2
    }
  }
}

This is my docker-compose file for caddy:

  caddy:
    container_name: caddy
    image: caddy-route53:2.0.0
    restart: unless-stopped
    hostname: caddy
    ports:
      - "192.168.1.21:2019:2019"
      - "192.168.1.21:80:80"
      - "192.168.1.21:443:443"
    environment:
      AWS_ACCESS_KEY_ID: 'someId'
      AWS_SECRET_ACCESS_KEY: 'someKey'
      AWS_HOSTED_ZONE_ID: 'someZoneId'
    volumes:
       - './caddy/Caddyfile:/etc/caddy/Caddyfile'
       - './caddy/resolv.conf:/etc/resolv.conf'
       - './caddy/certs/:/data'
    dns:
      - 1.1.1.1
      - 1.0.0.1
      - 8.8.8.8
      - 8.4.4.8
    cap_add:
      - NET_ADMIN

Caddy boots up and successfully deployes the record inside my dns domain at aws.
Then it tries to retrieve the record and fails:

...
2020/06/08 18:26:44 [INFO] [unifi.karstensiemer.de] acme: Waiting for DNS record propagation.
2020/06/08 18:26:46 [INFO] [unifi.karstensiemer.de] acme: Waiting for DNS record propagation.
2020/06/08 18:26:48 [INFO] [unifi.karstensiemer.de] acme: Waiting for DNS record propagation.
2020/06/08 18:26:50 [INFO] [unifi.karstensiemer.de] acme: Waiting for DNS record propagation.
2020/06/08 18:26:53 [INFO] [unifi.karstensiemer.de] acme: Waiting for DNS record propagation.
2020/06/08 18:26:55 [INFO] [unifi.karstensiemer.de] acme: Waiting for DNS record propagation.
2020/06/08 18:26:57 [INFO] [unifi.karstensiemer.de] acme: Waiting for DNS record propagation.
2020/06/08 18:26:59 [INFO] [unifi.karstensiemer.de] acme: Waiting for DNS record propagation.
2020/06/08 18:27:01 [INFO] [unifi.karstensiemer.de] acme: Cleaning DNS-01 challenge
2020/06/08 18:28:04 [INFO] Deactivating auth: https://acme-v02.api.letsencrypt.org/acme/authz-v3/5104633946
2020/06/08 18:28:05 [ERROR] error: one or more domains had a problem:
[unifi.karstensiemer.de] time limit exceeded: last error: NS ns-558.awsdns-05.net. did not return the expected TXT record [fqdn: _acme-challenge.unifi.karstensiemer.de., value: value]:
 (challenge=dns-01 remaining=[])
2020/06/08 18:28:07 [ERROR] attempt 1: [unifi.karstensiemer.de] Obtain: [unifi.karstensiemer.de] error: one or more domains had a problem:
[unifi.karstensiemer.de] time limit exceeded: last error: NS ns-558.awsdns-05.net. did not return the expected TXT record [fqdn: _acme-challenge.unifi.karstensiemer.de., value: value]:
 - retrying in 1m0s (3m14.100346839s/720h0m0s elapsed)...
...

The aws record is absolutely correct and in the way I’d expect it.

What I have tried to fix this:

  1. At first I didn’t supply any extra dns settings via docker. It failed.
  2. I tried supplying dns servers via docker-compose (those in the docker-compose file). Caddy still used 127.0.0.11 because thats the way docker works. But the docker internal resolver should have used those. It failed.
  3. I execed into the container and installed bind-tools and did a dig on the record while caddy failed to retrieve it, and I could successfully resolve the record via the dns servers caddy asked.
  4. I swapped the resolv.conf from caddys container with one where the dns servers where in directly.
    Inside caddys log it showed that it tries to connect to them directly, bypassing the docker internal one, still caddy failed with retrieving the record.

karstensiemer.de is hosted at a other provider than aws but unifi.karstensiemer.de is delegated to aws where I already created an A record for it. But this shoudn’t hinder caddy from getting a cert should it?

I’d be super happy if any of you could asist me.
Thanks for taking the Time!

One thing I will ask right away is that you post what the actual domain is – it’s a forum rule since we cannot help as much otherwise. Often, redacting the domain name (which is public information) only hinders any troubleshooting.

1 Like

I see, thats of course okay for me :slight_smile:
I did update the post

2 Likes

@KarstenSiemer, the karstensiemer.de domain must be hosted by AWS in your scenario. Your delegation will only work if you want to create a URL under unifi.karstensiemer.de, for example: demo.unifi.karstensiemer.de would work just fine.

1 Like
2020/06/09 08:56:07 [INFO] [unms.unifi.karstensiemer.de] acme: Preparing to solve DNS-01
2020/06/09 08:57:10 [INFO] [unms.unifi.karstensiemer.de] acme: Trying to solve DNS-01
2020/06/09 08:57:10 [INFO] [unms.unifi.karstensiemer.de] acme: Checking DNS record propagation using [1.1.1.1:53 1.0.0.1:53 8.8.8.8:53 8.4.4.8:53 9.9.9.9:53]
2020/06/09 08:57:10 [INFO] Wait for propagation [timeout: 1m0s, interval: 2s]
2020/06/09 08:57:11 [INFO] [unms.unifi.karstensiemer.de] acme: Waiting for DNS record propagation.
2020/06/09 08:57:13 [INFO] [unms.unifi.karstensiemer.de] acme: Waiting for DNS record propagation.
2020/06/09 08:57:16 [INFO] [unms.unifi.karstensiemer.de] acme: Waiting for DNS record propagation.
2020/06/09 08:57:18 [INFO] [unms.unifi.karstensiemer.de] acme: Waiting for DNS record propagation.
2020/06/09 08:57:20 [INFO] [unms.unifi.karstensiemer.de] acme: Waiting for DNS record propagation.
2020/06/09 08:57:22 [INFO] [unms.unifi.karstensiemer.de] acme: Waiting for DNS record propagation.
2020/06/09 08:57:25 [INFO] [unms.unifi.karstensiemer.de] acme: Waiting for DNS record propagation.
2020/06/09 08:57:27 [INFO] [unms.unifi.karstensiemer.de] acme: Waiting for DNS record propagation.
2020/06/09 08:57:29 [INFO] [unms.unifi.karstensiemer.de] acme: Waiting for DNS record propagation.
2020/06/09 08:57:31 [INFO] [unms.unifi.karstensiemer.de] acme: Waiting for DNS record propagation.
2020/06/09 08:57:33 [INFO] [unms.unifi.karstensiemer.de] acme: Waiting for DNS record propagation.
2020/06/09 08:57:35 [INFO] [unms.unifi.karstensiemer.de] acme: Waiting for DNS record propagation.
2020/06/09 08:57:38 [INFO] [unms.unifi.karstensiemer.de] acme: Waiting for DNS record propagation.
2020/06/09 08:57:40 [INFO] [unms.unifi.karstensiemer.de] acme: Waiting for DNS record propagation.
2020/06/09 08:57:42 [INFO] [unms.unifi.karstensiemer.de] acme: Waiting for DNS record propagation.
2020/06/09 08:57:44 [INFO] [unms.unifi.karstensiemer.de] acme: Waiting for DNS record propagation.
2020/06/09 08:57:46 [INFO] [unms.unifi.karstensiemer.de] acme: Waiting for DNS record propagation.
2020/06/09 08:57:48 [INFO] [unms.unifi.karstensiemer.de] acme: Waiting for DNS record propagation.
2020/06/09 08:57:50 [INFO] [unms.unifi.karstensiemer.de] acme: Waiting for DNS record propagation.
2020/06/09 08:57:53 [INFO] [unms.unifi.karstensiemer.de] acme: Waiting for DNS record propagation.
2020/06/09 08:57:55 [INFO] [unms.unifi.karstensiemer.de] acme: Waiting for DNS record propagation.
2020/06/09 08:57:57 [INFO] [unms.unifi.karstensiemer.de] acme: Waiting for DNS record propagation.
2020/06/09 08:57:59 [INFO] [unms.unifi.karstensiemer.de] acme: Waiting for DNS record propagation.
2020/06/09 08:58:01 [INFO] [unms.unifi.karstensiemer.de] acme: Waiting for DNS record propagation.
2020/06/09 08:58:03 [INFO] [unms.unifi.karstensiemer.de] acme: Waiting for DNS record propagation.
2020/06/09 08:58:06 [INFO] [unms.unifi.karstensiemer.de] acme: Waiting for DNS record propagation.
2020/06/09 08:58:08 [INFO] [unms.unifi.karstensiemer.de] acme: Waiting for DNS record propagation.
2020/06/09 08:58:10 [INFO] [unms.unifi.karstensiemer.de] acme: Waiting for DNS record propagation.
2020/06/09 08:58:12 [INFO] [unms.unifi.karstensiemer.de] acme: Cleaning DNS-01 challenge
2020/06/09 08:59:15 [INFO] Deactivating auth: https://acme-v02.api.letsencrypt.org/acme/authz-v3/5117443517
2020/06/09 08:59:16 [ERROR] error: one or more domains had a problem:
[unms.unifi.karstensiemer.de] time limit exceeded: last error: NS ns-1599.awsdns-07.co.uk. returned NXDOMAIN for _acme-challenge.unms.unifi.karstensiemer.de.
 (challenge=dns-01 remaining=[])
2020/06/09 08:59:18 [ERROR] attempt 1: [unms.unifi.karstensiemer.de] Obtain: [unms.unifi.karstensiemer.de] error: one or more domains had a problem:
[unms.unifi.karstensiemer.de] time limit exceeded: last error: NS ns-1599.awsdns-07.co.uk. returned NXDOMAIN for _acme-challenge.unms.unifi.karstensiemer.de.
 - retrying in 1m0s (3m14.206988805s/720h0m0s elapsed)...

I am afraid it doesn’t for some reason.
Record inside route53 was created though

The delegation should work I think.
Letsencrypt states this in their docs at least.

Since Let’s Encrypt follows the DNS standards when looking up TXT records for DNS-01 validation, you can use CNAME records or NS records to delegate answering the challenge to other DNS zones. This can be used to delegate the _acme-challenge subdomain to a validation-specific server or zone. It can also be used if your DNS provider is slow to update, and you want to delegate to a quicker-updating server.

digging it works fine as well:

dig txt _acme-challenge.unms.unifi.karstensiemer.de.

; <<>> DiG 9.10.6 <<>> txt _acme-challenge.unms.unifi.karstensiemer.de.
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 21930
;; flags: qr rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 4096
;; QUESTION SECTION:
;_acme-challenge.unms.unifi.karstensiemer.de. IN	TXT

;; ANSWER SECTION:
_acme-challenge.unms.unifi.karstensiemer.de. 0 IN TXT "removedThisValue"

;; Query time: 42 msec
;; SERVER: 192.168.1.1#53(192.168.1.1)
;; WHEN: Tue Jun 09 11:17:56 CEST 2020
;; MSG SIZE  rcvd: 128

From my understanding, the _acme-challenge TXT record must be created under unifi.karstensiemer.de, not under unms.unifi.karstensiemer.de. Something is not right! :slight_smile:

I edited my caddy file because of your comment:

unms.unifi.karstensiemer.de {
  reverse_proxy https://192.168.1.1:8443 {
       transport http {
           tls
           tls_insecure_skip_verify
       }
  }
  tls {
    dns route53 {
      max_retries 2
    }
  }
}

so the acme record is correct isn’t it?

Can you try having only a single DNS resolver? Also, make sure your container is able to reach out to the DNS servers you specify.

Hi!
I tried using more than one only after it wasn’t working.
Using dig inside that same container is able to resolve the record.

Can you try to build v2.1-beta1 and give it another try? I have a very similar setup as yours and it works for me.

I will update, but first take a look at the dns requests caddy is doing. This is a sample from a tcpdump:

20:48:00.514817 IP 192.168.1.20.52355 > 192.168.1.1.domain: 56155+ [1au] TXT? _acme-challenge.unms.unifi.karstensiemer.de. (72)
20:48:00.535849 IP 192.168.1.1.domain > 192.168.1.20.52355: 56155 1/0/1 TXT "rgZMIqgKHsmRd5f13yuTWcMmSyp1IyhG9pdR8OKoWRM" (128)
20:48:00.538511 IP 192.168.1.20.37189 > 192.168.1.1.domain: 13563+ [1au] NS? unifi.karstensiemer.de. (51)
20:48:00.558683 IP 192.168.1.1.domain > 192.168.1.20.37189: 13563 4/0/1 NS ns-558.awsdns-05.net., NS ns-1079.awsdns-06.org., NS ns-9.awsdns-01.com., NS ns-1599.awsdns-07.co.uk. (189)
20:48:00.566805 IP 192.168.1.20.59716 > 192.168.1.1.domain: 8748+ AAAA? ns-558.awsdns-05.net. (38)
20:48:00.566894 IP 192.168.1.20.52191 > 192.168.1.1.domain: 53302+ A? ns-558.awsdns-05.net. (38)
20:48:00.567265 IP 192.168.1.1.domain > 192.168.1.20.59716: 8748 1/0/0 AAAA 2600:9000:5302:2e00::1 (66)
20:48:00.567347 IP 192.168.1.1.domain > 192.168.1.20.52191: 53302 1/0/0 A 205.251.194.46 (54)
20:48:00.570593 IP 192.168.1.20.39584 > ns-558.awsdns-05.net.domain: 8588 [1au] TXT? _acme-challenge.unms.unifi.karstensiemer.de. (72)
20:48:00.590759 IP ns-558.awsdns-05.net.domain > 192.168.1.20.39584: 8588 0/0/1 (72)

apperantly caddy is successfully retrieving the record. Then asks for the nameservers of the domain.
Resolves one of the nameservers and asks it directly again and is not getting an answer?
Or do I misinterpret this?

here the same part with a higher verbosity:

21:00:15.186581 IP (tos 0x0, ttl 63, id 42665, offset 0, flags [DF], proto UDP (17), length 100)
    192.168.1.20.43546 > 192.168.1.1.domain: [bad udp cksum 0x83c7 -> 0x0472!] 25124+ [1au] TXT? _acme-challenge.unms.unifi.karstensiemer.de. ar: . OPT UDPsize=4096 (72)
21:00:15.209265 IP (tos 0x0, ttl 63, id 6425, offset 0, flags [DF], proto UDP (17), length 156)
    192.168.1.1.domain > 192.168.1.20.43546: [udp sum ok] 25124 q: TXT? _acme-challenge.unms.unifi.karstensiemer.de. 1/0/1 _acme-challenge.unms.unifi.karstensiemer.de. [0s] TXT "vK3gTOY5iSOgxCNqXdTVIdc-CjyBfhyE24db74_cG84" ar: . OPT UDPsize=4096 (128)
21:00:15.211961 IP (tos 0x0, ttl 63, id 42667, offset 0, flags [DF], proto UDP (17), length 79)
    192.168.1.20.40932 > 192.168.1.1.domain: [bad udp cksum 0x83b2 -> 0x98b7!] 20854+ [1au] NS? unifi.karstensiemer.de. ar: . OPT UDPsize=4096 (51)
21:00:15.232210 IP (tos 0x0, ttl 63, id 6427, offset 0, flags [DF], proto UDP (17), length 217)
    192.168.1.1.domain > 192.168.1.20.40932: [udp sum ok] 20854 q: NS? unifi.karstensiemer.de. 4/0/1 unifi.karstensiemer.de. [23h47m4s] NS ns-1599.awsdns-07.co.uk., unifi.karstensiemer.de. [23h47m4s] NS ns-9.awsdns-01.com., unifi.karstensiemer.de. [23h47m4s] NS ns-1079.awsdns-06.org., unifi.karstensiemer.de. [23h47m4s] NS ns-558.awsdns-05.net. ar: . OPT UDPsize=4096 (189)
21:00:15.235710 IP (tos 0x0, ttl 63, id 42668, offset 0, flags [DF], proto UDP (17), length 69)
    192.168.1.20.48675 > 192.168.1.1.domain: [bad udp cksum 0x83a8 -> 0x0787!] 17347+ AAAA? ns-1599.awsdns-07.co.uk. (41)
21:00:15.235738 IP (tos 0x0, ttl 63, id 42669, offset 0, flags [DF], proto UDP (17), length 69)
    192.168.1.20.49443 > 192.168.1.1.domain: [bad udp cksum 0x83a8 -> 0x0b65!] 22501+ A? ns-1599.awsdns-07.co.uk. (41)
21:00:15.236161 IP (tos 0x0, ttl 63, id 6428, offset 0, flags [DF], proto UDP (17), length 97)
    192.168.1.1.domain > 192.168.1.20.48675: [udp sum ok] 17347 q: AAAA? ns-1599.awsdns-07.co.uk. 1/0/0 ns-1599.awsdns-07.co.uk. [23h47m14s] AAAA 2600:9000:5306:3f00::1 (69)
21:00:15.236247 IP (tos 0x0, ttl 63, id 6429, offset 0, flags [DF], proto UDP (17), length 85)
    192.168.1.1.domain > 192.168.1.20.49443: [udp sum ok] 22501 q: A? ns-1599.awsdns-07.co.uk. 1/0/0 ns-1599.awsdns-07.co.uk. [21h30m2s] A 205.251.198.63 (57)
21:00:15.239110 IP (tos 0x0, ttl 63, id 49821, offset 0, flags [DF], proto UDP (17), length 100)
    192.168.1.20.52775 > ns-1599.awsdns-07.co.uk.domain: [bad udp cksum 0x5659 -> 0x8e54!] 58018 [1au] TXT? _acme-challenge.unms.unifi.karstensiemer.de. ar: . OPT UDPsize=4096 (72)
21:00:15.259567 IP (tos 0x0, ttl 63, id 6430, offset 0, flags [DF], proto UDP (17), length 100)
    ns-1599.awsdns-07.co.uk.domain > 192.168.1.20.52775: [udp sum ok] 58018 q: TXT? _acme-challenge.unms.unifi.karstensiemer.de. 0/0/1 ar: . OPT UDPsize=4096 (72)

maybe more info on my setup.
eth0 is the real interface with the ip of 192.168.1.20.
192.168.1.21 is the ip of an virtual inface bridged onto eth0. Caddy is binding on that bridge.
192.168.1.1 is my local network router and local dns server.

supplying docker images for v2.1-beta1 for the builder would be nice. I am using a raspberry pi with armv7hl architecture and would have to setup my golang enviroment first

Coming soon! The version was just released yesterday, we need the people responsible to find the time to do it, you know, people are busy.

You can use xcaddy to cross-compile to a different OS/arch though. Probably something like this (not certain if the arch string is the correct one):

GOOS=linux GOARCH=armv7 xcaddy build v2.1.0-beta.1 --with <whatever plugin>
1 Like

my sincierest apologies.
I got it working and it was all my fault.
Apperantly my local firewall blocked some dns requests and didn’t log anything about it.
When bypassing my firewall it just works like it should. So the homework has to be done on my side.
Not sure what is going on there but caddy at least is working like a charm now.
Didn’t thought about it being a firewalling issue since the record could be resolved, like seen inside the tcpdump.
Sorry guys! I’ll close this one.

3 Likes

Glad you figured it out! Thanks for following up.

1 Like

I have to follow up with saying how cool caddy is.
I easily reverse proxied all my domain local ui’s which all use my private ca.
Within 5 minutes everything is using valid letsencrypt certificates with a glimpse of what would be needed with nginx, apache or even haproxy.
Amazing!
Thanks for your work on this project!

3 Likes

This topic was automatically closed after 30 days. New replies are no longer allowed.