Failed to get certificate with route53 DNS provider

Eirik_Bell · April 12, 2018, 8:39am

We are settings up some new services in aws an decided to give Caddy a try as a reverse proxy to get automatic HTTPS. We are however having some difficulties getting it up and running and would very much appreciate some guidance on what may be the problem. The domain in hosted zone is replaced by DOMAIN.

In aws we have a public hosted zone DOMAIN, and we have a record caddy.DOMAIN. The domain itself has been verified working and we are able resolve dns.

Caddy is started in docker using abisoft/caddy image version 0.10.12 (Docker Hub). As we would like to host caddy behind an elastic load balancer we are trying to obtain certificate using the route53 DNS provider and have built the docker image with this command:

docker build -t our_registry.com/caddy-route53:latest --build-arg \ 
    plugins=route53 \
    github.com/abiosoft/caddy-docker.git

When starting the docker image, we expose ports 80 and 443, mount Caddyfile to /etc/Caddyfile and a certificate path to /etc/caddycerts. We also pass the following environment variables

CADDYPATH: /etc/caddycerts
AWS_ACCESS_KEY_ID: <MASKED>
AWS_SECRET_ACCESS_KEY: <MASKED>

The aws IAM user have the policy AmazonRoute53FullAccess

After starting the service, output is as following:

Activating privacy features... 2018/04/12 08:08:29 [INFO][caddy.DOMAIN] acme: Obtaining bundled SAN certificate
2018/04/12 08:08:30 [INFO][caddy.DOMAIN] AuthURL: https://acme-v02.api.letsencrypt.org/acme/authz/<MASKED>
2018/04/12 08:08:30 [INFO][caddy.DOMAIN] acme: Trying to solve DNS-01
2018/04/12 08:09:20 [INFO][caddy.DOMAIN] Checking DNS record propagation using [172.36.0.2:53]
2018/04/12 08:11:07 [caddy.DOMAIN] failed to get certificate: Time limit exceeded. Last error: read udp 172.17.0.3:34971->205.251.194.202:53: i/o timeout
exit status 1

This is our Caddyfile:

caddy.DOMAIN {
    proxy / <IP>:<PORT>

    tls {
        dns route53
    }
}

Any help or guidance to what we are doing wrong is very much appreciated.

Whitestrake · April 13, 2018, 4:55pm

You note that the domain can be resolved via DNS but I’m not sure if you mean publicly or for the Caddy host specifically - can Caddy inside the container access 172.36.0.2:53 and does that host serve a DNS response?

Eirik_Bell · April 16, 2018, 6:07am

Thank you for taking time to help resolve this issue.

Seems ok to me, this is dig from within the container. 172.36.0.2 is the nameserver setup by docker

bash-4.4# dig caddy.DOMAIN

; <<>> DiG 9.10.2 <<>> caddy.DOMAIN
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 25785
;; flags: qr rd ra; QUERY: 1, ANSWER: 3, AUTHORITY: 0, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 4096
;; QUESTION SECTION:
;caddy.DOMAIN.		IN	A

;; ANSWER SECTION:
caddy.DOMAIN.	60	IN	A	<ELB PUBLIC IP 1>
caddy.DOMAIN.	60	IN	A	<ELB PUBLIC IP 2>
caddy.DOMAIN.	60	IN	A	<ELB PUBLIC IP 3>

;; Query time: 2 msec
;; SERVER: 172.36.0.2#53(172.36.0.2)
;; WHEN: Mon Apr 16 06:02:36 UTC 2018
;; MSG SIZE  rcvd: 94

Is it something about the setup of DNS challenge I’m completely missing?

Whitestrake · April 16, 2018, 6:22am

Not sure. The error itself says it timed out reading UDP from 172.17.0.3:34971->205.251.194.202:53. The latter must be an external resolver?

Eirik_Bell · April 16, 2018, 6:28am

Yep that ip address is amazon.

Perhaps I should just play with certbot locally on the machine first

Eirik_Bell · April 16, 2018, 6:49am

So I just tested certbot in docker running on the same machine, this worked:

:~$ sudo docker run -it --rm --name certbot \
>             -v "/etc/letsencrypt:/etc/letsencrypt" \
>             -v "/var/lib/letsencrypt:/var/lib/letsencrypt" \
>             -e AWS_ACCESS_KEY_ID=<MASKED> \
>             -e AWS_SECRET_ACCESS_KEY=<MASKED> \
>             certbot/dns-route53 certonly --dns-route53 -d caddy.DOMAIN --register-unsafely-without-email
Saving debug log to /var/log/letsencrypt/letsencrypt.log
Found credentials in environment variables.
Plugins selected: Authenticator dns-route53, Installer None
Registering without email!
/usr/local/lib/python2.7/site-packages/josepy/jwa.py:107: CryptographyDeprecationWarning: signer and verifier have been deprecated. Please use sign and verify instead.
  signer = key.signer(self.padding, self.hash)

-------------------------------------------------------------------------------
Please read the Terms of Service at
https://letsencrypt.org/documents/LE-SA-v1.2-November-15-2017.pdf. You must
agree in order to register with the ACME server at
https://acme-v01.api.letsencrypt.org/directory
-------------------------------------------------------------------------------
(A)gree/(C)ancel: A
Obtaining a new certificate
Performing the following challenges:
dns-01 challenge for caddy.DOMAIN
Starting new HTTPS connection (1): route53.amazonaws.com
Waiting 10 seconds for DNS changes to propagate
Waiting for verification...
Cleaning up challenges
Resetting dropped connection: route53.amazonaws.com

IMPORTANT NOTES:
 - Congratulations! Your certificate and chain have been saved at:
   /etc/letsencrypt/live/caddy.DOMAIN/fullchain.pem
   Your key file has been saved at:
   /etc/letsencrypt/live/caddy.DOMAIN/privkey.pem
   Your cert will expire on 2018-07-15. To obtain a new or tweaked
   version of this certificate in the future, simply run certbot
   again. To non-interactively renew *all* of your certificates, run
   "certbot renew"
 - Your account credentials have been saved in your Certbot
   configuration directory at /etc/letsencrypt. You should make a
   secure backup of this folder now. This configuration directory will
   also contain certificates and private keys obtained by Certbot so
   making regular backups of this folder is ideal.
 - If you like Certbot, please consider supporting our work by:

   Donating to ISRG / Let's Encrypt:   https://letsencrypt.org/donate
   Donating to EFF:                    https://eff.org/donate-le

:~$ sudo ls -lah /etc/letsencrypt/live/caddy.DOMAIN/
total 12K
drwxr-xr-x 2 root root 4.0K Apr 16 06:42 .
drwx------ 3 root root 4.0K Apr 16 06:42 ..
-rw-r--r-- 1 root root  543 Apr 16 06:42 README
lrwxrwxrwx 1 root root   41 Apr 16 06:42 cert.pem -> ../../archive/caddy.DOMAIN/cert1.pem
lrwxrwxrwx 1 root root   42 Apr 16 06:42 chain.pem -> ../../archive/caddy.DOMAIN/chain1.pem
lrwxrwxrwx 1 root root   46 Apr 16 06:42 fullchain.pem -> ../../archive/caddy.DOMAIN/fullchain1.pem
lrwxrwxrwx 1 root root   44 Apr 16 06:42 privkey.pem -> ../../archive/caddy.DOMAIN/privkey1.pem

system · July 15, 2018, 6:49am

This topic was automatically closed 90 days after the last reply. New replies are no longer allowed.