Route53: updating the parent domain zone, instead of the subdomain zone

1. The problem I’m having:

Route53: Caddy attempts to update the wrong zone.

Our base domain, which I’m showing here as fleefleefloofloo.com, has a number of delegated subdomains in Route53. For whatever reason, Caddy is trying to update the base domain zone instead of the subdomain zone:

Obtain: [sub.fleefleefloofloo.com] solving challenges: presenting for challenge: adding temporary record for zone \"fleefleefloofloo.com.\": operation error Route 53: ListResourceRecordSets, https response error StatusCode: 403, RequestID: 306dd9ed-1570-4d36-b90c-16953e211de5, api error AccessDenied: User: arn:aws:iam::123456789012:user/CTFDNSUpdateUser is not authorized to perform: route53:ListResourceRecordSets on resource: arn:aws:route53:::hostedzone/SNIPSNIPSNIPSNIPSNIPSNIP because no identity-based policy allows the route53:ListResourceRecordSets action 

It’s just trying to update the wrong zone; the credentials provided have update rights for only sub.fleefleefloofloo.com for security reasons, and in any case, it should be updating the zone for sub.fleefleefloofloo.com, not fleefleefloofloo.com.

Is there a way to manually specify what zone the plugin should be trying to update?

For what it’s worth, the purpose of using the DNS challenge here is so that I can eventually request a wildcard for *.sub.fleefleefloofloo.com; if I remove the ‘tls’ key from the Caddyfile and just let Caddy attempt to get a cert based on an HTTP challenge, it actually works fine.

2. Error messages and/or full log output:

2024/03/08 17:23:14.581	INFO	using provided configuration	{"config_file": "/etc/caddy/Caddyfile", "config_adapter": "caddyfile"}
2024/03/08 17:23:14.584	WARN	Caddyfile input is not formatted; run 'caddy fmt --overwrite' to fix inconsistencies	{"adapter": "caddyfile", "file": "/etc/caddy/Caddyfile", "line": 2}
2024/03/08 17:23:14.585	INFO	redirected default logger	{"from": "stderr", "to": "stdout"}
2024/03/08 17:23:14.586	INFO	admin	admin endpoint started	{"address": "localhost:2019", "enforce_origin": false, "origins": ["//localhost:2019", "//[::1]:2019", "//127.0.0.1:2019"]}
2024/03/08 17:23:14.587	INFO	http.auto_https	server is listening only on the HTTPS port but has no TLS connection policies; adding one to enable TLS	{"server_name": "srv0", "https_port": 443}
2024/03/08 17:23:14.587	INFO	tls.cache.maintenance	started background certificate maintenance	{"cache": "0xc00036a500"}
2024/03/08 17:23:14.587	INFO	http.auto_https	enabling automatic HTTP->HTTPS redirects	{"server_name": "srv0"}
2024/03/08 17:23:14.588	WARN	http.auto_https	server is listening only on the HTTP port, so no automatic HTTPS will be applied to this server	{"server_name": "srv1", "http_port": 80}
2024/03/08 17:23:14.589	DEBUG	http.auto_https	adjusted config	{"tls": {"automation":{"policies":[{"subjects":["sub.fleefleefloofloo.com"]},{}]}}, "http": {"servers":{"srv0":{"listen":[":443"],"routes":[{"handle":[{"handler":"subroute","routes":[{"handle":[{"handler":"reverse_proxy","upstreams":[{"dial":"10.201.12.1:8000"}]}]}]}],"terminal":true}],"tls_connection_policies":[{}],"automatic_https":{}},"srv1":{"listen":[":80"],"routes":[{},{"handle":[{"handler":"static_response","headers":{"Location":["https://sub.fleefleefloofloo.com"]},"status_code":302}]},{}],"automatic_https":{"disable":true}}}}}
2024/03/08 17:23:14.590	INFO	http	enabling HTTP/3 listener	{"addr": ":443"}
2024/03/08 17:23:14.590	INFO	failed to sufficiently increase receive buffer size (was: 208 kiB, wanted: 2048 kiB, got: 416 kiB). See https://github.com/quic-go/quic-go/wiki/UDP-Buffer-Sizes for details.
2024/03/08 17:23:14.591	DEBUG	http	starting server loop	{"address": "[::]:443", "tls": true, "http3": true}
2024/03/08 17:23:14.591	INFO	http.log	server running	{"name": "srv0", "protocols": ["h1", "h2", "h3"]}
2024/03/08 17:23:14.591	DEBUG	http	starting server loop	{"address": "[::]:80", "tls": false, "http3": false}
2024/03/08 17:23:14.591	INFO	http.log	server running	{"name": "srv1", "protocols": ["h1", "h2", "h3"]}
2024/03/08 17:23:14.591	INFO	http	enabling automatic TLS certificate management	{"domains": ["sub.fleefleefloofloo.com"]}
2024/03/08 17:23:14.592	INFO	tls	cleaning storage unit	{"storage": "FileStorage:/data/caddy"}
2024/03/08 17:23:14.592	INFO	tls	finished cleaning storage units
2024/03/08 17:23:14.593	INFO	autosaved config (load with --resume flag)	{"file": "/config/caddy/autosave.json"}
2024/03/08 17:23:14.593	INFO	serving initial configuration
2024/03/08 17:23:14.594	INFO	tls.obtain	acquiring lock	{"identifier": "sub.fleefleefloofloo.com"}
2024/03/08 17:23:14.594	INFO	tls.obtain	lock acquired	{"identifier": "sub.fleefleefloofloo.com"}
2024/03/08 17:23:14.595	INFO	tls.obtain	obtaining certificate	{"identifier": "sub.fleefleefloofloo.com"}
2024/03/08 17:23:14.595	DEBUG	events	event	{"name": "cert_obtaining", "id": "2c605c7b-4275-4c42-a31a-380a0da71104", "origin": "tls", "data": {"identifier":"sub.fleefleefloofloo.com"}}
2024/03/08 17:23:14.595	DEBUG	tls.obtain	trying issuer 1/2	{"issuer": "acme-staging-v02.api.letsencrypt.org-directory"}
2024/03/08 17:23:14.662	DEBUG	tls.issuance.acme.acme_client	http request	{"method": "GET", "url": "https://acme-staging-v02.api.letsencrypt.org/directory", "headers": {"User-Agent":["Caddy/2.7.6 CertMagic acmez (linux; amd64)"]}, "response_headers": {"Cache-Control":["public, max-age=0, no-cache"],"Content-Length":["826"],"Content-Type":["application/json"],"Date":["Fri, 08 Mar 2024 17:23:14 GMT"],"Server":["nginx"],"Strict-Transport-Security":["max-age=604800"],"X-Frame-Options":["DENY"]}, "status_code": 200}
2024/03/08 17:23:14.672	DEBUG	tls.issuance.acme.acme_client	http request	{"method": "HEAD", "url": "https://acme-staging-v02.api.letsencrypt.org/acme/new-nonce", "headers": {"User-Agent":["Caddy/2.7.6 CertMagic acmez (linux; amd64)"]}, "response_headers": {"Cache-Control":["public, max-age=0, no-cache"],"Date":["Fri, 08 Mar 2024 17:23:14 GMT"],"Link":["<https://acme-staging-v02.api.letsencrypt.org/directory>;rel=\"index\""],"Replay-Nonce":["VeqK7FlMyQSlntY0BARviPUYyjXoLDTqp7C9OiiwwaWIjNiWhck"],"Server":["nginx"],"Strict-Transport-Security":["max-age=604800"],"X-Frame-Options":["DENY"]}, "status_code": 200}
2024/03/08 17:23:14.702	DEBUG	tls.issuance.acme.acme_client	http request	{"method": "POST", "url": "https://acme-staging-v02.api.letsencrypt.org/acme/new-acct", "headers": {"Content-Type":["application/jose+json"],"User-Agent":["Caddy/2.7.6 CertMagic acmez (linux; amd64)"]}, "response_headers": {"Boulder-Requester":["139520964"],"Cache-Control":["public, max-age=0, no-cache"],"Content-Length":["267"],"Content-Type":["application/json"],"Date":["Fri, 08 Mar 2024 17:23:14 GMT"],"Link":["<https://acme-staging-v02.api.letsencrypt.org/directory>;rel=\"index\"","<https://letsencrypt.org/documents/LE-SA-v1.3-September-21-2022.pdf>;rel=\"terms-of-service\""],"Location":["https://acme-staging-v02.api.letsencrypt.org/acme/acct/139520964"],"Replay-Nonce":["fwe4XdjY35nx6x-WRL_WDQHAHipRIYLaZqllrn26gUAEh0bEm5A"],"Server":["nginx"],"Strict-Transport-Security":["max-age=604800"],"X-Frame-Options":["DENY"]}, "status_code": 201}
2024/03/08 17:23:14.703	INFO	tls.issuance.acme	waiting on internal rate limiter	{"identifiers": ["sub.fleefleefloofloo.com"], "ca": "https://acme-staging-v02.api.letsencrypt.org/directory", "account": ""}
2024/03/08 17:23:14.703	INFO	tls.issuance.acme	done waiting on internal rate limiter	{"identifiers": ["sub.fleefleefloofloo.com"], "ca": "https://acme-staging-v02.api.letsencrypt.org/directory", "account": ""}
2024/03/08 17:23:14.757	DEBUG	tls.issuance.acme.acme_client	http request	{"method": "POST", "url": "https://acme-staging-v02.api.letsencrypt.org/acme/new-order", "headers": {"Content-Type":["application/jose+json"],"User-Agent":["Caddy/2.7.6 CertMagic acmez (linux; amd64)"]}, "response_headers": {"Boulder-Requester":["139520964"],"Cache-Control":["public, max-age=0, no-cache"],"Content-Length":["355"],"Content-Type":["application/json"],"Date":["Fri, 08 Mar 2024 17:23:14 GMT"],"Link":["<https://acme-staging-v02.api.letsencrypt.org/directory>;rel=\"index\""],"Location":["https://acme-staging-v02.api.letsencrypt.org/acme/order/139520964/15134713034"],"Replay-Nonce":["VeqK7FlMW9kiMkxGsFRalkW6nNbT_priL8hYxzSm35pCSYnIa10"],"Server":["nginx"],"Strict-Transport-Security":["max-age=604800"],"X-Frame-Options":["DENY"]}, "status_code": 201}
2024/03/08 17:23:14.774	DEBUG	tls.issuance.acme.acme_client	http request	{"method": "POST", "url": "https://acme-staging-v02.api.letsencrypt.org/acme/authz-v3/11578646034", "headers": {"Content-Type":["application/jose+json"],"User-Agent":["Caddy/2.7.6 CertMagic acmez (linux; amd64)"]}, "response_headers": {"Boulder-Requester":["139520964"],"Cache-Control":["public, max-age=0, no-cache"],"Content-Length":["821"],"Content-Type":["application/json"],"Date":["Fri, 08 Mar 2024 17:23:14 GMT"],"Link":["<https://acme-staging-v02.api.letsencrypt.org/directory>;rel=\"index\""],"Replay-Nonce":["VeqK7FlMwVCRC_LXTSQHiYg7YQK-0HvAc_EfifUeLmEu2hXgDpk"],"Server":["nginx"],"Strict-Transport-Security":["max-age=604800"],"X-Frame-Options":["DENY"]}, "status_code": 200}
2024/03/08 17:23:14.775	DEBUG	tls.issuance.acme.acme_client	no solver configured	{"challenge_type": "http-01"}
2024/03/08 17:23:14.775	DEBUG	tls.issuance.acme.acme_client	no solver configured	{"challenge_type": "tls-alpn-01"}
2024/03/08 17:23:14.775	INFO	tls.issuance.acme.acme_client	trying to solve challenge	{"identifier": "sub.fleefleefloofloo.com", "challenge_type": "dns-01", "ca": "https://acme-staging-v02.api.letsencrypt.org/directory"}
2024/03/08 17:23:15.226	ERROR	tls.issuance.acme.acme_client	cleaning up solver	{"identifier": "sub.fleefleefloofloo.com", "challenge_type": "dns-01", "error": "no memory of presenting a DNS record for \"_acme-challenge.sub.fleefleefloofloo.com\" (usually OK if presenting also failed)"}
2024/03/08 17:23:15.249	DEBUG	tls.issuance.acme.acme_client	http request	{"method": "POST", "url": "https://acme-staging-v02.api.letsencrypt.org/acme/authz-v3/11578646034", "headers": {"Content-Type":["application/jose+json"],"User-Agent":["Caddy/2.7.6 CertMagic acmez (linux; amd64)"]}, "response_headers": {"Boulder-Requester":["139520964"],"Cache-Control":["public, max-age=0, no-cache"],"Content-Length":["825"],"Content-Type":["application/json"],"Date":["Fri, 08 Mar 2024 17:23:15 GMT"],"Link":["<https://acme-staging-v02.api.letsencrypt.org/directory>;rel=\"index\""],"Replay-Nonce":["fwe4XdjYT9BlnjMhBE41IWanTxxR8tIzti3kIgG7LPehkCTAbos"],"Server":["nginx"],"Strict-Transport-Security":["max-age=604800"],"X-Frame-Options":["DENY"]}, "status_code": 200}
2024/03/08 17:23:15.249	ERROR	tls.obtain	could not get certificate from issuer	{"identifier": "sub.fleefleefloofloo.com", "issuer": "acme-staging-v02.api.letsencrypt.org-directory", "error": "[sub.fleefleefloofloo.com] solving challenges: presenting for challenge: adding temporary record for zone \"fleefleefloofloo.com.\": operation error Route 53: ListResourceRecordSets, https response error StatusCode: 403, RequestID: cea79b02-75af-432b-9564-ff8651a568ca, api error AccessDenied: User: arn:aws:iam::123456789012:user/CTFDNSUpdateUser is not authorized to perform: route53:ListResourceRecordSets on resource: arn:aws:route53:::hostedzone/SNIPSNIPSNIPSNIPSNIPSNIP because no identity-based policy allows the route53:ListResourceRecordSets action (order=https://acme-staging-v02.api.letsencrypt.org/acme/order/139520964/15134713034) (ca=https://acme-staging-v02.api.letsencrypt.org/directory)"}
2024/03/08 17:23:15.249	DEBUG	tls.obtain	trying issuer 2/2	{"issuer": "acme-staging-v02.api.letsencrypt.org-directory"}
2024/03/08 17:23:15.251	INFO	tls.issuance.zerossl	waiting on internal rate limiter	{"identifiers": ["sub.fleefleefloofloo.com"], "ca": "https://acme-staging-v02.api.letsencrypt.org/directory", "account": ""}
2024/03/08 17:23:15.252	INFO	tls.issuance.zerossl	done waiting on internal rate limiter	{"identifiers": ["sub.fleefleefloofloo.com"], "ca": "https://acme-staging-v02.api.letsencrypt.org/directory", "account": ""}
2024/03/08 17:23:15.304	DEBUG	tls.issuance.zerossl.acme_client	http request	{"method": "HEAD", "url": "https://acme-staging-v02.api.letsencrypt.org/acme/new-nonce", "headers": {"User-Agent":["Caddy/2.7.6 CertMagic acmez (linux; amd64)"]}, "response_headers": {"Cache-Control":["public, max-age=0, no-cache"],"Date":["Fri, 08 Mar 2024 17:23:15 GMT"],"Link":["<https://acme-staging-v02.api.letsencrypt.org/directory>;rel=\"index\""],"Replay-Nonce":["fwe4XdjY4vt9ByMqaq8nCxCNxsziE-V23yxHw05OoN4-xJkYeNk"],"Server":["nginx"],"Strict-Transport-Security":["max-age=604800"],"X-Frame-Options":["DENY"]}, "status_code": 200}
2024/03/08 17:23:15.348	DEBUG	tls.issuance.zerossl.acme_client	http request	{"method": "POST", "url": "https://acme-staging-v02.api.letsencrypt.org/acme/new-order", "headers": {"Content-Type":["application/jose+json"],"User-Agent":["Caddy/2.7.6 CertMagic acmez (linux; amd64)"]}, "response_headers": {"Boulder-Requester":["139520964"],"Cache-Control":["public, max-age=0, no-cache"],"Content-Length":["355"],"Content-Type":["application/json"],"Date":["Fri, 08 Mar 2024 17:23:15 GMT"],"Link":["<https://acme-staging-v02.api.letsencrypt.org/directory>;rel=\"index\""],"Location":["https://acme-staging-v02.api.letsencrypt.org/acme/order/139520964/15134713304"],"Replay-Nonce":["VeqK7FlM6szkgF1tocVjWWHpNg-fGwcG4qAaYIar04CYECqVjLM"],"Server":["nginx"],"Strict-Transport-Security":["max-age=604800"],"X-Frame-Options":["DENY"]}, "status_code": 201}
2024/03/08 17:23:15.362	DEBUG	tls.issuance.zerossl.acme_client	http request	{"method": "POST", "url": "https://acme-staging-v02.api.letsencrypt.org/acme/authz-v3/11578646194", "headers": {"Content-Type":["application/jose+json"],"User-Agent":["Caddy/2.7.6 CertMagic acmez (linux; amd64)"]}, "response_headers": {"Boulder-Requester":["139520964"],"Cache-Control":["public, max-age=0, no-cache"],"Content-Length":["821"],"Content-Type":["application/json"],"Date":["Fri, 08 Mar 2024 17:23:15 GMT"],"Link":["<https://acme-staging-v02.api.letsencrypt.org/directory>;rel=\"index\""],"Replay-Nonce":["VeqK7FlM1nvPTPnKzx4ftx3z6O7KGoZYeXqMA-69q_0a750hI60"],"Server":["nginx"],"Strict-Transport-Security":["max-age=604800"],"X-Frame-Options":["DENY"]}, "status_code": 200}
2024/03/08 17:23:15.363	DEBUG	tls.issuance.zerossl.acme_client	no solver configured	{"challenge_type": "http-01"}
2024/03/08 17:23:15.363	DEBUG	tls.issuance.zerossl.acme_client	no solver configured	{"challenge_type": "tls-alpn-01"}
2024/03/08 17:23:15.363	INFO	tls.issuance.zerossl.acme_client	trying to solve challenge	{"identifier": "sub.fleefleefloofloo.com", "challenge_type": "dns-01", "ca": "https://acme-staging-v02.api.letsencrypt.org/directory"}
2024/03/08 17:23:15.779	ERROR	tls.issuance.zerossl.acme_client	cleaning up solver	{"identifier": "sub.fleefleefloofloo.com", "challenge_type": "dns-01", "error": "no memory of presenting a DNS record for \"_acme-challenge.sub.fleefleefloofloo.com\" (usually OK if presenting also failed)"}
2024/03/08 17:23:15.799	DEBUG	tls.issuance.zerossl.acme_client	http request	{"method": "POST", "url": "https://acme-staging-v02.api.letsencrypt.org/acme/authz-v3/11578646194", "headers": {"Content-Type":["application/jose+json"],"User-Agent":["Caddy/2.7.6 CertMagic acmez (linux; amd64)"]}, "response_headers": {"Boulder-Requester":["139520964"],"Cache-Control":["public, max-age=0, no-cache"],"Content-Length":["825"],"Content-Type":["application/json"],"Date":["Fri, 08 Mar 2024 17:23:15 GMT"],"Link":["<https://acme-staging-v02.api.letsencrypt.org/directory>;rel=\"index\""],"Replay-Nonce":["VeqK7FlMIvpfxknjxuhRZEzm0bALHQsqR9bWJTiWz6G-0ECaBtU"],"Server":["nginx"],"Strict-Transport-Security":["max-age=604800"],"X-Frame-Options":["DENY"]}, "status_code": 200}
2024/03/08 17:23:15.800	ERROR	tls.obtain	could not get certificate from issuer	{"identifier": "sub.fleefleefloofloo.com", "issuer": "acme-staging-v02.api.letsencrypt.org-directory", "error": "[sub.fleefleefloofloo.com] solving challenges: presenting for challenge: adding temporary record for zone \"fleefleefloofloo.com.\": operation error Route 53: ListResourceRecordSets, https response error StatusCode: 403, RequestID: 306dd9ed-1570-4d36-b90c-16953e211de5, api error AccessDenied: User: arn:aws:iam::123456789012:user/CTFDNSUpdateUser is not authorized to perform: route53:ListResourceRecordSets on resource: arn:aws:route53:::hostedzone/SNIPSNIPSNIPSNIPSNIPSNIP because no identity-based policy allows the route53:ListResourceRecordSets action (order=https://acme-staging-v02.api.letsencrypt.org/acme/order/139520964/15134713304) (ca=https://acme-staging-v02.api.letsencrypt.org/directory)"}
2024/03/08 17:23:15.800	DEBUG	events	event	{"name": "cert_failed", "id": "b8ceebc9-e3c1-4262-9978-dadb13a5bc3e", "origin": "tls", "data": {"error":{},"identifier":"sub.fleefleefloofloo.com","issuers":["acme-staging-v02.api.letsencrypt.org-directory","acme-staging-v02.api.letsencrypt.org-directory"],"renewal":false}}
2024/03/08 17:23:15.800	ERROR	tls.obtain	will retry	{"error": "[sub.fleefleefloofloo.com] Obtain: [sub.fleefleefloofloo.com] solving challenges: presenting for challenge: adding temporary record for zone \"fleefleefloofloo.com.\": operation error Route 53: ListResourceRecordSets, https response error StatusCode: 403, RequestID: 306dd9ed-1570-4d36-b90c-16953e211de5, api error AccessDenied: User: arn:aws:iam::123456789012:user/CTFDNSUpdateUser is not authorized to perform: route53:ListResourceRecordSets on resource: arn:aws:route53:::hostedzone/SNIPSNIPSNIPSNIPSNIPSNIP because no identity-based policy allows the route53:ListResourceRecordSets action (order=https://acme-staging-v02.api.letsencrypt.org/acme/order/139520964/15134713304) (ca=https://acme-staging-v02.api.letsencrypt.org/directory)", "attempt": 1, "retrying_in": 60, "elapsed": 1.205483859, "max_duration": 2592000}


3. Caddy version:

v2.7.6 h1:w0NymbG2m9PcvKWsrXO6EEkY9Ru4FJK8uQbYcev1p3A=

4. How I installed and ran Caddy:

using the official Docker images, by using the following Dockerfile and command to build a version that includes the Route53 plugin:

FROM caddy:builder as builder

RUN xcaddy build \
  --with github.com/caddy-dns/route53

FROM caddy:latest

COPY --from=builder /usr/bin/caddy /usr/bin/caddy
docker build -t fleefleefloofloo/caddy .

a. System environment:

Debian 12, running as an lxc in proxmox
Docker info

Client: Docker Engine - Community
 Version:           25.0.0                                                                                                  API version:       1.44
 Go version:        go1.21.6
 Git commit:        e758fe5
 Built:             Thu Jan 18 17:09:59 2024
 OS/Arch:           linux/amd64
 Context:           default

Server: Docker Engine - Community
 Engine:
  Version:          25.0.0
  API version:      1.44 (minimum version 1.24)
  Go version:       go1.21.6
  Git commit:       615dfdf
  Built:            Thu Jan 18 17:09:59 2024
  OS/Arch:          linux/amd64                                                                                              Experimental:     false
 containerd:
  Version:          1.6.27
  GitCommit:        a1496014c916f9e62104b33d1bb5bd03b0858e59
 runc:
  Version:          1.1.11
  GitCommit:        v1.1.11-0-g4bccb38
 docker-init:
  Version:          0.19.0
  GitCommit:        de40ad0

b. Command:

docker run -it --rm -p 8080:80 -p 443:443 -v $PWD/Caddyfile:/etc/caddy/Caddyfile -v caddy_data_b:/data fleefleefloofloo/caddy       

c. Service/unit/compose file:

n/a

d. My complete Caddy config:


    acme_ca https://acme-staging-v02.api.letsencrypt.org/directory

    log default {
        output stdout
        format console
        level debug
    }
}

:80 {
    redir https://sub.fleefleefloofloo.com
}

https://sub.fleefleefloofloo.com:443 {
    tls {
        dns route53 {
                max_retries 10
                access_key_id "snip"
                secret_access_key "snip"
                region "us-west-2"
        }
    }

    reverse_proxy 10.201.12.1:8000
}

:man_shrugging:

Probably best if you ask for help on the plugin’s GitHub repo. It’s community maintained.

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.