Caddy x Route 53 reverse proxy

1. The problem I’m having:

Hello,

I am currently using a domain purchased through Route53 and am in the process of configuring Caddy for internal deployment of Seatable.

However, it seems that the reverse proxy is not functioning correctly, and I am unsure what needs to be fixed.

The main errors are as follows:
[Include the list of main errors here]

no certificate available for ‘whoami.mydomain.io
could not get certificate from issuer identifier=.mydomain.io issuer=acme.zerossl.com-v2-DV90 error=[.mydomain.io] solving challenges: presenting for challenge: adding temporary record for zone “mydomain.io.”: not found, ResolveEndpointV2 (order=https://acme.zerossl.com/v2/DV90/order/H4uW-oIUKbTN1cyh0rDSkg) (ca=https://acme.zerossl.com/v2/DV90)

Could you please assist me with this issue? Thank you very much for your help!

Best regards,
Damon

2. Error messages and/or full log output:

INF ts=1718711767.7110274 logger=docker-proxy msg=New Config JSON json={"logging":{"logs":{"default":{"level":"DEBUG"}}},"apps":{"http":{"servers":{"srv0":{"listen":[":443"],"routes":[{"match":[{"host":["seatable.mydomain.io"]}],"handle":[{"handler":"subroute","routes":[{"handle":[{"handler":"reverse_proxy","upstreams":[{"dial":":80"}]}]}]}],"terminal":true}]}}},"tls":{"automation":{"policies":[{"subjects":["seatable.mydomain.io"],"issuers":[{"challenges":{"dns":{"propagation_timeout":-1,"provider":{"access_key_id":"access_key_id","max_retries":3,"name":"route53","region":"region","secret_access_key":"secret_access_key"},"resolvers":["1.1.1.1","8.8.8.8"]}},"email":"myemail@mydomain.io","module":"acme"}]}]}}}}
INF ts=1718711767.7110636 logger=docker-proxy msg=Sending configuration to server=localhost
INF ts=1718711767.711548 logger=admin.api msg=received request method=POST host=localhost:2019 uri=/load remote_ip=127.0.0.1 remote_port=58382 headers={"Accept-Encoding":["gzip"],"Content-Length":["750"],"Content-Type":["application/json"],"User-Agent":["Go-http-client/1.1"]}
INF ts=1718711767.7120156 logger=admin msg=admin endpoint started address=localhost:2019 enforce_origin=false origins=["//localhost:2019","//[::1]:2019","//127.0.0.1:2019"]
INF ts=1718711767.7122111 logger=http.auto_https msg=server is listening only on the HTTPS port but has no TLS connection policies; adding one to enable TLS server_name=srv0 https_port=443
INF ts=1718711767.7122424 logger=http.auto_https msg=enabling automatic HTTP->HTTPS redirects server_name=srv0
DBG ts=1718711767.7122662 logger=http.auto_https msg=adjusted config tls={"automation":{"policies":[{"subjects":["seatable.mydomain.io"]},{}]}} http={"servers":{"remaining_auto_https_redirects":{"listen":[":80"],"routes":[{},{}]},"srv0":{"listen":[":443"],"routes":[{"handle":[{"handler":"subroute","routes":[{"handle":[{"handler":"reverse_proxy","upstreams":[{"dial":":80"}]}]}]}],"terminal":true}],"tls_connection_policies":[{}],"automatic_https":{}}}}
INF ts=1718711767.7124074 logger=http msg=enabling HTTP/3 listener addr=:443
DBG ts=1718711767.7124252 logger=http msg=starting server loop address=[::]:443 tls=true http3=true
INF ts=1718711767.7124305 logger=http.log msg=server running name=srv0 protocols=["h1","h2","h3"]
DBG ts=1718711767.7124534 logger=http msg=starting server loop address=[::]:80 tls=false http3=false
INF ts=1718711767.712463 logger=http.log msg=server running name=remaining_auto_https_redirects protocols=["h1","h2","h3"]
INF ts=1718711767.7124667 logger=http msg=enabling automatic TLS certificate management domains=["seatable.mydomain.io"]
INF ts=1718711767.7125268 logger=http msg=servers shutting down with eternal grace period
INF ts=1718711767.712716 msg=autosaved config (load with --resume flag) file=/config/caddy/autosave.json
INF ts=1718711767.7127247 logger=admin.api msg=load complete
INF ts=1718711767.7127526 logger=tls.obtain msg=releasing lock identifier=seatable.mydomain.io
INF ts=1718711767.7128258 logger=docker-proxy msg=Successfully configured server=localhost
INF ts=1718711767.7139182 logger=admin msg=stopped previous server address=localhost:2019
ERR ts=1718711767.7165534 logger=tls msg=job failed error=seatable.mydomain.io: obtaining certificate: context canceled
INF ts=1718711767.7424927 logger=tls.obtain msg=acquiring lock identifier=seatable.mydomain.io
INF ts=1718711767.7564337 logger=tls.obtain msg=lock acquired identifier=seatable.mydomain.io
INF ts=1718711767.7565138 logger=tls.obtain msg=obtaining certificate identifier=seatable.mydomain.io
DBG ts=1718711767.7565632 logger=events msg=event name=cert_obtaining id=cb07a3dc-a8e5-44a9-b5b8-678cc24be577 origin=tls data={"identifier":"seatable.mydomain.io"}
DBG ts=1718711767.7567446 logger=tls.obtain msg=trying issuer 1/1 issuer=acme-v02.api.letsencrypt.org-directory
INF ts=1718711767.7614386 logger=tls.issuance.acme msg=waiting on internal rate limiter identifiers=["seatable.mydomain.io"] ca=https://acme-v02.api.letsencrypt.org/directory account=myemail@mydomain.io
INF ts=1718711767.7614598 logger=tls.issuance.acme msg=done waiting on internal rate limiter identifiers=["seatable.mydomain.io"] ca=https://acme-v02.api.letsencrypt.org/directory account=myemail@mydomain.io
INF ts=1718711767.7614696 logger=tls.issuance.acme msg=using ACME account account_id=https://acme-v02.api.letsencrypt.org/acme/acct/1789009567 account_contact=["mailto:myemail@mydomain.io"]
DBG ts=1718711767.761478 logger=tls.issuance.acme.acme_client msg=creating order account=https://acme-v02.api.letsencrypt.org/acme/acct/1789009567 identifiers=["seatable.mydomain.io"]
DBG ts=1718711767.9743378 logger=http.stdlib msg=http: TLS handshake error from 10.75.60.82:29700: EOF
DBG ts=1718711768.3706481 logger=tls.issuance.acme.acme_client msg=http request method=HEAD url=https://acme-v02.api.letsencrypt.org/acme/new-nonce headers={"User-Agent":["Caddy/2.8.4 CertMagic acmez (linux; amd64)"]} response_headers={"Cache-Control":["public, max-age=0, no-cache"],"Date":["Tue, 18 Jun 2024 11:56:08 GMT"],"Link":["<https://acme-v02.api.letsencrypt.org/directory>;rel=\"index\""],"Replay-Nonce":["5yfKMBJJq6v2b6GIrSHAxXcE7vCbOm5-0_qtYLL7cPv6JVqR0FI"],"Server":["nginx"],"Strict-Transport-Security":["max-age=604800"],"X-Frame-Options":["DENY"]} status_code=200
DBG ts=1718711768.6106393 logger=tls.issuance.acme.acme_client msg=http request method=POST url=https://acme-v02.api.letsencrypt.org/acme/new-order headers={"Content-Type":["application/jose+json"],"User-Agent":["Caddy/2.8.4 CertMagic acmez (linux; amd64)"]} response_headers={"Boulder-Requester":["1789009567"],"Cache-Control":["public, max-age=0, no-cache"],"Content-Length":["346"],"Content-Type":["application/json"],"Date":["Tue, 18 Jun 2024 11:56:08 GMT"],"Link":["<https://acme-v02.api.letsencrypt.org/directory>;rel=\"index\""],"Location":["https://acme-v02.api.letsencrypt.org/acme/order/1789009567/279485473657"],"Replay-Nonce":["pOeJQAlBaQ1pgk-SrrdbZIOLpQaBg6Ivc0xZ-n6ZdBsHQ5RxPR0"],"Server":["nginx"],"Strict-Transport-Security":["max-age=604800"],"X-Frame-Options":["DENY"]} status_code=201
DBG ts=1718711768.8139734 logger=tls.issuance.acme.acme_client msg=http request method=POST url=https://acme-v02.api.letsencrypt.org/acme/authz-v3/365543603347 headers={"Content-Type":["application/jose+json"],"User-Agent":["Caddy/2.8.4 CertMagic acmez (linux; amd64)"]} response_headers={"Boulder-Requester":["1789009567"],"Cache-Control":["public, max-age=0, no-cache"],"Content-Length":["804"],"Content-Type":["application/json"],"Date":["Tue, 18 Jun 2024 11:56:08 GMT"],"Link":["<https://acme-v02.api.letsencrypt.org/directory>;rel=\"index\""],"Replay-Nonce":["pOeJQAlBXkkpGuJKwwAFxXizjOKQdL-qm6gr18PkMiwd6YDMEYA"],"Server":["nginx"],"Strict-Transport-Security":["max-age=604800"],"X-Frame-Options":["DENY"]} status_code=200
DBG ts=1718711768.814082 logger=tls.issuance.acme.acme_client msg=no solver configured challenge_type=http-01
DBG ts=1718711768.8140924 logger=tls.issuance.acme.acme_client msg=no solver configured challenge_type=tls-alpn-01
INF ts=1718711768.814096 logger=tls.issuance.acme.acme_client msg=trying to solve challenge identifier=seatable.mydomain.io challenge_type=dns-01 ca=https://acme-v02.api.letsencrypt.org/directory
ERR ts=1718711769.2356842 logger=tls.issuance.acme.acme_client msg=cleaning up solver identifier=seatable.mydomain.io challenge_type=dns-01 error=no memory of presenting a DNS record for "_acme-challenge.seatable.mydomain.io" (usually OK if presenting also failed)
DBG ts=1718711769.4421864 logger=tls.issuance.acme.acme_client msg=http request method=POST url=https://acme-v02.api.letsencrypt.org/acme/authz-v3/365543603347 headers={"Content-Type":["application/jose+json"],"User-Agent":["Caddy/2.8.4 CertMagic acmez (linux; amd64)"]} response_headers={"Boulder-Requester":["1789009567"],"Cache-Control":["public, max-age=0, no-cache"],"Content-Length":["808"],"Content-Type":["application/json"],"Date":["Tue, 18 Jun 2024 11:56:09 GMT"],"Link":["<https://acme-v02.api.letsencrypt.org/directory>;rel=\"index\""],"Replay-Nonce":["pOeJQAlB4vZoUOKCGrc5H_YJDjII8VhOW9FzhUECPXhypafElnI"],"Server":["nginx"],"Strict-Transport-Security":["max-age=604800"],"X-Frame-Options":["DENY"]} status_code=200
ERR ts=1718711769.442301 logger=tls.obtain msg=could not get certificate from issuer identifier=seatable.mydomain.io issuer=acme-v02.api.letsencrypt.org-directory error=[seatable.mydomain.io] solving challenges: presenting for challenge: adding temporary record for zone "mydomain.io.": not found, ResolveEndpointV2 (order=https://acme-v02.api.letsencrypt.org/acme/order/1789009567/279485473657) (ca=https://acme-v02.api.letsencrypt.org/directory)
DBG ts=1718711769.442354 logger=events msg=event name=cert_failed id=76c60e03-d03f-4c35-ba58-f6338f6dc6c5 origin=tls data={"error":{},"identifier":"seatable.mydomain.io","issuers":["acme-v02.api.letsencrypt.org-directory"],"renewal":false}
ERR ts=1718711769.4423635 logger=tls.obtain msg=will retry error=[seatable.mydomain.io] Obtain: [seatable.mydomain.io] solving challenges: presenting for challenge: adding temporary record for zone "mydomain.io.": not found, ResolveEndpointV2 (order=https://acme-v02.api.letsencrypt.org/acme/order/1789009567/279485473657) (ca=https://acme-v02.api.letsencrypt.org/directory) attempt=1 retrying_in=60 elapsed=1.685915835 max_duration=2592000
DBG ts=1718636271.4610305 logger=http.stdlib msg=http: TLS handshake error from 10.75.60.24:53067: no certificate available for 'whoami.mydomain.io'

3. Caddy version:

2.8.4

4. How I installed and ran Caddy:

a. System environment:

docker swarm(linux)

b. Command:

docker stack deploy ./caddy.yml caddy

c. Service/unit/compose file:

services:
  caddy:
    image: caddy-route53
    ports:
      - target: 80
        published: 80
        mode: host
      - target: 443
        published: 443
        mode: host
    environment:
      - CADDY_INGRESS_NETWORKS=caddy
      - CADDY_DOCKER_CADDYFILE_PATH=/etc/caddy/caddyfile/Caddyfile
      - TZ=Asia/Seoul
    networks:
      - caddy
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock
      - .caddy/data:/data
      - .caddyfile:/etc/caddy/caddyfile
    dns:
      - 1.1.1.1
      - 8.8.8.8
    deploy:
      labels: # Global options
        caddy.email: myemail@example.com
      placement:
        constraints:
          - node.role == manager
      replicas: 1
      restart_policy:
        condition: any
      resources:
        reservations:
          cpus: "0.1"
          memory: 200M

networks:
  caddy:
    external: true

whoami
version: "3.8"

services:
  whoami:
    image: containous/whoami
    networks:
      - caddy
    labels:
      caddy: whoami.mydomain.io
      caddy.reverse_proxy: "{{upstreams 80}}"

networks: 
  caddy:


d. My complete Caddy config:

# global settings
{
  debug
}

seatable.mydomain.io {
  reverse_proxy :80
  tls {
    dns route53 {
        max_retries 3
        access_key_id "<redacted>"
        secret_access_key "<redacted>"
        region "ap-northeast-2"
        token "<redacted>"
    }
    propagation_timeout -1
    resolvers 1.1.1.1 8.8.8.8
  }
}

5. Links to relevant resources:

There’s an open issue about this I think on the libdns/route53 repo. I’m mobile right now so it’s hard to link it but it sounds like AWS updated their SDK and had a breaking change that needs updating.

2 Likes

Hello, Matt!

It seems there was an issue with the AWS SDK.
I am able to obtain the certificate from acme successfully.

Although I defined the docker-compose.yml file as follows,

yml

caddy: https://seatable.mydomain.io
caddy.reverse_proxy: "{{upstreams 80}}"

Caddyfile

seatable.mydomain.io {
        reverse_proxy seatable-server:80
        tls {
                dns route53 {
                        max_retries 3
                        access_key_id "access_key_id"
                        secret_access_key "secret_access_key"
                        region "my_region"
                }
        }

        encode zstd gzip

}

it did not automatically configure in the Caddyfile. I was able to access seatable.mydomain.io after manually adding it.

What configuration should I use to have it automatically added to the Caddyfile?

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.