Problem with ssl challenge dns-01

kwypchlo · March 1, 2022, 11:18pm

1. Caddy version (`caddy version`):

2.4.6 with dns route53 plugin

2. How I run Caddy:

SkynetLabs/skynet-webportal/blob/master/docker/caddy/Dockerfile

FROM caddy:2.4.6-builder AS caddy-builder

# available dns resolvers: https://github.com/caddy-dns
RUN xcaddy build --with github.com/caddy-dns/route53

FROM caddy:2.4.6-alpine

COPY --from=caddy-builder /usr/bin/caddy /usr/bin/caddy

# bash required for mo to work (mo is mustache templating engine - https://github.com/tests-always-included/mo)
RUN apk add --no-cache bash

COPY caddy.json.template mo /etc/caddy/

CMD [ "sh", "-c", \
      "/etc/caddy/mo < /etc/caddy/caddy.json.template > /etc/caddy/caddy.json ; \
       caddy run --config /etc/caddy/caddy.json" \
    ]

a. System environment:

Docker

b. Command:

using in docker-compose context, default entry point

c. Service/unit/compose file:

github.com

SkynetLabs/skynet-webportal/blob/master/docker-compose.yml#L42-L56

      
        
            caddy:
              build:
                context: ./docker/caddy
                dockerfile: Dockerfile
              container_name: caddy
              restart: unless-stopped
              logging: *default-logging
              env_file:
                - .env
              volumes:
                - ./docker/data/caddy/data:/data
                - ./docker/data/caddy/config:/config
              networks:
                shared:
                  ipv4_address: 10.10.10.20

d. My complete Caddyfile or JSON config:

{
  "apps": {
    "tls": {
      "certificates": {
        "automate": [
              "skynetfree.net", "*.skynetfree.net", "*.hns.skynetfree.net",
              "eu-lv-101.skynetfree.net", "*.eu-lv-101.skynetfree.net", "*.hns.eu-lv-101.skynetfree.net"
        ]
      },
      "automation": {
        "policies": [
          {
            "issuers": [
              {
                "module": "acme",
                "email": "devs@siasky.net",
                "challenges": {
                  "dns": {
                    "provider": {
                      "name": "route53"
                    },
                    "ttl": "30m"
                  }
                }
              }
            ]
          }
        ]
      }
    }
  }
}

3. The problem I’m having:

I am using caddy only to fetch and manage ssl certificates. Most of the time when I’m fetching new certificates, caddy seems to fail somehow - it creates acme records in route53 correctly but then it fails and it fails to clean up those records and then every time it restarts the process it says the records are already there and cannot do anything.

4. Error messages and/or full log output:

Attaching to caddy
caddy            | {"level":"info","ts":1646175973.865099,"msg":"using provided configuration","config_file":"/etc/caddy/caddy.json","config_adapter":""}
caddy            | {"level":"info","ts":1646175973.866614,"logger":"admin","msg":"admin endpoint started","address":"tcp/localhost:2019","enforce_origin":false,"origins":["localhost:2019","[::1]:2019","127.0.0.1:2019"]}
caddy            | {"level":"info","ts":1646175973.8668988,"logger":"tls.cache.maintenance","msg":"started background certificate maintenance","cache":"0xc000904fc0"}
caddy            | {"level":"info","ts":1646175973.8679416,"msg":"autosaved config (load with --resume flag)","file":"/config/caddy/autosave.json"}
caddy            | {"level":"info","ts":1646175973.8679512,"msg":"serving initial configuration"}
caddy            | {"level":"info","ts":1646175973.8679683,"logger":"tls","msg":"cleaning storage unit","description":"FileStorage:/data/caddy"}
caddy            | {"level":"info","ts":1646175973.8685193,"logger":"tls","msg":"finished cleaning storage units"}
caddy            | {"level":"info","ts":1646175973.86868,"logger":"tls.obtain","msg":"acquiring lock","identifier":"skynetfree.net"}
caddy            | {"level":"info","ts":1646175973.8693416,"logger":"tls.obtain","msg":"lock acquired","identifier":"skynetfree.net"}
caddy            | {"level":"info","ts":1646175973.8695743,"logger":"tls.obtain","msg":"acquiring lock","identifier":"*.skynetfree.net"}
caddy            | {"level":"info","ts":1646175973.8697376,"logger":"tls.issuance.acme","msg":"waiting on internal rate limiter","identifiers":["skynetfree.net"],"ca":"https://acme-v02.api.letsencrypt.org/directory","account":"devs@siasky.net"}
caddy            | {"level":"info","ts":1646175973.8697467,"logger":"tls.issuance.acme","msg":"done waiting on internal rate limiter","identifiers":["skynetfree.net"],"ca":"https://acme-v02.api.letsencrypt.org/directory","account":"devs@siasky.net"}
caddy            | {"level":"info","ts":1646175973.8700404,"logger":"tls.obtain","msg":"acquiring lock","identifier":"*.hns.skynetfree.net"}
caddy            | {"level":"info","ts":1646175973.8702877,"logger":"tls.obtain","msg":"lock acquired","identifier":"*.skynetfree.net"}
caddy            | {"level":"info","ts":1646175973.8706462,"logger":"tls.obtain","msg":"lock acquired","identifier":"*.hns.skynetfree.net"}
caddy            | {"level":"info","ts":1646175973.870674,"logger":"tls.issuance.acme","msg":"waiting on internal rate limiter","identifiers":["*.skynetfree.net"],"ca":"https://acme-v02.api.letsencrypt.org/directory","account":"devs@siasky.net"}
caddy            | {"level":"info","ts":1646175973.8706818,"logger":"tls.issuance.acme","msg":"done waiting on internal rate limiter","identifiers":["*.skynetfree.net"],"ca":"https://acme-v02.api.letsencrypt.org/directory","account":"devs@siasky.net"}
caddy            | {"level":"info","ts":1646175973.8710628,"logger":"tls.issuance.acme","msg":"waiting on internal rate limiter","identifiers":["*.hns.skynetfree.net"],"ca":"https://acme-v02.api.letsencrypt.org/directory","account":"devs@siasky.net"}
caddy            | {"level":"info","ts":1646175973.8710725,"logger":"tls.issuance.acme","msg":"done waiting on internal rate limiter","identifiers":["*.hns.skynetfree.net"],"ca":"https://acme-v02.api.letsencrypt.org/directory","account":"devs@siasky.net"}
caddy            | {"level":"info","ts":1646175975.0144036,"logger":"tls.issuance.acme.acme_client","msg":"trying to solve challenge","identifier":"skynetfree.net","challenge_type":"dns-01","ca":"https://acme-v02.api.letsencrypt.org/directory"}
caddy            | {"level":"info","ts":1646175975.4628952,"logger":"tls.issuance.acme.acme_client","msg":"trying to solve challenge","identifier":"*.skynetfree.net","challenge_type":"dns-01","ca":"https://acme-v02.api.letsencrypt.org/directory"}
caddy            | {"level":"info","ts":1646175975.645902,"logger":"tls.issuance.acme.acme_client","msg":"trying to solve challenge","identifier":"*.hns.skynetfree.net","challenge_type":"dns-01","ca":"https://acme-v02.api.letsencrypt.org/directory"}
caddy            | {"level":"error","ts":1646176040.616508,"logger":"tls.issuance.acme.acme_client","msg":"cleaning up solver","identifier":"skynetfree.net","challenge_type":"dns-01"}
caddy            | {"level":"info","ts":1646176040.6165626,"logger":"tls.issuance.acme.acme_client","msg":"validations succeeded; finalizing order","order":"https://acme-v02.api.letsencrypt.org/acme/order/431842630/68002779550"}
caddy            | {"level":"error","ts":1646176041.52123,"logger":"tls.issuance.acme.acme_client","msg":"cleaning up solver","identifier":"*.hns.skynetfree.net","challenge_type":"dns-01"}
caddy            | {"level":"info","ts":1646176041.5212748,"logger":"tls.issuance.acme.acme_client","msg":"validations succeeded; finalizing order","order":"https://acme-v02.api.letsencrypt.org/acme/order/431842630/68002781670"}
caddy            | {"level":"error","ts":1646176041.5950713,"logger":"tls.issuance.acme.acme_client","msg":"cleaning up solver","identifier":"*.skynetfree.net","challenge_type":"dns-01","error":"no memory of presenting a DNS record for skynetfree.net (probably OK if presenting failed)"}
caddy            | {"level":"error","ts":1646176041.774566,"logger":"tls.obtain","msg":"could not get certificate from issuer","identifier":"*.skynetfree.net","issuer":"acme-v02.api.letsencrypt.org-directory","error":"[*.skynetfree.net] solving challenges: presenting for challenge: adding temporary record for zone skynetfree.net.: InvalidChangeBatch: InvalidChangeBatch: [Tried to create resource record set [name='_acme-challenge.skynetfree.net.', type='TXT'] but it already exists]\n\tstatus code: 400, request id: cea7b376-a1b9-4608-b047-3f3c595a374f (order=https://acme-v02.api.letsencrypt.org/acme/order/431842630/68002781280) (ca=https://acme-v02.api.letsencrypt.org/directory)"}
caddy            | {"level":"error","ts":1646176041.774632,"logger":"tls.obtain","msg":"will retry","error":"[*.skynetfree.net] Obtain: [*.skynetfree.net] solving challenges: presenting for challenge: adding temporary record for zone skynetfree.net.: InvalidChangeBatch: InvalidChangeBatch: [Tried to create resource record set [name='_acme-challenge.skynetfree.net.', type='TXT'] but it already exists]\n\tstatus code: 400, request id: cea7b376-a1b9-4608-b047-3f3c595a374f (order=https://acme-v02.api.letsencrypt.org/acme/order/431842630/68002781280) (ca=https://acme-v02.api.letsencrypt.org/directory)","attempt":1,"retrying_in":60,"elapsed":67.90433396,"max_duration":2592000}
caddy            | {"level":"info","ts":1646176041.779712,"logger":"tls.issuance.acme.acme_client","msg":"successfully downloaded available certificate chains","count":2,"first_url":"https://acme-v02.api.letsencrypt.org/acme/cert/0310ab580c2243e63ffdaa7b9e5a046e2670"}
caddy            | {"level":"info","ts":1646176041.7803743,"logger":"tls.obtain","msg":"certificate obtained successfully","identifier":"skynetfree.net"}
caddy            | {"level":"info","ts":1646176041.7803993,"logger":"tls.obtain","msg":"releasing lock","identifier":"skynetfree.net"}
caddy            | {"level":"info","ts":1646176042.7881117,"logger":"tls.issuance.acme.acme_client","msg":"successfully downloaded available certificate chains","count":2,"first_url":"https://acme-v02.api.letsencrypt.org/acme/cert/04a677ac9b9824f7d0316e4791639a0ac79e"}
caddy            | {"level":"info","ts":1646176042.7889297,"logger":"tls.obtain","msg":"certificate obtained successfully","identifier":"*.hns.skynetfree.net"}
caddy            | {"level":"info","ts":1646176042.7889636,"logger":"tls.obtain","msg":"releasing lock","identifier":"*.hns.skynetfree.net"}
caddy            | {"level":"info","ts":1646176103.1420152,"logger":"tls.issuance.acme.acme_client","msg":"trying to solve challenge","identifier":"*.skynetfree.net","challenge_type":"dns-01","ca":"https://acme-staging-v02.api.letsencrypt.org/directory"}
caddy            | {"level":"error","ts":1646176103.6177459,"logger":"tls.issuance.acme.acme_client","msg":"cleaning up solver","identifier":"*.skynetfree.net","challenge_type":"dns-01","error":"no memory of presenting a DNS record for skynetfree.net (probably OK if presenting failed)"}
caddy            | {"level":"error","ts":1646176103.7966042,"logger":"tls.obtain","msg":"could not get certificate from issuer","identifier":"*.skynetfree.net","issuer":"acme-v02.api.letsencrypt.org-directory","error":"[*.skynetfree.net] solving challenges: presenting for challenge: adding temporary record for zone skynetfree.net.: InvalidChangeBatch: InvalidChangeBatch: [Tried to create resource record set [name='_acme-challenge.skynetfree.net.', type='TXT'] but it already exists]\n\tstatus code: 400, request id: 760f84f8-7da6-4f89-9f47-302c2098f240 (order=https://acme-staging-v02.api.letsencrypt.org/acme/order/45741628/1916393478) (ca=https://acme-staging-v02.api.letsencrypt.org/directory)"}
caddy            | {"level":"error","ts":1646176103.7966654,"logger":"tls.obtain","msg":"will retry","error":"[*.skynetfree.net] Obtain: [*.skynetfree.net] solving challenges: presenting for challenge: adding temporary record for zone skynetfree.net.: InvalidChangeBatch: InvalidChangeBatch: [Tried to create resource record set [name='_acme-challenge.skynetfree.net.', type='TXT'] but it already exists]\n\tstatus code: 400, request id: 760f84f8-7da6-4f89-9f47-302c2098f240 (order=https://acme-staging-v02.api.letsencrypt.org/acme/order/45741628/1916393478) (ca=https://acme-staging-v02.api.letsencrypt.org/directory)","attempt":2,"retrying_in":120,"elapsed":129.926367323,"max_duration":2592000}
caddy            | {"level":"info","ts":1646176224.943264,"logger":"tls.issuance.acme.acme_client","msg":"trying to solve challenge","identifier":"*.skynetfree.net","challenge_type":"dns-01","ca":"https://acme-staging-v02.api.letsencrypt.org/directory"}
caddy            | {"level":"error","ts":1646176225.7464094,"logger":"tls.issuance.acme.acme_client","msg":"cleaning up solver","identifier":"*.skynetfree.net","challenge_type":"dns-01","error":"no memory of presenting a DNS record for skynetfree.net (probably OK if presenting failed)"}
caddy            | {"level":"error","ts":1646176225.9244406,"logger":"tls.obtain","msg":"could not get certificate from issuer","identifier":"*.skynetfree.net","issuer":"acme-v02.api.letsencrypt.org-directory","error":"[*.skynetfree.net] solving challenges: presenting for challenge: adding temporary record for zone skynetfree.net.: InvalidChangeBatch: InvalidChangeBatch: [Tried to create resource record set [name='_acme-challenge.skynetfree.net.', type='TXT'] but it already exists]\n\tstatus code: 400, request id: f5cbc9fe-b415-4059-8cbd-10bc3b214cb1 (order=https://acme-staging-v02.api.letsencrypt.org/acme/order/45741628/1916410268) (ca=https://acme-staging-v02.api.letsencrypt.org/directory)"}
caddy            | {"level":"error","ts":1646176225.9244585,"logger":"tls.obtain","msg":"will retry","error":"[*.skynetfree.net] Obtain: [*.skynetfree.net] solving challenges: presenting for challenge: adding temporary record for zone skynetfree.net.: InvalidChangeBatch: InvalidChangeBatch: [Tried to create resource record set [name='_acme-challenge.skynetfree.net.', type='TXT'] but it already exists]\n\tstatus code: 400, request id: f5cbc9fe-b415-4059-8cbd-10bc3b214cb1 (order=https://acme-staging-v02.api.letsencrypt.org/acme/order/45741628/1916410268) (ca=https://acme-staging-v02.api.letsencrypt.org/directory)","attempt":3,"retrying_in":120,"elapsed":252.054161198,"max_duration":2592000}
caddy            | {"level":"info","ts":1646176346.8539615,"logger":"tls.issuance.acme.acme_client","msg":"trying to solve challenge","identifier":"*.skynetfree.net","challenge_type":"dns-01","ca":"https://acme-staging-v02.api.letsencrypt.org/directory"}
caddy            | {"level":"error","ts":1646176347.6377158,"logger":"tls.issuance.acme.acme_client","msg":"cleaning up solver","identifier":"*.skynetfree.net","challenge_type":"dns-01","error":"no memory of presenting a DNS record for skynetfree.net (probably OK if presenting failed)"}
caddy            | {"level":"error","ts":1646176347.9910886,"logger":"tls.obtain","msg":"could not get certificate from issuer","identifier":"*.skynetfree.net","issuer":"acme-v02.api.letsencrypt.org-directory","error":"[*.skynetfree.net] solving challenges: presenting for challenge: adding temporary record for zone skynetfree.net.: InvalidChangeBatch: InvalidChangeBatch: [Tried to create resource record set [name='_acme-challenge.skynetfree.net.', type='TXT'] but it already exists]\n\tstatus code: 400, request id: 519afad6-14c3-4e84-bbd6-66a50dc161c4 (order=https://acme-staging-v02.api.letsencrypt.org/acme/order/45741628/1916424408) (ca=https://acme-staging-v02.api.letsencrypt.org/directory)"}
caddy            | {"level":"error","ts":1646176347.991109,"logger":"tls.obtain","msg":"will retry","error":"[*.skynetfree.net] Obtain: [*.skynetfree.net] solving challenges: presenting for challenge: adding temporary record for zone skynetfree.net.: InvalidChangeBatch: InvalidChangeBatch: [Tried to create resource record set [name='_acme-challenge.skynetfree.net.', type='TXT'] but it already exists]\n\tstatus code: 400, request id: 519afad6-14c3-4e84-bbd6-66a50dc161c4 (order=https://acme-staging-v02.api.letsencrypt.org/acme/order/45741628/1916424408) (ca=https://acme-staging-v02.api.letsencrypt.org/directory)","attempt":4,"retrying_in":300,"elapsed":374.120811071,"max_duration":2592000}

I always end up with dangling records:

5. What I already tried:

I cleared all records in aws and restarted caddy. Didn’t help.

6. Links to relevant resources:

francislavoie · March 2, 2022, 12:04am

You should probably ask for help on the route53 plugin’s repo. Seems like an issue with that plugin in particular.

kwypchlo · March 2, 2022, 12:08am

Which one do you think it would be ?

Those have not been changed for months and it feels like it’s a new issue for me. I think it has something to do with wildcard maybe.

matt · March 2, 2022, 12:28am

Most likely the libdns one, where the actual DNS record logic goes. The caddy-dns repo is only the configuration surface.

system · March 31, 2022, 11:19pm

This topic was automatically closed after 30 days. New replies are no longer allowed.