Caddy not using dns-01 challenge

Help
1

1. The problem I’m having:

I’ve been running caddy 2.8.4 for years, and recently upgraded to caddy 2.10.0, and everything broke down.
I’m using caddy with modules:

  • cloudflare
  • caddy-docker-proxy

In a Docker Swarm, I have 5 nodes running caddy. They do not have a shared certificate storage.
My root config now has cloudflare setup as a dns challenge. I would really like to just switch everything to use dns as challenges, but none of the challenge attempts use it, just http-01 or tls-alpn-01

{
  order rate_limit before basic_auth
  email {env.ADMIN_EMAIL}
  dns cloudflare {env.CLOUDFLARE_API}
  acme_dns cloudflare {env.CLOUDFLARE_API}
}

What I want it to do

  • Use dns-01 challenge. Lets Debug shows this working for every site.
  • Work with http-01 challenge

http-01 challenges do not work. Caddy is redirecting to https, which fails with a tls: internal error. This is with cloudflare gray-cloud (non proxied)

curl -I http://app.sitehydrator.com                                                                                                                                                                      apps -> master — ! ?
HTTP/1.1 308 Permanent Redirect
Connection: close
Location: https://app.sitehydrator.com/
Server: Caddy
Date: Thu, 24 Jul 2025 16:43:48 GMT

(Let’s Debug)

ANotWorking
Error
app.sitehydrator.com has an A (IPv4) record (143.42.2.10) but a request to this address over port 80 did not succeed. Your web server must have at least one working IPv4 or IPv6 address.
Get "https://app.sitehydrator.com/.well-known/acme-challenge/letsdebug-test": remote error: tls: internal error

Trace:
@0ms: Making a request to http://app.sitehydrator.com/.well-known/acme-challenge/letsdebug-test (using initial IP 143.42.2.10)
@0ms: Dialing 143.42.2.10
@198ms: Server response: HTTP 308 Permanent Redirect
@198ms: Received redirect to https://app.sitehydrator.com/.well-known/acme-challenge/letsdebug-test
@198ms: Dialing 143.42.2.10
@397ms: Experienced error: remote error: tls: internal error

Secondly, even though the autosave.json shows cloudflare as a dns challenger, it never tries it

{
    "admin": {
        "listen": "tcp/10.200.200.24:2019"
    },
    "apps": {
        "http": {
            "servers": {
                "srv0": {
                    "listen": [
                        ":443"
                    ],
                    "logs": {
                        "skip_hosts": [
                            "app.sitehydrator.com"
                        ]
                    },
                    "routes": [
                        {
                            "handle": [
                                {
                                    "handler": "subroute",
                                    "routes": [
                                        {
                                            "handle": [
                                                {
                                                    "handler": "reverse_proxy",
                                                    "upstreams": [
                                                        {
                                                            "dial": "10.0.1.47:8055"
                                                        }
                                                    ]
                                                }
                                            ]
                                        }
                                    ]
                                }
                            ],
                            "match": [
                                {
                                    "host": [
                                        "app.sitehydrator.com"
                                    ]
                                }
                            ],
                            "terminal": true
                        }
                    ]
                }
            }
        },
        "tls": {
            "automation": {
                "policies": [
                    {
                        "issuers": [
                            {
                                "challenges": {
                                    "dns": {
                                        "provider": {
                                            "api_token": "123-token",
                                            "name": "cloudflare"
                                        },
                                        "resolvers": [
                                            "1.1.1.1"
                                        ]
                                    }
                                },
                                "module": "acme"
                            }
                        ]
                    },
                    {
                        "subjects": [
                            "app.sitehydrator.com"
                        ]
                    }
                ]
            }
        }
    }
}

What I want it to do is

2. Error messages and/or full log output:

{"level":"error","ts":1753373727.502432,"msg":"challenge failed","identifier":"app.sitehydrator.com","challenge_type":"tls-alpn-01","problem":{"type":"urn:ietf:params:acme:error:unauthorized","title":"","detail":"Incorrect validation certificate for tls-alpn-01 challenge. Requested app.sitehydrator.com from 143.42.2.10:443. Received certificate with acmeValidationV1 extension value 4aca03a77d9d8f1cefe2c6bbbe4d4cf6fedcad1475ddcc74d4312447f22447f9 but expected bd2da8338c09bf020654913ddf4e993b4f16acac9f57601dd2e631aeda0b27e9.","instance":"","subproblems":null},"stacktrace":"github.com/mholt/acmez/v3.(*Client).pollAuthorization\n\tgithub.com/mholt/acmez/v3@v3.1.2/client.go:557\ngithub.com/mholt/acmez/v3.(*Client).solveChallenges\n\tgithub.com/mholt/acmez/v3@v3.1.2/client.go:378\ngithub.com/mholt/acmez/v3.(*Client).ObtainCertificate\n\tgithub.com/mholt/acmez/v3@v3.1.2/client.go:136\ngithub.com/caddyserver/certmagic.(*ACMEIssuer).doIssue\n\tgithub.com/caddyserver/certmagic@v0.23.0/acmeissuer.go:489\ngithub.com/caddyserver/certmagic.(*ACMEIssuer).Issue\n\tgithub.com/caddyserver/certmagic@v0.23.0/acmeissuer.go:382\ngithub.com/caddyserver/caddy/v2/modules/caddytls.(*ACMEIssuer).Issue\n\tgithub.com/caddyserver/caddy/v2@v2.10.0/modules/caddytls/acmeissuer.go:288\ngithub.com/caddyserver/certmagic.(*Config).obtainCert.func2\n\tgithub.com/caddyserver/certmagic@v0.23.0/config.go:626\ngithub.com/caddyserver/certmagic.doWithRetry\n\tgithub.com/caddyserver/certmagic@v0.23.0/async.go:104\ngithub.com/caddyserver/certmagic.(*Config).obtainCert\n\tgithub.com/caddyserver/certmagic@v0.23.0/config.go:700\ngithub.com/caddyserver/certmagic.(*Config).ObtainCertAsync\n\tgithub.com/caddyserver/certmagic@v0.23.0/config.go:505\ngithub.com/caddyserver/certmagic.(*Config).manageOne.func1\n\tgithub.com/caddyserver/certmagic@v0.23.0/config.go:415\ngithub.com/caddyserver/certmagic.(*jobManager).worker\n\tgithub.com/caddyserver/certmagic@v0.23.0/async.go:73"}
{"level":"error","ts":1753374377.2532115,"msg":"validating authorization","identifier":"app.sitehydrator.com","problem":{"type":"urn:ietf:params:acme:error:tls","title":"","detail":"143.42.2.10: Fetching https://app.sitehydrator.com/.well-known/acme-challenge/pBv05sKg5ip8NpHkV2B6yCbdCJdr_o4jKH1JLnXGCmA: remote error: tls: internal error","instance":"","subproblems":null},"order":"https://acme-staging-v02.api.letsencrypt.org/acme/order/111886334/26252476574","attempt":2,"max_attempts":3,"stacktrace":"github.com/mholt/acmez/v3.(*Client).ObtainCertificate\n\tgithub.com/mholt/acmez/v3@v3.1.2/client.go:152\ngithub.com/caddyserver/certmagic.(*ACMEIssuer).doIssue\n\tgithub.com/caddyserver/certmagic@v0.23.0/acmeissuer.go:489\ngithub.com/caddyserver/certmagic.(*ACMEIssuer).Issue\n\tgithub.com/caddyserver/certmagic@v0.23.0/acmeissuer.go:382\ngithub.com/caddyserver/caddy/v2/modules/caddytls.(*ACMEIssuer).Issue\n\tgithub.com/caddyserver/caddy/v2@v2.10.0/modules/caddytls/acmeissuer.go:288\ngithub.com/caddyserver/certmagic.(*Config).obtainCert.func2\n\tgithub.com/caddyserver/certmagic@v0.23.0/config.go:626\ngithub.com/caddyserver/certmagic.doWithRetry\n\tgithub.com/caddyserver/certmagic@v0.23.0/async.go:104\ngithub.com/caddyserver/certmagic.(*Config).obtainCert\n\tgithub.com/caddyserver/certmagic@v0.23.0/config.go:700\ngithub.com/caddyserver/certmagic.(*Config).ObtainCertAsync\n\tgithub.com/caddyserver/certmagic@v0.23.0/config.go:505\ngithub.com/caddyserver/certmagic.(*Config).manageOne.func1\n\tgithub.com/caddyserver/certmagic@v0.23.0/config.go:415\ngithub.com/caddyserver/certmagic.(*jobManager).worker\n\tgithub.com/caddyserver/certmagic@v0.23.0/async.go:73"}
swarm_caddy_caddy_server.0.7lxwkg2cyk17@173.255.227.29    | {"level":"error","ts":1753374377.253697,"logger":"tls.obtain","msg":"could not get certificate from issuer","identifier":"app.sitehydrator.com","issuer":"acme-v02.api.letsencrypt.org-directory","error":"HTTP 400 urn:ietf:params:acme:error:tls - 143.42.2.10: Fetching https://app.sitehydrator.com/.well-known/acme-challenge/pBv05sKg5ip8NpHkV2B6yCbdCJdr_o4jKH1JLnXGCmA: remote error: tls: internal error"}
swarm_caddy_caddy_server.0.7lxwkg2cyk17@173.255.227.29    | {"level":"error","ts":1753374377.2541964,"logger":"tls.obtain","msg":"will retry","error":"[app.sitehydrator.com] Obtain: [app.sitehydrator.com] solving challenge: app.sitehydrator.com: [app.sitehydrator.com] authorization failed: HTTP 400 urn:ietf:params:acme:error:tls - 143.42.2.10: Fetching https://app.sitehydrator.com/.well-known/acme-challenge/pBv05sKg5ip8NpHkV2B6yCbdCJdr_o4jKH1JLnXGCmA: remote error: tls: internal error (ca=https://acme-staging-v02.api.letsencrypt.org/directory)","attempt":5,"retrying_in":600,"elapsed":619.796684747,"max_duration":2592000}
swarm_caddy_caddy_server.0.7lxwkg2cyk17@173.255.227.29    | {"level":"error","ts":1753374377.9972188,"logger":"tls","msg":"tls-alpn challenge","remote_addr":"10.0.0.2:58802","server_name":"app.sitehydrator.com","error":"no information found to solve challenge for identifier: app.sitehydrator.com"}

3. Caddy version:

v2.10.0 h1:fonubSaQKF1YANl8TXqGcn4IbIRUDdfAkpcsfI/vX5U=

4. How I installed and ran Caddy:

Through a custom docker image.

a. System environment:

Linux, Docker Swarm, Debian base

b. Command:

PASTE OVER THIS, BETWEEN THE ``` LINES.
Please use the preview pane to ensure it looks nice.

c. Service/unit/compose file:

services:
  caddy_controller:
    image: "${DOCKER_IMAGE}"
    environment:
#     ADMIN_EMAIL: "${ADMIN_EMAIL}"
      CADDY_DOCKER_MODE: controller
      CADDY_CONTROLLER_NETWORK: 10.200.200.0/24
      CADDY_INGRESS_NETWORKS: public
#     CADDY_DOCKER_POLLING_INTERVAL: 30s
#     CLOUDFLARE_API: "${CLOUDFLARE_API}"
    volumes:
      - '/var/run/docker.sock:/var/run/docker.sock'
    networks:
      - public
      - caddy_controller
    deploy:
      replicas: 1
      resources:
        limits:
          memory: 128M
      placement:
        constraints:
          - node.platform.os == linux
          - node.role == manager
  caddy_server:
    image: "${DOCKER_IMAGE}"
    environment:
      ADMIN_EMAIL: "${ADMIN_EMAIL}"
      CADDY_DOCKER_MODE: server
      CADDY_CONTROLLER_NETWORK: 10.200.200.0/24
      CADDY_INGRESS_NETWORKS: public
      CADDY_DOCKER_POLLING_INTERVAL: 30s
      CADDY_DOCKER_CADDYFILE_PATH: /config/caddy/Caddyfile
      CLOUDFLARE_API: "${CLOUDFLARE_API}"
    ports:
      - '80:80'
      - '443:443'
      - '443:443/udp'
    volumes:
      - caddy_data:/data
    networks:
      - public
      - caddy_controller
    deploy:
      mode: global
      labels:
        caddy_controlled_server:
      resources:
        limits:
          memory: 128M
      placement:
        constraints:
          - node.platform.os == linux
#         - node.role == worker

networks:
  public:
    name: public
    external: true
  caddy_controller:
    driver: overlay
    ipam:
      driver: default
      config:
        - subnet: "10.200.200.0/24"

volumes:
  caddy_data:

d. My complete Caddy config:

{
  order rate_limit before basic_auth
  email {env.ADMIN_EMAIL}
  dns cloudflare {env.CLOUDFLARE_API}
  acme_dns cloudflare {env.CLOUDFLARE_API}
}

5. Links to relevant resources:

Let’s Debug:
http-01 (doesn’t work)
dns-01 (works)

2

I don’t use Cloudflare module, so I could be wrong, but I believe dns is a per-site configuration directive while acme_dns is a global option configuration directive. You shouldn’t be using both or at least not the per-site one in the global options.

Also, is that really your complete Caddyfile?

3

I’ve removed the dns config, to try that again.

Also, is that really your complete Caddyfile?

It is, the rest is auto generated from docker labels.

    deploy:
      mode: replicated
      replicas: 1
      labels:
        caddy: "${HOSTNAME}"
        caddy.reverse_proxy: "{{ upstreams 8055 }}"

Here is the autosave.json for this host

{
    "admin": {
        "listen": "tcp/10.200.200.24:2019"
    },
    "apps": {
        "http": {
            "servers": {
                "srv0": {
                    "listen": [
                        ":443"
                    ],
                    "logs": {
                        "skip_hosts": [
                            "app.sitehydrator.com"
                        ]
                    },
                    "routes": [
                        {
                            "handle": [
                                {
                                    "handler": "subroute",
                                    "routes": [
                                        {
                                            "handle": [
                                                {
                                                    "handler": "reverse_proxy",
                                                    "upstreams": [
                                                        {
                                                            "dial": "10.0.1.47:8055"
                                                        }
                                                    ]
                                                }
                                            ]
                                        }
                                    ]
                                }
                            ],
                            "match": [
                                {
                                    "host": [
                                        "app.sitehydrator.com"
                                    ]
                                }
                            ],
                            "terminal": true
                        }
                    ]
                }
            }
        },
        "tls": {
            "automation": {
                "policies": [
                    {
                        "issuers": [
                            {
                                "challenges": {
                                    "dns": {
                                        "provider": {
                                            "api_token": "123-token",
                                            "name": "cloudflare"
                                        },
                                        "resolvers": [
                                            "1.1.1.1"
                                        ]
                                    }
                                },
                                "module": "acme"
                            }
                        ]
                    },
                    {
                        "subjects": [
                            "app.sitehydrator.com"
                        ]
                    }
                ]
            }
        }
    }
}
4

More logs:
(all of them: dpaste: 6FUNEE4QB)

swarm.0.odxitvobw5ys@*.79.141.245    | {"level":"info","ts":1753378985.0866601,"logger":"tls.obtain","msg":"acquiring lock","identifier":"app.sitehydrator.com"}
swarm.0.odxitvobw5ys@*.79.141.245    | {"level":"info","ts":1753378985.0876732,"logger":"tls.obtain","msg":"lock acquired","identifier":"app.sitehydrator.com"}
swarm.0.odxitvobw5ys@*.79.141.245    | {"level":"info","ts":1753378985.087865,"logger":"tls.obtain","msg":"obtaining certificate","identifier":"app.sitehydrator.com"}
swarm.0.odxitvobw5ys@*.79.141.245    | {"level":"info","ts":1753378985.098776,"logger":"tls","msg":"waiting on internal rate limiter","identifiers":["app.sitehydrator.com"],"ca":"https://acme-v02.api.letsencrypt.org/directory","account":"caddy@zerossl.com"}
swarm.0.odxitvobw5ys@*.79.141.245    | {"level":"info","ts":1753378985.0988023,"logger":"tls","msg":"done waiting on internal rate limiter","identifiers":["app.sitehydrator.com"],"ca":"https://acme-v02.api.letsencrypt.org/directory","account":"caddy@zerossl.com"}
swarm.0.odxitvobw5ys@*.79.141.245    | {"level":"info","ts":1753378985.5968826,"msg":"trying to solve challenge","identifier":"app.sitehydrator.com","challenge_type":"http-01","ca":"https://acme-v02.api.letsencrypt.org/directory"}
swarm.0.odxitvobw5ys@*.79.141.245    | {"level":"error","ts":1753378985.9346452,"msg":"challenge failed","identifier":"app.sitehydrator.com","challenge_type":"http-01","problem":{"type":"urn:ietf:params:acme:error:connection","title":"","detail":"143.42.2.10: Fetching http://app.sitehydrator.com/.well-known/acme-challenge/gr_xNIG86Mn4pxieMnPUYXR1OyF0zq6kUd-3IiN9FLA: Connection refused","instance":"","subproblems":null},"stacktrace":"github.com/mholt/acmez/v3.(*Client).pollAuthorization\n\tgithub.com/mholt/acmez/v3@v3.1.2/client.go:557\ngithub.com/mholt/acmez/v3.(*Client).solveChallenges\n\tgithub.com/mholt/acmez/v3@v3.1.2/client.go:378\ngithub.com/mholt/acmez/v3.(*Client).ObtainCertificate\n\tgithub.com/mholt/acmez/v3@v3.1.2/client.go:136\ngithub.com/caddyserver/certmagic.(*ACMEIssuer).doIssue\n\tgithub.com/caddyserver/certmagic@v0.23.0/acmeissuer.go:489\ngithub.com/caddyserver/certmagic.(*ACMEIssuer).Issue\n\tgithub.com/caddyserver/certmagic@v0.23.0/acmeissuer.go:382\ngithub.com/caddyserver/caddy/v2/modules/caddytls.(*ACMEIssuer).Issue\n\tgithub.com/caddyserver/caddy/v2@v2.10.0/modules/caddytls/acmeissuer.go:288\ngithub.com/caddyserver/certmagic.(*Config).obtainCert.func2\n\tgithub.com/caddyserver/certmagic@v0.23.0/config.go:626\ngithub.com/caddyserver/certmagic.doWithRetry\n\tgithub.com/caddyserver/certmagic@v0.23.0/async.go:104\ngithub.com/caddyserver/certmagic.(*Config).obtainCert\n\tgithub.com/caddyserver/certmagic@v0.23.0/config.go:700\ngithub.com/caddyserver/certmagic.(*Config).ObtainCertAsync\n\tgithub.com/caddyserver/certmagic@v0.23.0/config.go:505\ngithub.com/caddyserver/certmagic.(*Config).manageOne.func1\n\tgithub.com/caddyserver/certmagic@v0.23.0/config.go:415\ngithub.com/caddyserver/certmagic.(*jobManager).worker\n\tgithub.com/caddyserver/certmagic@v0.23.0/async.go:73"}
swarm.0.odxitvobw5ys@*.79.141.245    | {"level":"error","ts":1753378985.9363914,"msg":"validating authorization","identifier":"app.sitehydrator.com","problem":{"type":"urn:ietf:params:acme:error:connection","title":"","detail":"143.42.2.10: Fetching http://app.sitehydrator.com/.well-known/acme-challenge/gr_xNIG86Mn4pxieMnPUYXR1OyF0zq6kUd-3IiN9FLA: Connection refused","instance":"","subproblems":null},"order":"https://acme-v02.api.letsencrypt.org/acme/order/1221654447/410450454901","attempt":1,"max_attempts":3,"stacktrace":"github.com/mholt/acmez/v3.(*Client).ObtainCertificate\n\tgithub.com/mholt/acmez/v3@v3.1.2/client.go:152\ngithub.com/caddyserver/certmagic.(*ACMEIssuer).doIssue\n\tgithub.com/caddyserver/certmagic@v0.23.0/acmeissuer.go:489\ngithub.com/caddyserver/certmagic.(*ACMEIssuer).Issue\n\tgithub.com/caddyserver/certmagic@v0.23.0/acmeissuer.go:382\ngithub.com/caddyserver/caddy/v2/modules/caddytls.(*ACMEIssuer).Issue\n\tgithub.com/caddyserver/caddy/v2@v2.10.0/modules/caddytls/acmeissuer.go:288\ngithub.com/caddyserver/certmagic.(*Config).obtainCert.func2\n\tgithub.com/caddyserver/certmagic@v0.23.0/config.go:626\ngithub.com/caddyserver/certmagic.doWithRetry\n\tgithub.com/caddyserver/certmagic@v0.23.0/async.go:104\ngithub.com/caddyserver/certmagic.(*Config).obtainCert\n\tgithub.com/caddyserver/certmagic@v0.23.0/config.go:700\ngithub.com/caddyserver/certmagic.(*Config).ObtainCertAsync\n\tgithub.com/caddyserver/certmagic@v0.23.0/config.go:505\ngithub.com/caddyserver/certmagic.(*Config).manageOne.func1\n\tgithub.com/caddyserver/certmagic@v0.23.0/config.go:415\ngithub.com/caddyserver/certmagic.(*jobManager).worker\n\tgithub.com/caddyserver/certmagic@v0.23.0/async.go:73"}
swarm.0.odxitvobw5ys@*.79.141.245    | {"level":"info","ts":1753378987.055881,"msg":"trying to solve challenge","identifier":"app.sitehydrator.com","challenge_type":"tls-alpn-01","ca":"https://acme-v02.api.letsencrypt.org/directory"}
swarm.0.odxitvobw5ys@*.79.141.245    | {"level":"error","ts":1753378987.4064052,"msg":"validating authorization","identifier":"app.sitehydrator.com","problem":{"type":"urn:ietf:params:acme:error:connection","title":"","detail":"143.42.2.10: Connection refused","instance":"","subproblems":null},"order":"https://acme-v02.api.letsencrypt.org/acme/order/1221654447/410450462061","attempt":2,"max_attempts":3,"stacktrace":"github.com/mholt/acmez/v3.(*Client).ObtainCertificate\n\tgithub.com/mholt/acmez/v3@v3.1.2/client.go:152\ngithub.com/caddyserver/certmagic.(*ACMEIssuer).doIssue\n\tgithub.com/caddyserver/certmagic@v0.23.0/acmeissuer.go:489\ngithub.com/caddyserver/certmagic.(*ACMEIssuer).Issue\n\tgithub.com/caddyserver/certmagic@v0.23.0/acmeissuer.go:382\ngithub.com/caddyserver/caddy/v2/modules/caddytls.(*ACMEIssuer).Issue\n\tgithub.com/caddyserver/caddy/v2@v2.10.0/modules/caddytls/acmeissuer.go:288\ngithub.com/caddyserver/certmagic.(*Config).obtainCert.func2\n\tgithub.com/caddyserver/certmagic@v0.23.0/config.go:626\ngithub.com/caddyserver/certmagic.doWithRetry\n\tgithub.com/caddyserver/certmagic@v0.23.0/async.go:104\ngithub.com/caddyserver/certmagic.(*Config).obtainCert\n\tgithub.com/caddyserver/certmagic@v0.23.0/config.go:700\ngithub.com/caddyserver/certmagic.(*Config).ObtainCertAsync\n\tgithub.com/caddyserver/certmagic@v0.23.0/config.go:505\ngithub.com/caddyserver/certmagic.(*Config).manageOne.func1\n\tgithub.com/caddyserver/certmagic@v0.23.0/config.go:415\ngithub.com/caddyserver/certmagic.(*jobManager).worker\n\tgithub.com/caddyserver/certmagic@v0.23.0/async.go:73"}
swarm.0.odxitvobw5ys@*.79.141.245    | {"level":"error","ts":1753378987.4064827,"logger":"tls.obtain","msg":"could not get certificate from issuer","identifier":"app.sitehydrator.com","issuer":"acme-v02.api.letsencrypt.org-directory","error":"HTTP 400 urn:ietf:params:acme:error:connection - 143.42.2.10: Connection refused"}
swarm.0.odxitvobw5ys@*.79.141.245    | {"level":"error","ts":1753378987.4065187,"logger":"tls.obtain","msg":"will retry","error":"[app.sitehydrator.com] Obtain: [app.sitehydrator.com] solving challenge: app.sitehydrator.com: [app.sitehydrator.com] authorization failed: HTTP 400 urn:ietf:params:acme:error:connection - 143.42.2.10: Connection refused (ca=https://acme-v02.api.letsencrypt.org/directory)","attempt":1,"retrying_in":60,"elapsed":2.318724096,"max_duration":2592000}
swarm.0.odxitvobw5ys@*.79.141.245    | {"level":"warn","ts":1753379017.5307662,"logger":"tls","msg":"looking up info for HTTP challenge","host":"app.sitehydrator.com","remote_addr":"10.0.0.2:34203","user_agent":"Mozilla/5.0 (compatible; Let's Encrypt validation server; +https://www.letsencrypt.org)","error":"no information found to solve challenge for identifier: app.sitehydrator.com"}
swarm.0.odxitvobw5ys@*.79.141.245    | {"level":"info","ts":1753379047.40733,"logger":"tls.obtain","msg":"obtaining certificate","identifier":"app.sitehydrator.com"}
swarm.0.odxitvobw5ys@*.79.141.245    | {"level":"error","ts":1753379048.0936644,"msg":"challenge failed","identifier":"app.sitehydrator.com","challenge_type":"tls-alpn-01","problem":{"type":"urn:ietf:params:acme:error:tls","title":"","detail":"143.42.2.10: remote error: tls: internal error","instance":"","subproblems":null},"stacktrace":"github.com/mholt/acmez/v3.(*Client).pollAuthorization\n\tgithub.com/mholt/acmez/v3@v3.1.2/client.go:557\ngithub.com/mholt/acmez/v3.(*Client).solveChallenges\n\tgithub.com/mholt/acmez/v3@v3.1.2/client.go:378\ngithub.com/mholt/acmez/v3.(*Client).ObtainCertificate\n\tgithub.com/mholt/acmez/v3@v3.1.2/client.go:136\ngithub.com/caddyserver/certmagic.(*ACMEIssuer).doIssue\n\tgithub.com/caddyserver/certmagic@v0.23.0/acmeissuer.go:489\ngithub.com/caddyserver/certmagic.(*ACMEIssuer).Issue\n\tgithub.com/caddyserver/certmagic@v0.23.0/acmeissuer.go:382\ngithub.com/caddyserver/caddy/v2/modules/caddytls.(*ACMEIssuer).Issue\n\tgithub.com/caddyserver/caddy/v2@v2.10.0/modules/caddytls/acmeissuer.go:288\ngithub.com/caddyserver/certmagic.(*Config).obtainCert.func2\n\tgithub.com/caddyserver/certmagic@v0.23.0/config.go:626\ngithub.com/caddyserver/certmagic.doWithRetry\n\tgithub.com/caddyserver/certmagic@v0.23.0/async.go:104\ngithub.com/caddyserver/certmagic.(*Config).obtainCert\n\tgithub.com/caddyserver/certmagic@v0.23.0/config.go:700\ngithub.com/caddyserver/certmagic.(*Config).ObtainCertAsync\n\tgithub.com/caddyserver/certmagic@v0.23.0/config.go:505\ngithub.com/caddyserver/certmagic.(*Config).manageOne.func1\n\tgithub.com/caddyserver/certmagic@v0.23.0/config.go:415\ngithub.com/caddyserver/certmagic.(*jobManager).worker\n\tgithub.com/caddyserver/certmagic@v0.23.0/async.go:73"}
swarm.0.odxitvobw5ys@*.79.141.245    | {"level":"error","ts":1753379048.0947769,"msg":"validating authorization","identifier":"app.sitehydrator.com","problem":{"type":"urn:ietf:params:acme:error:tls","title":"","detail":"143.42.2.10: remote error: tls: internal error","instance":"","subproblems":null},"order":"https://acme-staging-v02.api.letsencrypt.org/acme/order/111982664/26253595824","attempt":1,"max_attempts":3,"stacktrace":"github.com/mholt/acmez/v3.(*Client).ObtainCertificate\n\tgithub.com/mholt/acmez/v3@v3.1.2/client.go:152\ngithub.com/caddyserver/certmagic.(*ACMEIssuer).doIssue\n\tgithub.com/caddyserver/certmagic@v0.23.0/acmeissuer.go:489\ngithub.com/caddyserver/certmagic.(*ACMEIssuer).Issue\n\tgithub.com/caddyserver/certmagic@v0.23.0/acmeissuer.go:382\ngithub.com/caddyserver/caddy/v2/modules/caddytls.(*ACMEIssuer).Issue\n\tgithub.com/caddyserver/caddy/v2@v2.10.0/modules/caddytls/acmeissuer.go:288\ngithub.com/caddyserver/certmagic.(*Config).obtainCert.func2\n\tgithub.com/caddyserver/certmagic@v0.23.0/config.go:626\ngithub.com/caddyserver/certmagic.doWithRetry\n\tgithub.com/caddyserver/certmagic@v0.23.0/async.go:104\ngithub.com/caddyserver/certmagic.(*Config).obtainCert\n\tgithub.com/caddyserver/certmagic@v0.23.0/config.go:700\ngithub.com/caddyserver/certmagic.(*Config).ObtainCertAsync\n\tgithub.com/caddyserver/certmagic@v0.23.0/config.go:505\ngithub.com/caddyserver/certmagic.(*Config).manageOne.func1\n\tgithub.com/caddyserver/certmagic@v0.23.0/config.go:415\ngithub.com/caddyserver/certmagic.(*jobManager).worker\n\tgithub.com/caddyserver/certmagic@v0.23.0/async.go:73"}
swarm.0.odxitvobw5ys@*.79.141.245    | {"level":"info","ts":1753379049.206461,"msg":"trying to solve challenge","identifier":"app.sitehydrator.com","challenge_type":"http-01","ca":"https://acme-staging-v02.api.letsencrypt.org/directory"}
swarm.0.odxitvobw5ys@*.79.141.245    | {"level":"error","ts":1753379049.869552,"msg":"validating authorization","identifier":"app.sitehydrator.com","problem":{"type":"urn:ietf:params:acme:error:tls","title":"","detail":"143.42.2.10: Fetching https://app.sitehydrator.com/.well-known/acme-challenge/kB0guoZ51v0n7aBIGjZlXe_RBiYi1gTup5ueQVBNFz4: remote error: tls: internal error","instance":"","subproblems":null},"order":"https://acme-staging-v02.api.letsencrypt.org/acme/order/111982664/26253596194","attempt":2,"max_attempts":3,"stacktrace":"github.com/mholt/acmez/v3.(*Client).ObtainCertificate\n\tgithub.com/mholt/acmez/v3@v3.1.2/client.go:152\ngithub.com/caddyserver/certmagic.(*ACMEIssuer).doIssue\n\tgithub.com/caddyserver/certmagic@v0.23.0/acmeissuer.go:489\ngithub.com/caddyserver/certmagic.(*ACMEIssuer).Issue\n\tgithub.com/caddyserver/certmagic@v0.23.0/acmeissuer.go:382\ngithub.com/caddyserver/caddy/v2/modules/caddytls.(*ACMEIssuer).Issue\n\tgithub.com/caddyserver/caddy/v2@v2.10.0/modules/caddytls/acmeissuer.go:288\ngithub.com/caddyserver/certmagic.(*Config).obtainCert.func2\n\tgithub.com/caddyserver/certmagic@v0.23.0/config.go:626\ngithub.com/caddyserver/certmagic.doWithRetry\n\tgithub.com/caddyserver/certmagic@v0.23.0/async.go:104\ngithub.com/caddyserver/certmagic.(*Config).obtainCert\n\tgithub.com/caddyserver/certmagic@v0.23.0/config.go:700\ngithub.com/caddyserver/certmagic.(*Config).ObtainCertAsync\n\tgithub.com/caddyserver/certmagic@v0.23.0/config.go:505\ngithub.com/caddyserver/certmagic.(*Config).manageOne.func1\n\tgithub.com/caddyserver/certmagic@v0.23.0/config.go:415\ngithub.com/caddyserver/certmagic.(*jobManager).worker\n\tgithub.com/caddyserver/certmagic@v0.23.0/async.go:73"}
swarm.0.odxitvobw5ys@*.79.141.245    | {"level":"error","ts":1753379049.8698134,"logger":"tls.obtain","msg":"could not get certificate from issuer","identifier":"app.sitehydrator.com","issuer":"acme-v02.api.letsencrypt.org-directory","error":"HTTP 400 urn:ietf:params:acme:error:tls - 143.42.2.10: Fetching https://app.sitehydrator.com/.well-known/acme-challenge/kB0guoZ51v0n7aBIGjZlXe_RBiYi1gTup5ueQVBNFz4: remote error: tls: internal error"}
swarm.0.odxitvobw5ys@*.79.141.245    | {"level":"error","ts":1753379049.869884,"logger":"tls.obtain","msg":"will retry","error":"[app.sitehydrator.com] Obtain: [app.sitehydrator.com] solving challenge: app.sitehydrator.com: [app.sitehydrator.com] authorization failed: HTTP 400 urn:ietf:params:acme:error:tls - 143.42.2.10: Fetching https://app.sitehydrator.com/.well-known/acme-challenge/kB0guoZ51v0n7aBIGjZlXe_RBiYi1gTup5ueQVBNFz4: remote error: tls: internal error (ca=https://acme-staging-v02.api.letsencrypt.org/directory)","attempt":2,"retrying_in":120,"elapsed":64.782089357,"max_duration":2592000}
swarm.0.odxitvobw5ys@*.79.141.245    | {"level":"warn","ts":1753379078.5453293,"logger":"tls","msg":"looking up info for HTTP challenge","host":"app.sitehydrator.com","remote_addr":"10.0.0.2:49962","user_agent":"Mozilla/5.0 (compatible; Let's Encrypt validation server; +https://www.letsencrypt.org)","error":"no information found to solve challenge for identifier: app.sitehydrator.com"}
swarm.0.odxitvobw5ys@*.79.141.245    | {"level":"warn","ts":1753379078.573912,"logger":"tls","msg":"looking up info for HTTP challenge","host":"app.sitehydrator.com","remote_addr":"10.0.0.2:58380","user_agent":"Mozilla/5.0 (compatible; Let's Encrypt validation server; +https://www.letsencrypt.org)","error":"no information found to solve challenge for identifier: app.sitehydrator.com"}
swarm.0.odxitvobw5ys@*.79.141.245    | {"level":"error","ts":1753379170.7065487,"msg":"challenge failed","identifier":"app.sitehydrator.com","challenge_type":"tls-alpn-01","problem":{"type":"urn:ietf:params:acme:error:tls","title":"","detail":"143.42.2.10: remote error: tls: internal error","instance":"","subproblems":null},"stacktrace":"github.com/mholt/acmez/v3.(*Client).pollAuthorization\n\tgithub.com/mholt/acmez/v3@v3.1.2/client.go:557\ngithub.com/mholt/acmez/v3.(*Client).solveChallenges\n\tgithub.com/mholt/acmez/v3@v3.1.2/client.go:378\ngithub.com/mholt/acmez/v3.(*Client).ObtainCertificate\n\tgithub.com/mholt/acmez/v3@v3.1.2/client.go:136\ngithub.com/caddyserver/certmagic.(*ACMEIssuer).doIssue\n\tgithub.com/caddyserver/certmagic@v0.23.0/acmeissuer.go:489\ngithub.com/caddyserver/certmagic.(*ACMEIssuer).Issue\n\tgithub.com/caddyserver/certmagic@v0.23.0/acmeissuer.go:382\ngithub.com/caddyserver/caddy/v2/modules/caddytls.(*ACMEIssuer).Issue\n\tgithub.com/caddyserver/caddy/v2@v2.10.0/modules/caddytls/acmeissuer.go:288\ngithub.com/caddyserver/certmagic.(*Config).obtainCert.func2\n\tgithub.com/caddyserver/certmagic@v0.23.0/config.go:626\ngithub.com/caddyserver/certmagic.doWithRetry\n\tgithub.com/caddyserver/certmagic@v0.23.0/async.go:104\ngithub.com/caddyserver/certmagic.(*Config).obtainCert\n\tgithub.com/caddyserver/certmagic@v0.23.0/config.go:700\ngithub.com/caddyserver/certmagic.(*Config).ObtainCertAsync\n\tgithub.com/caddyserver/certmagic@v0.23.0/config.go:505\ngithub.com/caddyserver/certmagic.(*Config).manageOne.func1\n\tgithub.com/caddyserver/certmagic@v0.23.0/config.go:415\ngithub.com/caddyserver/certmagic.(*jobManager).worker\n\tgithub.com/caddyserver/certmagic@v0.23.0/async.go:73"}
swarm.0.odxitvobw5ys@*.79.141.245    | {"level":"error","ts":1753379170.7068043,"msg":"validating authorization","identifier":"app.sitehydrator.com","problem":{"type":"urn:ietf:params:acme:error:tls","title":"","detail":"143.42.2.10: remote error: tls: internal error","instance":"","subproblems":null},"order":"https://acme-staging-v02.api.letsencrypt.org/acme/order/111982664/26253624244","attempt":1,"max_attempts":3,"stacktrace":"github.com/mholt/acmez/v3.(*Client).ObtainCertificate\n\tgithub.com/mholt/acmez/v3@v3.1.2/client.go:152\ngithub.com/caddyserver/certmagic.(*ACMEIssuer).doIssue\n\tgithub.com/caddyserver/certmagic@v0.23.0/acmeissuer.go:489\ngithub.com/caddyserver/certmagic.(*ACMEIssuer).Issue\n\tgithub.com/caddyserver/certmagic@v0.23.0/acmeissuer.go:382\ngithub.com/caddyserver/caddy/v2/modules/caddytls.(*ACMEIssuer).Issue\n\tgithub.com/caddyserver/caddy/v2@v2.10.0/modules/caddytls/acmeissuer.go:288\ngithub.com/caddyserver/certmagic.(*Config).obtainCert.func2\n\tgithub.com/caddyserver/certmagic@v0.23.0/config.go:626\ngithub.com/caddyserver/certmagic.doWithRetry\n\tgithub.com/caddyserver/certmagic@v0.23.0/async.go:104\ngithub.com/caddyserver/certmagic.(*Config).obtainCert\n\tgithub.com/caddyserver/certmagic@v0.23.0/config.go:700\ngithub.com/caddyserver/certmagic.(*Config).ObtainCertAsync\n\tgithub.com/caddyserver/certmagic@v0.23.0/config.go:505\ngithub.com/caddyserver/certmagic.(*Config).manageOne.func1\n\tgithub.com/caddyserver/certmagic@v0.23.0/config.go:415\ngithub.com/caddyserver/certmagic.(*jobManager).worker\n\tgithub.com/caddyserver/certmagic@v0.23.0/async.go:73"}
swarm.0.odxitvobw5ys@*.79.141.245    | {"level":"info","ts":1753379172.1241772,"logger":"tls","msg":"served key authentication","identifier":"app.sitehydrator.com","challenge":"http-01","remote":"10.0.0.2:40041","distributed":false}
swarm.0.odxitvobw5ys@*.79.141.245    | {"level":"info","ts":1753379172.7851264,"logger":"tls","msg":"served key authentication","identifier":"app.sitehydrator.com","challenge":"http-01","remote":"10.0.0.2:25896","distributed":false}
swarm.0.odxitvobw5ys@*.79.141.245    | {"level":"error","ts":1753379173.0855997,"msg":"validating authorization","identifier":"app.sitehydrator.com","problem":{"type":"urn:ietf:params:acme:error:tls","title":"","detail":"During secondary validation: 143.42.2.10: Fetching https://app.sitehydrator.com/.well-known/acme-challenge/YXPCzl1085D8Z0NCJyQbdeNwMPnpxCo7lh9mloYBfek: remote error: tls: internal error","instance":"","subproblems":null},"order":"https://acme-staging-v02.api.letsencrypt.org/acme/order/111982664/26253624624","attempt":2,"max_attempts":3,"stacktrace":"github.com/mholt/acmez/v3.(*Client).ObtainCertificate\n\tgithub.com/mholt/acmez/v3@v3.1.2/client.go:152\ngithub.com/caddyserver/certmagic.(*ACMEIssuer).doIssue\n\tgithub.com/caddyserver/certmagic@v0.23.0/acmeissuer.go:489\ngithub.com/caddyserver/certmagic.(*ACMEIssuer).Issue\n\tgithub.com/caddyserver/certmagic@v0.23.0/acmeissuer.go:382\ngithub.com/caddyserver/caddy/v2/modules/caddytls.(*ACMEIssuer).Issue\n\tgithub.com/caddyserver/caddy/v2@v2.10.0/modules/caddytls/acmeissuer.go:288\ngithub.com/caddyserver/certmagic.(*Config).obtainCert.func2\n\tgithub.com/caddyserver/certmagic@v0.23.0/config.go:626\ngithub.com/caddyserver/certmagic.doWithRetry\n\tgithub.com/caddyserver/certmagic@v0.23.0/async.go:104\ngithub.com/caddyserver/certmagic.(*Config).obtainCert\n\tgithub.com/caddyserver/certmagic@v0.23.0/config.go:700\ngithub.com/caddyserver/certmagic.(*Config).ObtainCertAsync\n\tgithub.com/caddyserver/certmagic@v0.23.0/config.go:505\ngithub.com/caddyserver/certmagic.(*Config).manageOne.func1\n\tgithub.com/caddyserver/certmagic@v0.23.0/config.go:415\ngithub.com/caddyserver/certmagic.(*jobManager).worker\n\tgithub.com/caddyserver/certmagic@v0.23.0/async.go:73"}
swarm.0.odxitvobw5ys@*.79.141.245    | {"level":"error","ts":1753379173.0857434,"logger":"tls.obtain","msg":"could not get certificate from issuer","identifier":"app.sitehydrator.com","issuer":"acme-v02.api.letsencrypt.org-directory","error":"HTTP 400 urn:ietf:params:acme:error:tls - During secondary validation: 143.42.2.10: Fetching https://app.sitehydrator.com/.well-known/acme-challenge/YXPCzl1085D8Z0NCJyQbdeNwMPnpxCo7lh9mloYBfek: remote error: tls: internal error"}
swarm.0.odxitvobw5ys@*.79.141.245    | {"level":"error","ts":1753379173.0857666,"logger":"tls.obtain","msg":"will retry","error":"[app.sitehydrator.com] Obtain: [app.sitehydrator.com] solving challenge: app.sitehydrator.com: [app.sitehydrator.com] authorization failed: HTTP 400 urn:ietf:params:acme:error:tls - During secondary validation: 143.42.2.10: Fetching https://app.sitehydrator.com/.well-known/acme-challenge/YXPCzl1085D8Z0NCJyQbdeNwMPnpxCo7lh9mloYBfek: remote error: tls: internal error (ca=https://acme-staging-v02.api.letsencrypt.org/directory)","attempt":3,"retrying_in":120,"elapsed":187.997972042,"max_duration":2592000}
swarm.0.odxitvobw5ys@*.79.141.245    | {"level":"error","ts":1753379203.0895376,"logger":"tls","msg":"tls-alpn challenge","remote_addr":"10.0.0.2:32475","server_name":"app.sitehydrator.com","error":"no information found to solve challenge for identifier: app.sitehydrator.com"}
swarm.0.odxitvobw5ys@*.79.141.245    | {"level":"info","ts":1753379293.0860026,"logger":"tls.obtain","msg":"obtaining certificate","identifier":"app.sitehydrator.com"}
swarm.0.odxitvobw5ys@*.79.141.245    | {"level":"error","ts":1753379293.9197788,"msg":"validating authorization","identifier":"app.sitehydrator.com","problem":{"type":"urn:ietf:params:acme:error:tls","title":"","detail":"143.42.2.10: remote error: tls: internal error","instance":"","subproblems":null},"order":"https://acme-staging-v02.api.letsencrypt.org/acme/order/111982664/26253650584","attempt":1,"max_attempts":3,"stacktrace":"github.com/mholt/acmez/v3.(*Client).ObtainCertificate\n\tgithub.com/mholt/acmez/v3@v3.1.2/client.go:152\ngithub.com/caddyserver/certmagic.(*ACMEIssuer).doIssue\n\tgithub.com/caddyserver/certmagic@v0.23.0/acmeissuer.go:489\ngithub.com/caddyserver/certmagic.(*ACMEIssuer).Issue\n\tgithub.com/caddyserver/certmagic@v0.23.0/acmeissuer.go:382\ngithub.com/caddyserver/caddy/v2/modules/caddytls.(*ACMEIssuer).Issue\n\tgithub.com/caddyserver/caddy/v2@v2.10.0/modules/caddytls/acmeissuer.go:288\ngithub.com/caddyserver/certmagic.(*Config).obtainCert.func2\n\tgithub.com/caddyserver/certmagic@v0.23.0/config.go:626\ngithub.com/caddyserver/certmagic.doWithRetry\n\tgithub.com/caddyserver/certmagic@v0.23.0/async.go:104\ngithub.com/caddyserver/certmagic.(*Config).obtainCert\n\tgithub.com/caddyserver/certmagic@v0.23.0/config.go:700\ngithub.com/caddyserver/certmagic.(*Config).ObtainCertAsync\n\tgithub.com/caddyserver/certmagic@v0.23.0/config.go:505\ngithub.com/caddyserver/certmagic.(*Config).manageOne.func1\n\tgithub.com/caddyserver/certmagic@v0.23.0/config.go:415\ngithub.com/caddyserver/certmagic.(*jobManager).worker\n\tgithub.com/caddyserver/certmagic@v0.23.0/async.go:73"}
swarm.0.odxitvobw5ys@*.79.141.245    | {"level":"info","ts":1753379295.757492,"logger":"tls","msg":"served key authentication","identifier":"app.sitehydrator.com","challenge":"http-01","remote":"10.0.0.2:39003","distributed":false}
swarm.0.odxitvobw5ys@*.79.141.245    | {"level":"error","ts":1753379297.2069538,"msg":"validating authorization","identifier":"app.sitehydrator.com","problem":{"type":"urn:ietf:params:acme:error:tls","title":"","detail":"During secondary validation: 143.42.2.10: Fetching https://app.sitehydrator.com/.well-known/acme-challenge/l2vubmF1BcFqzVybkw943beJqKUobAr1KSp8AAvf3dc: remote error: tls: internal error","instance":"","subproblems":null},"order":"https://acme-staging-v02.api.letsencrypt.org/acme/order/111982664/26253650964","attempt":2,"max_attempts":3,"stacktrace":"github.com/mholt/acmez/v3.(*Client).ObtainCertificate\n\tgithub.com/mholt/acmez/v3@v3.1.2/client.go:152\ngithub.com/caddyserver/certmagic.(*ACMEIssuer).doIssue\n\tgithub.com/caddyserver/certmagic@v0.23.0/acmeissuer.go:489\ngithub.com/caddyserver/certmagic.(*ACMEIssuer).Issue\n\tgithub.com/caddyserver/certmagic@v0.23.0/acmeissuer.go:382\ngithub.com/caddyserver/caddy/v2/modules/caddytls.(*ACMEIssuer).Issue\n\tgithub.com/caddyserver/caddy/v2@v2.10.0/modules/caddytls/acmeissuer.go:288\ngithub.com/caddyserver/certmagic.(*Config).obtainCert.func2\n\tgithub.com/caddyserver/certmagic@v0.23.0/config.go:626\ngithub.com/caddyserver/certmagic.doWithRetry\n\tgithub.com/caddyserver/certmagic@v0.23.0/async.go:104\ngithub.com/caddyserver/certmagic.(*Config).obtainCert\n\tgithub.com/caddyserver/certmagic@v0.23.0/config.go:700\ngithub.com/caddyserver/certmagic.(*Config).ObtainCertAsync\n\tgithub.com/caddyserver/certmagic@v0.23.0/config.go:505\ngithub.com/caddyserver/certmagic.(*Config).manageOne.func1\n\tgithub.com/caddyserver/certmagic@v0.23.0/config.go:415\ngithub.com/caddyserver/certmagic.(*jobManager).worker\n\tgithub.com/caddyserver/certmagic@v0.23.0/async.go:73"}
swarm.0.odxitvobw5ys@*.79.141.245    | {"level":"error","ts":1753379297.207,"logger":"tls.obtain","msg":"could not get certificate from issuer","identifier":"app.sitehydrator.com","issuer":"acme-v02.api.letsencrypt.org-directory","error":"HTTP 400 urn:ietf:params:acme:error:tls - During secondary validation: 143.42.2.10: Fetching https://app.sitehydrator.com/.well-known/acme-challenge/l2vubmF1BcFqzVybkw943beJqKUobAr1KSp8AAvf3dc: remote error: tls: internal error"}
swarm.0.odxitvobw5ys@*.79.141.245    | {"level":"error","ts":1753379297.207041,"logger":"tls.obtain","msg":"will retry","error":"[app.sitehydrator.com] Obtain: [app.sitehydrator.com] solving challenge: app.sitehydrator.com: [app.sitehydrator.com] authorization failed: HTTP 400 urn:ietf:params:acme:error:tls - During secondary validation: 143.42.2.10: Fetching https://app.sitehydrator.com/.well-known/acme-challenge/l2vubmF1BcFqzVybkw943beJqKUobAr1KSp8AAvf3dc: remote error: tls: internal error (ca=https://acme-staging-v02.api.letsencrypt.org/directory)","attempt":4,"retrying_in":300,"elapsed":312.11924636,"max_duration":2592000}
swarm.0.odxitvobw5ys@*.79.141.245    | {"level":"error","ts":1753379325.4275837,"logger":"tls","msg":"tls-alpn challenge","remote_addr":"10.0.0.2:36023","server_name":"app.sitehydrator.com","error":"no information found to solve challenge for identifier: app.sitehydrator.com"}
swarm.0.bhtwxqxdu3en@*.228.42.26    | {"level":"info","ts":1753379015.0719967,"logger":"tls.obtain","msg":"acquiring lock","identifier":"app.sitehydrator.com"}
swarm.0.bhtwxqxdu3en@*.228.42.26    | {"level":"info","ts":1753379015.0838852,"logger":"tls.obtain","msg":"lock acquired","identifier":"app.sitehydrator.com"}
swarm.0.bhtwxqxdu3en@*.228.42.26    | {"level":"info","ts":1753379015.08416,"logger":"tls.obtain","msg":"obtaining certificate","identifier":"app.sitehydrator.com"}
swarm.0.bhtwxqxdu3en@*.228.42.26    | {"level":"info","ts":1753379015.0874996,"logger":"tls","msg":"waiting on internal rate limiter","identifiers":["app.sitehydrator.com"],"ca":"https://acme-v02.api.letsencrypt.org/directory","account":"caddy@zerossl.com"}
swarm.0.bhtwxqxdu3en@*.228.42.26    | {"level":"warn","ts":1753379015.8466866,"logger":"tls","msg":"looking up info for HTTP challenge","host":"app.sitehydrator.com","remote_addr":"10.0.0.2:43427","user_agent":"Mozilla/5.0 (compatible; Let's Encrypt validation server; +https://www.letsencrypt.org)","error":"no information found to solve challenge for identifier: app.sitehydrator.com"}
swarm.0.bhtwxqxdu3en@*.228.42.26    | {"level":"error","ts":1753379015.93504,"logger":"tls","msg":"tls-alpn challenge","remote_addr":"10.0.0.2:49123","server_name":"app.sitehydrator.com","error":"no information found to solve challenge for identifier: app.sitehydrator.com"}
swarm.0.n3kbjmj3w0xv@*.107.134.41    | {"level":"info","ts":1753379015.05347,"logger":"tls.obtain","msg":"acquiring lock","identifier":"app.sitehydrator.com"}
swarm.0.n3kbjmj3w0xv@*.107.134.41    | {"level":"info","ts":1753379015.072526,"logger":"tls.obtain","msg":"lock acquired","identifier":"app.sitehydrator.com"}
swarm.0.n3kbjmj3w0xv@*.107.134.41    | {"level":"info","ts":1753379015.072903,"logger":"tls.obtain","msg":"obtaining certificate","identifier":"app.sitehydrator.com"}
swarm.0.n3kbjmj3w0xv@*.107.134.41    | {"level":"info","ts":1753379015.073691,"logger":"tls","msg":"waiting on internal rate limiter","identifiers":["app.sitehydrator.com"],"ca":"https://acme-v02.api.letsencrypt.org/directory","account":"caddy@zerossl.com"}
swarm.0.n3kbjmj3w0xv@*.107.134.41    | {"level":"info","ts":1753379015.0749648,"logger":"tls","msg":"done waiting on internal rate limiter","identifiers":["app.sitehydrator.com"],"ca":"https://acme-v02.api.letsencrypt.org/directory","account":"caddy@zerossl.com"}
swarm.0.n3kbjmj3w0xv@*.107.134.41    | {"level":"info","ts":1753379015.5318663,"msg":"trying to solve challenge","identifier":"app.sitehydrator.com","challenge_type":"http-01","ca":"https://acme-v02.api.letsencrypt.org/directory"}
swarm.0.n3kbjmj3w0xv@*.107.134.41    | {"level":"error","ts":1753379016.199817,"msg":"challenge failed","identifier":"app.sitehydrator.com","challenge_type":"http-01","problem":{"type":"urn:ietf:params:acme:error:tls","title":"","detail":"143.42.2.10: Fetching https://app.sitehydrator.com/.well-known/acme-challenge/kTSpJrMKzZoi49o44cMzBiz4LYpBBzjacUsbdfz9B2I: remote error: tls: internal error","instance":"","subproblems":null},"stacktrace":"github.com/mholt/acmez/v3.(*Client).pollAuthorization\n\tgithub.com/mholt/acmez/v3@v3.1.2/client.go:557\ngithub.com/mholt/acmez/v3.(*Client).solveChallenges\n\tgithub.com/mholt/acmez/v3@v3.1.2/client.go:378\ngithub.com/mholt/acmez/v3.(*Client).ObtainCertificate\n\tgithub.com/mholt/acmez/v3@v3.1.2/client.go:136\ngithub.com/caddyserver/certmagic.(*ACMEIssuer).doIssue\n\tgithub.com/caddyserver/certmagic@v0.23.0/acmeissuer.go:489\ngithub.com/caddyserver/certmagic.(*ACMEIssuer).Issue\n\tgithub.com/caddyserver/certmagic@v0.23.0/acmeissuer.go:382\ngithub.com/caddyserver/caddy/v2/modules/caddytls.(*ACMEIssuer).Issue\n\tgithub.com/caddyserver/caddy/v2@v2.10.0/modules/caddytls/acmeissuer.go:288\ngithub.com/caddyserver/certmagic.(*Config).obtainCert.func2\n\tgithub.com/caddyserver/certmagic@v0.23.0/config.go:626\ngithub.com/caddyserver/certmagic.doWithRetry\n\tgithub.com/caddyserver/certmagic@v0.23.0/async.go:104\ngithub.com/caddyserver/certmagic.(*Config).obtainCert\n\tgithub.com/caddyserver/certmagic@v0.23.0/config.go:700\ngithub.com/caddyserver/certmagic.(*Config).ObtainCertAsync\n\tgithub.com/caddyserver/certmagic@v0.23.0/config.go:505\ngithub.com/caddyserver/certmagic.(*Config).manageOne.func1\n\tgithub.com/caddyserver/certmagic@v0.23.0/config.go:415\ngithub.com/caddyserver/certmagic.(*jobManager).worker\n\tgithub.com/caddyserver/certmagic@v0.23.0/async.go:73"}
swarm.0.n3kbjmj3w0xv@*.107.134.41    | {"level":"error","ts":1753379016.1999302,"msg":"validating authorization","identifier":"app.sitehydrator.com","problem":{"type":"urn:ietf:params:acme:error:tls","title":"","detail":"143.42.2.10: Fetching https://app.sitehydrator.com/.well-known/acme-challenge/kTSpJrMKzZoi49o44cMzBiz4LYpBBzjacUsbdfz9B2I: remote error: tls: internal error","instance":"","subproblems":null},"order":"https://acme-v02.api.letsencrypt.org/acme/order/1676612997/410450582911","attempt":1,"max_attempts":3,"stacktrace":"github.com/mholt/acmez/v3.(*Client).ObtainCertificate\n\tgithub.com/mholt/acmez/v3@v3.1.2/client.go:152\ngithub.com/caddyserver/certmagic.(*ACMEIssuer).doIssue\n\tgithub.com/caddyserver/certmagic@v0.23.0/acmeissuer.go:489\ngithub.com/caddyserver/certmagic.(*ACMEIssuer).Issue\n\tgithub.com/caddyserver/certmagic@v0.23.0/acmeissuer.go:382\ngithub.com/caddyserver/caddy/v2/modules/caddytls.(*ACMEIssuer).Issue\n\tgithub.com/caddyserver/caddy/v2@v2.10.0/modules/caddytls/acmeissuer.go:288\ngithub.com/caddyserver/certmagic.(*Config).obtainCert.func2\n\tgithub.com/caddyserver/certmagic@v0.23.0/config.go:626\ngithub.com/caddyserver/certmagic.doWithRetry\n\tgithub.com/caddyserver/certmagic@v0.23.0/async.go:104\ngithub.com/caddyserver/certmagic.(*Config).obtainCert\n\tgithub.com/caddyserver/certmagic@v0.23.0/config.go:700\ngithub.com/caddyserver/certmagic.(*Config).ObtainCertAsync\n\tgithub.com/caddyserver/certmagic@v0.23.0/config.go:505\ngithub.com/caddyserver/certmagic.(*Config).manageOne.func1\n\tgithub.com/caddyserver/certmagic@v0.23.0/config.go:415\ngithub.com/caddyserver/certmagic.(*jobManager).worker\n\tgithub.com/caddyserver/certmagic@v0.23.0/async.go:73"}
swarm.0.n3kbjmj3w0xv@*.107.134.41    | {"level":"error","ts":1753379017.2557652,"logger":"tls.obtain","msg":"could not get certificate from issuer","identifier":"app.sitehydrator.com","issuer":"acme-v02.api.letsencrypt.org-directory","error":"HTTP 429 urn:ietf:params:acme:error:rateLimited - too many failed authorizations (5) for \"app.sitehydrator.com\" in the last 1h0m0s, retry after 2025-07-24 17:44:44 UTC: see https://letsencrypt.org/docs/rate-limits/#authorization-failures-per-hostname-per-account"}
swarm.0.n3kbjmj3w0xv@*.107.134.41    | {"level":"error","ts":1753379017.2558448,"logger":"tls.obtain","msg":"will retry","error":"[app.sitehydrator.com] Obtain: [app.sitehydrator.com] creating new order: attempt 1: https://acme-v02.api.letsencrypt.org/acme/new-order: HTTP 429 urn:ietf:params:acme:error:rateLimited - too many failed authorizations (5) for \"app.sitehydrator.com\" in the last 1h0m0s, retry after 2025-07-24 17:44:44 UTC: see https://letsencrypt.org/docs/rate-limits/#authorization-failures-per-hostname-per-account (ca=https://acme-v02.api.letsencrypt.org/directory)","attempt":1,"retrying_in":60,"elapsed":2.183178921,"max_duration":2592000}
swarm.0.n3kbjmj3w0xv@*.107.134.41    | {"level":"info","ts":1753379080.8106115,"msg":"got renewal info","names":["app.sitehydrator.com"],"window_start":1758483011,"window_end":1758638460,"selected_time":1758578009,"recheck_after":1753400680.8106039,"explanation_url":""}
swarm.0.n3kbjmj3w0xv@*.107.134.41    | {"level":"info","ts":1753379080.918318,"msg":"got renewal info","names":["app.sitehydrator.com"],"window_start":1758483011,"window_end":1758638460,"selected_time":1758511825,"recheck_after":1753400680.9183109,"explanation_url":""}
swarm.0.n3kbjmj3w0xv@*.107.134.41    | {"level":"info","ts":1753379080.919997,"logger":"tls","msg":"waiting on internal rate limiter","identifiers":["app.sitehydrator.com"],"ca":"https://acme-v02.api.letsencrypt.org/directory","account":"caddy@zerossl.com"}
swarm.0.n3kbjmj3w0xv@*.107.134.41    | {"level":"info","ts":1753379080.9205968,"logger":"tls","msg":"done waiting on internal rate limiter","identifiers":["app.sitehydrator.com"],"ca":"https://acme-v02.api.letsencrypt.org/directory","account":"caddy@zerossl.com"}
swarm.0.n3kbjmj3w0xv@*.107.134.41    | {"level":"error","ts":1753379081.0238419,"logger":"tls.obtain","msg":"could not get certificate from issuer","identifier":"app.sitehydrator.com","issuer":"acme-v02.api.letsencrypt.org-directory","error":"HTTP 429 urn:ietf:params:acme:error:rateLimited - too many failed authorizations (5) for \"app.sitehydrator.com\" in the last 1h0m0s, retry after 2025-07-24 17:44:42 UTC: see https://letsencrypt.org/docs/rate-limits/#authorization-failures-per-hostname-per-account"}
swarm.0.n3kbjmj3w0xv@*.107.134.41    | {"level":"error","ts":1753379081.0238903,"logger":"tls.obtain","msg":"will retry","error":"[app.sitehydrator.com] Obtain: [app.sitehydrator.com] creating new order: attempt 1: https://acme-v02.api.letsencrypt.org/acme/new-order: HTTP 429 urn:ietf:params:acme:error:rateLimited - too many failed authorizations (5) for \"app.sitehydrator.com\" in the last 1h0m0s, retry after 2025-07-24 17:44:42 UTC: see https://letsencrypt.org/docs/rate-limits/#authorization-failures-per-hostname-per-account (ca=https://acme-v02.api.letsencrypt.org/directory)","attempt":2,"retrying_in":120,"elapsed":65.95122433,"max_duration":2592000}
swarm.0.n3kbjmj3w0xv@*.107.134.41    | {"level":"info","ts":1753379201.024272,"logger":"tls.obtain","msg":"obtaining certificate","identifier":"app.sitehydrator.com"}
swarm.0.n3kbjmj3w0xv@*.107.134.41    | {"level":"info","ts":1753379201.1906025,"msg":"authorization finalized","identifier":"app.sitehydrator.com","authz_status":"valid"}
swarm.0.n3kbjmj3w0xv@*.107.134.41    | {"level":"warn","ts":1753379201.530075,"logger":"tls","msg":"looking up info for HTTP challenge","host":"app.sitehydrator.com","remote_addr":"10.0.0.2:49433","user_agent":"Mozilla/5.0 (compatible; Let's Encrypt validation server; +https://www.letsencrypt.org)","error":"no information found to solve challenge for identifier: app.sitehydrator.com"}
swarm.0.n3kbjmj3w0xv@*.107.134.41    | {"level":"info","ts":1753379204.4097471,"msg":"got renewal info","names":["app.sitehydrator.com"],"window_start":1758483135,"window_end":1758638585,"selected_time":1758569939,"recheck_after":1753400804.4097385,"explanation_url":""}
swarm.0.n3kbjmj3w0xv@*.107.134.41    | {"level":"info","ts":1753379204.5175812,"msg":"got renewal info","names":["app.sitehydrator.com"],"window_start":1758483135,"window_end":1758638585,"selected_time":1758561122,"recheck_after":1753400804.5175724,"explanation_url":""}
swarm.0.n3kbjmj3w0xv@*.107.134.41    | {"level":"info","ts":1753379204.5196683,"logger":"tls","msg":"waiting on internal rate limiter","identifiers":["app.sitehydrator.com"],"ca":"https://acme-v02.api.letsencrypt.org/directory","account":"caddy@zerossl.com"}
swarm.0.n3kbjmj3w0xv@*.107.134.41    | {"level":"info","ts":1753379204.5197709,"logger":"tls","msg":"done waiting on internal rate limiter","identifiers":["app.sitehydrator.com"],"ca":"https://acme-v02.api.letsencrypt.org/directory","account":"caddy@zerossl.com"}
swarm.0.n3kbjmj3w0xv@*.107.134.41    | {"level":"info","ts":1753379204.6989589,"msg":"trying to solve challenge","identifier":"app.sitehydrator.com","challenge_type":"http-01","ca":"https://acme-v02.api.letsencrypt.org/directory"}
swarm.0.n3kbjmj3w0xv@*.107.134.41    | {"level":"error","ts":1753379205.6330407,"msg":"challenge failed","identifier":"app.sitehydrator.com","challenge_type":"http-01","problem":{"type":"urn:ietf:params:acme:error:tls","title":"","detail":"143.42.2.10: Fetching https://app.sitehydrator.com/.well-known/acme-challenge/cSb9hk8ldPzxKEjsOCKtZXXuwLgPSxaH-9cmjRq4OB4: remote error: tls: internal error","instance":"","subproblems":null},"stacktrace":"github.com/mholt/acmez/v3.(*Client).pollAuthorization\n\tgithub.com/mholt/acmez/v3@v3.1.2/client.go:557\ngithub.com/mholt/acmez/v3.(*Client).solveChallenges\n\tgithub.com/mholt/acmez/v3@v3.1.2/client.go:378\ngithub.com/mholt/acmez/v3.(*Client).ObtainCertificate\n\tgithub.com/mholt/acmez/v3@v3.1.2/client.go:136\ngithub.com/caddyserver/certmagic.(*ACMEIssuer).doIssue\n\tgithub.com/caddyserver/certmagic@v0.23.0/acmeissuer.go:489\ngithub.com/caddyserver/certmagic.(*ACMEIssuer).Issue\n\tgithub.com/caddyserver/certmagic@v0.23.0/acmeissuer.go:410\ngithub.com/caddyserver/caddy/v2/modules/caddytls.(*ACMEIssuer).Issue\n\tgithub.com/caddyserver/caddy/v2@v2.10.0/modules/caddytls/acmeissuer.go:288\ngithub.com/caddyserver/certmagic.(*Config).obtainCert.func2\n\tgithub.com/caddyserver/certmagic@v0.23.0/config.go:626\ngithub.com/caddyserver/certmagic.doWithRetry\n\tgithub.com/caddyserver/certmagic@v0.23.0/async.go:104\ngithub.com/caddyserver/certmagic.(*Config).obtainCert\n\tgithub.com/caddyserver/certmagic@v0.23.0/config.go:700\ngithub.com/caddyserver/certmagic.(*Config).ObtainCertAsync\n\tgithub.com/caddyserver/certmagic@v0.23.0/config.go:505\ngithub.com/caddyserver/certmagic.(*Config).manageOne.func1\n\tgithub.com/caddyserver/certmagic@v0.23.0/config.go:415\ngithub.com/caddyserver/certmagic.(*jobManager).worker\n\tgithub.com/caddyserver/certmagic@v0.23.0/async.go:73"}
swarm.0.n3kbjmj3w0xv@*.107.134.41    | {"level":"error","ts":1753379205.633191,"msg":"validating authorization","identifier":"app.sitehydrator.com","problem":{"type":"urn:ietf:params:acme:error:tls","title":"","detail":"143.42.2.10: Fetching https://app.sitehydrator.com/.well-known/acme-challenge/cSb9hk8ldPzxKEjsOCKtZXXuwLgPSxaH-9cmjRq4OB4: remote error: tls: internal error","instance":"","subproblems":null},"order":"https://acme-v02.api.letsencrypt.org/acme/order/1676612997/410451395441","attempt":1,"max_attempts":3,"stacktrace":"github.com/mholt/acmez/v3.(*Client).ObtainCertificate\n\tgithub.com/mholt/acmez/v3@v3.1.2/client.go:152\ngithub.com/caddyserver/certmagic.(*ACMEIssuer).doIssue\n\tgithub.com/caddyserver/certmagic@v0.23.0/acmeissuer.go:489\ngithub.com/caddyserver/certmagic.(*ACMEIssuer).Issue\n\tgithub.com/caddyserver/certmagic@v0.23.0/acmeissuer.go:410\ngithub.com/caddyserver/caddy/v2/modules/caddytls.(*ACMEIssuer).Issue\n\tgithub.com/caddyserver/caddy/v2@v2.10.0/modules/caddytls/acmeissuer.go:288\ngithub.com/caddyserver/certmagic.(*Config).obtainCert.func2\n\tgithub.com/caddyserver/certmagic@v0.23.0/config.go:626\ngithub.com/caddyserver/certmagic.doWithRetry\n\tgithub.com/caddyserver/certmagic@v0.23.0/async.go:104\ngithub.com/caddyserver/certmagic.(*Config).obtainCert\n\tgithub.com/caddyserver/certmagic@v0.23.0/config.go:700\ngithub.com/caddyserver/certmagic.(*Config).ObtainCertAsync\n\tgithub.com/caddyserver/certmagic@v0.23.0/config.go:505\ngithub.com/caddyserver/certmagic.(*Config).manageOne.func1\n\tgithub.com/caddyserver/certmagic@v0.23.0/config.go:415\ngithub.com/caddyserver/certmagic.(*jobManager).worker\n\tgithub.com/caddyserver/certmagic@v0.23.0/async.go:73"}
swarm.0.n3kbjmj3w0xv@*.107.134.41    | {"level":"warn","ts":1753379206.4423306,"logger":"tls","msg":"looking up info for HTTP challenge","host":"app.sitehydrator.com","remote_addr":"10.0.0.2:51705","user_agent":"Mozilla/5.0 (compatible; Let's Encrypt validation server; +https://www.letsencrypt.org)","error":"no information found to solve challenge for identifier: app.sitehydrator.com"}
swarm.0.n3kbjmj3w0xv@*.107.134.41    | {"level":"error","ts":1753379206.689995,"logger":"tls.obtain","msg":"could not get certificate from issuer","identifier":"app.sitehydrator.com","issuer":"acme-v02.api.letsencrypt.org-directory","error":"HTTP 429 urn:ietf:params:acme:error:rateLimited - too many failed authorizations (5) for \"app.sitehydrator.com\" in the last 1h0m0s, retry after 2025-07-24 17:56:43 UTC: see https://letsencrypt.org/docs/rate-limits/#authorization-failures-per-hostname-per-account"}
swarm.0.n3kbjmj3w0xv@*.107.134.41    | {"level":"error","ts":1753379206.6900723,"logger":"tls.obtain","msg":"will retry","error":"[app.sitehydrator.com] Obtain: [app.sitehydrator.com] creating new order: attempt 1: https://acme-v02.api.letsencrypt.org/acme/new-order: HTTP 429 urn:ietf:params:acme:error:rateLimited - too many failed authorizations (5) for \"app.sitehydrator.com\" in the last 1h0m0s, retry after 2025-07-24 17:56:43 UTC: see https://letsencrypt.org/docs/rate-limits/#authorization-failures-per-hostname-per-account (ca=https://acme-v02.api.letsencrypt.org/directory)","attempt":3,"retrying_in":120,"elapsed":191.617406443,"max_duration":2592000}
swarm.0.n3kbjmj3w0xv@*.107.134.41    | {"level":"error","ts":1753379208.237425,"logger":"tls","msg":"tls-alpn challenge","remote_addr":"10.0.0.2:33265","server_name":"app.sitehydrator.com","error":"no information found to solve challenge for identifier: app.sitehydrator.com"}
swarm.0.n3kbjmj3w0xv@*.107.134.41    | {"level":"warn","ts":1753379295.9987612,"logger":"tls","msg":"looking up info for HTTP challenge","host":"app.sitehydrator.com","remote_addr":"10.0.0.2:64356","user_agent":"Mozilla/5.0 (compatible; Let's Encrypt validation server; +https://www.letsencrypt.org)","error":"no information found to solve challenge for identifier: app.sitehydrator.com"}
swarm.0.bhtwxqxdu3en@*.228.42.26    | {"level":"info","ts":1753379025.0631964,"logger":"tls","msg":"done waiting on internal rate limiter","identifiers":["app.sitehydrator.com"],"ca":"https://acme-v02.api.letsencrypt.org/directory","account":"caddy@zerossl.com"}
swarm.0.bhtwxqxdu3en@*.228.42.26    | {"level":"error","ts":1753379025.5955942,"logger":"tls.obtain","msg":"could not get certificate from issuer","identifier":"app.sitehydrator.com","issuer":"acme-v02.api.letsencrypt.org-directory","error":"HTTP 404 urn:ietf:params:acme:error:malformed - No such authorization"}
swarm.0.bhtwxqxdu3en@*.228.42.26    | {"level":"error","ts":1753379025.596167,"logger":"tls.obtain","msg":"will retry","error":"[app.sitehydrator.com] Obtain: [app.sitehydrator.com] solving challenge: getting authorization at https://acme-v02.api.letsencrypt.org/acme/authz/1560261627/558273559021: attempt 1: https://acme-v02.api.letsencrypt.org/acme/authz/1560261627/558273559021: HTTP 404 urn:ietf:params:acme:error:malformed - No such authorization (ca=https://acme-v02.api.letsencrypt.org/directory)","attempt":1,"retrying_in":60,"elapsed":10.512045328,"max_duration":2592000}
swarm.0.bhtwxqxdu3en@*.228.42.26    | {"level":"warn","ts":1753379078.6282747,"logger":"tls","msg":"looking up info for HTTP challenge","host":"app.sitehydrator.com","remote_addr":"10.0.0.2:47622","user_agent":"Mozilla/5.0 (compatible; Let's Encrypt validation server; +https://www.letsencrypt.org)","error":"no information found to solve challenge for identifier: app.sitehydrator.com"}
swarm.0.bhtwxqxdu3en@*.228.42.26    | {"level":"warn","ts":1753379078.856043,"logger":"tls","msg":"looking up info for HTTP challenge","host":"app.sitehydrator.com","remote_addr":"10.0.0.2:48896","user_agent":"Mozilla/5.0 (compatible; Let's Encrypt validation server; +https://www.letsencrypt.org)","error":"no information found to solve challenge for identifier: app.sitehydrator.com"}
swarm.0.bhtwxqxdu3en@*.228.42.26    | {"level":"error","ts":1753379086.1277406,"msg":"challenge failed","identifier":"app.sitehydrator.com","challenge_type":"tls-alpn-01","problem":{"type":"urn:ietf:params:acme:error:tls","title":"","detail":"143.42.2.10: remote error: tls: internal error","instance":"","subproblems":null},"stacktrace":"github.com/mholt/acmez/v3.(*Client).pollAuthorization\n\tgithub.com/mholt/acmez/v3@v3.1.2/client.go:557\ngithub.com/mholt/acmez/v3.(*Client).solveChallenges\n\tgithub.com/mholt/acmez/v3@v3.1.2/client.go:378\ngithub.com/mholt/acmez/v3.(*Client).ObtainCertificate\n\tgithub.com/mholt/acmez/v3@v3.1.2/client.go:136\ngithub.com/caddyserver/certmagic.(*ACMEIssuer).doIssue\n\tgithub.com/caddyserver/certmagic@v0.23.0/acmeissuer.go:489\ngithub.com/caddyserver/certmagic.(*ACMEIssuer).Issue\n\tgithub.com/caddyserver/certmagic@v0.23.0/acmeissuer.go:382\ngithub.com/caddyserver/caddy/v2/modules/caddytls.(*ACMEIssuer).Issue\n\tgithub.com/caddyserver/caddy/v2@v2.10.0/modules/caddytls/acmeissuer.go:288\ngithub.com/caddyserver/certmagic.(*Config).obtainCert.func2\n\tgithub.com/caddyserver/certmagic@v0.23.0/config.go:626\ngithub.com/caddyserver/certmagic.doWithRetry\n\tgithub.com/caddyserver/certmagic@v0.23.0/async.go:104\ngithub.com/caddyserver/certmagic.(*Config).obtainCert\n\tgithub.com/caddyserver/certmagic@v0.23.0/config.go:700\ngithub.com/caddyserver/certmagic.(*Config).ObtainCertAsync\n\tgithub.com/caddyserver/certmagic@v0.23.0/config.go:505\ngithub.com/caddyserver/certmagic.(*Config).manageOne.func1\n\tgithub.com/caddyserver/certmagic@v0.23.0/config.go:415\ngithub.com/caddyserver/certmagic.(*jobManager).worker\n\tgithub.com/caddyserver/certmagic@v0.23.0/async.go:73"}
swarm.0.bhtwxqxdu3en@*.228.42.26    | {"level":"error","ts":1753379086.1279304,"msg":"validating authorization","identifier":"app.sitehydrator.com","problem":{"type":"urn:ietf:params:acme:error:tls","title":"","detail":"143.42.2.10: remote error: tls: internal error","instance":"","subproblems":null},"order":"https://acme-staging-v02.api.letsencrypt.org/acme/order/135604333/26253603614","attempt":1,"max_attempts":3,"stacktrace":"github.com/mholt/acmez/v3.(*Client).ObtainCertificate\n\tgithub.com/mholt/acmez/v3@v3.1.2/client.go:152\ngithub.com/caddyserver/certmagic.(*ACMEIssuer).doIssue\n\tgithub.com/caddyserver/certmagic@v0.23.0/acmeissuer.go:489\ngithub.com/caddyserver/certmagic.(*ACMEIssuer).Issue\n\tgithub.com/caddyserver/certmagic@v0.23.0/acmeissuer.go:382\ngithub.com/caddyserver/caddy/v2/modules/caddytls.(*ACMEIssuer).Issue\n\tgithub.com/caddyserver/caddy/v2@v2.10.0/modules/caddytls/acmeissuer.go:288\ngithub.com/caddyserver/certmagic.(*Config).obtainCert.func2\n\tgithub.com/caddyserver/certmagic@v0.23.0/config.go:626\ngithub.com/caddyserver/certmagic.doWithRetry\n\tgithub.com/caddyserver/certmagic@v0.23.0/async.go:104\ngithub.com/caddyserver/certmagic.(*Config).obtainCert\n\tgithub.com/caddyserver/certmagic@v0.23.0/config.go:700\ngithub.com/caddyserver/certmagic.(*Config).ObtainCertAsync\n\tgithub.com/caddyserver/certmagic@v0.23.0/config.go:505\ngithub.com/caddyserver/certmagic.(*Config).manageOne.func1\n\tgithub.com/caddyserver/certmagic@v0.23.0/config.go:415\ngithub.com/caddyserver/certmagic.(*jobManager).worker\n\tgithub.com/caddyserver/certmagic@v0.23.0/async.go:73"}
swarm.0.bhtwxqxdu3en@*.228.42.26    | {"level":"error","ts":1753379087.9020586,"msg":"challenge failed","identifier":"app.sitehydrator.com","challenge_type":"http-01","problem":{"type":"urn:ietf:params:acme:error:tls","title":"","detail":"143.42.2.10: Fetching https://app.sitehydrator.com/.well-known/acme-challenge/QBHPM1Z6AzEA2xrTNU7qVfZ55qCLb6c9a5UQPvvk9J8: remote error: tls: internal error","instance":"","subproblems":null},"stacktrace":"github.com/mholt/acmez/v3.(*Client).pollAuthorization\n\tgithub.com/mholt/acmez/v3@v3.1.2/client.go:557\ngithub.com/mholt/acmez/v3.(*Client).solveChallenges\n\tgithub.com/mholt/acmez/v3@v3.1.2/client.go:378\ngithub.com/mholt/acmez/v3.(*Client).ObtainCertificate\n\tgithub.com/mholt/acmez/v3@v3.1.2/client.go:136\ngithub.com/caddyserver/certmagic.(*ACMEIssuer).doIssue\n\tgithub.com/caddyserver/certmagic@v0.23.0/acmeissuer.go:489\ngithub.com/caddyserver/certmagic.(*ACMEIssuer).Issue\n\tgithub.com/caddyserver/certmagic@v0.23.0/acmeissuer.go:382\ngithub.com/caddyserver/caddy/v2/modules/caddytls.(*ACMEIssuer).Issue\n\tgithub.com/caddyserver/caddy/v2@v2.10.0/modules/caddytls/acmeissuer.go:288\ngithub.com/caddyserver/certmagic.(*Config).obtainCert.func2\n\tgithub.com/caddyserver/certmagic@v0.23.0/config.go:626\ngithub.com/caddyserver/certmagic.doWithRetry\n\tgithub.com/caddyserver/certmagic@v0.23.0/async.go:104\ngithub.com/caddyserver/certmagic.(*Config).obtainCert\n\tgithub.com/caddyserver/certmagic@v0.23.0/config.go:700\ngithub.com/caddyserver/certmagic.(*Config).ObtainCertAsync\n\tgithub.com/caddyserver/certmagic@v0.23.0/config.go:505\ngithub.com/caddyserver/certmagic.(*Config).manageOne.func1\n\tgithub.com/caddyserver/certmagic@v0.23.0/config.go:415\ngithub.com/caddyserver/certmagic.(*jobManager).worker\n\tgithub.com/caddyserver/certmagic@v0.23.0/async.go:73"}
swarm.0.bhtwxqxdu3en@*.228.42.26    | {"level":"error","ts":1753379087.9035597,"msg":"validating authorization","identifier":"app.sitehydrator.com","problem":{"type":"urn:ietf:params:acme:error:tls","title":"","detail":"143.42.2.10: Fetching https://app.sitehydrator.com/.well-known/acme-challenge/QBHPM1Z6AzEA2xrTNU7qVfZ55qCLb6c9a5UQPvvk9J8: remote error: tls: internal error","instance":"","subproblems":null},"order":"https://acme-staging-v02.api.letsencrypt.org/acme/order/135604333/26253603984","attempt":2,"max_attempts":3,"stacktrace":"github.com/mholt/acmez/v3.(*Client).ObtainCertificate\n\tgithub.com/mholt/acmez/v3@v3.1.2/client.go:152\ngithub.com/caddyserver/certmagic.(*ACMEIssuer).doIssue\n\tgithub.com/caddyserver/certmagic@v0.23.0/acmeissuer.go:489\ngithub.com/caddyserver/certmagic.(*ACMEIssuer).Issue\n\tgithub.com/caddyserver/certmagic@v0.23.0/acmeissuer.go:382\ngithub.com/caddyserver/caddy/v2/modules/caddytls.(*ACMEIssuer).Issue\n\tgithub.com/caddyserver/caddy/v2@v2.10.0/modules/caddytls/acmeissuer.go:288\ngithub.com/caddyserver/certmagic.(*Config).obtainCert.func2\n\tgithub.com/caddyserver/certmagic@v0.23.0/config.go:626\ngithub.com/caddyserver/certmagic.doWithRetry\n\tgithub.com/caddyserver/certmagic@v0.23.0/async.go:104\ngithub.com/caddyserver/certmagic.(*Config).obtainCert\n\tgithub.com/caddyserver/certmagic@v0.23.0/config.go:700\ngithub.com/caddyserver/certmagic.(*Config).ObtainCertAsync\n\tgithub.com/caddyserver/certmagic@v0.23.0/config.go:505\ngithub.com/caddyserver/certmagic.(*Config).manageOne.func1\n\tgithub.com/caddyserver/certmagic@v0.23.0/config.go:415\ngithub.com/caddyserver/certmagic.(*jobManager).worker\n\tgithub.com/caddyserver/certmagic@v0.23.0/async.go:73"}
swarm.0.bhtwxqxdu3en@*.228.42.26    | {"level":"error","ts":1753379087.904904,"logger":"tls.obtain","msg":"could not get certificate from issuer","identifier":"app.sitehydrator.com","issuer":"acme-v02.api.letsencrypt.org-directory","error":"HTTP 400 urn:ietf:params:acme:error:tls - 143.42.2.10: Fetching https://app.sitehydrator.com/.well-known/acme-challenge/QBHPM1Z6AzEA2xrTNU7qVfZ55qCLb6c9a5UQPvvk9J8: remote error: tls: internal error"}
swarm.0.bhtwxqxdu3en@*.228.42.26    | {"level":"error","ts":1753379087.905045,"logger":"tls.obtain","msg":"will retry","error":"[app.sitehydrator.com] Obtain: [app.sitehydrator.com] solving challenge: app.sitehydrator.com: [app.sitehydrator.com] authorization failed: HTTP 400 urn:ietf:params:acme:error:tls - 143.42.2.10: Fetching https://app.sitehydrator.com/.well-known/acme-challenge/QBHPM1Z6AzEA2xrTNU7qVfZ55qCLb6c9a5UQPvvk9J8: remote error: tls: internal error (ca=https://acme-staging-v02.api.letsencrypt.org/directory)","attempt":2,"retrying_in":120,"elapsed":72.82092229,"max_duration":2592000}
swarm.0.bhtwxqxdu3en@*.228.42.26    | {"level":"error","ts":1753379170.3553221,"logger":"tls","msg":"tls-alpn challenge","remote_addr":"10.0.0.2:53669","server_name":"app.sitehydrator.com","error":"no information found to solve challenge for identifier: app.sitehydrator.com"}
swarm.0.bhtwxqxdu3en@*.228.42.26    | {"level":"error","ts":1753379208.4301722,"msg":"challenge failed","identifier":"app.sitehydrator.com","challenge_type":"tls-alpn-01","problem":{"type":"urn:ietf:params:acme:error:tls","title":"","detail":"143.42.2.10: remote error: tls: internal error","instance":"","subproblems":null},"stacktrace":"github.com/mholt/acmez/v3.(*Client).pollAuthorization\n\tgithub.com/mholt/acmez/v3@v3.1.2/client.go:557\ngithub.com/mholt/acmez/v3.(*Client).solveChallenges\n\tgithub.com/mholt/acmez/v3@v3.1.2/client.go:378\ngithub.com/mholt/acmez/v3.(*Client).ObtainCertificate\n\tgithub.com/mholt/acmez/v3@v3.1.2/client.go:136\ngithub.com/caddyserver/certmagic.(*ACMEIssuer).doIssue\n\tgithub.com/caddyserver/certmagic@v0.23.0/acmeissuer.go:489\ngithub.com/caddyserver/certmagic.(*ACMEIssuer).Issue\n\tgithub.com/caddyserver/certmagic@v0.23.0/acmeissuer.go:382\ngithub.com/caddyserver/caddy/v2/modules/caddytls.(*ACMEIssuer).Issue\n\tgithub.com/caddyserver/caddy/v2@v2.10.0/modules/caddytls/acmeissuer.go:288\ngithub.com/caddyserver/certmagic.(*Config).obtainCert.func2\n\tgithub.com/caddyserver/certmagic@v0.23.0/config.go:626\ngithub.com/caddyserver/certmagic.doWithRetry\n\tgithub.com/caddyserver/certmagic@v0.23.0/async.go:104\ngithub.com/caddyserver/certmagic.(*Config).obtainCert\n\tgithub.com/caddyserver/certmagic@v0.23.0/config.go:700\ngithub.com/caddyserver/certmagic.(*Config).ObtainCertAsync\n\tgithub.com/caddyserver/certmagic@v0.23.0/config.go:505\ngithub.com/caddyserver/certmagic.(*Config).manageOne.func1\n\tgithub.com/caddyserver/certmagic@v0.23.0/config.go:415\ngithub.com/caddyserver/certmagic.(*jobManager).worker\n\tgithub.com/caddyserver/certmagic@v0.23.0/async.go:73"}
swarm.0.bhtwxqxdu3en@*.228.42.26    | {"level":"error","ts":1753379208.431631,"msg":"validating authorization","identifier":"app.sitehydrator.com","problem":{"type":"urn:ietf:params:acme:error:tls","title":"","detail":"143.42.2.10: remote error: tls: internal error","instance":"","subproblems":null},"order":"https://acme-staging-v02.api.letsencrypt.org/acme/order/135604333/26253631504","attempt":1,"max_attempts":3,"stacktrace":"github.com/mholt/acmez/v3.(*Client).ObtainCertificate\n\tgithub.com/mholt/acmez/v3@v3.1.2/client.go:152\ngithub.com/caddyserver/certmagic.(*ACMEIssuer).doIssue\n\tgithub.com/caddyserver/certmagic@v0.23.0/acmeissuer.go:489\ngithub.com/caddyserver/certmagic.(*ACMEIssuer).Issue\n\tgithub.com/caddyserver/certmagic@v0.23.0/acmeissuer.go:382\ngithub.com/caddyserver/caddy/v2/modules/caddytls.(*ACMEIssuer).Issue\n\tgithub.com/caddyserver/caddy/v2@v2.10.0/modules/caddytls/acmeissuer.go:288\ngithub.com/caddyserver/certmagic.(*Config).obtainCert.func2\n\tgithub.com/caddyserver/certmagic@v0.23.0/config.go:626\ngithub.com/caddyserver/certmagic.doWithRetry\n\tgithub.com/caddyserver/certmagic@v0.23.0/async.go:104\ngithub.com/caddyserver/certmagic.(*Config).obtainCert\n\tgithub.com/caddyserver/certmagic@v0.23.0/config.go:700\ngithub.com/caddyserver/certmagic.(*Config).ObtainCertAsync\n\tgithub.com/caddyserver/certmagic@v0.23.0/config.go:505\ngithub.com/caddyserver/certmagic.(*Config).manageOne.func1\n\tgithub.com/caddyserver/certmagic@v0.23.0/config.go:415\ngithub.com/caddyserver/certmagic.(*jobManager).worker\n\tgithub.com/caddyserver/certmagic@v0.23.0/async.go:73"}
swarm.0.bhtwxqxdu3en@*.228.42.26    | {"level":"error","ts":1753379209.9055119,"msg":"validating authorization","identifier":"app.sitehydrator.com","problem":
5

Can you share output of this?

docker exec YOUR-CADDY-DOCKER-PROXY-CONTAINER cat /config/caddy/Caddyfile.autosave

and this:

docker exec YOUR-CADDY-DOCKER-PROXY-CONTAINER caddy list-modules
6

cat: can't open '/config/caddy/Caddyfile.autosave': No such file or directory

I do have a /config/caddy/autosave.json file. I cannot share it directly due to there being credentials:
docker exec CONTAINER cat /config/caddy/autosave.json

{
    "admin": {
        "listen": "tcp/10.200.200.8:2019"
    },
    "apps": {
        "http": {
            "servers": {
                "srv0": {
                    "listen": [
                        ":443"
                    ],
                    "routes": [
                        {
                            "handle": [
                                {
                                    "handler": "subroute",
                                    "routes": [
                                        {
                                            "handle": [
                                                {
                                                    "handler": "reverse_proxy",
                                                    "upstreams": [
                                                        {
                                                            "dial": "10.0.1.47:8055"
                                                        }
                                                    ]
                                                }
                                            ]
                                        }
                                    ]
                                }
                            ],
                            "match": [
                                {
                                    "host": [
                                        "app.sitehydrator.com"
                                    ]
                                }
                            ],
                            "terminal": true
                        }
                    ]
                }
            }
        },
        "tls": {
            "automation": {
                "policies": [
                    {
                        "issuers": [
                            {
                                "challenges": {
                                    "dns": {
                                        "provider": {
                                            "api_token": "1234-token",
                                            "name": "cloudflare"
                                        },
                                        "resolvers": [
                                            "1.1.1.1"
                                        ]
                                    }
                                },
                                "module": "acme"
                            }
                        ],
                        "subjects": [
                            "*.auth.sitehydrator.com"
                        ]
                    },
                    {
                        "subjects": [
                            "app.sitehydrator.com"
                        ]
                    }
                ]
            }
        }
    }
}

caddy list-modules
admin.api.load
admin.api.metrics
admin.api.pki
admin.api.reverse_proxy
caddy.adapters.caddyfile
caddy.config_loaders.http
caddy.filesystems
caddy.listeners.http_redirect
caddy.listeners.proxy_protocol
caddy.listeners.tls
caddy.logging.cores.mock
caddy.logging.encoders.append
caddy.logging.encoders.console
caddy.logging.encoders.filter
caddy.logging.encoders.filter.cookie
caddy.logging.encoders.filter.delete
caddy.logging.encoders.filter.hash
caddy.logging.encoders.filter.ip_mask
caddy.logging.encoders.filter.query
caddy.logging.encoders.filter.regexp
caddy.logging.encoders.filter.rename
caddy.logging.encoders.filter.replace
caddy.logging.encoders.json
caddy.logging.writers.discard
caddy.logging.writers.file
caddy.logging.writers.net
caddy.logging.writers.stderr
caddy.logging.writers.stdout
caddy.network_proxy.none
caddy.network_proxy.url
caddy.storage.file_system
events
http
http.authentication.hashes.bcrypt
http.authentication.providers.http_basic
http.encoders.gzip
http.encoders.zstd
http.handlers.acme_server
http.handlers.authentication
http.handlers.copy_response
http.handlers.copy_response_headers
http.handlers.encode
http.handlers.error
http.handlers.file_server
http.handlers.headers
http.handlers.intercept
http.handlers.invoke
http.handlers.log_append
http.handlers.map
http.handlers.metrics
http.handlers.push
http.handlers.request_body
http.handlers.reverse_proxy
http.handlers.rewrite
http.handlers.static_response
http.handlers.subroute
http.handlers.templates
http.handlers.tracing
http.handlers.vars
http.ip_sources.static
http.matchers.client_ip
http.matchers.expression
http.matchers.file
http.matchers.header
http.matchers.header_regexp
http.matchers.host
http.matchers.method
http.matchers.not
http.matchers.path
http.matchers.path_regexp
http.matchers.protocol
http.matchers.query
http.matchers.remote_ip
http.matchers.tls
http.matchers.vars
http.matchers.vars_regexp
http.precompressed.br
http.precompressed.gzip
http.precompressed.zstd
http.reverse_proxy.selection_policies.client_ip_hash
http.reverse_proxy.selection_policies.cookie
http.reverse_proxy.selection_policies.first
http.reverse_proxy.selection_policies.header
http.reverse_proxy.selection_policies.ip_hash
http.reverse_proxy.selection_policies.least_conn
http.reverse_proxy.selection_policies.query
http.reverse_proxy.selection_policies.random
http.reverse_proxy.selection_policies.random_choose
http.reverse_proxy.selection_policies.round_robin
http.reverse_proxy.selection_policies.uri_hash
http.reverse_proxy.selection_policies.weighted_round_robin
http.reverse_proxy.transport.fastcgi
http.reverse_proxy.transport.http
http.reverse_proxy.upstreams.a
http.reverse_proxy.upstreams.multi
http.reverse_proxy.upstreams.srv
pki
tls
tls.ca_pool.source.file
tls.ca_pool.source.http
tls.ca_pool.source.inline
tls.ca_pool.source.pki_intermediate
tls.ca_pool.source.pki_root
tls.ca_pool.source.storage
tls.certificates.automate
tls.certificates.load_files
tls.certificates.load_folders
tls.certificates.load_pem
tls.certificates.load_storage
tls.client_auth.verifier.leaf
tls.ech.publishers.dns
tls.get_certificate.http
tls.get_certificate.tailscale
tls.handshake_match.local_ip
tls.handshake_match.remote_ip
tls.handshake_match.sni
tls.handshake_match.sni_regexp
tls.issuance.acme
tls.issuance.internal
tls.issuance.zerossl
tls.leaf_cert_loader.file
tls.leaf_cert_loader.folder
tls.leaf_cert_loader.pem
tls.leaf_cert_loader.storage
tls.permission.http
tls.stek.distributed
tls.stek.standard

  Standard modules: 127

admin.api.souin
cache
dns.providers.cloudflare
docker_proxy
http.authentication.providers.authorizer
http.handlers.authenticator
http.handlers.cache
http.handlers.rate_limit
security

  Non-standard modules: 9

  Unknown modules: 0
7 
docker exec YOUR-CADDY-DOCKER-PROXY-CONTAINER caddy environ

Alright, the module is there :+1:

1 Like
8 
caddy.HomeDir=/root
caddy.AppDataDir=/data/caddy
caddy.AppConfigDir=/config/caddy
caddy.ConfigAutosavePath=/config/caddy/autosave.json
caddy.Version=v2.10.0 h1:fonubSaQKF1YANl8TXqGcn4IbIRUDdfAkpcsfI/vX5U=
runtime.GOOS=linux
runtime.GOARCH=amd64
runtime.Compiler=gc
runtime.NumCPU=1
runtime.GOMAXPROCS=1
runtime.Version=go1.24.5
os.Getwd=/srv

PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
HOSTNAME=5ff0ff10a74b
ADMIN_EMAIL=myemail@example.com
CADDY_CONTROLLER_NETWORK=10.200.200.0/24
CADDY_DOCKER_CADDYFILE_PATH=/config/caddy/Caddyfile
CADDY_DOCKER_MODE=server
CADDY_DOCKER_POLLING_INTERVAL=30s
CADDY_INGRESS_NETWORKS=public
CLOUDFLARE_API=1234-private-token
CADDY_VERSION=v2.10.0
XDG_CONFIG_HOME=/config
XDG_DATA_HOME=/data
HOME=/root
9

It’s interesting that you don’t have a Caddyfile.autosave in your caddy.AppConfigDir folder. caddy-docker-proxy usually generates the resulting Caddyfile there under that name.

For example, here’s what I see when I spin up lucaslorentz/caddy-docker-proxy on my laptop:

$ docker exec caddy-docker-proxy-caddy-1 ls -lsa /config/caddy/
total 8
     0 drwx------    1 root     root            62 Jul 24 18:32 .
     0 drwx------    1 root     root            10 Jul 24 18:32 ..
     4 -rw-r--r--    1 root     root            43 Jul 24 18:32 Caddyfile.autosave
     4 -rw-------    1 root     root           312 Jul 24 18:32 autosave.json
1 Like
10

Also the TLS-ALPN-01 challenge works shown here https://letsdebug.net/app.sitehydrator.com/2509381
And Caddy does support TLS-ALPN-01.

1 Like
11

It does show that it would work with Let’s Debug, but the logs shows it failing:

This seems to show the following to me:

  1. got renewal info from (?)
  2. Waits for rate limiter
  3. It tries tls-alpn, gets “no information found to solve challenge for identifier: app.sitehydrator.com”
  4. It tries http-01, which fails with a tls internal error (no valid cert)
swarm.0.zdjfho6@***.42.2.10    | {"level":"info","ts":1753382923.1875298,"msg":"got renewal info","names":["app.sitehydrator.com"],"window_start":1758486853,"window_end":1758642303,"selected_time":1758628315,"recheck_after":1753404523.1875174,"explanation_url":""}
swarm.0.zdjfho6@***.42.2.10    | {"level":"info","ts":1753382923.1881351,"logger":"tls","msg":"waiting on internal rate limiter","identifiers":["app.sitehydrator.com"],"ca":"https://acme-v02.api.letsencrypt.org/directory","account":"caddy@zerossl.com"}
swarm.0.zdjfho6@***.42.2.10    | {"level":"info","ts":1753382923.1881475,"logger":"tls","msg":"done waiting on internal rate limiter","identifiers":["app.sitehydrator.com"],"ca":"https://acme-v02.api.letsencrypt.org/directory","account":"caddy@zerossl.com"}
swarm.0.zdjfho6@***.42.2.10    | {"level":"error","ts":1753382923.3075523,"logger":"tls.obtain","msg":"will retry","error":"[app.sitehydrator.com] Obtain: [app.sitehydrator.com] creating new order: attempt 1: https://acme-v02.api.letsencrypt.org/acme/new-order: HTTP 429 urn:ietf:params:acme:error:rateLimited - too many failed authorizations (5) for \"app.sitehydrator.com\" in the last 1h0m0s, retry after 2025-07-24 18:58:22 UTC: see https://letsencrypt.org/docs/rate-limits/#authorization-failures-per-hostname-per-account (ca=https://acme-v02.api.letsencrypt.org/directory)","attempt":2,"retrying_in":120,"elapsed":66.866439865,"max_duration":2592000}
swarm.0.zdjfho6@***.42.2.10    | {"level":"warn","ts":1753382924.8827648,"logger":"tls","msg":"looking up info for HTTP challenge","host":"app.sitehydrator.com","remote_addr":"10.0.0.2:49417","user_agent":"Mozilla/5.0 (compatible; Let's Encrypt validation server; +https://www.letsencrypt.org)","error":"no information found to solve challenge for identifier: app.sitehydrator.com"}
swarm.0.iyv8uab@***.42.2.10    | {"level":"info","ts":1753382334.5770226,"logger":"tls","msg":"waiting on internal rate limiter","identifiers":["app.sitehydrator.com"],"ca":"https://acme-v02.api.letsencrypt.org/directory","account":"caddy@zerossl.com"}
swarm.0.iyv8uab@***.42.2.10    | {"level":"error","ts":1753382339.3468933,"logger":"tls","msg":"tls-alpn challenge","remote_addr":"10.0.0.2:17918","server_name":"app.sitehydrator.com","error":"no information found to solve challenge for identifier: app.sitehydrator.com"}
swarm.0.iyv8uab@***.42.2.10    | {"level":"info","ts":1753382344.5434904,"logger":"tls","msg":"done waiting on internal rate limiter","identifiers":["app.sitehydrator.com"],"ca":"https://acme-v02.api.letsencrypt.org/directory","account":"caddy@zerossl.com"}
swarm.0.iyv8uab@***.42.2.10    | {"level":"error","ts":1753382345.824077,"msg":"challenge failed","identifier":"app.sitehydrator.com","challenge_type":"http-01","problem":{"type":"urn:ietf:params:acme:error:tls","title":"","detail":"143.42.2.10: Fetching https://app.sitehydrator.com/.well-known/acme-challenge/28ooLHutHwZmacmOR7mcSxzLY5E5R8_T6OhaahaU2g0: remote error: tls: internal error","instance":"","subproblems":null},"stacktrace":"github.com/mholt/acmez/v3.(*Client).pollAuthorization\n\tgithub.com/mholt/acmez/v3@v3.1.2/client.go:557\ngithub.com/mholt/acmez/v3.(*Client).solveChallenges\n\tgithub.com/mholt/acmez/v3@v3.1.2/client.go:378\ngithub.com/mholt/acmez/v3.(*Client).ObtainCertificate\n\tgithub.com/mholt/acmez/v3@v3.1.2/client.go:136\ngithub.com/caddyserver/certmagic.(*ACMEIssuer).doIssue\n\tgithub.com/caddyserver/certmagic@v0.23.0/acmeissuer.go:489\ngithub.com/caddyserver/certmagic.(*ACMEIssuer).Issue\n\tgithub.com/caddyserver/certmagic@v0.23.0/acmeissuer.go:382\ngithub.com/caddyserver/caddy/v2/modules/caddytls.(*ACMEIssuer).Issue\n\tgithub.com/caddyserver/caddy/v2@v2.10.0/modules/caddytls/acmeissuer.go:288\ngithub.com/caddyserver/certmagic.(*Config).obtainCert.func2\n\tgithub.com/caddyserver/certmagic@v0.23.0/config.go:626\ngithub.com/caddyserver/certmagic.doWithRetry\n\tgithub.com/caddyserver/certmagic@v0.23.0/async.go:104\ngithub.com/caddyserver/certmagic.(*Config).obtainCert\n\tgithub.com/caddyserver/certmagic@v0.23.0/config.go:700\ngithub.com/caddyserver/certmagic.(*Config).ObtainCertAsync\n\tgithub.com/caddyserver/certmagic@v0.23.0/config.go:505\ngithub.com/caddyserver/certmagic.(*Config).manageOne.func1\n\tgithub.com/caddyserver/certmagic@v0.23.0/config.go:415\ngithub.com/caddyserver/certmagic.(*jobManager).worker\n\tgithub.com/caddyserver/certmagic@v0.23.0/async.go:73"}
12

It seems presently there are no certificates, thus TLS (i.e. SSL) will fail.
Without a certificate there is nothing for TLS to work with.

Checking HTTP; seeing a redirect to HTTPS

$ curl -Ii http://app.sitehydrator.com/.well-known/acme-challenge/sometestfile
HTTP/1.1 308 Permanent Redirect
Connection: close
Location: https://app.sitehydrator.com/.well-known/acme-challenge/sometestfile
Server: Caddy
Date: Thu, 24 Jul 2025 19:25:27 GMT

Checking HTTPS gets an error

$ curl -k -i https://app.sitehydrator.com/.well-known/acme-challenge/sometestfile
curl: (35) OpenSSL/3.0.13: error:0A000438:SSL routines::tlsv1 alert internal error

Checking HTTPS with openssl indicates there are no certificates (for Let’s Encrypt an expired or self signed is fine for HTTP-01)

$ openssl s_client -showcerts -servername app.sitehydrator.com -connect app.sitehydrator.com:443 < /dev/null
CONNECTED(00000003)
40A7C6F5AC760000:error:0A000438:SSL routines:ssl3_read_bytes:tlsv1 alert internal error:../ssl/record/rec_layer_s3.c:1599:SSL alert number 80
---
no peer certificate available
---
No client certificate CA names sent
---
SSL handshake has read 7 bytes and written 322 bytes
Verification: OK
---
New, (NONE), Cipher is (NONE)
Secure Renegotiation IS NOT supported
Compression: NONE
Expansion: NONE
No ALPN negotiated
Early data was not sent
Verify return code: 0 (ok)
---
13

What you’ve got here is that the issuer is only defined for *.auth.sitehydrator.com:

Notice that app.sitehydrator.com has no issuer:

                    {
                        "subjects": [
                            "app.sitehydrator.com"
                        ]
                    }

Or, to be explicit with jq:

$ jq '.apps.tls.automation.policies[] | { hasIssuer: has("issuers"), subjects }' autosave.json
{
  "hasIssuer": true,
  "subjects": [
    "*.auth.sitehydrator.com"
  ]
}
{
  "hasIssuer": false,
  "subjects": [
    "app.sitehydrator.com"
  ]
}

Could you try a per-site configuration instead? For example, set your global options:

{
  order rate_limit before basic_auth
  email {env.ADMIN_EMAIL}
}

and labels for caddy-docker-proxy:

  labels:
    caddy: "${HOSTNAME}"
    caddy.reverse_proxy: "{{ upstreams 8055 }}"
    caddy.tls: "dns cloudflare {env.CLOUDFLARE_API}"

Just to see if that makes any difference.

1 Like
14

EDIT: After spending a lot of time collecting this post, the page does load!
I’ve left the below in for reference.

I’ve included the autosave below. It seems to show an issuer in the correct spot, but no dice. It still doesn’t load.

swarm.0.seryc4dg7me5@**.228.42.26    | {"level":"info","ts":1753389682.3317988,"msg":"authorization finalized","identifier":"app.sitehydrator.com","authz_status":"valid"}
swarm.0.seryc4dg7me5@**.228.42.26    | {"level":"info","ts":1753389682.7937806,"msg":"got renewal info","names":["app.sitehydrator.com"],"window_start":1758493616,"window_end":1758649065,"selected_time":1758625838,"recheck_after":1753411282.7937713,"explanation_url":""}
swarm.0.seryc4dg7me5@**.228.42.26    | {"level":"info","ts":1753389682.7966728,"logger":"tls.obtain","msg":"certificate obtained successfully","identifier":"app.sitehydrator.com","issuer":"acme-v02.api.letsencrypt.org-directory"}
swarm.0.seryc4dg7me5@**.228.42.26    | {"level":"info","ts":1753389682.7968311,"logger":"tls.obtain","msg":"releasing lock","identifier":"app.sitehydrator.com"}
swarm.0.seryc4dg7me5@**.228.42.26    | {"level":"warn","ts":1753389682.7973576,"logger":"tls","msg":"stapling OCSP","error":"no OCSP stapling for [app.sitehydrator.com]: no OCSP server specified in certificate","identifiers":["app.sitehydrator.com"]}
swarm.0.7zbrnaiec4pk@**3.42.2.10     | {"level":"info","ts":1753389496.025229,"msg":"authorization finalized","identifier":"app.sitehydrator.com","authz_status":"valid"}
swarm.0.4yqb7s2ahv81@**3.255.227.29  | {"level":"error","ts":1753389523.7432573,"msg":"validating authorization","identifier":"app.sitehydrator.com","problem":{"type":"urn:ietf:params:acme:error:unauthorized","title":"","detail":"During secondary validation: Incorrect TXT record \"qTWrhfERo2yYMrE-iWcAbH0vgDrzstfMqevqIhd1KEc\" (and 3 more) found at _acme-challenge.app.sitehydrator.com","instance":"","subproblems":null},"order":"https://acme-staging-v02.api.letsencrypt.org/acme/order/111886334/26255872064","attempt":1,"max_attempts":3,"stacktrace":"github.com/mholt/acmez/v3.(*Client).ObtainCertificate\n\tgithub.com/mholt/acmez/v3@v3.1.2/client.go:152\ngithub.com/caddyserver/certmagic.(*ACMEIssuer).doIssue\n\tgithub.com/caddyserver/certmagic@v0.23.0/acmeissuer.go:489\ngithub.com/caddyserver/certmagic.(*ACMEIssuer).Issue\n\tgithub.com/caddyserver/certmagic@v0.23.0/acmeissuer.go:382\ngithub.com/caddyserver/caddy/v2/modules/caddytls.(*ACMEIssuer).Issue\n\tgithub.com/caddyserver/caddy/v2@v2.10.0/modules/caddytls/acmeissuer.go:288\ngithub.com/caddyserver/certmagic.(*Config).obtainCert.func2\n\tgithub.com/caddyserver/certmagic@v0.23.0/config.go:626\ngithub.com/caddyserver/certmagic.doWithRetry\n\tgithub.com/caddyserver/certmagic@v0.23.0/async.go:104\ngithub.com/caddyserver/certmagic.(*Config).obtainCert\n\tgithub.com/caddyserver/certmagic@v0.23.0/config.go:700\ngithub.com/caddyserver/certmagic.(*Config).ObtainCertAsync\n\tgithub.com/caddyserver/certmagic@v0.23.0/config.go:505\ngithub.com/caddyserver/certmagic.(*Config).manageOne.func1\n\tgithub.com/caddyserver/certmagic@v0.23.0/config.go:415\ngithub.com/caddyserver/certmagic.(*jobManager).worker\n\tgithub.com/caddyserver/certmagic@v0.23.0/async.go:73"}
swarm.0.4yqb7s2ahv81@**3.255.227.29  | {"level":"error","ts":1753389523.7435231,"logger":"tls.obtain","msg":"could not get certificate from issuer","identifier":"app.sitehydrator.com","issuer":"acme-v02.api.letsencrypt.org-directory","error":"HTTP 403 urn:ietf:params:acme:error:unauthorized - During secondary validation: Incorrect TXT record \"qTWrhfERo2yYMrE-iWcAbH0vgDrzstfMqevqIhd1KEc\" (and 3 more) found at _acme-challenge.app.sitehydrator.com"}
swarm.0.4yqb7s2ahv81@**3.255.227.29  | {"level":"error","ts":1753389523.7437453,"logger":"tls.obtain","msg":"will retry","error":"[app.sitehydrator.com] Obtain: [app.sitehydrator.com] solving challenge: app.sitehydrator.com: [app.sitehydrator.com] authorization failed: HTTP 403 urn:ietf:params:acme:error:unauthorized - During secondary validation: Incorrect TXT record \"qTWrhfERo2yYMrE-iWcAbH0vgDrzstfMqevqIhd1KEc\" (and 3 more) found at _acme-challenge.app.sitehydrator.com (ca=https://acme-staging-v02.api.letsencrypt.org/directory)","attempt":2,"retrying_in":120,"elapsed":88.428720449,"max_duration":2592000}
swarm.0.4yqb7s2ahv81@**3.255.227.29  | {"level":"info","ts":1753389661.4501374,"msg":"authorization finalized","identifier":"app.sitehydrator.com","authz_status":"valid"}
swarm.0.4yqb7s2ahv81@**3.255.227.29  | {"level":"info","ts":1753389664.7844741,"msg":"got renewal info","names":["app.sitehydrator.com"],"window_start":1758493595,"window_end":1758649044,"selected_time":1758634748,"recheck_after":1753411264.7844646,"explanation_url":""}
swarm.0.4yqb7s2ahv81@**3.255.227.29  | {"level":"info","ts":1753389664.7856238,"logger":"tls.issuance.acme","msg":"waiting on internal rate limiter","identifiers":["app.sitehydrator.com"],"ca":"https://acme-v02.api.letsencrypt.org/directory","account":"caddy@zerossl.com"}
swarm.0.4yqb7s2ahv81@**3.255.227.29  | {"level":"info","ts":1753389664.7857075,"logger":"tls.issuance.acme","msg":"done waiting on internal rate limiter","identifiers":["app.sitehydrator.com"],"ca":"https://acme-v02.api.letsencrypt.org/directory","account":"caddy@zerossl.com"}
swarm.0.4yqb7s2ahv81@**3.255.227.29  | {"level":"error","ts":1753389665.0822256,"logger":"tls.obtain","msg":"could not get certificate from issuer","identifier":"app.sitehydrator.com","issuer":"acme-v02.api.letsencrypt.org-directory","error":"HTTP 404 urn:ietf:params:acme:error:malformed - No such authorization"}
swarm.0.4yqb7s2ahv81@**3.255.227.29  | {"level":"info","ts":1753389665.082417,"logger":"tls.obtain","msg":"releasing lock","identifier":"app.sitehydrator.com"}
swarm.0.4yqb7s2ahv81@**3.255.227.29  | {"level":"error","ts":1753389665.083768,"logger":"tls","msg":"job failed","error":"app.sitehydrator.com: obtaining certificate: [app.sitehydrator.com] Obtain: [app.sitehydrator.com] solving challenge: getting authorization at https://acme-v02.api.letsencrypt.org/acme/authz/1218427617/558336693071: attempt 1: https://acme-v02.api.letsencrypt.org/acme/authz/1218427617/558336693071: HTTP 404 urn:ietf:params:acme:error:malformed - No such authorization (ca=https://acme-v02.api.letsencrypt.org/directory)"}

I found this interesting:

solving challenge: app.sitehydrator.com: [app.sitehydrator.com] authorization failed: HTTP 403 urn:ietf:params:acme:error:unauthorized - During secondary validation: Incorrect TXT record "qTWrhfERo2yYMrE-iWcAbH0vgDrzstfMqevqIhd1KEc" (and 3 more) found at _acme-challenge.app.sitehydrator.com

This makes me think there’s a race condition with DNS validation, where multiple nodes are trying to validate the txt record at the same time.

{
    "admin": {
        "listen": "tcp/10.200.200.8:2019"
    },
    "apps": {
        "http": {
            "servers": {
                "srv0": {
                    "listen": [
                        ":443"
                    ]
                    "routes": [
                        {
                            "handle": [
                                {
                                    "handler": "subroute",
                                    "routes": [
                                        {
                                            "handle": [
                                                {
                                                    "handler": "reverse_proxy",
                                                    "upstreams": [
                                                        {
                                                            "dial": "10.0.1.128:8055"
                                                        }
                                                    ]
                                                }
                                            ]
                                        }
                                    ]
                                }
                            ],
                            "match": [
                                {
                                    "host": [
                                        "app.sitehydrator.com"
                                    ]
                                }
                            ],
                            "terminal": true
                        }
                    ]
                }
            }
        },
        "tls": {
            "automation": {
                "policies": [
                    {
                        "issuers": [
                            {
                                "challenges": {
                                    "dns": {
                                        "provider": {
                                            "api_token": "{env.CLOUDFLARE_API}",
                                            "name": "cloudflare"
                                        }
                                    }
                                },
                                "module": "acme"
                            }
                        ],
                        "subjects": [
                            "app.sitehydrator.com"
                        ]
                    }
                ]
            }
        }
    }
}
2 Likes
15

Let’s Encrypt uses Multi-Perspective Validation Improves Domain Validation Security - Let’s Encrypt

2 Likes
16

Hi @vaclench,

Some supplemental information
Here SSL Checker shows the presently being served certificate as

image
image1180×614 21.2 KB

image
image1465×1126 28.4 KB

and that is this certificate crt.sh | 19032821490

openssl s_client -showcerts -servername app.sitehydrator.com -connect app.sitehydrator.com:443 
$ openssl s_client -showcerts -servername app.sitehydrator.com -connect app.sitehydrator.com:443 < /dev/null
CONNECTED(00000003)
depth=2 C = US, O = Google Trust Services LLC, CN = GTS Root R4
verify return:1
depth=1 C = US, O = Google Trust Services, CN = WE1
verify return:1
depth=0 CN = sitehydrator.com
verify return:1
---
Certificate chain
 0 s:CN = sitehydrator.com
   i:C = US, O = Google Trust Services, CN = WE1
   a:PKEY: id-ecPublicKey, 256 (bit); sigalg: ecdsa-with-SHA256
   v:NotBefore: Jun 15 07:26:20 2025 GMT; NotAfter: Sep 13 08:24:50 2025 GMT
-----BEGIN CERTIFICATE-----
MIIDtDCCA1mgAwIBAgIQJMM2OJR7H78TP7Zr6dmZjzAKBggqhkjOPQQDAjA7MQsw
CQYDVQQGEwJVUzEeMBwGA1UEChMVR29vZ2xlIFRydXN0IFNlcnZpY2VzMQwwCgYD
VQQDEwNXRTEwHhcNMjUwNjE1MDcyNjIwWhcNMjUwOTEzMDgyNDUwWjAbMRkwFwYD
VQQDExBzaXRlaHlkcmF0b3IuY29tMFkwEwYHKoZIzj0CAQYIKoZIzj0DAQcDQgAE
4RGiqbimsXkTW3rdhmnrK4bau+K7bHpmIoulD62wHgyE+uzssDF2Oh91onrowFbh
vMgfhuhgWb6Fjir/XB0r46OCAl0wggJZMA4GA1UdDwEB/wQEAwIHgDATBgNVHSUE
DDAKBggrBgEFBQcDATAMBgNVHRMBAf8EAjAAMB0GA1UdDgQWBBSMRT92AxNiXqT/
zWE9g3FnVh6WQDAfBgNVHSMEGDAWgBSQd5I1Z8T/qMyp5nvZgHl7zJP5ODBeBggr
BgEFBQcBAQRSMFAwJwYIKwYBBQUHMAGGG2h0dHA6Ly9vLnBraS5nb29nL3Mvd2Ux
L0pNTTAlBggrBgEFBQcwAoYZaHR0cDovL2kucGtpLmdvb2cvd2UxLmNydDAvBgNV
HREEKDAmghBzaXRlaHlkcmF0b3IuY29tghIqLnNpdGVoeWRyYXRvci5jb20wEwYD
VR0gBAwwCjAIBgZngQwBAgEwNgYDVR0fBC8wLTAroCmgJ4YlaHR0cDovL2MucGtp
Lmdvb2cvd2UxL0REVlNMaEw4Wk9VLmNybDCCAQQGCisGAQQB1nkCBAIEgfUEgfIA
8AB2AN3cyjSV1+EWBeeVMvrHn/g9HFDf2wA6FBJ2Ciysu8gqAAABl3KyWggAAAQD
AEcwRQIgF/tfq+ooxv4K8gbLdyRBVIjWMxXhV6RwPTrHyGxP68UCIQCsX/fDZJWU
iTvE7tlZjc7KGxi0LIVesBmEZPf1/4vwbgB2AMz7D2qFcQll/pWbU87psnwi6YVc
DZeNtql+VMD+TA2wAAABl3KyWmoAAAQDAEcwRQIgAId6AkfyWVX3BEpibACfZ5jZ
K8Ryg3uubv8T76KL878CIQC2zZT86LuFUMPzLQi2kjpTAY6K1O8tsLJUkP28QBnV
nzAKBggqhkjOPQQDAgNJADBGAiEArUjFsufsvRDUmWXz+VTFvT3v9LgU/cEMNZ1k
SrSoSFYCIQDaQTn/ObVn9mv3coJdHg1Y4XBYgyBR/Kz5fylaYpebsQ==
-----END CERTIFICATE-----
 1 s:C = US, O = Google Trust Services, CN = WE1
   i:C = US, O = Google Trust Services LLC, CN = GTS Root R4
   a:PKEY: id-ecPublicKey, 256 (bit); sigalg: ecdsa-with-SHA384
   v:NotBefore: Dec 13 09:00:00 2023 GMT; NotAfter: Feb 20 14:00:00 2029 GMT
-----BEGIN CERTIFICATE-----
MIICnzCCAiWgAwIBAgIQf/MZd5csIkp2FV0TttaF4zAKBggqhkjOPQQDAzBHMQsw
CQYDVQQGEwJVUzEiMCAGA1UEChMZR29vZ2xlIFRydXN0IFNlcnZpY2VzIExMQzEU
MBIGA1UEAxMLR1RTIFJvb3QgUjQwHhcNMjMxMjEzMDkwMDAwWhcNMjkwMjIwMTQw
MDAwWjA7MQswCQYDVQQGEwJVUzEeMBwGA1UEChMVR29vZ2xlIFRydXN0IFNlcnZp
Y2VzMQwwCgYDVQQDEwNXRTEwWTATBgcqhkjOPQIBBggqhkjOPQMBBwNCAARvzTr+
Z1dHTCEDhUDCR127WEcPQMFcF4XGGTfn1XzthkubgdnXGhOlCgP4mMTG6J7/EFmP
LCaY9eYmJbsPAvpWo4H+MIH7MA4GA1UdDwEB/wQEAwIBhjAdBgNVHSUEFjAUBggr
BgEFBQcDAQYIKwYBBQUHAwIwEgYDVR0TAQH/BAgwBgEB/wIBADAdBgNVHQ4EFgQU
kHeSNWfE/6jMqeZ72YB5e8yT+TgwHwYDVR0jBBgwFoAUgEzW63T/STaj1dj8tT7F
avCUHYwwNAYIKwYBBQUHAQEEKDAmMCQGCCsGAQUFBzAChhhodHRwOi8vaS5wa2ku
Z29vZy9yNC5jcnQwKwYDVR0fBCQwIjAgoB6gHIYaaHR0cDovL2MucGtpLmdvb2cv
ci9yNC5jcmwwEwYDVR0gBAwwCjAIBgZngQwBAgEwCgYIKoZIzj0EAwMDaAAwZQIx
AOcCq1HW90OVznX+0RGU1cxAQXomvtgM8zItPZCuFQ8jSBJSjz5keROv9aYsAm5V
sQIwJonMaAFi54mrfhfoFNZEfuNMSQ6/bIBiNLiyoX46FohQvKeIoJ99cx7sUkFN
7uJW
-----END CERTIFICATE-----
 2 s:C = US, O = Google Trust Services LLC, CN = GTS Root R4
   i:C = BE, O = GlobalSign nv-sa, OU = Root CA, CN = GlobalSign Root CA
   a:PKEY: id-ecPublicKey, 384 (bit); sigalg: RSA-SHA256
   v:NotBefore: Nov 15 03:43:21 2023 GMT; NotAfter: Jan 28 00:00:42 2028 GMT
-----BEGIN CERTIFICATE-----
MIIDejCCAmKgAwIBAgIQf+UwvzMTQ77dghYQST2KGzANBgkqhkiG9w0BAQsFADBX
MQswCQYDVQQGEwJCRTEZMBcGA1UEChMQR2xvYmFsU2lnbiBudi1zYTEQMA4GA1UE
CxMHUm9vdCBDQTEbMBkGA1UEAxMSR2xvYmFsU2lnbiBSb290IENBMB4XDTIzMTEx
NTAzNDMyMVoXDTI4MDEyODAwMDA0MlowRzELMAkGA1UEBhMCVVMxIjAgBgNVBAoT
GUdvb2dsZSBUcnVzdCBTZXJ2aWNlcyBMTEMxFDASBgNVBAMTC0dUUyBSb290IFI0
MHYwEAYHKoZIzj0CAQYFK4EEACIDYgAE83Rzp2iLYK5DuDXFgTB7S0md+8Fhzube
Rr1r1WEYNa5A3XP3iZEwWus87oV8okB2O6nGuEfYKueSkWpz6bFyOZ8pn6KY019e
WIZlD6GEZQbR3IvJx3PIjGov5cSr0R2Ko4H/MIH8MA4GA1UdDwEB/wQEAwIBhjAd
BgNVHSUEFjAUBggrBgEFBQcDAQYIKwYBBQUHAwIwDwYDVR0TAQH/BAUwAwEB/zAd
BgNVHQ4EFgQUgEzW63T/STaj1dj8tT7FavCUHYwwHwYDVR0jBBgwFoAUYHtmGkUN
l8qJUC99BM00qP/8/UswNgYIKwYBBQUHAQEEKjAoMCYGCCsGAQUFBzAChhpodHRw
Oi8vaS5wa2kuZ29vZy9nc3IxLmNydDAtBgNVHR8EJjAkMCKgIKAehhxodHRwOi8v
Yy5wa2kuZ29vZy9yL2dzcjEuY3JsMBMGA1UdIAQMMAowCAYGZ4EMAQIBMA0GCSqG
SIb3DQEBCwUAA4IBAQAYQrsPBtYDh5bjP2OBDwmkoWhIDDkic574y04tfzHpn+cJ
odI2D4SseesQ6bDrarZ7C30ddLibZatoKiws3UL9xnELz4ct92vID24FfVbiI1hY
+SW6FoVHkNeWIP0GCbaM4C6uVdF5dTUsMVs/ZbzNnIdCp5Gxmx5ejvEau8otR/Cs
kGN+hr/W5GvT1tMBjgWKZ1i4//emhA1JG1BbPzoLJQvyEotc03lXjTaCzv8mEbep
8RqZ7a2CPsgRbuvTPBwcOMBBmuFeU88+FSBX6+7iP0il8b4Z0QFqIwwMHfs/L6K1
vepuoxtGzi4CZ68zJpiq1UvSqTbFJjtbD4seiMHl
-----END CERTIFICATE-----
---
Server certificate
subject=CN = sitehydrator.com
issuer=C = US, O = Google Trust Services, CN = WE1
---
No client certificate CA names sent
Peer signing digest: SHA256
Peer signature type: ECDSA
Server Temp Key: X25519, 253 bits
---
SSL handshake has read 2841 bytes and written 402 bytes
Verification: OK
---
New, TLSv1.3, Cipher is TLS_AES_256_GCM_SHA384
Server public key is 256 bit
Secure Renegotiation IS NOT supported
Compression: NONE
Expansion: NONE
No ALPN negotiated
Early data was not sent
Verify return code: 0 (ok)
---
DONE
curl -v -Ii https://app.sitehydrator.com/.well-known/acme-challenge/sometestfile 
$ curl -v -Ii https://app.sitehydrator.com/.well-known/acme-challenge/sometestfile
* Host app.sitehydrator.com:443 was resolved.
* IPv6: 2606:4700:3030::6815:3001, 2606:4700:3030::6815:7001, 2606:4700:3030::6815:1001, 2606:4700:3030::6815:2001, 2606:4700:3030::6815:5001, 2606:4700:3030::6815:4001, 2606:4700:3030::6815:6001
* IPv4: 104.21.80.1, 104.21.48.1, 104.21.16.1, 104.21.32.1, 104.21.112.1, 104.21.96.1, 104.21.64.1
*   Trying 104.21.80.1:443...
* Connected to app.sitehydrator.com (104.21.80.1) port 443
* ALPN: curl offers h2,http/1.1
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
*  CAfile: /etc/ssl/certs/ca-certificates.crt
*  CApath: /etc/ssl/certs
* TLSv1.3 (IN), TLS handshake, Server hello (2):
* TLSv1.3 (IN), TLS handshake, Encrypted Extensions (8):
* TLSv1.3 (IN), TLS handshake, Certificate (11):
* TLSv1.3 (IN), TLS handshake, CERT verify (15):
* TLSv1.3 (IN), TLS handshake, Finished (20):
* TLSv1.3 (OUT), TLS change cipher, Change cipher spec (1):
* TLSv1.3 (OUT), TLS handshake, Finished (20):
* SSL connection using TLSv1.3 / TLS_AES_256_GCM_SHA384 / X25519 / id-ecPublicKey
* ALPN: server accepted h2
* Server certificate:
*  subject: CN=sitehydrator.com
*  start date: Jun 15 07:26:20 2025 GMT
*  expire date: Sep 13 08:24:50 2025 GMT
*  subjectAltName: host "app.sitehydrator.com" matched cert's "*.sitehydrator.com"
*  issuer: C=US; O=Google Trust Services; CN=WE1
*  SSL certificate verify ok.
*   Certificate level 0: Public key type EC/prime256v1 (256/128 Bits/secBits), signed using ecdsa-with-SHA256
*   Certificate level 1: Public key type EC/prime256v1 (256/128 Bits/secBits), signed using ecdsa-with-SHA384
*   Certificate level 2: Public key type EC/secp384r1 (384/192 Bits/secBits), signed using ecdsa-with-SHA384
* using HTTP/2
* [HTTP/2] [1] OPENED stream for https://app.sitehydrator.com/.well-known/acme-challenge/sometestfile
* [HTTP/2] [1] [:method: HEAD]
* [HTTP/2] [1] [:scheme: https]
* [HTTP/2] [1] [:authority: app.sitehydrator.com]
* [HTTP/2] [1] [:path: /.well-known/acme-challenge/sometestfile]
* [HTTP/2] [1] [user-agent: curl/8.5.0]
* [HTTP/2] [1] [accept: */*]
> HEAD /.well-known/acme-challenge/sometestfile HTTP/2
> Host: app.sitehydrator.com
> User-Agent: curl/8.5.0
> Accept: */*
>
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
* old SSL session ID is stale, removing
< HTTP/2 404
HTTP/2 404
< date: Thu, 24 Jul 2025 23:35:55 GMT
date: Thu, 24 Jul 2025 23:35:55 GMT
< content-type: application/json; charset=utf-8
content-type: application/json; charset=utf-8
< content-length: 178
content-length: 178
< alt-svc: h3=":443"; ma=86400
alt-svc: h3=":443"; ma=86400
< content-security-policy: script-src 'self' 'unsafe-eval';worker-src 'self' blob:;child-src 'self' blob:;img-src 'self' https://ci.sandcatdev.com blob: https://raw.githubusercontent.com https://avatars.githubusercontent.com;media-src 'self';connect-src 'self' https://* wss://*;default-src 'self';base-uri 'self';font-src 'self' https: data:;form-action 'self';frame-ancestors 'self';object-src 'none';script-src-attr 'none';style-src 'self' https: 'unsafe-inline'
content-security-policy: script-src 'self' 'unsafe-eval';worker-src 'self' blob:;child-src 'self' blob:;img-src 'self' https://ci.sandcatdev.com blob: https://raw.githubusercontent.com https://avatars.githubusercontent.com;media-src 'self';connect-src 'self' https://* wss://*;default-src 'self';base-uri 'self';font-src 'self' https: data:;form-action 'self';frame-ancestors 'self';object-src 'none';script-src-attr 'none';style-src 'self' https: 'unsafe-inline'
< etag: W/"b2-XiGg3X1DzfckHmePfpz6FbCWCEg"
etag: W/"b2-XiGg3X1DzfckHmePfpz6FbCWCEg"
< via: 1.1 Caddy
via: 1.1 Caddy
< x-powered-by: Directus
x-powered-by: Directus
< cf-cache-status: DYNAMIC
cf-cache-status: DYNAMIC
< report-to: {"group":"cf-nel","max_age":604800,"endpoints":[{"url":"https://a.nel.cloudflare.com/report/v4?s=QocBiYgiVGfFtspIIF%2Bb682KoDL1zN8PNrSHfJtfd1uT1KUspbwpMs5U3UqP51wU8WN%2Flv5fkowtfOsPQaBhr5m7gZmp18fL4bbCPSqk8OmmMw%3D%3D"}]}
report-to: {"group":"cf-nel","max_age":604800,"endpoints":[{"url":"https://a.nel.cloudflare.com/report/v4?s=QocBiYgiVGfFtspIIF%2Bb682KoDL1zN8PNrSHfJtfd1uT1KUspbwpMs5U3UqP51wU8WN%2Flv5fkowtfOsPQaBhr5m7gZmp18fL4bbCPSqk8OmmMw%3D%3D"}]}
< nel: {"report_to":"cf-nel","success_fraction":0.0,"max_age":604800}
nel: {"report_to":"cf-nel","success_fraction":0.0,"max_age":604800}
< server: cloudflare
server: cloudflare
< cf-ray: 96473c9cd815fef8-PDX
cf-ray: 96473c9cd815fef8-PDX

<

Not one of the recently issued certificates https://crt.sh/?q=sitehydrator.com

Also here https://unboundtest.com/m/TXT/_acme-challenge.app.sitehydrator.com/RHQYVB7E
we see that the DNS-01 _acme-challenge. DNS Records were not cleaned up (i.e. removed)
after the certificates being issued.

Query results for TXT _acme-challenge.app.sitehydrator.com

Response:
;; opcode: QUERY, status: NOERROR, id: 17293
;; flags: qr rd ra; QUERY: 1, ANSWER: 3, AUTHORITY: 0, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version 0; flags: do; udp: 1232

;; QUESTION SECTION:
;_acme-challenge.app.sitehydrator.com.	IN	 TXT

;; ANSWER SECTION:
_acme-challenge.app.sitehydrator.com.	0	IN	TXT	"4p8sSgGx0t9OkrasD-LnslJMQfduRkE8JHRo2G6YbLA"
_acme-challenge.app.sitehydrator.com.	0	IN	TXT	"8lkPOdQMdxupPSxA6l0ehqgsP-WMBE2OIDV7Zgk8Zs8"
_acme-challenge.app.sitehydrator.com.	0	IN	TXT	"qTWrhfERo2yYMrE-iWcAbH0vgDrzstfMqevqIhd1KEc"

----- Unbound logs -----
1 Like
17

Is this my responsibility? Or a bug with the cloudflare dns plugin for Caddy?

18

Generally ACME clients do the proper cleanup.
Now some clients don’t cleanup well if an error happens during the ACME challenge.

19

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.