How to stop Caddy from trying to renew a particular certificate in on demand mode

Help
1

1. The problem I’m having:

We use Caddy at our SAAS as a HTTPS reverse proxy, handling a great many (700+) domains (belonging to our end users and not directly under our control).

One of these domains is due for renewal, but the renewal fails, as the user’s DNS records no longer point to our Caddy server.

However, Caddy appears to be continuing to retry renewing this domain every few minutes and failing.

We want to stop Caddy from trying to renew this certificate, as we know it will never work.

Our “ask” endpoint no longer returns a 200 response for the domain, we’ve deleted the local certificates and the OCSP staple file for the domains (and restarting Caddy after each change), but Caddy still tries to renew the certificate every few minutes.

Is there a way to tell Caddy to stop trying to renew the certificate for a specific domain and forget it, when using on demand TLS?

2. Error messages and/or full log output:

The log message output - this is a snippet from the last 15 minutes, but the output has been continuous for the last 48 hours:

Date,Service,Message
"2024-02-13T14:04:40.011Z","caddy","releasing lock"
"2024-02-13T14:04:37.009Z","caddy","will retry"
"2024-02-13T14:04:37.009Z","caddy","could not get certificate from issuer"
"2024-02-13T14:04:31.005Z","caddy","done waiting on internal rate limiter"
"2024-02-13T14:04:31.005Z","caddy","waiting on internal rate limiter"
"2024-02-13T14:04:31.005Z","caddy","could not get certificate from issuer"
"2024-02-13T14:04:31.005Z","caddy","done waiting on internal rate limiter"
"2024-02-13T14:04:31.005Z","caddy","waiting on internal rate limiter"
"2024-02-13T14:04:31.005Z","caddy","obtaining certificate"
"2024-02-13T14:04:31.005Z","caddy","lock acquired"
"2024-02-13T14:04:30.369Z","caddy","releasing lock"
"2024-02-13T14:02:35.306Z","caddy","will retry"
"2024-02-13T14:02:35.306Z","caddy","could not get certificate from issuer"
"2024-02-13T14:02:34.305Z","caddy","could not get certificate from issuer"
"2024-02-13T14:02:34.305Z","caddy","validating authorization"
"2024-02-13T14:02:34.305Z","caddy","challenge failed"
"2024-02-13T14:02:34.305Z","caddy","trying to solve challenge"
"2024-02-13T14:02:32.304Z","caddy","validating authorization"
"2024-02-13T14:02:32.304Z","caddy","challenge failed"
"2024-02-13T14:02:31.303Z","caddy","trying to solve challenge"
"2024-02-13T14:02:31.303Z","caddy","obtaining certificate"
"2024-02-13T14:01:39.895Z","caddy","acquiring lock"
"2024-02-13T14:01:39.895Z","caddy","obtaining new certificate"
"2024-02-13T14:01:31.269Z","caddy","will retry"
"2024-02-13T14:01:31.269Z","caddy","could not get certificate from issuer"
"2024-02-13T14:01:30.268Z","caddy","done waiting on internal rate limiter"
"2024-02-13T14:01:30.268Z","caddy","waiting on internal rate limiter"
"2024-02-13T14:01:30.268Z","caddy","could not get certificate from issuer"
"2024-02-13T14:01:30.268Z","caddy","done waiting on internal rate limiter"
"2024-02-13T14:01:30.268Z","caddy","waiting on internal rate limiter"
"2024-02-13T14:01:30.268Z","caddy","obtaining certificate"
"2024-02-13T14:01:30.268Z","caddy","lock acquired"
"2024-02-13T14:01:30.268Z","caddy","acquiring lock"
"2024-02-13T14:01:30.268Z","caddy","obtaining new certificate"

This is the actual error encountered when attempting the renewal (with ZeroSSL):

[eot.hello.agorapulse.com] solving challenges: authz https://acme.zerossl.com/v2/DV90/authz/g-5_lcKQSQToJNhrv4d9RA has unexpected status; order will fail: invalid (order=https://acme.zerossl.com/v2/DV90/order/TYUsO44AFxB9aetfVIep7w) (ca=https://acme.zerossl.com/v2/DV90)

…and the error with Let’s Encrypt:

[eot.hello.agorapulse.com] solving challenges: authz https://acme.zerossl.com/v2/DV90/authz/g-5_lcKQSQToJNhrv4d9RA has unexpected status; order will fail: invalid (order=https://acme.zerossl.com/v2/DV90/order/TYUsO44AFxB9aetfVIep7w) (ca=https://acme.zerossl.com/v2/DV90)

3. Caddy version:

v2.7.6

4. How I installed and ran Caddy:

a. System environment:

Amazon Linux 2 running on an AWS EC2 instance.

b. Command:

yum install yum-plugin-copr -y
yum copr enable @caddy/caddy epel-8-aarch64 -y
yum install caddy -y

d. My complete Caddy config:

{
    on_demand_tls {
        ask https://eomail5.com/sender-tls-check
    }

    storage file_system {
        root /home/caddy/storage
    }

    email admin@emailoctopus.com

    log {
        output file /var/log/caddy/caddy.log {
            roll_size     100
            roll_keep     5
        }
        format json
        level  INFO
    }
}

http:// {
    header -server

    redir https://{host}{uri}
}

https:// {
    tls {
        on_demand
    }

    header -server

    handle /f/a* {
        reverse_proxy http://eu.spgo.io
    }

    handle /q* {
        reverse_proxy http://eu.spgo.io
    }

    handle /v2* {
        reverse_proxy http://eu.spgo.io
    }

    handle * {
        root * /etc/caddy/html
        rewrite * /index.html
        file_server
    }
}
2

So the way on-demand TLS works, is a TLS handshake for a domain (specifically: the ClientHello contains the domain in its ServerName field) comes in will trigger an obtain/renewal operation.

If Caddy is still trying to renew it then it’s because it’s still getting TLS handshakes for that domain.

Typically, that doesn’t happen because once the DNS goes away then clients stop resolving it to your server, because the DNS entry changed. Sometimes that can take a couple days depending on propagation.

Are you sure you adjusted your ‘ask’ endpoint properly? If it was returning non-200, then Caddy will emit a log saying so, and it won’t try to get a cert from the CA.

3

Hi Matt,

Thanks for the response!

I’m fairly confident that the DNS changes had fully propagated at this point - but I agree that potentially some requests could have come through?

I did spot after your point that I was incorrect about my statement about our ask endpoint returning a non-200 response for this domain - apologies! I’ve corrected this and I’ll monitor.

1 Like
4

Thanks @matt - that seemed to have been the case!

1 Like
5

Great :slight_smile: Glad it’s fixed.

6

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.