How to renew a lot of ssl certificates fast?

Help
1

1. Caddy version (caddy version):

v2.4.3 h1:Y1FaV2N4WO3rBqxSYA8UZsZTQdN+PwcoOcAiZTM8C0I=

2. How I run Caddy:

a. System environment:

No LSB modules are available.
Distributor ID: Ubuntu
Description: Ubuntu 20.04.2 LTS
Release: 20.04
Codename: focal

b. Command:

Paste command here.

c. Service/unit/compose file:

Paste full file contents here.
Make sure backticks stay on their own lines,
and the post looks nice in the preview pane.

d. My complete Caddyfile or JSON config:

{
  "apps": {
    "http": {
      "servers": {
        "myserver": {
          "listen": [
            ":443"
          ],
          "logs": {},
          "routes": [
            {
              "handle": [
                {
                  "handler": "reverse_proxy",
                  "upstreams": [
                    {
                      "dial": "localhost:8001"
                    }
                  ]
                }
              ]
            }
          ]
        }
      }
    },
    "tls": {
      "automation": {
        "on_demand": {
          "ask": "http://localhost:8001/proxyCheck"
        },
        "policies": [
          {
            "issuers": [
              {
                "api_key": "...",
                "email": "",
                "module": "zerossl"
              },
              {
                "email": "",
                "module": "acme"
              }
            ],
            "on_demand": true
          }
        ]
      }
    }
  },
  "logging": {
    "logs": {
      "default": {
        "writer": {
          "filename": "/var/log/caddy/caddy.log",
          "output": "file"
        }
      }
    }
  }
}


Paste config here, replacing this text.
Use `caddy fmt` to make it readable.
DO NOT REDACT anything except credentials.
LEAVE DOMAIN NAMES INTACT.
Make sure the backticks stay on their own lines.

3. The problem I’m having:

We got a really serious issue. Because of this Questions about Renewing before TLS-ALPN-01 Revocations - Help - Let's Encrypt Community Support
About 7500 of our SSL certificates have been revoked.
We try to renew it asap but it taking a lot of time. Is it anyway how we can speed up this process?

4. Error messages and/or full log output:

5. What I already tried:

I was trying to add multiple issuers in config

6. Links to relevant resources:

2

Please upgrade to v2.4.6!

The latest version of Caddy has an increased speed for the internal certificate event rate limiter from 20 per minute up to 10 per 10 seconds (effectively triple the speed).

So at 10/10s, it should take 125 minutes (2h5min) to re-issue 7500 certs.

3

Sorry to hear about the trouble you’re having.

Like Francis said, the latest version of Caddy lets things move along a little faster. However, you’d still be subject to any CA rate limits (ZeroSSL doesn’t have any that I know of; Let’s Encrypt might limit to 300 orders every 3 hours.)

Can you please post your full logs here as the help template asks for? (Not just the access logs.) I can only guess otherwise. We’ll need the logs for the last 3-4 days or so. (If too long, please link to them instead.)

The easiest way to force certs to renew is to simply move them into a different folder (don’t delete, in case you need to get them back for some reason).

4

It’s possible that on-demand certificates don’t currently benefit from auto-renewal in the case of revocation.

When I first implemented the auto-replace feature it was not clear whether on-demand TLS is important, because Caddy does not have telemetry (primarily due to public backlash) to tell us how widely the feature is being used. When I saw that integrating this niche feature into a niche mode of operation was going to be non-trivial, I ended up shelving the idea for another time, if there was some demand for it.

Fast forward to today, and based on feedback over the last few months it seems clear to me that on-demand TLS is of greater importance than I had anticipated. Sorry I didn’t implement this sooner. With the lack of telemetry, this is why it is helpful for businesses to sponsor Caddy’s development and to have a relationship with the maintainers so that we can know how they are relying on the software to better serve their businesses. Back then, I only implemented this feature as a party trick, not because any businesses were actually relying on it.

Here is the untested patch:

5

I update caddy to the latest - v2.4.6 h1:HGkGICFGvyrodcqOOclHKfvJC0qTU7vny/7FhYp9hNw=

Here are the latest logs

https://drive.google.com/file/d/1C7M-0qDnji8shMR0B1BwiNCR_oI89wFx/view?usp=sharing

6

I think that I forgot to add Caddy doesn`t update our renewal certificates. And i force it by deleting this certificates from a file system. Now for us important to renew it asap.

7

What do you mean by this? I don’t understand.

That’ll force a renewal, but in general it’s safer to simply rename the folder on disk so you can still have them if you need them. For future reference.

Caddy will renew ASAP, but only as fast as CA rate limits and other networking constraints allow.

8

We still have an issue that some certificates are still in the queue that is not updated. We use zerossl it seems that it shouldn’t have any limmitation.

Is it possible to prioritize certificate creation for some domains?

Matt it will be possible to have a call with you to help us optimize the config?

9

Is it still constantly issuing certs? How does it look in your logs?

If you went the route of deleting all of them, then no, you’ll essentially be limited by Caddy’s internal rate limiter.

Are you a sponsor? Sponsor @mholt on GitHub Sponsors · GitHub

10

Hey, I saw your sponsorship. Thank you very much. Yes, we can do a call. But I’m getting married tomorrow…

If we’re going to do it can we do it today? I won’t be available for the next 4 days, probably.

11

Since today - yes

I found that some domains don`t propagate for a few days but all logs that I found it is it

Feb 02 14:41:44 api1-on-demend-fixed-latest caddy[3993579]: 2022/02/02 14:41:44 [INFO][FileStorage:/var/lib/caddy/.local/share/caddy] Lock for 'issue_cert_DOMAIN' is stale (created: 2022-02-02 14:39:49.539145176 +0000 UTC, last update: 2022-02-02 14:41:34.616606671 +0000 UTC); removing then retrying: /var/lib/caddy/.local/share/caddy/locks/issue_cert_DOMAIN.lock

and lock file constantly in share/caddy/locks that don`t seem right

12

it can be great if we can have a call asap, how I can reach you for a call?

p.s. congrats on the wedding

13

I’ll DM you soon, thanks

14

Haha that’s dedication working on this issue the day before your marriage! Thank you!

2 Likes
15

This topic was automatically closed after 30 days. New replies are no longer allowed.