I just got this email from Let’s Encrypt:
Please immediately renew your TLS certificate(s) that were issued from
Let’s Encrypt using the TLS-ALPN-01 validation method and the following
ACME registration (account) ID(s):
We’ve determined that an error made it possible for TLS-ALPN-01
challenges, completed before today, to not comply with certificate
issuance requirements. We have remediated this problem and will revoke
all unexpired certificates that used this validation method at 16:00 UTC
on 28 January 2022. Please renew your certificates now to ensure an
uninterrupted experience for your site visitors.
Unfortunately I use the same email across several servers so I’m not sure which server this is messages is referencing.
Can anyone tell me where I look to find the hostnames tied to the above Let’s Encrypt account ids?
Or how to ask Caddy to force renew all certificates?
Many thanks for any pointers.
No action required. Caddy takes care of it on its own.
I’m bit confused whether certs will definitely be renewed before they expire? From the twitter thread, it sounds like it could be some days after?
Do we have to do something else to ensure they renew in time? We are using the DynamoDB storage plugin, in case relevant
You can force renewal by deleting the record in the storage then reloading or restarting Caddy.
Thanks for the reply, but delete what record exactly? Do you have any recommendations on how to do that with the dynamodb storage option? It doesn’t seem as straightforward as deleting a “ocsp” folder say, which I’ve seen mentioned elsewhere
Delete the records corresponding to the domain name whose certificate and private key are stored in it as its content. I’m not familiar with DynamoDB, but the way to force Caddy to obtain new certificates is to remove them from its storage backend.
We use Caddy’s dynamic TLS for a SaaS app, so we have 100s of certs. So this approach doesn’t seem practical for us (and I’m guessing many others), even if I knew exactly what records to delete from DynamoDB (and I’m very hesitant to just go in and delete random records).
Is there a safer, more comprehensive solution to this?
Could you be a bit more specific? When exactly will caddy renew the certs exactly?
I assume it would be better to renew the certs before the revocation to avoid problems.
I’m on the same position. I really hate the whole thing, including communication from Let’s encrypt.
We use version 2.3 with custom modules and redis storage. Upgrading to 2.4 when doing 10 req/s in 2 days without doesn’t let much room for testing if all.
If we upgrade and restart, will all certificates renew automatically… what if we have 1000 certificates? will that triggers some rate limits on Let’s encrypt?
we have multiple servers, we want to try take one server out of the load balance, upgade to 2.4.3 and test it with a few accounts. after upgrading (i guess it’s just replacing executable and restarting the service) will all 1000+ certificates start to renew at once? If yes, will the 2.3.0 version on the other servers will work with certificates issues by 2.4.3 version as they share the same redis storage?
Regarding deleting the certs, this means that our users will have a delay until they got renewed?
Also, in our redis storage we have entries for a domain:
Should we delete all of them? Isn’t a way to force renewal without deleting and having downtime?
I would recommend taking a look at JSON Config Structure › apps › tls › automation › policies › renewal_window_ratio.
apps->tls->automation->policies[...]->renewal_window_ratio to a high value you can start renewing all your certificates immediately. As always YMMV but I managed to start renewing a bunch of certificates we happen to use.
what value should we set to renew them right away?
I wish there is a command on caddy to manually trigger renewal.
I think this should be the best approach as it won’t take our users website down until it renews… some users have lots of domains with traffic on them, it’s not easy to delete all of them and hope everything works.
if we can trigger or force a renewal without messing with existing certs will be amazing.
Same here. Running Caddy
v2.2.2 h1:Ha3bvEvkb/GLGEX648/qI5zTt6uJCnfQhZHmZBxhzDY= for a SaaS business with some custom-built modules and a little bit below 100 custom domains going through caddy. Some domains are not owned by us. Certificates are stored on AWS EFS to share it between loadbalanced caddy instances.
$ ls -la /mnt/efs/fs1/caddy/certificates/acme-v02.api.letsencrypt.org-directory/ | wc -l
Luckily we use “on_demand_tls” so I do not expect any downtime except an initial slow request per domain. In our case those single domains are not frequently visited (5-10 reqs/min).
I tested one domain and it worked like a charm.
$ mv /mnt/efs/fs1/caddy/certificates/acme-v02.api.letsencrypt.org-directory/random-domain-from.com /backup/caddy/random-domain-from.com
$ service caddy stop
$ service caddy start
The first request against this (and only this domain) performed the certificate renewal.
Will delete all certificates now and let caddy do it’s amazing job!
@matt thank you so much for bringing this amazing piece of stable and reliable software!
Everyone… it can take a few days for Caddy to detect the revocation and do the replacement. Caddy will DEFINITELY not let them expire, it won’t even let their OCSP staple expire. OCSP staples are usually valid for just a few days to a week, and Caddy refreshes them halfway through, so if will see the Revoked status in at most a few days once they are revoked. In the meantime Caddy will continue to serve Good OCSP staples.
You should be fine even if it takes a few days. Remember that Let’s Encrypt won’t revoke for another day or so yet.
It a-okay to be proactive and take extra initiative, of course. Especially if your users have clients that may not honor valid, signed OCSP responses. In that case just delete the certificates from Caddy’s storage and reload, and Caddy will right away get new certificates.
(I’m of course only talking about Caddy instances from the last ~6 months or so since this feature was released. V2.4.2 or higher. If you’re not already on the latest version, please upgrade!)
That is one way to do it, but the right value depends on you. You can set to 1 and it will always renew every time it scans certificates. Of course in production this will quickly run you up against duplicate cert rate limits enforced by Let’s Encrypt. But 1 will guarantee it renews them all. Just be sure to remove the custom value right away afterwards.
Thanks for chiming in.
Our clients require a high uptime rate and having lots of certificates, we can’t risk anything. We saw many times that we get too many requests error and new certificates don’t issue for hours, we can’t let this happen for clients where their sales are directly affected by this.
How often the certificates are scanned? So if i set it to 1 when can we expect to trigger the renewal?
We have like 1000+ certificates which should be renewed until Friday. we also can’t afford downtime.
Hard to say, but it’s unlikely. They’ll renew as their OCSP staples become stale. OCSP staples are scanned every hour. If they were all obtained at the same time or stapled OCSP at the same time, then yes they’d all renew at the same time, but I figure this is unlikely.
Btw some LE rate limits don’t apply to renewals.
Yes, but they won’t know to renew the cert so they won’t even look in storage. You can reload them though and they’ll get the new certs. Still, please keep Caddy up to date in the future.
So this is what I’ll do:
- take one server out of load balancer an upgrade caddy to latest version
- do some tests to make sure everything works, including custom modules
- if it works, upgrade caddy on all servers
- once that’s done, set renewal_window_ratio=1 for one of the servers , reload caddy and wait 1-2 hours
- pray that 1000+ clients websites that host their checkout pages don’t go offline and won’t get myself and our clients screwed
OCSP staples are scanned every hour. But if you change config (by setting that value to 1, for example) Caddy will immediately renew the certificate regardless of scan time, because it always checks when the config is first loaded.
Only that many? That’s fine. Remember that downtime is a function of both server, client, and CA when it comes to certificates. Caddy will do its best to not let your sites go down. There are a number of factors though that affect this which you may not be able to control, to name just two for example:
- CA rate limits (1000 won’t be too much of a problem, maybe 300 orders every 3 hours, but Caddy will just retry until it can keep getting more, including trying ZeroSSL as a fallback)
- Clients that may ignore or reject signed, valid OCSP staples – not much you can do about their trust decisions unless they make it configurable (unlikely)
Seeing a lot of users that rely on Caddy for their business. That’s awesome! I’d recommend please sponsoring the project so we can better help you. This feature was only possible in the first place thanks to sponsors like ZeroSSL and several others:
thanks. just to be extra cautious, if i change the value to 1 on renewal_window_ratio and do a sudo systemctl caddy reload , is there any chance to break existing certificates?
I mean, “any chance” is a pretty wide descriptor so I hesitate to say absolutely “no” but I think in the general case, you’d be fine; just be sure to remove that value afterward.