Let's Encrypt hits rate limit: "Too many registrations for this IP"

Help
1

1. The problem I’m having:

I have a Caddy installation with circa 40 domains configured via Caddyfile. Since upgrading to v2.8.x, I am unable to access any of my sites due to an SSL error. For example:

$ curl -vL https://thomaspreece.net
*   Trying 132.145.12.162:443...
* Connected to thomaspreece.net (132.145.12.162) port 443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
*  CAfile: /etc/ssl/certs/ca-certificates.crt
*  CApath: /etc/ssl/certs
* TLSv1.0 (OUT), TLS header, Certificate Status (22):
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* TLSv1.2 (IN), TLS header, Unknown (21):
* TLSv1.3 (IN), TLS alert, internal error (592):
* error:0A000438:SSL routines::tlsv1 alert internal error
* Closing connection 0
curl: (35) error:0A000438:SSL routines::tlsv1 alert internal error

I downgraded to v2.7.6 in order to get all of my sites working again, so you won’t see this error if you try now.

Investigation showed that the problem seems to be that Caddy is unable to obtain Let’s Encrypt certificates. On v2.7.6 and below, it instead gets a certification from ZeroSSL and continues to function, but on v2.8.x this no longer works. I will look at configuring ZeroSSL on v2.8.x, but I want to resolve the issue with Let’s Encrypt first.

Looking at the logs shows repeated error messages of the following form:

caddy[150320]: {"level":"error","ts":1717338648.9697998,"logger":"tls.obtain","msg":"will retry","error":"[thomaspreece.net] Obtain: registering account [mailto:thomas@thomaspreece.net] with server: attempt 1: https://acme-v02.api.letsencrypt.org/acme/new-acct: HTTP 429  urn:ietf:params:acme:error:rateLimited - Error creating new account :: too many registrations for this IP: see https://letsencrypt.org/docs/too-many-registrations-for-this-ip/","attempt":1,"retrying_in":60,"elapsed":0.855544075,"max_duration":2592000}

It appears that the first 10 certificate requests succeed, and then all of the other ones (approx 30) fail. This is consistent with the document linked in the error message, which specifies that the rate limit is 10 accounts registered within 3 hours.

Presumably then Caddy is trying to register a new account for every certificate request, rather than creating one account and using it for every request.

To confirm that the SSL errors are caused by Caddy not being able to get certificates from Let’s Encrypt, I waited several hours for the rate limit timeout to expire, then added the following to my Caddyfile on v2.7.6 to force it to use Let’s Encrypt and not ZeroSSL:

{
       acme_ca https://acme-v02.api.letsencrypt.org/directory
}

After restarting Caddy, I get the same SSL errors as above, and the same errors in the log file about the “too many registrations”. It seems that by asking it only to obtain certificates from Let’s Encrypt, it won’t use the existing ZeroSSL certificates, which I assume is expected behaviour.

I commented out the acme_ca line and restarted Caddy again, and all sites were once again available using the ZeroSSL certificates.

On searching for information about this error, I found a recommendation to look at the .caddy directory containing all of the certificates etc (which on my system is in fact /var/lib/caddy/.local/share/caddy). I found the subdirectories certificates/acme.zerossl.com-v2-dv90 (containing certificates for all sites) and certificates/acme-v02.api.letsencrypt.org-directory (containing the 10 certificates that Caddy was able to obtain before being rate limited). So it seems that Caddy is able to save the certificates correctly.

I also found a recommendation to delete the .caddy directory, to force Caddy to re-issue every certificate from scratch. After waiting several hours again for the Let’s Encrypt timeout to expire, I did this (actually, renamed the directory caddy to caddy-old) and restarted Caddy. Again I got the SSL errors on all sites - it seems it was now unable to fail over to ZeroSSL and obtain certificates that way, so all of the sites were unavailable.

I put the old caddy directory back in place and once again restarted Caddy, and confirmed that all sites were back up using the previously saved certificates.

I have also found other references online to people having issues with Caddy trying to create a new account for every certificate request, but all of these appear to be related to bugs that were fixed upwards of 5 years ago.

2. Error messages and/or full log output:

The log extract was too long to include in a post. It is available here.

The logs are with the acme_ca line mentioned above in place in the Caddyfile, to force it to request certificates from Let’s Encrypt. The timestamps don’t line up with the narrative of waiting several hours since then before trying other steps, because I re-ran that step today to get a fresh set of logs.

3. Caddy version:

$ caddy version
v2.7.6 h1:w0NymbG2m9PcvKWsrXO6EEkY9Ru4FJK8uQbYcev1p3A=

4. How I installed and ran Caddy:

a. System environment:

$ lsb_release -a
No LSB modules are available.
Distributor ID: Ubuntu
Description:    Ubuntu 22.04.4 LTS
Release:        22.04
Codename:       jammy

Caddy is installed via apt from the following repository:

$ cat /etc/apt/sources.list.d/caddy-stable.list 
# Source: Caddy
# Site: https://github.com/caddyserver/caddy
# Repository: Caddy / stable
# Description: Fast, multi-platform web server with automatic HTTPS


deb [signed-by=/usr/share/keyrings/caddy-stable-archive-keyring.gpg] https://dl.cloudsmith.io/public/caddy/stable/deb/debian any-version main

deb-src [signed-by=/usr/share/keyrings/caddy-stable-archive-keyring.gpg] https://dl.cloudsmith.io/public/caddy/stable/deb/debian any-version main

b. Command:

$ sudo systemctl start caddy

c. Service/unit/compose file:

$ cat /usr/lib/systemd/system/caddy.service 
# caddy.service
#
# For using Caddy with a config file.
#
# Make sure the ExecStart and ExecReload commands are correct
# for your installation.
#
# See https://caddyserver.com/docs/install for instructions.
#
# WARNING: This service does not use the --resume flag, so if you
# use the API to make changes, they will be overwritten by the
# Caddyfile next time the service is restarted. If you intend to
# use Caddy's API to configure it, add the --resume flag to the
# `caddy run` command or use the caddy-api.service file instead.

[Unit]
Description=Caddy
Documentation=https://caddyserver.com/docs/
After=network.target network-online.target
Requires=network-online.target

[Service]
Type=notify
User=caddy
Group=caddy
ExecStart=/usr/bin/caddy run --environ --config /etc/caddy/Caddyfile
ExecReload=/usr/bin/caddy reload --config /etc/caddy/Caddyfile --force
TimeoutStopSec=5s
LimitNOFILE=1048576
LimitNPROC=512
PrivateTmp=true
ProtectSystem=full
AmbientCapabilities=CAP_NET_ADMIN CAP_NET_BIND_SERVICE

[Install]
WantedBy=multi-user.target

d. My complete Caddy config:

Available here.

1 Like
2

Hi @tepreece,

Is your IP Address being use by more than you?

Please see Rate Limits - Let’s Encrypt and Failed Validation Limit - Let’s Encrypt

Rate Limits - Let’s Encrypt states:
"We have two other limits that you’re very unlikely to run into.

You can create a maximum of 10 Accounts per IP Address per 3 hours. You can create a maximum of 500 Accounts per IP Range within an IPv6 /48 per 3 hours. Hitting either account rate limit is very rare, and we recommend that large integrators prefer a design using one account for many customers. Exceeding these limits is reported with the error message too many registrations for this IP or too many registrations for this IP range.

You can have a maximum of 300 Pending Authorizations on your account. Hitting this rate limit is rare, and happens most often when developing ACME clients. It usually means that your client is creating authorizations and not fulfilling them. Please utilize our staging environment if you’re developing an ACME client. Exceeding the Pending Authorizations limit is reported with the error message too many currently pending authorizations."

1 Like
3

I’m running Caddy on a VPS with a public IP address, so as far as I can see it’s the only machine with that IP address.

I did wonder whether it might be hitting the IP range account limit, but given that every time I try it, it successfully obtains exactly 10 certificates and then the rest all fail, it would need a huge set of coincidences for it to be anything other than the 10 account limit per IP address.

But either way, if Caddy was re-using the same account for each certificate request, it shouldn’t hit either of those limits.

4

@tepreece,

I also find all of these Domain Names map to the IPv4 Address of 132.145.12.162, thus any one of the could be the problem too.

a-night-in.com
copingresources.org
dreamologypictures.com
eomusic.thomaspreece.net
eomusichistory.com
familytree.thomaspreece.net
fearlessbillyjenkins.com
fedimedia.thomaspreece.net
gagazoid.com
lemmy.thomaspreece.net
mastodon.thomaspreece.net
maxdoescoverage.com
preecemusic.com
safeharbourexeter.org.uk
safeharbourmcc.co.uk
sass.thomaspreece.net
slayersfilm.com
stalledtheshortfilm.com
thefearlessbillyjenkins.com
thomaspreece.co.uk
thomaspreece.net
tpreece.net
5

Yes, those are all domains being hosted by my Caddy server.

1 Like
6

Hi @tepreece,

Not sure if this is part of the issue or not, but additional information.

Also Let’s Debug has added (and changed locations) the remote perspectives for domain validation.
Let’s Encrypt uses Multi-Perspective Validation Improves Domain Validation Security - Let’s Encrypt

Please read these:

7

@tepreece

And here SSL Server Test: thomaspreece.net (Powered by Qualys SSL Labs) I see something I have not seen before (that could just be my experience) “CRL ERROR: IOException occurred”.

Expanding the Hide Certification PathsCertification Paths

image
image1000×887 15.4 KB

8

That error doesn’t mean anything to me either unfortunately. I thought it might be useful to do the same test on one of the domains on the same server that did get a Let’s Encrypt certificate though: SSL Server Test: preecemusic.com (Powered by Qualys SSL Labs)

Same error, also on the number 2 certificate.

1 Like
9

I tried a sample of my domains on Let’s Debug. All of them say “All OK!” for all three methods.

1 Like
10

I’ve replaced the log extract with an unredacted version, and I’ve also uploaded my full unredacted Caddyfile, as neither of them contain anything you’ve not found out already.

1 Like
11

@tepreece,

Thank you for suppling the logs and configuration Caddyfile, I believe others will find the information useful.

Kindly wait for more knowledgeable Caddy community volunteers to assist.

1 Like
12

Hopefully someone will have an answer - thanks for your help so far!

1 Like
13

I’ll look at this in the morning… I’ve been absolutely exhausted this weekend. But I definitely want to figure this out. Not sure why that would be happening.

Thank you for the very thorough documentation up to this point.

2 Likes
14

Just a thought that may help with the timeline of when my Caddy installation started failing to get Let’s Encrypt certificates - I had two emails from the Let’s Encrypt Expiry Bot last month, stating that the certificate for fedimedia.thomaspreece.net would expire on 2024-05-10, and that the certificate for mastodon.thomaspreece.net would expire on 2024-05-11.

So that must mean that Caddy successfully obtained a Let’s Encrypt certificate on 11th February 2024, and wasn’t able to obtain one when it tried to renew the fedimedia.thomaspreece.net certificate (presumably on 10th April 2024?).

15

Please read Unexpected renewal failures since April 2024? Please read this! - Help - Let's Encrypt Community Support

2 Likes
16

To start, I just added more logging around ACME account creation, which previously had no logging to speak of (it’s some of the oldest code in Caddy/CertMagic at this point).

Still investigating. But if you do/can run with that commit (maybe using the LE Staging environment to avoid rate limits in production) and experience the behavior, it’s likely to reveal more clues as to why.

1 Like
17

@tepreece
The Staging Environment - Let's Encrypt information.
" The ACME URL for our ACME v2 staging environment is:

https://acme-staging-v02.api.letsencrypt.org/directory"

1 Like
18

Thanks for adding the additional logging. I built the modified version and ran it with the following additional lines in my Caddyfile:

{
        acme_ca https://acme-staging-v02.api.letsencrypt.org/directory
        debug
}

Here is the resulting log.

The following message appears 43 times, which I believe is the same as the number of configured domains:

Jun 04 12:28:04 instance-20230213-1453 caddy[247046]: {"level":"info","ts":1717504084.4736488,"logger":"tls.issuance.acme","msg":"creating new account because no account for configured email is known to us","email":"thomas@thomaspreece.net","ca":"https://acme-staging-v02.api.letsencrypt.org/directory","error":"open /var/lib/caddy/.local/share/caddy/acme/acme-staging-v02.api.letsencrypt.org-directory/users/thomas@thomaspreece.net/thomas.json: no such file or directory"}

I then looked to see whether that file was present:

caddy@instance-20230213-1453:~$ stat /var/lib/caddy/.local/share/caddy/acme/acme-staging-v02.api.letsencrypt.org-directory/users/thomas@thomaspreece.net/thomas.json
  File: /var/lib/caddy/.local/share/caddy/acme/acme-staging-v02.api.letsencrypt.org-directory/users/thomas@thomaspreece.net/thomas.json
  Size: 202             Blocks: 8          IO Block: 4096   regular file
Device: 801h/2049d      Inode: 1561890     Links: 1
Access: (0600/-rw-------)  Uid: (  998/   caddy)   Gid: (  999/   caddy)
Access: 2024-06-04 12:33:44.985384526 +0000
Modify: 2024-06-04 12:28:05.475124611 +0000
Change: 2024-06-04 12:28:05.475124611 +0000
 Birth: 2024-06-04 12:28:05.219122907 +0000
caddy@instance-20230213-1453:~$ cat /var/lib/caddy/.local/share/caddy/acme/acme-staging-v02.api.letsencrypt.org-directory/users/thomas@thomaspreece.net/thomas.json
{
        "status": "valid",
        "contact": [
                "mailto:thomas@thomaspreece.net"
        ],
        "termsOfServiceAgreed": true,
        "orders": "",
        "location": "https://acme-staging-v02.api.letsencrypt.org/acme/acct/150739144"

Having verified that the file was indeed present, I configured a new domain as follows:

test.thomaspreece.net {
        respond "Hello world"
}

I then restarted Caddy, to see whether it would find the account file. The log from this run is here.

Jun 04 12:51:19 instance-20230213-1453 caddy[248145]: {"level":"debug","ts":1717505479.481711,"logger":"tls.issuance.acme","msg":"using existing ACME account because key found in storage associated with email","email":"thomas@thomaspreece.net","ca":"https://acme-staging-v02.api.letsencrypt.org/directory"}

I then deleted the certificates directory to force it to re-issue all certificates, using the account file that I had just confirmed it could read. This time it was able to issue all the certificates, and the log file contained the expected 44 instances of “using existing ACME account”.

I then decided to try to issue certificates from the main Let’s Encrypt endpoint. I removed all but one site from my Caddyfile, restarted Caddy and verified that it was able to issue that certificate and create the account file. Of course the log for this shows that a new account was created, as expected. I then added one additional site to the Caddyfile and restarted Caddy, and the log file shows that it was able to use the existing account. I then added all of the remaining sites to the Caddyfile, restarted Caddy, and certificates were issued successfully for all of them, using the existing account.

So it looks to me that the problem is that when the account is created, it takes longer for the account file to get saved to disk than it does for the next 42 domains to attempt to read it and fail to find it.

I’ve got it working, at least for now. How long-lived are Let’s Encrypt accounts? ie, will the account expire at some point, and then I’ll have the same problem again?

I’m of course happy to help with any further investigation into the cause and any possible fixes for this. Many thanks for your help!

2 Likes
19

Wow, thank you for the deep dive and the details/commentary! That extra context is really useful and will make this go much quicker.

They never expire, so that’s good news.

Let me digest what’s going on here and see why it’s not happening for everyone (including me) and what we can do about it…

2 Likes
20

@tepreece Would you be able to try a branch of CertMagic?

I am not sure why this has not come up before… whether a recent change just exposed this bug or you’re incredibly unlucky… but I decided to explicitly implement a lock around account registration kind of like we do around cert obtaining, so that it only happens once.

Would you be able to try CertMagic at the account-lock branch?

2 Likes