Caddy2 ratelimit by letsencrypt

Help
1

1. My Caddy version (caddy -version):

v2.0.0-beta6 h1:tGZaM3NfxlBZhllJYKEehYYY9SMOyz8UNjMBoYALaT4=

2. How I run Caddy:

caddy is installed in /usr/local/bin/caddy

a. System environment:

centos-release-7-7.1908.0.el7.centos.x86_64

systemd 219

b. Command:

caddy start -adapter caddyfile -config /caddy/caddy.conf

c. Service/unit/compose file:

> no compose file

d. My complete Caddyfile:

www.domain2.com:443, domain2.com:443 {
	reverse_proxy / ip-address:port
}

3. The problem I’m having:

I have installed caddy version 2 yesterday. Experimented with domain1.com as a load balancer / reverse proxy and it was working great. Happy with it.

Today I tried to use with domain2.com and noticed the error rate limit applied by letsencrypt . One thing I forgot was pointing the domain to the IP address when I tried to start caddy.

4. Error messages and/or full log output:

2019/10/22 05:21:19 Caddy 2 admin endpoint listening on localhost:2019
2019/10/22 05:21:19 [INFO] Enabling automatic TLS certificate management for [www.domain2.com domain2.com]
2019/10/22 05:21:19 [INFO][cache:0xc000099a40] Started certificate maintenance routine
2019/10/22 05:21:19 [INFO][www.domain2.com] Obtain certificate
2019/10/22 05:21:19 [INFO] [www.domain2.com] acme: Obtaining bundled SAN certificate
2019/10/22 05:21:19 [ERROR][www.domain2.com] failed to obtain certificate: acme: error: 429 :: POST :: https://acme-v02.api.letsencrypt.org/acme/new-order :: urn:ietf:params:acme:error:rateLimited :: Error creating new order :: too many failed authorizations recently: see Rate Limits - Let's Encrypt, url: (attempt 1/3; challenge=http-01)
2019/10/22 05:21:20 [INFO] [www.domain2.com] acme: Obtaining bundled SAN certificate
2019/10/22 05:21:20 [ERROR][www.domain2.com] failed to obtain certificate: acme: error: 429 :: POST :: https://acme-v02.api.letsencrypt.org/acme/new-order :: urn:ietf:params:acme:error:rateLimited :: Error creating new order :: too many failed authorizations recently: see Rate Limits - Let's Encrypt, url: (attempt 2/3; challenge=http-01)
2019/10/22 05:21:21 [INFO] [www.domain2.com] acme: Obtaining bundled SAN certificate
2019/10/22 05:21:21 [ERROR][www.domain2.com] failed to obtain certificate: acme: error: 429 :: POST :: https://acme-v02.api.letsencrypt.org/acme/new-order :: urn:ietf:params:acme:error:rateLimited :: Error creating new order :: too many failed authorizations recently: see Rate Limits - Let's Encrypt, url: (attempt 3/3; challenge=http-01)
2019/10/22 05:21:22 [INFO] [www.domain2.com] acme: Obtaining bundled SAN certificate
2019/10/22 05:21:23 [ERROR][www.domain2.com] failed to obtain certificate: acme: error: 429 :: POST :: https://acme-v02.api.letsencrypt.org/acme/new-order :: urn:ietf:params:acme:error:rateLimited :: Error creating new order :: too many failed authorizations recently: see Rate Limits - Let's Encrypt, url: (attempt 1/3; challenge=tls-alpn-01)
2019/10/22 05:21:24 [INFO] [www.domain2.com] acme: Obtaining bundled SAN certificate
2019/10/22 05:21:24 [ERROR][www.domain2.com] failed to obtain certificate: acme: error: 429 :: POST :: https://acme-v02.api.letsencrypt.org/acme/new-order :: urn:ietf:params:acme:error:rateLimited :: Error creating new order :: too many failed authorizations recently: see Rate Limits - Let's Encrypt, url: (attempt 2/3; challenge=tls-alpn-01)
2019/10/22 05:21:25 [INFO] [www.domain2.com] acme: Obtaining bundled SAN certificate
2019/10/22 05:21:25 [ERROR][www.domain2.com] failed to obtain certificate: acme: error: 429 :: POST :: https://acme-v02.api.letsencrypt.org/acme/new-order :: urn:ietf:params:acme:error:rateLimited :: Error creating new order :: too many failed authorizations recently: see Rate Limits - Let's Encrypt, url: (attempt 3/3; challenge=tls-alpn-01)

5. What I already tried:

I pointed the domain domain2.com to an entirely new ip address.

Tried starting domain2.com via caddy. But it still shows the letsencrypt rate limit has occurred. On further checks I noticed a message as

2019/10/22 05:45:53 http: TLS handshake error from ip-address:46502: tls: first record does not look like a TLS handshake

I have noticed this error message many times when I started caddy yesterday. I wonder whether this has something to do with rate limit or due to the domain was not pointed to the ip when I tried to start caddy caused the rate limit issue.

It would have been nice if caddy don’t exceed too many requests when we issue only a single command like caddy start if the domain ip is not pointed.

6. Links to relevant resources:

One last question is what will happen if we have multiple domains in single caddyfile and an ssl certificate renewal failed due to some reason, does all servers goes down / fail to reload ?

2

Thanks for trying the Caddy 2 beta!

It looks like you attempted to get certificates for one or more hostname that weren’t properly configured:

2019/10/22 05:21:19 [ERROR][www.domain2.com] failed to obtain certificate: acme: error: 429 :: POST :: https://acme-v02.api.letsencrypt.org/acme/new-order :: urn:ietf:params:acme:error:rateLimited :: Error creating new order :: too many failed authorizations recently: see https://letsencrypt.org/docs/rate-limits/, url: (attempt 1/3; challenge=http-01)

You can avoid this in the future by using the staging endpoint for testing: https://github.com/caddyserver/caddy/tree/v2#how-do-i-avoid-lets-encrypt-rate-limits-with-caddy-2

Unfortunately because you redacted the domain names (against our instructions) there is nothing more we can do to help you.

3

Just to pop in on this one, too - junk traffic is not unexpected when you host web-accessible services. It’s likely just someone trying to connect to random ports on the internet. It certainly won’t interfere with the ACME challenge process. As Matt said, this one’s a good old fashioned misconfiguration. Staging endpoint and don’t auto-restart Caddy on a failure will prevent this happening if misconfigurations happen again in future (happens to the best of us).

1 Like
4

FWIW, the next beta has async cert operations and even smarter throttling, but it still won’t save you from CA-enforced rate limits if your DNS is misconfigured – you’ll have to check the logs and be sure there are no errors, and if there are, get them fixed ASAP. (Exponential backoff will help some, but it won’t be as fast as just getting the configurations right in the first place.)

5

Just curious, what information can you get if the domain name is real?

From a different thread I have checked the website https://check-your-website.server-daten.de/ and https://crt.sh . But didn’t noticed any issues.

  1. Is there a way we can check whether the domain is misconfigured ?
  2. Do we have an option to set the number of times caddy can request for certificates when it fails?

The problem is once it gets rate limit applied, we need to wait for 7 days.

A flag would be really nice than requesting multiple times when the failure happens.

6

Yes, I haven’t thought I would get a rate limit like this very fast :frowning: . There is only 2 domains I tried from the IP.

In version 1, there is a CADDY_PATH , in version 2 is there something like that ?

Regarding auto-restart I am not sure what information we need to pass to turn it off in version 2.

I am wondering where is Caddy storing the certificates in version 2. Is there a way we can tell caddy to list the certificates ?

7

Pretty much everything we need to diagnose the problem: DNS records, IP lookups, HTTP responses, packet routing info, and (for now) order/authz details, etc. Basically any and all “Network Engineering 101” troubleshooting steps to perform an actual investigation, whereas right now all we can do is say, “Yep, that sucks, something is misconfigured.”

Also, when domains are redacted, it’s impossible for us to know which code paths were executed and which rate limits actually apply. It basically full-stops any rigorous troubleshooting process.

Make sure your DNS and firewalls are properly set up for the ACME challenges. There are docs at https://caddyserver.com/v1/docs/automatic-https which explain what is needed.

If you can’t verify your setup manually for whatever good reasons, then automated checks will require some significant effort on your end, which are still not guaranteed to be accurate predictors. For that reason we don’t recommend this. But if you insist, you’ll need one or more (preferably multiple) external vantage points, or in other words, servers outside your site’s network (and preferably on a different provider and in a different region). They’ll need to run ACME servers. Then from each of the machines you want to validate, you’ll need to conduct an ACME transaction with each vantage point. This causes your ACME servers to perform authoritative DNS lookups for your domains and, depending on the challenge type, establish special TLS or HTTP connections with your machines (or none at all if using the DNS challenge). If all of that works and your own ACME servers can issue a certificate, then it is likely (but not guaranteed) that Let’s Encrypt will, too.

This gets even more tricky considering the nuances of individual DNS providers, which have different propagation times and other guarantees (or lack thereof).

That’s a lot of work for very little reward, so we recommend just making sure that your ports are configured properly and your DNS records, too.

No, since the context in which a request fails varies, and the context changes the error handling, and this is too difficult to configure manually.

Not always, depends on the rate limit: Rate Limits - Let's Encrypt

Always test with some staging environment, never with their production endpoint. That’s in our docs too. :slight_smile:

Yes and no. Caddy 2 doesn’t use env variables except for HOME or XDG_DATA_DIR (if set). But you can customize the storage location, of course. It’s just not via env vars.

This actually has nothing to do with Caddy. As our docs say, if the process exits with a status code of 1, don’t automatically restart. It’s up to however you are running the caddy process.

Again, this is in the documentation: GitHub - caddyserver/caddy: Fast and extensible multi-platform HTTP/1-2-3 web server with automatic HTTPS

Configures Caddy’s default storage module. A storage module defines how and where Caddy stores assets (such as TLS certificates).

Default: The local file system (caddy.storage.file_system). If XDG_DATA_HOME is set, then $XDG_DATA_HOME/caddy is the folder. Otherwise, $HOME/.local/share/caddy is the folder.

:slight_smile: Hope this helps!

1 Like
8

Thank you for the detailed response. I am sorry that I am unable to share the necessary details for debugging it. If it was my domain I would have.

9

This topic was automatically closed 90 days after the last reply. New replies are no longer allowed.