Seeing "Too many new orders recently" occasionally

Hi

Caddy is awesome and I have been using it for my startup Hashnode for 1 year now. Never had any issues. Kudos to the awesome work by the team.

However, for the last couple of days, I am facing a weird issue. Here are the details:

1. Caddy version (caddy version):

1.0

2. How I run Caddy:

We start Caddy using the following command:

nohup caddy -log access.log

a. System environment:

OS: Ubuntu 16.04

b. Command:

nohup caddy -log access.log

c. Service/unit/compose file:

None

d. My complete Caddyfile or JSON config:

* {
    proxy / localhost:3000 {
        transparent
    }
    gzip
    tls {
        ask <ask-url>
    }
}

*.hashnode.dev:443 {
    proxy / localhost:3000 {
        transparent
    }
    gzip
    tls fullchain.pem privkey.pem
}

3. The problem I’m having:

We are running Caddy in fleet mode (total 5 servers). A common disk is mounted on .caddy/acme via SSHFS to share SSL certs. Since we have a multi-tenant system, there are around 2500 different domains. Everything was smooth so far, but lately, I have been seeing the error “Too many new orders recently” while requesting certs for new domains. This happens occasionally. As per LE, this error comes when you exceed 300 orders / 3 hrs limit. But we are not doing that many requests in 3 hours. I checked our server logs to verify this.

What could be the reason?

4. Error messages and/or full log output:

2020/08/03 18:34:42 http: TLS handshake error from 127.0.0.1:43364: EOF
2020/08/03 18:34:44 [ERROR] Renewing [hn.werick.codes]: acme: error: 429 :: POST :: https://acme-v02.api.letsencrypt.org/acme/new-order :: urn:ietf:params:acme:error:rateLimited :: Error creating new order :: too many failed authorizations recently: see https://letsencrypt.org/docs/rate-limits/, url: ; trying again in 10s
2020/08/03 18:34:55 [ERROR] Renewing [hn.werick.codes]: acme: error: 429 :: POST :: https://acme-v02.api.letsencrypt.org/acme/new-order :: urn:ietf:params:acme:error:rateLimited :: Error creating new order :: too many failed authorizations recently: see https://letsencrypt.org/docs/rate-limits/, url: ; trying again in 10s
2020/08/03 18:35:05 [ERROR] too many renewal attempts; last error: acme: error: 429 :: POST :: https://acme-v02.api.letsencrypt.org/acme/new-order :: urn:ietf:params:acme:error:rateLimited :: Error creating new order :: too many failed authorizations recently: see https://letsencrypt.org/docs/rate-limits/, url: 
2020/08/03 18:35:05 [INFO] Certificate for [blog.amnota.dev] expires in -1140h50m18.035994548s; attempting renewal
2020/08/03 18:35:09 http: TLS handshake error from 127.0.0.1:43372: EOF
2020/08/03 18:35:12 [ERROR] Renewing [blog.amnota.dev]: acme: error: 429 :: POST :: https://acme-v02.api.letsencrypt.org/acme/new-order :: urn:ietf:params:acme:error:rateLimited :: Error creating new order :: too many failed authorizations recently: see https://letsencrypt.org/docs/rate-limits/, url: ; trying again in 10s
2020/08/03 18:35:22 [ERROR] Renewing [blog.amnota.dev]: acme: error: 429 :: POST :: https://acme-v02.api.letsencrypt.org/acme/new-order :: urn:ietf:params:acme:error:rateLimited :: Error creating new order :: too many failed authorizations recently: see https://letsencrypt.org/docs/rate-limits/, url: ; trying again in 10s
2020/08/03 18:35:32 [ERROR] too many renewal attempts; last error: acme: error: 429 :: POST :: https://acme-v02.api.letsencrypt.org/acme/new-order :: urn:ietf:params:acme:error:rateLimited :: Error creating new order :: too many failed authorizations recently: see https://letsencrypt.org/docs/rate-limits/, url: 
2020/08/03 18:35:32 [INFO][cache:0xc0000b87d0] Done scanning certificates
2020/08/03 18:35:32 [INFO][cache:0xc0000b87d0] Scanning for stale OCSP staples
2020/08/03 18:35:32 [INFO] Advancing OCSP staple for [blog.herce.co] from 2020-08-07 06:00:00 +0000 UTC to 2020-08-10 06:00:00 +0000 UTC
2020/08/03 18:35:33 [INFO] Advancing OCSP staple for [blog.bytefly.io] from 2020-08-07 06:00:00 +0000 UTC to 2020-08-10 06:00:00 +0000 UTC
2020/08/03 18:35:33 [INFO] Advancing OCSP staple for [juantobias.dev] from 2020-08-07 06:00:00 +0000 UTC to 2020-08-10 06:00:00 +0000 UTC
2020/08/03 18:35:33 [INFO] Advancing OCSP staple for [til.agentofuser.com] from 2020-08-07 06:00:00 +0000 UTC to 2020-08-10 06:00:00 +0000 UTC
2020/08/03 18:35:34 [INFO] Advancing OCSP staple for [cloudy.achakladar.com] from 2020-08-07 06:00:00 +0000 UTC to 2020-08-10 06:00:00 +0000 UTC
2020/08/03 18:35:34 [INFO] Advancing OCSP staple for [jordanfarrer.com] from 2020-08-07 06:00:00 +0000 UTC to 2020-08-10 06:00:00 +0000 UTC
2020/08/03 18:35:34 [INFO] Advancing OCSP staple for [risav.dev] from 2020-08-07 06:00:00 +0000 UTC to 2020-08-10 06:00:00 +0000 UTC
2020/08/03 18:35:35 [INFO] Advancing OCSP staple for [blog.levippaul.com] from 2020-08-07 06:00:00 +0000 UTC to 2020-08-10 06:00:00 +0000 UTC
2020/08/03 18:35:35 [INFO] Advancing OCSP staple for [blog.joostvandergaag.com] from 2020-08-07 06:00:00 +0000 UTC to 2020-08-10 06:00:00 +0000 UTC
2020/08/03 18:35:35 [INFO] Advancing OCSP staple for [blog.collinsruto.com] from 2020-08-07 06:00:00 +0000 UTC to 2020-08-10 06:00:00 +0000 UTC
2020/08/03 18:35:36 [INFO] Advancing OCSP staple for [blog.jarda.it] from 2020-08-07 06:00:00 +0000 UTC to 2020-08-10 06:00:00 +0000 UTC
2020/08/03 18:35:36 [INFO] Advancing OCSP staple for [devblog.qcobjects.org] from 2020-08-07 06:00:00 +0000 UTC to 2020-08-10 06:00:00 +0000 UTC
2020/08/03 18:35:36 [INFO] Advancing OCSP staple for [dev.liuweifeng.net] from 2020-08-07 06:00:00 +0000 UTC to 2020-08-10 06:00:00 +0000 UTC
2020/08/03 18:35:37 [INFO] Advancing OCSP staple for [blog.prokop.dev] from 2020-08-07 06:00:00 +0000 UTC to 2020-08-10 06:00:00 +0000 UTC
2020/08/03 18:35:37 [INFO] Advancing OCSP staple for [blog.leojacon.tech] from 2020-08-07 06:00:00 +0000 UTC to 2020-08-10 06:00:00 +0000 UTC
2020/08/03 18:35:37 [INFO] Advancing OCSP staple for [blog.athreyapatel.codes] from 2020-08-07 06:00:00 +0000 UTC to 2020-08-10 06:00:00 +0000 UTC
2020/08/03 18:35:37 [INFO] Advancing OCSP staple for [blog.cloudnative-labs.nl] from 2020-08-07 06:00:00 +0000 UTC to 2020-08-10 06:00:00 +0000 UTC
2020/08/03 18:35:38 [INFO] Advancing OCSP staple for [blog.mikeattara.com] from 2020-08-07 06:00:00 +0000 UTC to 2020-08-10 06:00:00 +0000 UTC
2020/08/03 18:35:38 [INFO] Advancing OCSP staple for [techoodle.vidhyaranganathan.com] from 2020-08-07 06:00:00 +0000 UTC to 2020-08-10 06:00:00 +0000 UTC
2020/08/03 18:35:38 [INFO] Advancing OCSP staple for [blog.akush.in] from 2020-08-07 06:00:00 +0000 UTC to 2020-08-10 06:00:00 +0000 UTC
2020/08/03 18:35:39 [INFO] Advancing OCSP staple for [blog.romanjasek.com] from 2020-08-07 06:00:00 +0000 UTC to 2020-08-10 06:00:00 +0000 UTC
2020/08/03 18:35:39 [INFO] Advancing OCSP staple for [blog.chrisjohn.digital] from 2020-08-07 06:00:00 +0000 UTC to 2020-08-10 06:00:00 +0000 UTC
2020/08/03 18:35:39 [INFO] Advancing OCSP staple for [fiq.me] from 2020-08-07 06:00:00 +0000 UTC to 2020-08-10 06:00:00 +0000 UTC
2020/08/03 18:35:40 [INFO] Advancing OCSP staple for [blog.yeshu.in] from 2020-08-07 06:00:00 +0000 UTC to 2020-08-10 06:00:00 +0000 UTC
2020/08/03 18:35:40 [ERROR] Checking OCSP: invalid: OCSP response for [hash.imfancy.cn] valid after certificate expiration (-24h6m36s)
2020/08/03 18:35:40 [INFO] Advancing OCSP staple for [abrahamnm.com] from 2020-08-07 06:00:00 +0000 UTC to 2020-08-10 06:00:00 +0000 UTC
2020/08/03 18:35:41 [INFO] Advancing OCSP staple for [blog.vaishnavibaliga.com] from 2020-08-07 06:00:00 +0000 UTC to 2020-08-10 06:00:00 +0000 UTC
2020/08/03 18:35:41 [INFO] Advancing OCSP staple for [saltgen.dev] from 2020-08-07 06:00:00 +0000 UTC to 2020-08-10 06:00:00 +0000 UTC
2020/08/03 18:35:41 [INFO][cache:0xc0000b87d0] Done checking OCSP staples
2020/08/03 18:35:44 [ERROR] Sending telemetry: Post https://telemetry.caddyserver.com/v1/update/eb54bcd5-4dc0-42f4-9575-5a23cea6efe7: dial tcp: lookup telemetry.caddyserver.com on 67.207.67.3:53: no such host
2020/08/03 18:38:38 http: TLS handshake error from 18.212.228.7:47866: certificate for hostname '' not allowed, non-2xx status code 404 returned from <ask-url>
2020/08/03 19:14:32 [INFO][cache:0xc0000b87d0] Scanning for stale OCSP staples
2020/08/03 19:14:32 [INFO] Advancing OCSP staple for [blog.rakeshyadav.info] from 2020-08-07 07:00:00 +0000 UTC to 2020-08-10 07:00:00 +0000 UTC
2020/08/03 19:14:32 [INFO] Advancing OCSP staple for [blog.sireto.io] from 2020-08-07 07:00:00 +0000 UTC to 2020-08-10 07:00:00 +0000 UTC
2020/08/03 19:14:32 [ERROR] Checking OCSP: invalid: OCSP response for [hash.imfancy.cn] valid after certificate expiration (-24h6m36s)
2020/08/03 19:14:33 [INFO] Advancing OCSP staple for [blog.rafedraper.de] from 2020-08-07 07:00:00 +0000 UTC to 2020-08-10 07:00:00 +0000 UTC
2020/08/03 19:14:33 [INFO] Advancing OCSP staple for [hn.werick.codes] from 2020-08-07 07:00:00 +0000 UTC to 2020-08-10 07:00:00 +0000 UTC
2020/08/03 19:14:33 [INFO][cache:0xc0000b87d0] Done checking OCSP staples
2020/08/03 19:21:29 [INFO] Obtaining new certificate for 20vueapps.com
2020/08/03 19:21:30 http: TLS handshake error from 127.0.0.1:43834: EOF
2020/08/03 19:21:32 http: TLS handshake error from 106.51.24.241:33508: [20vueapps.com] failed to obtain certificate: acme: error: 429 :: POST :: https://acme-v02.api.letsencrypt.org/acme/new-order :: urn:ietf:params:acme:error:rateLimited :: Error creating new order :: too many new orders recently: see https://letsencrypt.org/docs/rate-limits/, url: 
2020/08/03 19:21:33 [INFO] Obtaining new certificate for 20vueapps.com
2020/08/03 19:21:33 http: TLS handshake error from 127.0.0.1:43840: EOF
2020/08/03 19:21:34 http: TLS handshake error from 106.51.24.241:33513: [20vueapps.com] failed to obtain certificate: acme: error: 429 :: POST :: https://acme-v02.api.letsencrypt.org/acme/new-order :: urn:ietf:params:acme:error:rateLimited :: Error creating new order :: too many new orders recently: see https://letsencrypt.org/docs/rate-limits/, url: 
2020/08/03 19:21:34 http: TLS handshake error from 106.51.24.241:33516: tls: client offered only unsupported versions: [301 300]
2020/08/03 19:21:37 [INFO] Obtaining new certificate for 20vueapps.com
2020/08/03 19:21:37 http: TLS handshake error from 127.0.0.1:43844: EOF
2020/08/03 19:21:38 http: TLS handshake error from 106.51.24.241:33519: [20vueapps.com] failed to obtain certificate: acme: error: 429 :: POST :: https://acme-v02.api.letsencrypt.org/acme/new-order :: urn:ietf:params:acme:error:rateLimited :: Error creating new order :: too many new orders recently: see https://letsencrypt.org/docs/rate-limits/, url: 
2020/08/03 19:21:38 [INFO] Obtaining new certificate for 20vueapps.com
2020/08/03 19:21:39 http: TLS handshake error from 127.0.0.1:43848: EOF
2020/08/03 19:21:40 http: TLS handshake error from 106.51.24.241:33520: [20vueapps.com] failed to obtain certificate: acme: error: 429 :: POST :: https://acme-v02.api.letsencrypt.org/acme/new-order :: urn:ietf:params:acme:error:rateLimited :: Error creating new order :: too many new orders recently: see https://letsencrypt.org/docs/rate-limits/, url: 
2020/08/03 19:21:40 http: TLS handshake error from 106.51.24.241:33521: tls: client offered only unsupported versions: [301 300]
2020/08/03 19:21:40 [INFO] Obtaining new certificate for 20vueapps.com
2020/08/03 19:21:41 http: TLS handshake error from 127.0.0.1:43852: EOF
2020/08/03 19:21:41 http: TLS handshake error from 106.51.24.241:33522: [20vueapps.com] failed to obtain certificate: acme: error: 429 :: POST :: https://acme-v02.api.letsencrypt.org/acme/new-order :: urn:ietf:params:acme:error:rateLimited :: Error creating new order :: too many new orders recently: see https://letsencrypt.org/docs/rate-limits/, url: 
2020/08/03 19:21:42 [INFO] Obtaining new certificate for 20vueapps.com
2020/08/03 19:21:43 http: TLS handshake error from 127.0.0.1:43856: EOF
2020/08/03 19:21:43 http: TLS handshake error from 106.51.24.241:33523: [20vueapps.com] failed to obtain certificate: acme: error: 429 :: POST :: https://acme-v02.api.letsencrypt.org/acme/new-order :: urn:ietf:params:acme:error:rateLimited :: Error creating new order :: too many new orders recently: see https://letsencrypt.org/docs/rate-limits/, url: 
2020/08/03 19:21:43 http: TLS handshake error from 106.51.24.241:33525: tls: client offered only unsupported versions: [301 300]

If you look at the end of the above logs (domain 20vueapps.com), you will see a bunch of “too many new orders recently” errors.

5. What I already tried:

I tried finding out if we by chance request a lot of new certs in 3 hr window, but that’s not the case. I tried doing some research and realized that renewals might also be counting against my quota. But I think Caddy must be backing off before attempting to renew a lot of certs. Please correct me if I am wrong.

Also, I know that Caddy is capable of handling tens of thousands of certs easily.

So, I am not fully sure what’s going on. Any help will be highly appreciated.

Thank You!

Please upgrade to Caddy v2. Caddy v1 is no longer actively supported. Caddy v2 also brings much more robust TLS automation where this issue would be very unlikely to happen in the first place.

Your Caddyfile is quite simple, so it should be a breeze to upgrade:

Note that ask is now a global option instead of on the tls server directive:

Thanks! I’ll try upgrading tonight.

Quick question: Caddy 1 stores the certs inside $HOME/.caddy directory. But it looks like Caddy 2 uses a new format to store the certs. After upgrading to Caddy 2, if I keep my data dir same i.e. $HOME/.caddy, will Caddy automatically read the old certs and move them into correct format for me?

That’s a good question. I don’t believe it will. Caddy v2 uses a new storage location:

I think if you specify an email in your config’s global options, you could avoid the orders rate limit as you’d be using a different ACME account. I’m not certain of that though. @matt could clarify later (when he wakes up etc)

Okay, so if need to upgrade, do I have to manually copy all the certs and move them into the new format: $HOME/.local/share/caddy?

I don’t think that’s a good idea because the internal storage format is different. I think it’s too risky to do that, I think it would be better to let the domains get reissued with Caddy v2.

Caddy v2 takes much more care to avoid hitting rate limits (but it can’t know at what point your account currently is at with the rate limits at Let’s Encrypt).

Again I’d let @matt give you a more definitive answer here, I don’t have as much expertise in the area of the cert management behaviour as he does (since he wrote the thing :sweat_smile:).

Thanks for your help. Really appreciate it. I will wait for @matt to respond.

Caddy 2 migrates the certificates to the new location… for now. Eventually I want to rip out that code though. Time to upgrade! :slight_smile:

Make sure to follow the upgrade guide, test locally, and back up anything important (especially if you have a lot of certificates) before deploying the new version (but this should all go without saying).

1 Like

To clarify, Caddy 2 is smarter about avoiding rate limits due to validation errors, but ultimately it’s impossible to reproduce the CA’s internal state precisely so it can still get rate limited, say, if there are simply a lot of certificates to issue (i.e. 300 new orders per 3 hours) – Caddy will backoff and retry but in the case of a lot of certificates it absolutely can hit those rate limits (but it will retry until the rate limit is lifted).

2 Likes

Hey Matt,

Thanks for the response! So, if I set my data dir to $HOME/.caddy (it has around 2500 certs) after upgrading to Caddy 2, it should just reuse the old certs, right?

Another issue is I am on Caddy 1.0 and I am seeing “too many new orders” error even though we are not requesting that many certs in 3hrs. I read somewhere that renewals also count against the quota. How does Caddy 2 handle this compared to 1.0?

Caddy 2’s data directory is in a different location, depending on your OS. I suggest reading the docs on this:

I don’t think Caddy has used $HOME/.caddy for years though, it has used $HOME/.local/share/caddy on Linux for a long time, so I don’t think it will automatically migrate from .caddy for you anymore; we dropped that years ago. If you want the migration to happen automatically I believe the folder path needs to be in .local/share.

You can read more about previous auto-migrations here:

https://github.com/caddyserver/caddy/issues/2955

https://github.com/caddyserver/caddy/issues/3124

If an error occurs using Let’s Encrypt’s production endpoint, CertMagic will retry with its staging endpoint which is not subject to rate limits. But, once it succeeds in staging it will retry in production. This doesn’t help if you’ve already been rate limited of course, and is mostly to avoid hitting rate limits due to validation failures (the most common reason).

(Switching to staging doesn’t help when you’ve already been rate limited. Just last week I wrote a new underlying ACME library from scratch that replaces lego, so we have more control over error handling and can be even smarter about retries in the face of rate limits, but that’s not finished being implemented yet.)

Unfortunately the ACME spec (allegedly) requires a whole new order even if one challenge type fails: Refactor challenge retry logic to create new orders (sigh) · mholt/acmez@80adb6d · GitHub – I will be emailing the IETF WG about this to see if it can be changed, when I get around to it.

The #1 thing is to check your logs and prevent / fix validation errors as soon as possible (usually means correcting DNS records or network configuration).

I don’t think that’s correct. Caddy v1.0.5 definitely still used $HOME/.caddy. The change only happened in Caddy v2.

You could always just try it and let us know what happens then :smile: (Obviously, test it in a local or staging environment, on a copy of the data and not the actual data. And configure a staging ACME endpoint instead of a production one too.)

1 Like

This topic was automatically closed after 30 days. New replies are no longer allowed.