Seeing "Too many new orders recently" occasionally

sandeep · August 4, 2020, 9:25am

Hi

Caddy is awesome and I have been using it for my startup Hashnode for 1 year now. Never had any issues. Kudos to the awesome work by the team.

However, for the last couple of days, I am facing a weird issue. Here are the details:

1. Caddy version (`caddy version`):

1.0

2. How I run Caddy:

We start Caddy using the following command:

nohup caddy -log access.log

a. System environment:

OS: Ubuntu 16.04

b. Command:

nohup caddy -log access.log

c. Service/unit/compose file:

None

d. My complete Caddyfile or JSON config:

* {
    proxy / localhost:3000 {
        transparent
    }
    gzip
    tls {
        ask <ask-url>
    }
}

*.hashnode.dev:443 {
    proxy / localhost:3000 {
        transparent
    }
    gzip
    tls fullchain.pem privkey.pem
}

3. The problem I’m having:

We are running Caddy in fleet mode (total 5 servers). A common disk is mounted on .caddy/acme via SSHFS to share SSL certs. Since we have a multi-tenant system, there are around 2500 different domains. Everything was smooth so far, but lately, I have been seeing the error “Too many new orders recently” while requesting certs for new domains. This happens occasionally. As per LE, this error comes when you exceed 300 orders / 3 hrs limit. But we are not doing that many requests in 3 hours. I checked our server logs to verify this.

What could be the reason?

4. Error messages and/or full log output:

2020/08/03 18:34:42 http: TLS handshake error from 127.0.0.1:43364: EOF
2020/08/03 18:34:44 [ERROR] Renewing [hn.werick.codes]: acme: error: 429 :: POST :: https://acme-v02.api.letsencrypt.org/acme/new-order :: urn:ietf:params:acme:error:rateLimited :: Error creating new order :: too many failed authorizations recently: see https://letsencrypt.org/docs/rate-limits/, url: ; trying again in 10s
2020/08/03 18:34:55 [ERROR] Renewing [hn.werick.codes]: acme: error: 429 :: POST :: https://acme-v02.api.letsencrypt.org/acme/new-order :: urn:ietf:params:acme:error:rateLimited :: Error creating new order :: too many failed authorizations recently: see https://letsencrypt.org/docs/rate-limits/, url: ; trying again in 10s
2020/08/03 18:35:05 [ERROR] too many renewal attempts; last error: acme: error: 429 :: POST :: https://acme-v02.api.letsencrypt.org/acme/new-order :: urn:ietf:params:acme:error:rateLimited :: Error creating new order :: too many failed authorizations recently: see https://letsencrypt.org/docs/rate-limits/, url: 
2020/08/03 18:35:05 [INFO] Certificate for [blog.amnota.dev] expires in -1140h50m18.035994548s; attempting renewal
2020/08/03 18:35:09 http: TLS handshake error from 127.0.0.1:43372: EOF
2020/08/03 18:35:12 [ERROR] Renewing [blog.amnota.dev]: acme: error: 429 :: POST :: https://acme-v02.api.letsencrypt.org/acme/new-order :: urn:ietf:params:acme:error:rateLimited :: Error creating new order :: too many failed authorizations recently: see https://letsencrypt.org/docs/rate-limits/, url: ; trying again in 10s
2020/08/03 18:35:22 [ERROR] Renewing [blog.amnota.dev]: acme: error: 429 :: POST :: https://acme-v02.api.letsencrypt.org/acme/new-order :: urn:ietf:params:acme:error:rateLimited :: Error creating new order :: too many failed authorizations recently: see https://letsencrypt.org/docs/rate-limits/, url: ; trying again in 10s
2020/08/03 18:35:32 [ERROR] too many renewal attempts; last error: acme: error: 429 :: POST :: https://acme-v02.api.letsencrypt.org/acme/new-order :: urn:ietf:params:acme:error:rateLimited :: Error creating new order :: too many failed authorizations recently: see https://letsencrypt.org/docs/rate-limits/, url: 
2020/08/03 18:35:32 [INFO][cache:0xc0000b87d0] Done scanning certificates
2020/08/03 18:35:32 [INFO][cache:0xc0000b87d0] Scanning for stale OCSP staples
2020/08/03 18:35:32 [INFO] Advancing OCSP staple for [blog.herce.co] from 2020-08-07 06:00:00 +0000 UTC to 2020-08-10 06:00:00 +0000 UTC
2020/08/03 18:35:33 [INFO] Advancing OCSP staple for [blog.bytefly.io] from 2020-08-07 06:00:00 +0000 UTC to 2020-08-10 06:00:00 +0000 UTC
2020/08/03 18:35:33 [INFO] Advancing OCSP staple for [juantobias.dev] from 2020-08-07 06:00:00 +0000 UTC to 2020-08-10 06:00:00 +0000 UTC
2020/08/03 18:35:33 [INFO] Advancing OCSP staple for [til.agentofuser.com] from 2020-08-07 06:00:00 +0000 UTC to 2020-08-10 06:00:00 +0000 UTC
2020/08/03 18:35:34 [INFO] Advancing OCSP staple for [cloudy.achakladar.com] from 2020-08-07 06:00:00 +0000 UTC to 2020-08-10 06:00:00 +0000 UTC
2020/08/03 18:35:34 [INFO] Advancing OCSP staple for [jordanfarrer.com] from 2020-08-07 06:00:00 +0000 UTC to 2020-08-10 06:00:00 +0000 UTC
2020/08/03 18:35:34 [INFO] Advancing OCSP staple for [risav.dev] from 2020-08-07 06:00:00 +0000 UTC to 2020-08-10 06:00:00 +0000 UTC
2020/08/03 18:35:35 [INFO] Advancing OCSP staple for [blog.levippaul.com] from 2020-08-07 06:00:00 +0000 UTC to 2020-08-10 06:00:00 +0000 UTC
2020/08/03 18:35:35 [INFO] Advancing OCSP staple for [blog.joostvandergaag.com] from 2020-08-07 06:00:00 +0000 UTC to 2020-08-10 06:00:00 +0000 UTC
2020/08/03 18:35:35 [INFO] Advancing OCSP staple for [blog.collinsruto.com] from 2020-08-07 06:00:00 +0000 UTC to 2020-08-10 06:00:00 +0000 UTC
2020/08/03 18:35:36 [INFO] Advancing OCSP staple for [blog.jarda.it] from 2020-08-07 06:00:00 +0000 UTC to 2020-08-10 06:00:00 +0000 UTC
2020/08/03 18:35:36 [INFO] Advancing OCSP staple for [devblog.qcobjects.org] from 2020-08-07 06:00:00 +0000 UTC to 2020-08-10 06:00:00 +0000 UTC
2020/08/03 18:35:36 [INFO] Advancing OCSP staple for [dev.liuweifeng.net] from 2020-08-07 06:00:00 +0000 UTC to 2020-08-10 06:00:00 +0000 UTC
2020/08/03 18:35:37 [INFO] Advancing OCSP staple for [blog.prokop.dev] from 2020-08-07 06:00:00 +0000 UTC to 2020-08-10 06:00:00 +0000 UTC
2020/08/03 18:35:37 [INFO] Advancing OCSP staple for [blog.leojacon.tech] from 2020-08-07 06:00:00 +0000 UTC to 2020-08-10 06:00:00 +0000 UTC
2020/08/03 18:35:37 [INFO] Advancing OCSP staple for [blog.athreyapatel.codes] from 2020-08-07 06:00:00 +0000 UTC to 2020-08-10 06:00:00 +0000 UTC
2020/08/03 18:35:37 [INFO] Advancing OCSP staple for [blog.cloudnative-labs.nl] from 2020-08-07 06:00:00 +0000 UTC to 2020-08-10 06:00:00 +0000 UTC
2020/08/03 18:35:38 [INFO] Advancing OCSP staple for [blog.mikeattara.com] from 2020-08-07 06:00:00 +0000 UTC to 2020-08-10 06:00:00 +0000 UTC
2020/08/03 18:35:38 [INFO] Advancing OCSP staple for [techoodle.vidhyaranganathan.com] from 2020-08-07 06:00:00 +0000 UTC to 2020-08-10 06:00:00 +0000 UTC
2020/08/03 18:35:38 [INFO] Advancing OCSP staple for [blog.akush.in] from 2020-08-07 06:00:00 +0000 UTC to 2020-08-10 06:00:00 +0000 UTC
2020/08/03 18:35:39 [INFO] Advancing OCSP staple for [blog.romanjasek.com] from 2020-08-07 06:00:00 +0000 UTC to 2020-08-10 06:00:00 +0000 UTC
2020/08/03 18:35:39 [INFO] Advancing OCSP staple for [blog.chrisjohn.digital] from 2020-08-07 06:00:00 +0000 UTC to 2020-08-10 06:00:00 +0000 UTC
2020/08/03 18:35:39 [INFO] Advancing OCSP staple for [fiq.me] from 2020-08-07 06:00:00 +0000 UTC to 2020-08-10 06:00:00 +0000 UTC
2020/08/03 18:35:40 [INFO] Advancing OCSP staple for [blog.yeshu.in] from 2020-08-07 06:00:00 +0000 UTC to 2020-08-10 06:00:00 +0000 UTC
2020/08/03 18:35:40 [ERROR] Checking OCSP: invalid: OCSP response for [hash.imfancy.cn] valid after certificate expiration (-24h6m36s)
2020/08/03 18:35:40 [INFO] Advancing OCSP staple for [abrahamnm.com] from 2020-08-07 06:00:00 +0000 UTC to 2020-08-10 06:00:00 +0000 UTC
2020/08/03 18:35:41 [INFO] Advancing OCSP staple for [blog.vaishnavibaliga.com] from 2020-08-07 06:00:00 +0000 UTC to 2020-08-10 06:00:00 +0000 UTC
2020/08/03 18:35:41 [INFO] Advancing OCSP staple for [saltgen.dev] from 2020-08-07 06:00:00 +0000 UTC to 2020-08-10 06:00:00 +0000 UTC
2020/08/03 18:35:41 [INFO][cache:0xc0000b87d0] Done checking OCSP staples
2020/08/03 18:35:44 [ERROR] Sending telemetry: Post https://telemetry.caddyserver.com/v1/update/eb54bcd5-4dc0-42f4-9575-5a23cea6efe7: dial tcp: lookup telemetry.caddyserver.com on 67.207.67.3:53: no such host
2020/08/03 18:38:38 http: TLS handshake error from 18.212.228.7:47866: certificate for hostname '' not allowed, non-2xx status code 404 returned from <ask-url>
2020/08/03 19:14:32 [INFO][cache:0xc0000b87d0] Scanning for stale OCSP staples
2020/08/03 19:14:32 [INFO] Advancing OCSP staple for [blog.rakeshyadav.info] from 2020-08-07 07:00:00 +0000 UTC to 2020-08-10 07:00:00 +0000 UTC
2020/08/03 19:14:32 [INFO] Advancing OCSP staple for [blog.sireto.io] from 2020-08-07 07:00:00 +0000 UTC to 2020-08-10 07:00:00 +0000 UTC
2020/08/03 19:14:32 [ERROR] Checking OCSP: invalid: OCSP response for [hash.imfancy.cn] valid after certificate expiration (-24h6m36s)
2020/08/03 19:14:33 [INFO] Advancing OCSP staple for [blog.rafedraper.de] from 2020-08-07 07:00:00 +0000 UTC to 2020-08-10 07:00:00 +0000 UTC
2020/08/03 19:14:33 [INFO] Advancing OCSP staple for [hn.werick.codes] from 2020-08-07 07:00:00 +0000 UTC to 2020-08-10 07:00:00 +0000 UTC
2020/08/03 19:14:33 [INFO][cache:0xc0000b87d0] Done checking OCSP staples
2020/08/03 19:21:29 [INFO] Obtaining new certificate for 20vueapps.com
2020/08/03 19:21:30 http: TLS handshake error from 127.0.0.1:43834: EOF
2020/08/03 19:21:32 http: TLS handshake error from 106.51.24.241:33508: [20vueapps.com] failed to obtain certificate: acme: error: 429 :: POST :: https://acme-v02.api.letsencrypt.org/acme/new-order :: urn:ietf:params:acme:error:rateLimited :: Error creating new order :: too many new orders recently: see https://letsencrypt.org/docs/rate-limits/, url: 
2020/08/03 19:21:33 [INFO] Obtaining new certificate for 20vueapps.com
2020/08/03 19:21:33 http: TLS handshake error from 127.0.0.1:43840: EOF
2020/08/03 19:21:34 http: TLS handshake error from 106.51.24.241:33513: [20vueapps.com] failed to obtain certificate: acme: error: 429 :: POST :: https://acme-v02.api.letsencrypt.org/acme/new-order :: urn:ietf:params:acme:error:rateLimited :: Error creating new order :: too many new orders recently: see https://letsencrypt.org/docs/rate-limits/, url: 
2020/08/03 19:21:34 http: TLS handshake error from 106.51.24.241:33516: tls: client offered only unsupported versions: [301 300]
2020/08/03 19:21:37 [INFO] Obtaining new certificate for 20vueapps.com
2020/08/03 19:21:37 http: TLS handshake error from 127.0.0.1:43844: EOF
2020/08/03 19:21:38 http: TLS handshake error from 106.51.24.241:33519: [20vueapps.com] failed to obtain certificate: acme: error: 429 :: POST :: https://acme-v02.api.letsencrypt.org/acme/new-order :: urn:ietf:params:acme:error:rateLimited :: Error creating new order :: too many new orders recently: see https://letsencrypt.org/docs/rate-limits/, url: 
2020/08/03 19:21:38 [INFO] Obtaining new certificate for 20vueapps.com
2020/08/03 19:21:39 http: TLS handshake error from 127.0.0.1:43848: EOF
2020/08/03 19:21:40 http: TLS handshake error from 106.51.24.241:33520: [20vueapps.com] failed to obtain certificate: acme: error: 429 :: POST :: https://acme-v02.api.letsencrypt.org/acme/new-order :: urn:ietf:params:acme:error:rateLimited :: Error creating new order :: too many new orders recently: see https://letsencrypt.org/docs/rate-limits/, url: 
2020/08/03 19:21:40 http: TLS handshake error from 106.51.24.241:33521: tls: client offered only unsupported versions: [301 300]
2020/08/03 19:21:40 [INFO] Obtaining new certificate for 20vueapps.com
2020/08/03 19:21:41 http: TLS handshake error from 127.0.0.1:43852: EOF
2020/08/03 19:21:41 http: TLS handshake error from 106.51.24.241:33522: [20vueapps.com] failed to obtain certificate: acme: error: 429 :: POST :: https://acme-v02.api.letsencrypt.org/acme/new-order :: urn:ietf:params:acme:error:rateLimited :: Error creating new order :: too many new orders recently: see https://letsencrypt.org/docs/rate-limits/, url: 
2020/08/03 19:21:42 [INFO] Obtaining new certificate for 20vueapps.com
2020/08/03 19:21:43 http: TLS handshake error from 127.0.0.1:43856: EOF
2020/08/03 19:21:43 http: TLS handshake error from 106.51.24.241:33523: [20vueapps.com] failed to obtain certificate: acme: error: 429 :: POST :: https://acme-v02.api.letsencrypt.org/acme/new-order :: urn:ietf:params:acme:error:rateLimited :: Error creating new order :: too many new orders recently: see https://letsencrypt.org/docs/rate-limits/, url: 
2020/08/03 19:21:43 http: TLS handshake error from 106.51.24.241:33525: tls: client offered only unsupported versions: [301 300]

If you look at the end of the above logs (domain 20vueapps.com), you will see a bunch of “too many new orders recently” errors.

5. What I already tried:

I tried finding out if we by chance request a lot of new certs in 3 hr window, but that’s not the case. I tried doing some research and realized that renewals might also be counting against my quota. But I think Caddy must be backing off before attempting to renew a lot of certs. Please correct me if I am wrong.

Also, I know that Caddy is capable of handling tens of thousands of certs easily.

So, I am not fully sure what’s going on. Any help will be highly appreciated.

Thank You!

francislavoie · August 4, 2020, 9:41am

Please upgrade to Caddy v2. Caddy v1 is no longer actively supported. Caddy v2 also brings much more robust TLS automation where this issue would be very unlikely to happen in the first place.

Your Caddyfile is quite simple, so it should be a breeze to upgrade:

Note that ask is now a global option instead of on the tls server directive:

sandeep · August 4, 2020, 9:47am

Thanks! I’ll try upgrading tonight.

Quick question: Caddy 1 stores the certs inside $HOME/.caddy directory. But it looks like Caddy 2 uses a new format to store the certs. After upgrading to Caddy 2, if I keep my data dir same i.e. $HOME/.caddy, will Caddy automatically read the old certs and move them into correct format for me?

francislavoie · August 4, 2020, 9:53am

That’s a good question. I don’t believe it will. Caddy v2 uses a new storage location:

I think if you specify an email in your config’s global options, you could avoid the orders rate limit as you’d be using a different ACME account. I’m not certain of that though. @matt could clarify later (when he wakes up etc)

sandeep · August 4, 2020, 9:58am

Okay, so if need to upgrade, do I have to manually copy all the certs and move them into the new format: $HOME/.local/share/caddy?

francislavoie · August 4, 2020, 10:03am

I don’t think that’s a good idea because the internal storage format is different. I think it’s too risky to do that, I think it would be better to let the domains get reissued with Caddy v2.

Caddy v2 takes much more care to avoid hitting rate limits (but it can’t know at what point your account currently is at with the rate limits at Let’s Encrypt).

Again I’d let @matt give you a more definitive answer here, I don’t have as much expertise in the area of the cert management behaviour as he does (since he wrote the thing ).

sandeep · August 4, 2020, 10:13am

Thanks for your help. Really appreciate it. I will wait for @matt to respond.

matt · August 4, 2020, 3:53pm

Caddy 2 migrates the certificates to the new location… for now. Eventually I want to rip out that code though. Time to upgrade!

Make sure to follow the upgrade guide, test locally, and back up anything important (especially if you have a lot of certificates) before deploying the new version (but this should all go without saying).

matt · August 4, 2020, 3:55pm

To clarify, Caddy 2 is smarter about avoiding rate limits due to validation errors, but ultimately it’s impossible to reproduce the CA’s internal state precisely so it can still get rate limited, say, if there are simply a lot of certificates to issue (i.e. 300 new orders per 3 hours) – Caddy will backoff and retry but in the case of a lot of certificates it absolutely can hit those rate limits (but it will retry until the rate limit is lifted).

sandeep · August 4, 2020, 5:59pm

Hey Matt,

Thanks for the response! So, if I set my data dir to $HOME/.caddy (it has around 2500 certs) after upgrading to Caddy 2, it should just reuse the old certs, right?

Another issue is I am on Caddy 1.0 and I am seeing “too many new orders” error even though we are not requesting that many certs in 3hrs. I read somewhere that renewals also count against the quota. How does Caddy 2 handle this compared to 1.0?

matt · August 4, 2020, 6:22pm

Caddy 2’s data directory is in a different location, depending on your OS. I suggest reading the docs on this:

I don’t think Caddy has used $HOME/.caddy for years though, it has used $HOME/.local/share/caddy on Linux for a long time, so I don’t think it will automatically migrate from .caddy for you anymore; we dropped that years ago. If you want the migration to happen automatically I believe the folder path needs to be in .local/share.

You can read more about previous auto-migrations here:

github.com/caddyserver/caddy

v2: Honor OS-specific file system storage conventions and migrate all assets to new locations

opened 10:29PM - 31 Dec 19 UTC

closed 12:01AM - 01 Jan 20 UTC

mholt

I typically prefer simplicity and consistency when trying to make an app cross-p…latform, and it is difficult. I think it may be best to adhere to OS conventions when choosing where to store data. (This issue arises as I finish implementing a feature that auto-saves configs so that the last-saved config can be resumed after the server is restarted; since a good choice must be made as to where to store the config file.) Currently (Caddy 2.0 up to and including beta 11), data is stored in [`$XDG_DATA_HOME`](https://specifications.freedesktop.org/basedir-spec/basedir-spec-latest.html) or: ``` $HOME/.local/share/caddy/ ``` And if $HOME is empty, the current working directory is used in place of $HOME. But this is a little awkward on Mac, Windows, Plan 9, Android, and others. In fact, this only really makes sense on Linux and BSD (which is, granted, probably 99% of deployments). It can also lead to unintended consequences when data is not stored in an expected place for the OS, especially when you consider file backups, sysadmin experience, etc. Mac typically wants things stored in `~/Library` and Windows has `%AppData%` and `%LocalAppData%` folders. [Newer Go versions have](https://golang.org/pkg/os/#UserCacheDir) `os.UserCacheDir()`, `os.UserConfigDir()`, and `os.UserHomeDir()` which account for these differences. Although I will be using `UserConfigDir()`, it is not implemented quite how I want; and unfortunately there is no` os.UserDataDir()` at all. We can't use a cache directory for the data directory because Caddy data _is not ephemeral_ -- for example, certificates **must not** be deleted arbitrarily. (We may use UserCacheDir in the future, but we don't need it today. Caddy 2 currently needs UserConfigDir and UserDataDir). I will be implementing my own `dataDir()` function which essentially looks like this: ```go func dataDir() string { if basedir := os.Getenv("XDG_DATA_HOME"); basedir != "" { return filepath.Join(basedir, "caddy") } switch runtime.GOOS { case "windows": appData := os.Getenv("AppData") if appData != "" { return filepath.Join(appData, "Caddy") } case "darwin": home := homeDir() if home != "" { return filepath.Join(home, "Library", "Application Support", "Caddy") } case "plan9": home := homeDir() if home != "" { return filepath.Join(home, "lib", "caddy") } case "android": home := homeDir() if home != "" { return filepath.Join(home, "caddy") } default: home := homeDir() if home != "" { return filepath.Join(home, ".local", "share", "caddy") } } return "./caddy" } ``` Notice that the XDG spec is honored regardless of the OS. I think this is reasonable and useful. For a config directory, I will use `os.UserConfigDir()`, but only if `$XDG_CONFIG_HOME` is not set. The standard lib function only accepts that variable on Unix systems, but I want it to be able to work on any OS, for consistency's sake. This will take effect starting in Caddy 2 beta 12. From now until the release candidates, Caddy 2 will attempt to migrate the data folder to the new location for you until the release candidates. This will just be a simple "rename" syscall, so if the old and new locations are on different devices, the move will fail. Also, if the new destination already exists, Caddy will not perform any migration out of an abundance of caution. That'd be unusual, but I'll post instructions here in a reply for moving the files over yourself. You'll see an error message in the logs that links you to the instructions. Errors with migration will not prevent the server from running. I'll also be adding the locations to our new documentation, and relevant logs will also indicate file/folder paths where that is useful. I've also updated the `--environ` flag and `environ` subcommand to include information about the Go runtime and the directories that Caddy uses, as well as an error in the logs at startup if necessary env variables are missing. ### tl;dr Where Caddy stores stuff is changing for non-Linux/BSD systems to conform to OS specifications. They will be moved for you, but if there's an error, you can move them manually using the instructions below. If it succeeds, this is a transparent change.

github.com/caddyserver/caddy

v2: Beta 17 will auto-migrate certificate assets to a new folder path

opened 05:42PM - 07 Mar 20 UTC

closed 05:13PM - 13 Mar 20 UTC

mholt

documentation

In beta 16 (starting with commit 4d18587192e4fffe5b34b714eaabcfc212914c1e with a… couple of fixes added in 217419f6d97885daf0fb8832a407626e578523f0), Caddy will store certificates in a new path, and will attempt to automatically move them for you. - **Old paths: `<caddy data dir>/acme/<ca>/sites/<domain>`** - **New paths: `<caddy data dir>/certificates/<ca>/<domain>`** Where `<ca>` is derived from the CA's directory endpoint. For CAs containing `api.letsencrypt.org`, the new path appends `-directory` to the end, for example: `acme-v02.api.letsencrypt.org-directory`. This move is necessary to better support non-ACME certificate sources. (We have done similar moves in the past and it's always worked out alright.) The structure of the `.json` resource file associated with each certificate has also changed slightly, and it will be re-encoded for you. (This file is non-critical.) The parent data directory has _not_ changed, so this is a move of subfolders within a singular parent directory. ## If you have configured custom storage If you are not using the file system for storage, you will need to move those certificates yourself, or just let Caddy get new ones. Up to a moderate number of certificates (~50) this should be totally fine. If you are using the file system but have configured your own path for storage, Caddy will operate the migration within that custom path. ## If there are errors Errors during the auto-migration are non-critical. The worst thing that will probably happen as a resul of errors is that Caddy will attempt to get new certificates instead of using existing ones. Unless you have hundreds of certificates, this shouldn't be a problem. ## tl;dr This post is mostly informational, not a call to action. Most users will not have to do anything. You'll see in the logs if a migration occurs. If there are errors, they will be printed to the log. Errors are non-critical. If you have problems, feel free to [post on the forums](https://caddy.community) or reply to this thread. Closing this issue but feel free to comment if necessary.

If an error occurs using Let’s Encrypt’s production endpoint, CertMagic will retry with its staging endpoint which is not subject to rate limits. But, once it succeeds in staging it will retry in production. This doesn’t help if you’ve already been rate limited of course, and is mostly to avoid hitting rate limits due to validation failures (the most common reason).

(Switching to staging doesn’t help when you’ve already been rate limited. Just last week I wrote a new underlying ACME library from scratch that replaces lego, so we have more control over error handling and can be even smarter about retries in the face of rate limits, but that’s not finished being implemented yet.)

Unfortunately the ACME spec (allegedly) requires a whole new order even if one challenge type fails: https://github.com/mholt/acmez/commit/80adb6d5e64a3d36a56c58c66965b131ea366b8c – I will be emailing the IETF WG about this to see if it can be changed, when I get around to it.

The #1 thing is to check your logs and prevent / fix validation errors as soon as possible (usually means correcting DNS records or network configuration).

francislavoie · August 4, 2020, 7:32pm

I don’t think that’s correct. Caddy v1.0.5 definitely still used $HOME/.caddy. The change only happened in Caddy v2.

matt · August 4, 2020, 8:03pm

You could always just try it and let us know what happens then (Obviously, test it in a local or staging environment, on a copy of the data and not the actual data. And configure a staging ACME endpoint instead of a production one too.)

system · September 3, 2020, 9:25am

This topic was automatically closed after 30 days. New replies are no longer allowed.

1. Caddy version (caddy version):