How to trigger workaround for "New Certificates per Exact Set of Hostnames" rate limit?

Help
1

1. The problem I’m having:

Hi, I’m trying to use Caddy in a new project. The project is a web app hosted in AWS EC2, managed with terraform. My Caddyfile initially was:

dev.sophistree.app {
  reverse_proxy localhost:3000
  log {
    output file /var/log/caddy/access.log
  }
}

In the course of developing the web app, I triggered the “New Certificates per Exact Set of Hostnames” because I am recreating my instance and so reinstalling Caddy via user-data script multiple times per day.

Excerpt of the error message:

...
"logger":"tls.obtain","msg":"could not get certificate from issuer","identifier":"dev.sophistree.app","issuer":"acme-v02.api.letsencrypt.org-directory","error":"HTTP 429 urn:ietf:params:acme:error:rateLimited - too many certificates (5) already issued for this exact set of domains in the last 168h0m0s, retry after 2024-11-18 17:57:51 UTC: see https://letsencrypt.org/docs/rate-limits/#new-certificates-per-exact-set-of-hostnames"
...

I looked into using the letsencrypt staging endpoint, but unfortunately that doesn’t work because the TLD I picked for my project (.app) does not allow browsers to disable HSTS for them:

The entire TLD app is preloaded for HSTS and individual domain names cannot be removed.

https://hstspreload.org/removal/?domain=sophistree.app

So it’s not possible to connect to a site hosted under the .app TLD using an insecure cert.

So back to the rate limit docs, I notice that it mentions a workaround:

If you’ve hit this limit, you can change the set of hostnames by adding blog.example.com, to request additional certificates. Be aware that these new orders would not be considered renewals. Therefore, they would be subject to the New Orders per Account and New Certificates per Registered Domain rate limits.")

So I update my Caddyfile like so:

dev.sophistree.app, test.sophistree.app {
  reverse_proxy localhost:3000
  log {
    output file /var/log/caddy/access.log
  }
}

But this still doesn’t work, and I get similar error message as above. (See next section.)

2. Error messages and/or full log output:

Nov 18 03:31:28 ip-10-0-1-150.us-west-2.compute.internal systemd[1]: Starting caddy.service - Caddy...
Nov 18 03:31:29 ip-10-0-1-150.us-west-2.compute.internal caddy[21318]: caddy.HomeDir=/var/lib/caddy
Nov 18 03:31:29 ip-10-0-1-150.us-west-2.compute.internal caddy[21318]: caddy.AppDataDir=/var/lib/caddy/.local/share/caddy
Nov 18 03:31:29 ip-10-0-1-150.us-west-2.compute.internal caddy[21318]: caddy.AppConfigDir=/var/lib/caddy/.config/caddy
Nov 18 03:31:29 ip-10-0-1-150.us-west-2.compute.internal caddy[21318]: caddy.ConfigAutosavePath=/var/lib/caddy/.config/caddy/autosave.json
Nov 18 03:31:29 ip-10-0-1-150.us-west-2.compute.internal caddy[21318]: caddy.Version=v2.7.6 h1:w0NymbG2m9PcvKWsrXO6EEkY9Ru4FJK8uQbYcev1p3A=
Nov 18 03:31:29 ip-10-0-1-150.us-west-2.compute.internal caddy[21318]: runtime.GOOS=linux
Nov 18 03:31:29 ip-10-0-1-150.us-west-2.compute.internal caddy[21318]: runtime.GOARCH=amd64
Nov 18 03:31:29 ip-10-0-1-150.us-west-2.compute.internal caddy[21318]: runtime.Compiler=gc
Nov 18 03:31:29 ip-10-0-1-150.us-west-2.compute.internal caddy[21318]: runtime.NumCPU=2
Nov 18 03:31:29 ip-10-0-1-150.us-west-2.compute.internal caddy[21318]: runtime.GOMAXPROCS=2
Nov 18 03:31:29 ip-10-0-1-150.us-west-2.compute.internal caddy[21318]: runtime.Version=go1.20.10
Nov 18 03:31:29 ip-10-0-1-150.us-west-2.compute.internal caddy[21318]: os.Getwd=/
Nov 18 03:31:29 ip-10-0-1-150.us-west-2.compute.internal caddy[21318]: LANG=C.UTF-8
Nov 18 03:31:29 ip-10-0-1-150.us-west-2.compute.internal caddy[21318]: PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin
Nov 18 03:31:29 ip-10-0-1-150.us-west-2.compute.internal caddy[21318]: NOTIFY_SOCKET=/run/systemd/notify
Nov 18 03:31:29 ip-10-0-1-150.us-west-2.compute.internal caddy[21318]: HOME=/var/lib/caddy
Nov 18 03:31:29 ip-10-0-1-150.us-west-2.compute.internal caddy[21318]: LOGNAME=caddy
Nov 18 03:31:29 ip-10-0-1-150.us-west-2.compute.internal caddy[21318]: USER=caddy
Nov 18 03:31:29 ip-10-0-1-150.us-west-2.compute.internal caddy[21318]: INVOCATION_ID=9df21c68127f41b782ab6297cd4b3667
Nov 18 03:31:29 ip-10-0-1-150.us-west-2.compute.internal caddy[21318]: JOURNAL_STREAM=8:150309
Nov 18 03:31:29 ip-10-0-1-150.us-west-2.compute.internal caddy[21318]: SYSTEMD_EXEC_PID=21318
Nov 18 03:31:29 ip-10-0-1-150.us-west-2.compute.internal caddy[21318]: {"level":"info","ts":1731900689.018547,"msg":"using provided configuration","config_file":"/etc/caddy/Caddyfile","config_adapter":""}
Nov 18 03:31:29 ip-10-0-1-150.us-west-2.compute.internal caddy[21318]: {"level":"warn","ts":1731900689.020228,"msg":"Caddyfile input is not formatted; run 'caddy fmt --overwrite' to fix inconsistencies","adapter":"caddyfile","file":"/etc/caddy/Caddyfile"
,"line":2}
Nov 18 03:31:29 ip-10-0-1-150.us-west-2.compute.internal caddy[21318]: {"level":"info","ts":1731900689.021601,"logger":"admin","msg":"admin endpoint started","address":"localhost:2019","enforce_origin":false,"origins":["//127.0.0.1:2019","//localhost:201
9","//[::1]:2019"]}
Nov 18 03:31:29 ip-10-0-1-150.us-west-2.compute.internal caddy[21318]: {"level":"info","ts":1731900689.0217624,"logger":"http.auto_https","msg":"server is listening only on the HTTPS port but has no TLS connection policies; adding one to enable TLS","ser
ver_name":"srv0","https_port":443}
Nov 18 03:31:29 ip-10-0-1-150.us-west-2.compute.internal caddy[21318]: {"level":"info","ts":1731900689.02179,"logger":"http.auto_https","msg":"enabling automatic HTTP->HTTPS redirects","server_name":"srv0"}
Nov 18 03:31:29 ip-10-0-1-150.us-west-2.compute.internal caddy[21318]: {"level":"debug","ts":1731900689.021833,"logger":"http.auto_https","msg":"adjusted config","tls":{"automation":{"policies":[{}]}},"http":{"servers":{"remaining_auto_https_redirects":{
"listen":[":80"],"routes":[{},{}],"logs":{"logger_names":{"dev.sophistree.app":"log0","test.sophistree.app":"log0"}}},"srv0":{"listen":[":443"],"routes":[{"handle":[{"handler":"subroute","routes":[{"handle":[{"handler":"reverse_proxy","upstreams":[{"dial
":"localhost:3000"}]}]}]}],"terminal":true}],"tls_connection_policies":[{}],"automatic_https":{},"logs":{"logger_names":{"dev.sophistree.app":"log0","test.sophistree.app":"log0"}}}}}}
Nov 18 03:31:29 ip-10-0-1-150.us-west-2.compute.internal caddy[21318]: {"level":"debug","ts":1731900689.0228915,"logger":"http","msg":"starting server loop","address":"[::]:80","tls":false,"http3":false}
Nov 18 03:31:29 ip-10-0-1-150.us-west-2.compute.internal caddy[21318]: {"level":"info","ts":1731900689.0229201,"logger":"http.log","msg":"server running","name":"remaining_auto_https_redirects","protocols":["h1","h2","h3"]}
Nov 18 03:31:29 ip-10-0-1-150.us-west-2.compute.internal caddy[21318]: {"level":"info","ts":1731900689.0229623,"logger":"http","msg":"enabling HTTP/3 listener","addr":":443"}
Nov 18 03:31:29 ip-10-0-1-150.us-west-2.compute.internal caddy[21318]: {"level":"debug","ts":1731900689.0235527,"logger":"http","msg":"starting server loop","address":"[::]:443","tls":true,"http3":true}
Nov 18 03:31:29 ip-10-0-1-150.us-west-2.compute.internal caddy[21318]: {"level":"info","ts":1731900689.0235693,"logger":"http.log","msg":"server running","name":"srv0","protocols":["h1","h2","h3"]}
Nov 18 03:31:29 ip-10-0-1-150.us-west-2.compute.internal caddy[21318]: {"level":"info","ts":1731900689.023577,"logger":"http","msg":"enabling automatic TLS certificate management","domains":["dev.sophistree.app","test.sophistree.app"]}
Nov 18 03:31:29 ip-10-0-1-150.us-west-2.compute.internal caddy[21318]: {"level":"info","ts":1731900689.0240204,"msg":"autosaved config (load with --resume flag)","file":"/var/lib/caddy/.config/caddy/autosave.json"}
Nov 18 03:31:29 ip-10-0-1-150.us-west-2.compute.internal caddy[21318]: {"level":"info","ts":1731900689.0240834,"msg":"serving initial configuration"}
Nov 18 03:31:29 ip-10-0-1-150.us-west-2.compute.internal caddy[21318]: {"level":"info","ts":1731900689.0244203,"logger":"tls.obtain","msg":"acquiring lock","identifier":"dev.sophistree.app"}
Nov 18 03:31:29 ip-10-0-1-150.us-west-2.compute.internal caddy[21318]: {"level":"info","ts":1731900689.0247872,"logger":"tls.cache.maintenance","msg":"started background certificate maintenance","cache":"0xc0003ed580"}
Nov 18 03:31:29 ip-10-0-1-150.us-west-2.compute.internal systemd[1]: Started caddy.service - Caddy.
Nov 18 03:31:29 ip-10-0-1-150.us-west-2.compute.internal caddy[21318]: {"level":"info","ts":1731900689.0271451,"logger":"tls.obtain","msg":"lock acquired","identifier":"dev.sophistree.app"}
Nov 18 03:31:29 ip-10-0-1-150.us-west-2.compute.internal caddy[21318]: {"level":"info","ts":1731900689.0273025,"logger":"tls.obtain","msg":"obtaining certificate","identifier":"dev.sophistree.app"}
Nov 18 03:31:29 ip-10-0-1-150.us-west-2.compute.internal caddy[21318]: {"level":"debug","ts":1731900689.0273273,"logger":"events","msg":"event","name":"cert_obtaining","id":"1b38a78d-8409-4531-a00b-80eeaa8e3d37","origin":"tls","data":{"identifier":"dev.s
ophistree.app"}}
Nov 18 03:31:29 ip-10-0-1-150.us-west-2.compute.internal caddy[21318]: {"level":"debug","ts":1731900689.0276637,"logger":"tls.obtain","msg":"trying issuer 1/2","issuer":"acme-v02.api.letsencrypt.org-directory"}
Nov 18 03:31:29 ip-10-0-1-150.us-west-2.compute.internal caddy[21318]: {"level":"info","ts":1731900689.0282495,"logger":"http","msg":"waiting on internal rate limiter","identifiers":["dev.sophistree.app"],"ca":"https://acme-v02.api.letsencrypt.org/direct
ory","account":""}
Nov 18 03:31:29 ip-10-0-1-150.us-west-2.compute.internal caddy[21318]: {"level":"info","ts":1731900689.0282648,"logger":"http","msg":"done waiting on internal rate limiter","identifiers":["dev.sophistree.app"],"ca":"https://acme-v02.api.letsencrypt.org/d
irectory","account":""}
Nov 18 03:31:29 ip-10-0-1-150.us-west-2.compute.internal caddy[21318]: {"level":"info","ts":1731900689.0292134,"logger":"tls.obtain","msg":"acquiring lock","identifier":"test.sophistree.app"}
Nov 18 03:31:29 ip-10-0-1-150.us-west-2.compute.internal caddy[21318]: {"level":"warn","ts":1731900689.0306306,"logger":"tls","msg":"storage cleaning happened too recently; skipping for now","storage":"FileStorage:/var/lib/caddy/.local/share/caddy","inst
ance":"faa1e50c-a227-415a-9a47-888257ef578c","try_again":1731987089.0306277,"try_again_in":86399.999999453}
Nov 18 03:31:29 ip-10-0-1-150.us-west-2.compute.internal caddy[21318]: {"level":"info","ts":1731900689.031398,"logger":"tls","msg":"finished cleaning storage units"}
Nov 18 03:31:29 ip-10-0-1-150.us-west-2.compute.internal caddy[21318]: {"level":"info","ts":1731900689.0327473,"logger":"tls.obtain","msg":"lock acquired","identifier":"test.sophistree.app"}
Nov 18 03:31:29 ip-10-0-1-150.us-west-2.compute.internal caddy[21318]: {"level":"info","ts":1731900689.0329683,"logger":"tls.obtain","msg":"obtaining certificate","identifier":"test.sophistree.app"}
Nov 18 03:31:29 ip-10-0-1-150.us-west-2.compute.internal caddy[21318]: {"level":"debug","ts":1731900689.0330014,"logger":"events","msg":"event","name":"cert_obtaining","id":"02ad9e4d-fa9a-4b25-bc32-b07aab97a393","origin":"tls","data":{"identifier":"test.
sophistree.app"}}
Nov 18 03:31:29 ip-10-0-1-150.us-west-2.compute.internal caddy[21318]: {"level":"debug","ts":1731900689.0334213,"logger":"tls.obtain","msg":"trying issuer 1/2","issuer":"acme-v02.api.letsencrypt.org-directory"}
Nov 18 03:31:29 ip-10-0-1-150.us-west-2.compute.internal caddy[21318]: {"level":"info","ts":1731900689.0337565,"logger":"http","msg":"waiting on internal rate limiter","identifiers":["test.sophistree.app"],"ca":"https://acme-v02.api.letsencrypt.org/direc
tory","account":""}
Nov 18 03:31:29 ip-10-0-1-150.us-west-2.compute.internal caddy[21318]: {"level":"info","ts":1731900689.0337996,"logger":"http","msg":"done waiting on internal rate limiter","identifiers":["test.sophistree.app"],"ca":"https://acme-v02.api.letsencrypt.org/
directory","account":""}
Nov 18 03:31:29 ip-10-0-1-150.us-west-2.compute.internal caddy[21318]: {"level":"debug","ts":1731900689.225926,"logger":"http.acme_client","msg":"http request","method":"GET","url":"https://acme-v02.api.letsencrypt.org/directory","headers":{"User-Agent":
["Caddy/2.7.6 CertMagic acmez (linux; amd64)"]},"response_headers":{"Cache-Control":["public, max-age=0, no-cache"],"Content-Length":["746"],"Content-Type":["application/json"],"Date":["Mon, 18 Nov 2024 03:31:29 GMT"],"Server":["nginx"],"Strict-Transport
-Security":["max-age=604800"],"X-Frame-Options":["DENY"]},"status_code":200}
Nov 18 03:31:29 ip-10-0-1-150.us-west-2.compute.internal caddy[21318]: {"level":"debug","ts":1731900689.2719219,"logger":"http.acme_client","msg":"http request","method":"HEAD","url":"https://acme-v02.api.letsencrypt.org/acme/new-nonce","headers":{"User-
Agent":["Caddy/2.7.6 CertMagic acmez (linux; amd64)"]},"response_headers":{"Cache-Control":["public, max-age=0, no-cache"],"Date":["Mon, 18 Nov 2024 03:31:29 GMT"],"Link":["<https://acme-v02.api.letsencrypt.org/directory>;rel=\"index\""],"Replay-Nonce":[
"usf-jMgXlAxa7wjNAwkD5yEhVN7vocSTiZQpoSLNcsMIvTG1I34"],"Server":["nginx"],"Strict-Transport-Security":["max-age=604800"],"X-Frame-Options":["DENY"]},"status_code":200}
Nov 18 03:31:29 ip-10-0-1-150.us-west-2.compute.internal caddy[21318]: {"level":"debug","ts":1731900689.3223078,"logger":"http.acme_client","msg":"http request","method":"HEAD","url":"https://acme-v02.api.letsencrypt.org/acme/new-nonce","headers":{"User-
Agent":["Caddy/2.7.6 CertMagic acmez (linux; amd64)"]},"response_headers":{"Cache-Control":["public, max-age=0, no-cache"],"Date":["Mon, 18 Nov 2024 03:31:29 GMT"],"Link":["<https://acme-v02.api.letsencrypt.org/directory>;rel=\"index\""],"Replay-Nonce":[
"JNlwFRqrkJnHL-ZHrWEaqv-RSlBpSMwzjNFbzNlvvtFqAXo2o3M"],"Server":["nginx"],"Strict-Transport-Security":["max-age=604800"],"X-Frame-Options":["DENY"]},"status_code":200}
Nov 18 03:31:29 ip-10-0-1-150.us-west-2.compute.internal caddy[21318]: {"level":"debug","ts":1731900689.3703883,"logger":"http.acme_client","msg":"http request","method":"POST","url":"https://acme-v02.api.letsencrypt.org/acme/new-order","headers":{"Conte
nt-Type":["application/jose+json"],"User-Agent":["Caddy/2.7.6 CertMagic acmez (linux; amd64)"]},"response_headers":{"Boulder-Requester":["2062718477"],"Cache-Control":["public, max-age=0, no-cache"],"Content-Length":["306"],"Content-Type":["application/p
roblem+json"],"Date":["Mon, 18 Nov 2024 03:31:29 GMT"],"Link":["<https://acme-v02.api.letsencrypt.org/directory>;rel=\"index\"","<https://letsencrypt.org/docs/rate-limits>;rel=\"help\""],"Replay-Nonce":["JNlwFRqroo909L6GXp2DsIhrvyTGLaEF7-apqRvcf1hvF72FHu
I"],"Retry-After":["52555"],"Server":["nginx"]},"status_code":429}
Nov 18 03:31:29 ip-10-0-1-150.us-west-2.compute.internal caddy[21318]: {"level":"error","ts":1731900689.3705213,"logger":"tls.obtain","msg":"could not get certificate from issuer","identifier":"dev.sophistree.app","issuer":"acme-v02.api.letsencrypt.org-d
irectory","error":"HTTP 429 urn:ietf:params:acme:error:rateLimited - too many certificates (5) already issued for this exact set of domains in the last 168h0m0s, retry after 2024-11-18 18:07:24 UTC: see https://letsencrypt.org/docs/rate-limits/#new-certi
ficates-per-exact-set-of-hostnames"}
Nov 18 03:31:29 ip-10-0-1-150.us-west-2.compute.internal caddy[21318]: {"level":"debug","ts":1731900689.3705423,"logger":"tls.obtain","msg":"trying issuer 2/2","issuer":"acme.zerossl.com-v2-DV90"}
Nov 18 03:31:29 ip-10-0-1-150.us-west-2.compute.internal caddy[21318]: {"level":"warn","ts":1731900689.3707316,"logger":"http","msg":"missing email address for ZeroSSL; it is strongly recommended to set one for next time"}
Nov 18 03:31:29 ip-10-0-1-150.us-west-2.compute.internal caddy[21318]: {"level":"debug","ts":1731900689.5220962,"logger":"http.acme_client","msg":"http request","method":"POST","url":"https://acme-v02.api.letsencrypt.org/acme/new-order","headers":{"Conte
nt-Type":["application/jose+json"],"User-Agent":["Caddy/2.7.6 CertMagic acmez (linux; amd64)"]},"response_headers":{"Boulder-Requester":["2062718477"],"Cache-Control":["public, max-age=0, no-cache"],"Content-Length":["345"],"Content-Type":["application/j
son"],"Date":["Mon, 18 Nov 2024 03:31:29 GMT"],"Link":["<https://acme-v02.api.letsencrypt.org/directory>;rel=\"index\""],"Location":["https://acme-v02.api.letsencrypt.org/acme/order/2062718477/324207004377"],"Replay-Nonce":["JNlwFRqr4cl7IasIRQnCRZSOyB8a8
Caim9h4T2Y5E7s719VO_3k"],"Server":["nginx"],"Strict-Transport-Security":["max-age=604800"],"X-Frame-Options":["DENY"]},"status_code":201}
Nov 18 03:31:29 ip-10-0-1-150.us-west-2.compute.internal caddy[21318]: {"level":"debug","ts":1731900689.5770843,"logger":"http.acme_client","msg":"http request","method":"POST","url":"https://acme-v02.api.letsencrypt.org/acme/authz-v3/431739627647","head
ers":{"Content-Type":["application/jose+json"],"User-Agent":["Caddy/2.7.6 CertMagic acmez (linux; amd64)"]},"response_headers":{"Boulder-Requester":["2062718477"],"Cache-Control":["public, max-age=0, no-cache"],"Content-Length":["803"],"Content-Type":["a
pplication/json"],"Date":["Mon, 18 Nov 2024 03:31:29 GMT"],"Link":["<https://acme-v02.api.letsencrypt.org/directory>;rel=\"index\""],"Replay-Nonce":["usf-jMgXX3uFuXrrMkr-ANTrnjWeOuwtT7mPmcreQQ_33UdDcz8"],"Server":["nginx"],"Strict-Transport-Security":["m
ax-age=604800"],"X-Frame-Options":["DENY"]},"status_code":200}
Nov 18 03:31:29 ip-10-0-1-150.us-west-2.compute.internal caddy[21318]: {"level":"info","ts":1731900689.5786064,"logger":"http.acme_client","msg":"trying to solve challenge","identifier":"test.sophistree.app","challenge_type":"http-01","ca":"https://acme-
v02.api.letsencrypt.org/directory"}
Nov 18 03:31:29 ip-10-0-1-150.us-west-2.compute.internal caddy[21318]: {"level":"debug","ts":1731900689.580133,"logger":"http.acme_client","msg":"waiting for solver before continuing","identifier":"test.sophistree.app","challenge_type":"http-01"}
Nov 18 03:31:29 ip-10-0-1-150.us-west-2.compute.internal caddy[21318]: {"level":"debug","ts":1731900689.5801575,"logger":"http.acme_client","msg":"done waiting for solver","identifier":"test.sophistree.app","challenge_type":"http-01"}
Nov 18 03:31:29 ip-10-0-1-150.us-west-2.compute.internal caddy[21318]: {"level":"debug","ts":1731900689.6533482,"logger":"http.acme_client","msg":"http request","method":"POST","url":"https://acme-v02.api.letsencrypt.org/acme/chall-v3/431739627647/8kkO2Q
","headers":{"Content-Type":["application/jose+json"],"User-Agent":["Caddy/2.7.6 CertMagic acmez (linux; amd64)"]},"response_headers":{"Boulder-Requester":["2062718477"],"Cache-Control":["public, max-age=0, no-cache"],"Content-Length":["187"],"Content-Type":["application/json"],"Date":["Mon, 18 Nov 2024 03:31:29 GMT"],"Link":["<https://acme-v02.api.letsencrypt.org/directory>;rel=\"index\"","<https://acme-v02.api.letsencrypt.org/acme/authz-v3/431739627647>;rel=\"up\""],"Location":["https://acme-v02.api.letsencrypt.org/acme/chall-v3/431739627647/8kkO2Q"],"Replay-Nonce":["JNlwFRqr730xjiPbKW5wsEsE7FPddNMsZHVaU9LCveFa8yPQ9R0"],"Server":["nginx"],"Strict-Transport-Security":["max-age=604800"],"X-Frame-Options":["DENY"]},"status_code":200}
Nov 18 03:31:29 ip-10-0-1-150.us-west-2.compute.internal caddy[21318]: {"level":"debug","ts":1731900689.6534462,"logger":"http.acme_client","msg":"challenge accepted","identifier":"test.sophistree.app","challenge_type":"http-01"}
Nov 18 03:31:29 ip-10-0-1-150.us-west-2.compute.internal caddy[21318]: {"level":"error","ts":1731900689.7698932,"logger":"tls.obtain","msg":"could not get certificate from issuer","identifier":"dev.sophistree.app","issuer":"acme.zerossl.com-v2-DV90","error":"account pre-registration callback: failed getting EAB credentials: HTTP 422: caddy_legacy_user_removed (code 2977)"}
Nov 18 03:31:29 ip-10-0-1-150.us-west-2.compute.internal caddy[21318]: {"level":"debug","ts":1731900689.76995,"logger":"events","msg":"event","name":"cert_failed","id":"1a79b2e8-176b-4cd5-b95e-39a7485f571a","origin":"tls","data":{"error":{},"identifier":"dev.sophistree.app","issuers":["acme-v02.api.letsencrypt.org-directory","acme.zerossl.com-v2-DV90"],"renewal":false}}
Nov 18 03:31:29 ip-10-0-1-150.us-west-2.compute.internal caddy[21318]: {"level":"error","ts":1731900689.7699876,"logger":"tls.obtain","msg":"will retry","error":"[dev.sophistree.app] Obtain: account pre-registration callback: failed getting EAB credentials: HTTP 422: caddy_legacy_user_removed (code 2977)","attempt":1,"retrying_in":60,"elapsed":0.742820096,"max_duration":2592000}
Nov 18 03:31:29 ip-10-0-1-150.us-west-2.compute.internal caddy[21318]: {"level":"debug","ts":1731900689.9569058,"logger":"http.acme_client","msg":"http request","method":"POST","url":"https://acme-v02.api.letsencrypt.org/acme/authz-v3/431739627647","headers":{"Content-Type":["application/jose+json"],"User-Agent":["Caddy/2.7.6 CertMagic acmez (linux; amd64)"]},"response_headers":{"Boulder-Requester":["2062718477"],"Cache-Control":["public, max-age=0, no-cache"],"Content-Length":["756"],"Content-Type":["application/json"],"Date":["Mon, 18 Nov 2024 03:31:29 GMT"],"Link":["<https://acme-v02.api.letsencrypt.org/directory>;rel=\"index\""],"Replay-Nonce":["JNlwFRqrunBbAfI-K4B4T1pWbAWuHhBYsr5_00OiveU9JGAQP-s"],"Server":["nginx"],"Strict-Transport-Security":["max-age=604800"],"X-Frame-Options":["DENY"]},"status_code":200}
Nov 18 03:31:29 ip-10-0-1-150.us-west-2.compute.internal caddy[21318]: {"level":"error","ts":1731900689.9571407,"logger":"http.acme_client","msg":"challenge failed","identifier":"test.sophistree.app","challenge_type":"http-01","problem":{"type":"urn:ietf:params:acme:error:dns","title":"","detail":"DNS problem: NXDOMAIN looking up A for test.sophistree.app - check that a DNS record exists for this domain; DNS problem: NXDOMAIN looking up AAAA for test.sophistree.app - check that a DNS record exists for this domain","instance":"","subproblems":[]}}
Nov 18 03:31:29 ip-10-0-1-150.us-west-2.compute.internal caddy[21318]: {"level":"error","ts":1731900689.9571822,"logger":"http.acme_client","msg":"validating authorization","identifier":"test.sophistree.app","problem":{"type":"urn:ietf:params:acme:error:dns","title":"","detail":"DNS problem: NXDOMAIN looking up A for test.sophistree.app - check that a DNS record exists for this domain; DNS problem: NXDOMAIN looking up AAAA for test.sophistree.app - check that a DNS record exists for this domain","instance":"","subproblems":[]},"order":"https://acme-v02.api.letsencrypt.org/acme/order/2062718477/324207004377","attempt":1,"max_attempts":3}
Nov 18 03:31:31 ip-10-0-1-150.us-west-2.compute.internal caddy[21318]: {"level":"debug","ts":1731900691.1506338,"logger":"http.acme_client","msg":"http request","method":"POST","url":"https://acme-v02.api.letsencrypt.org/acme/new-order","headers":{"Content-Type":["application/jose+json"],"User-Agent":["Caddy/2.7.6 CertMagic acmez (linux; amd64)"]},"response_headers":{"Boulder-Requester":["2062718477"],"Cache-Control":["public, max-age=0, no-cache"],"Content-Length":["345"],"Content-Type":["application/json"],"Date":["Mon, 18 Nov 2024 03:31:31 GMT"],"Link":["<https://acme-v02.api.letsencrypt.org/directory>;rel=\"index\""],"Location":["https://acme-v02.api.letsencrypt.org/acme/order/2062718477/324207010367"],"Replay-Nonce":["JNlwFRqrqPSNF2fSi3E0vZwBCW9_mZAMn_8VhV0iv7lEORMI5HU"],"Server":["nginx"],"Strict-Transport-Security":["max-age=604800"],"X-Frame-Options":["DENY"]},"status_code":201}
Nov 18 03:31:31 ip-10-0-1-150.us-west-2.compute.internal caddy[21318]: {"level":"debug","ts":1731900691.2067866,"logger":"http.acme_client","msg":"http request","method":"POST","url":"https://acme-v02.api.letsencrypt.org/acme/authz-v3/431739635687","headers":{"Content-Type":["application/jose+json"],"User-Agent":["Caddy/2.7.6 CertMagic acmez (linux; amd64)"]},"response_headers":{"Boulder-Requester":["2062718477"],"Cache-Control":["public, max-age=0, no-cache"],"Content-Length":["803"],"Content-Type":["application/json"],"Date":["Mon, 18 Nov 2024 03:31:31 GMT"],"Link":["<https://acme-v02.api.letsencrypt.org/directory>;rel=\"index\""],"Replay-Nonce":["usf-jMgX_Fbgq_j__E8t8DqBafHTYQ6ZW7BiydET85PUrUbx2OY"],"Server":["nginx"],"Strict-Transport-Security":["max-age=604800"],"X-Frame-Options":["DENY"]},"status_code":200}
Nov 18 03:31:31 ip-10-0-1-150.us-west-2.compute.internal caddy[21318]: {"level":"debug","ts":1731900691.2069333,"logger":"http.acme_client","msg":"no solver configured","challenge_type":"dns-01"}
Nov 18 03:31:31 ip-10-0-1-150.us-west-2.compute.internal caddy[21318]: {"level":"info","ts":1731900691.2069435,"logger":"http.acme_client","msg":"trying to solve challenge","identifier":"test.sophistree.app","challenge_type":"tls-alpn-01","ca":"https://acme-v02.api.letsencrypt.org/directory"}
Nov 18 03:31:31 ip-10-0-1-150.us-west-2.compute.internal caddy[21318]: {"level":"debug","ts":1731900691.208131,"logger":"http.acme_client","msg":"waiting for solver before continuing","identifier":"test.sophistree.app","challenge_type":"tls-alpn-01"}
Nov 18 03:31:31 ip-10-0-1-150.us-west-2.compute.internal caddy[21318]: {"level":"debug","ts":1731900691.2081542,"logger":"http.acme_client","msg":"done waiting for solver","identifier":"test.sophistree.app","challenge_type":"tls-alpn-01"}
Nov 18 03:31:31 ip-10-0-1-150.us-west-2.compute.internal caddy[21318]: {"level":"debug","ts":1731900691.208402,"logger":"http.stdlib","msg":"http: TLS handshake error from 127.0.0.1:53954: EOF"}
Nov 18 03:31:31 ip-10-0-1-150.us-west-2.compute.internal caddy[21318]: {"level":"debug","ts":1731900691.26258,"logger":"http.acme_client","msg":"http request","method":"POST","url":"https://acme-v02.api.letsencrypt.org/acme/chall-v3/431739635687/kaNfDg","headers":{"Content-Type":["application/jose+json"],"User-Agent":["Caddy/2.7.6 CertMagic acmez (linux; amd64)"]},"response_headers":{"Boulder-Requester":["2062718477"],"Cache-Control":["public, max-age=0, no-cache"],"Content-Length":["191"],"Content-Type":["application/json"],"Date":["Mon, 18 Nov 2024 03:31:31 GMT"],"Link":["<https://acme-v02.api.letsencrypt.org/directory>;rel=\"index\"","<https://acme-v02.api.letsencrypt.org/acme/authz-v3/431739635687>;rel=\"up\""],"Location":["https://acme-v02.api.letsencrypt.org/acme/chall-v3/431739635687/kaNfDg"],"Replay-Nonce":["usf-jMgX6zfztMQoLX8wrmQJaGJ5XBzQa3G-YDbZh4QQ0U791ro"],"Server":["nginx"],"Strict-Transport-Security":["max-age=604800"],"X-Frame-Options":["DENY"]},"status_code":200}
Nov 18 03:31:31 ip-10-0-1-150.us-west-2.compute.internal caddy[21318]: {"level":"debug","ts":1731900691.2627342,"logger":"http.acme_client","msg":"challenge accepted","identifier":"test.sophistree.app","challenge_type":"tls-alpn-01"}
Nov 18 03:31:31 ip-10-0-1-150.us-west-2.compute.internal caddy[21318]: {"level":"debug","ts":1731900691.5639617,"logger":"http.acme_client","msg":"http request","method":"POST","url":"https://acme-v02.api.letsencrypt.org/acme/authz-v3/431739635687","headers":{"Content-Type":["application/jose+json"],"User-Agent":["Caddy/2.7.6 CertMagic acmez (linux; amd64)"]},"response_headers":{"Boulder-Requester":["2062718477"],"Cache-Control":["public, max-age=0, no-cache"],"Content-Length":["886"],"Content-Type":["application/json"],"Date":["Mon, 18 Nov 2024 03:31:31 GMT"],"Link":["<https://acme-v02.api.letsencrypt.org/directory>;rel=\"index\""],"Replay-Nonce":["usf-jMgXLS8oi_mCtxg_T2UwuFUgbDUckNdnRciIsjU1e_IbBiU"],"Server":["nginx"],"Strict-Transport-Security":["max-age=604800"],"X-Frame-Options":["DENY"]},"status_code":200}
Nov 18 03:31:31 ip-10-0-1-150.us-west-2.compute.internal caddy[21318]: {"level":"error","ts":1731900691.5641873,"logger":"http.acme_client","msg":"challenge failed","identifier":"test.sophistree.app","challenge_type":"tls-alpn-01","problem":{"type":"urn:ietf:params:acme:error:dns","title":"","detail":"DNS problem: NXDOMAIN looking up A for test.sophistree.app - check that a DNS record exists for this domain; DNS problem: NXDOMAIN looking up AAAA for test.sophistree.app - check that a DNS record exists for this domain","instance":"","subproblems":[]}}
Nov 18 03:31:31 ip-10-0-1-150.us-west-2.compute.internal caddy[21318]: {"level":"error","ts":1731900691.5642233,"logger":"http.acme_client","msg":"validating authorization","identifier":"test.sophistree.app","problem":{"type":"urn:ietf:params:acme:error:dns","title":"","detail":"DNS problem: NXDOMAIN looking up A for test.sophistree.app - check that a DNS record exists for this domain; DNS problem: NXDOMAIN looking up AAAA for test.sophistree.app - check that a DNS record exists for this domain","instance":"","subproblems":[]},"order":"https://acme-v02.api.letsencrypt.org/acme/order/2062718477/324207010367","attempt":2,"max_attempts":3}
Nov 18 03:31:31 ip-10-0-1-150.us-west-2.compute.internal caddy[21318]: {"level":"error","ts":1731900691.564257,"logger":"tls.obtain","msg":"could not get certificate from issuer","identifier":"test.sophistree.app","issuer":"acme-v02.api.letsencrypt.org-directory","error":"HTTP 400 urn:ietf:params:acme:error:dns - DNS problem: NXDOMAIN looking up A for test.sophistree.app - check that a DNS record exists for this domain; DNS problem: NXDOMAIN looking up AAAA for test.sophistree.app - check that a DNS record exists for this domain"}
Nov 18 03:31:31 ip-10-0-1-150.us-west-2.compute.internal caddy[21318]: {"level":"debug","ts":1731900691.5642753,"logger":"tls.obtain","msg":"trying issuer 2/2","issuer":"acme.zerossl.com-v2-DV90"}
Nov 18 03:31:31 ip-10-0-1-150.us-west-2.compute.internal caddy[21318]: {"level":"warn","ts":1731900691.5644484,"logger":"http","msg":"missing email address for ZeroSSL; it is strongly recommended to set one for next time"}
Nov 18 03:31:31 ip-10-0-1-150.us-west-2.compute.internal caddy[21318]: {"level":"error","ts":1731900691.7441924,"logger":"tls.obtain","msg":"could not get certificate from issuer","identifier":"test.sophistree.app","issuer":"acme.zerossl.com-v2-DV90","error":"account pre-registration callback: failed getting EAB credentials: HTTP 422: caddy_legacy_user_removed (code 2977)"}
Nov 18 03:31:31 ip-10-0-1-150.us-west-2.compute.internal caddy[21318]: {"level":"debug","ts":1731900691.7442496,"logger":"events","msg":"event","name":"cert_failed","id":"ddcceea4-0b80-4da4-922a-215bf13c0b90","origin":"tls","data":{"error":{},"identifier":"test.sophistree.app","issuers":["acme-v02.api.letsencrypt.org-directory","acme.zerossl.com-v2-DV90"],"renewal":false}}
Nov 18 03:31:31 ip-10-0-1-150.us-west-2.compute.internal caddy[21318]: {"level":"error","ts":1731900691.7442818,"logger":"tls.obtain","msg":"will retry","error":"[test.sophistree.app] Obtain: account pre-registration callback: failed getting EAB credenti
als: HTTP 422: caddy_legacy_user_removed (code 2977)","attempt":1,"retrying_in":60,"elapsed":2.71151537,"max_duration":2592000}

3. Caddy version:

$ caddy version
v2.7.6 h1:w0NymbG2m9PcvKWsrXO6EEkY9Ru4FJK8uQbYcev1p3A=

4. How I installed and ran Caddy:

a. System environment:

AWS EC2 Linux:

$ cat /etc/system-release
Amazon Linux release 2023.6.20241111 (Amazon Linux)

b. Command:

dnf install -y 'dnf-command(copr)'
dnf copr enable -y @caddy/caddy epel-7-$(arch)
dnf install -y caddy
systemctl enable caddy
systemctl start caddy

c. Service/unit/compose file:

d. My complete Caddy config:

{
  debug
}

dev.sophistree.app, test.sophistree.app {
  reverse_proxy localhost:3000
  log {
    output file /var/log/caddy/access.log
  }
}

5. Links to relevant resources:

Omitted because new users cannot post more than 4 links. See above.

2

Hello @sophistree,

There is no override for Let’s Encrypt Rate Limits; here is there Workaround documentation.

Please consider using the Staging Environment - Let’s Encrypt until you’ve debugged the error; as the limits are much higher.

Please see:

3

Thank you, Bruce. I tried the workaround and it didn’t work (see the part of my post mentioning dev.sophistree.app, test.sophistree.app. I might not have configured my Caddyfile correctly to trigger a different ‘exact set of hostnames’. Does it look correct?

Also please note that I can’t use the staging environment (this is also in my post) because the .app TLD does not allow disabling HSTS.

1 Like
4

The HTTP-01 challenge allows use of HTTPS.

“The HTTP-01 challenge can only be done on port 80.”

" Let’s Encrypt implementation of the HTTP-01 challenge follows redirects, up to 10 redirects deep. It only accepts redirects to “http:” or “https:”, and only to ports 80 or 443. It does not accept redirects to IP addresses. When redirected to an HTTPS URL, it does not validate certificates (since this challenge is intended to bootstrap valid certificates, it may encounter self-signed or expired certificates along the way)."

So it would seem that is more of configuration issue.

5

Or consider using TLS-ALPN-01.

6

Thanks, @Bruce5051. When I say that I can’t use the staging environment, what I mean is: requesting the certificate from the staging environment works. But clients cannot connect to the service because staging certs are not signed by a root cert.

For example, Firefox says this when I try to load https://dev.sophistree.app:

Did Not Connect: Potential Security Issue

Firefox detected a potential security threat and did not continue to dev.sophistree.app because this website requires a secure connection.

What can you do about it?

dev.sophistree.app has a security policy called HTTP Strict Transport Security (HSTS), which means that Firefox can only connect to it securely. You can’t add an exception to visit this site.

The issue is most likely with the website, and there is nothing you can do to resolve it.

Normally HSTS is configured by a server by sending an HTTP header Strict-Transport-Security. But some TLDs have ‘preloaded HSTS’, meaning that they always require a secure cert, and there’s no way to turn it off.

Although come to think of it, I guess I could install the staging root cert locally to try and get around this.

7

Correct, the staging environment is for testing and debugging.

When a rate limit is reached one can always wait until the expiration of the limit happens.

8

And here Let's Debug Toolkit is showing
“The next time this certificate can be issued is 22 Nov 2024 21:46:59 UTC”

9

Right, but I can’t really test or debug if a client can’t connect to the service.

Coming back to the workaround that mentions “set of hostnames” and blog.example.com, do you happen to know an example Caddyfile that triggers this workaround? I tried it, and it did not seem to work. Thanks!

10

Oh my. That’s a helpful tool. If this is correct, and I can’t iterate on this project for a week, then I guess I will need to look into requesting signed certificates myself and managing them.

1 Like
11

(For the record I opted not to do this, since at least on macOS Chrome it appeared to require installing the cert in my system certs in “keychain” which I think would affect many or all applications on my laptop, so wasn’t a safe alternative.)

12

Please use the latest version, v2.8.4

You really should not be wiping out Caddy’s storage like that. You should keep persistent storage between installations to get around this issue. You could backup/sync your storage with the S3 or Redis storage plugins.

1 Like
13

Thank you, @francislavoie. I am new to Caddy and was not aware of storage plugins. I will look into Modules - Caddy Documentation.

14

I think I have S3 storage working. My setup may be somewhat particular to my project, but for an example of configuring Caddy to use S3 storage while running in Docker, I have this commit. Thanks, @francislavoie & @Bruce5051.

1 Like
15

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.