Internal rate limiter & HTTP request error

Help
1

1. Caddy version (caddy version):

v2.3.0 h1:fnrqJLa3G5vfxcxmOH/+kJOcunPLhSBnjgIvjXV/QTA=

2. How I run Caddy:

a. System environment:

OS: CentOS 8
Systemd

b. Command:

sudo systemctl start caddy

c. Service/unit/compose file:

[Unit]
Description=Caddy
Documentation=https://caddyserver.com/docs/
After=network.target network-online.target
Requires=network-online.target

[Service]
User=caddy
Group=caddy
ExecStart=/usr/bin/caddy run --environ --config /etc/caddy/Caddyfile
ExecReload=/usr/bin/caddy reload --config /etc/caddy/Caddyfile
TimeoutStopSec=5s
LimitNOFILE=1048576
LimitNPROC=512
PrivateTmp=true
ProtectSystem=full
AmbientCapabilities=CAP_NET_BIND_SERVICE

[Install]
WantedBy=multi-user.target

d. My complete Caddyfile or JSON config:

{
	email support@mail.com
}
:443 {
	root * /var/www/project/public/
	file_server
	encode zstd gzip

	# Security headers
	import /etc/caddy/security_headers.conf

	# Logs
	log {
		format single_field common_log
		output file /var/log/caddy/access.log {
			roll_size 100mb
			roll_keep 10
			roll_keep_for 168h
		}
	}
	# SSL Settings
	tls {
		issuer acme
		issuer zerossl [API_KEY]
		on_demand
	}

	# PHP-FPM Configuration
	php_fastcgi unix//run/php-fpm/www.sock
}

3. The problem I’m having:

We are trying to issue SSL certs to our clients. We have thousands of users. We are doing around ~400 request in 1 minute. Even though we use multi-issuer, domains are waiting in internal limitter so long.

Our LE rate limits are like below, caddy’s internal rate limiter doesn’t allow for issuing new certs:
1300 certificates/registered domain/week; 2500 pending authorizations per account; 10,000 new orders/3 hours/account

And also most of the time, ZeroSSL challenges fails with “HTTP request failed” error message.

4. Error messages and/or full log output:

Mar  9 13:37:48 PSP caddy[270320]: {"level":"info","ts":1615297068.1177015,"logger":"tls.on_demand","msg":"obtaining new certificate","server_name":"hivestatus.usehive.com"}
Mar  9 13:37:48 PSP caddy[270320]: {"level":"info","ts":1615297068.1181648,"logger":"tls.obtain","msg":"acquiring lock","identifier":"hivestatus.usehive.com"}
Mar  9 13:37:48 PSP caddy[270320]: {"level":"info","ts":1615297068.118345,"logger":"tls.obtain","msg":"lock acquired","identifier":"hivestatus.usehive.com"}
Mar  9 13:37:48 PSP caddy[270320]: {"level":"info","ts":1615297068.1198642,"logger":"tls.issuance.acme","msg":"waiting on internal rate limiter","identifiers":["hivestatus.usehive.com"]}
Mar  9 13:37:48 PSP caddy[270320]: {"level":"info","ts":1615297068.622123,"logger":"tls.issuance.acme","msg":"waiting on internal rate limiter","identifiers":["webhealth.heymancpa.com"]}
Mar  9 13:37:48 PSP caddy[270320]: {"level":"info","ts":1615297068.6221607,"logger":"tls.obtain","msg":"releasing lock","identifier":"webhealth.heymancpa.com"}
Mar  9 13:37:48 PSP caddy[270320]: {"level":"info","ts":1615297068.841911,"logger":"tls.issuance.acme","msg":"waiting on internal rate limiter","identifiers":["status.bajaelping.com"]}
Mar  9 13:37:48 PSP caddy[270320]: {"level":"info","ts":1615297068.8419623,"logger":"tls.obtain","msg":"releasing lock","identifier":"status.bajaelping.com"}
Mar  9 13:37:50 PSP filebeat[259192]: 2021-03-09T13:37:50.342Z#011INFO#011[monitoring]#011log/log.go:145#011Non-zero metrics in the last 30s#011{"monitoring": {"metrics": {"beat":{"cgroup":{"memory":{"mem":{"usage":{"bytes":12288}}}},"cpu":{"system":{"ticks":141830,"time":{"ms":37}},"total":{"ticks":766050,"time":{"ms":211},"value":766050},"user":{"ticks":624220,"time":{"ms":174}}},"handles":{"limit":{"hard":262144,"soft":1024},"open":15},"info":{"ephemeral_id":"d9d16c2e-8493-4a7a-bd2a-9e1d4c7f46d2","uptime":{"ms":86702891}},"memstats":{"gc_next":21253120,"memory_alloc":12633880,"memory_total":89944141920},"runtime":{"goroutines":52}},"filebeat":{"events":{"active":-32,"added":1478,"done":1510},"harvester":{"open_files":1,"running":1}},"libbeat":{"config":{"module":{"running":1}},"output":{"events":{"acked":1510,"active":-16,"batches":42,"total":1494},"read":{"bytes":354904},"write":{"bytes":1833081}},"pipeline":{"clients":2,"events":{"active":52,"published":1478,"total":1478},"queue":{"acked":1510}}},"registrar":{"states":{"current":1,"update":1510},"writes":{"success":43,"total":43}},"system":{"load":{"1":1.02,"15":1.26,"5":1.2,"norm":{"1":0.255,"15":0.315,"5":0.3}}}}}}
Mar  9 13:37:52 PSP caddy[270320]: {"level":"error","ts":1615297072.791259,"logger":"tls.on_demand","msg":"renewing certificate on-demand failed","subjects":["systems.clevermessenger.com"],"not_after":1568030565,"error":"unable to acquire lock 'issue_cert_systems.clevermessenger.com': context deadline exceeded"}
Mar  9 13:37:52 PSP caddy[270320]: {"level":"error","ts":1615297072.7949288,"logger":"tls.on_demand","msg":"renewing certificate on-demand failed","subjects":["status.smartbuddy.vn"],"not_after":1605504357,"error":"unable to acquire lock 'issue_cert_status.smartbuddy.vn': context deadline exceeded"}
Mar  9 13:37:52 PSP caddy[270320]: {"level":"info","ts":1615297072.8122733,"logger":"tls.issuance.acme","msg":"waiting on internal rate limiter","identifiers":["www.colotti.net"]}
Mar  9 13:37:52 PSP caddy[270320]: {"level":"info","ts":1615297072.812308,"logger":"tls.issuance.acme","msg":"done waiting on internal rate limiter","identifiers":["www.colotti.net"]}
Mar  9 13:37:52 PSP caddy[270320]: {"level":"warn","ts":1615297072.8123689,"logger":"tls.issuance.acme.acme_client","msg":"HTTP request failed; retrying","url":"https://acme.zerossl.com/v2/DV90/newNonce","error":"performing request: Head \"https://acme.zerossl.com/v2/DV90/newNonce\": context deadline exceeded"}
Mar  9 13:37:52 PSP caddy[270320]: {"level":"error","ts":1615297072.812403,"logger":"tls.obtain","msg":"will retry","error":"[www.colotti.net] Obtain: [www.colotti.net] creating new order: fetching new nonce from server: context deadline exceeded (ca=https://acme.zerossl.com/v2/DV90)","attempt":1,"retrying_in":60,"elapsed":90.000261708,"max_duration":2592000}
Mar  9 13:37:52 PSP caddy[270320]: {"level":"info","ts":1615297072.8124192,"logger":"tls.obtain","msg":"releasing lock","identifier":"www.colotti.net"}
Mar  9 13:37:52 PSP caddy[270320]: {"level":"info","ts":1615297072.8433905,"logger":"tls.issuance.acme","msg":"waiting on internal rate limiter","identifiers":["hosting-status.codeinwp.com"]}
Mar  9 13:37:52 PSP caddy[270320]: {"level":"info","ts":1615297072.8434231,"logger":"tls.obtain","msg":"releasing lock","identifier":"hosting-status.codeinwp.com"}
Mar  9 13:37:52 PSP caddy[270320]: {"level":"info","ts":1615297072.854439,"logger":"tls.issuance.acme","msg":"waiting on internal rate limiter","identifiers":["status.upscri.be"]}
Mar  9 13:37:52 PSP caddy[270320]: {"level":"info","ts":1615297072.8544936,"logger":"tls.obtain","msg":"releasing lock","identifier":"status.upscri.be"}
Mar  9 13:37:52 PSP caddy[270320]: {"level":"info","ts":1615297072.8844128,"logger":"tls.issuance.acme","msg":"waiting on internal rate limiter","identifiers":["isitup.dberry2.com"]}
Mar  9 13:37:52 PSP caddy[270320]: {"level":"info","ts":1615297072.8844845,"logger":"tls.obtain","msg":"releasing lock","identifier":"isitup.dberry2.com"}
Mar  9 13:37:52 PSP caddy[270320]: {"level":"info","ts":1615297072.8869433,"logger":"tls.issuance.acme","msg":"waiting on internal rate limiter","identifiers":["status.garvitkothari.in"]}
Mar  9 13:37:52 PSP caddy[270320]: {"level":"info","ts":1615297072.88699,"logger":"tls.obtain","msg":"releasing lock","identifier":"status.garvitkothari.in"}
Mar  9 13:37:52 PSP caddy[270320]: {"level":"info","ts":1615297072.8910322,"logger":"tls.issuance.acme","msg":"waiting on internal rate limiter","identifiers":["monitorvmb.vivagroup.co.id"]}
Mar  9 13:37:52 PSP caddy[270320]: {"level":"info","ts":1615297072.8910656,"logger":"tls.obtain","msg":"releasing lock","identifier":"monitorvmb.vivagroup.co.id"}
Mar  9 13:37:52 PSP caddy[270320]: {"level":"info","ts":1615297072.8975718,"logger":"tls.issuance.acme","msg":"waiting on internal rate limiter","identifiers":["monitor.i99.com.br"]}
Mar  9 13:37:52 PSP caddy[270320]: {"level":"info","ts":1615297072.8976157,"logger":"tls.obtain","msg":"releasing lock","identifier":"monitor.i99.com.br"}
Mar  9 13:37:52 PSP caddy[270320]: {"level":"info","ts":1615297072.9006479,"logger":"tls.issuance.acme","msg":"waiting on internal rate limiter","identifiers":["statuspage.3on.se"]}
Mar  9 13:37:52 PSP caddy[270320]: {"level":"info","ts":1615297072.9006727,"logger":"tls.obtain","msg":"releasing lock","identifier":"statuspage.3on.se"}
Mar  9 13:37:52 PSP caddy[270320]: {"level":"info","ts":1615297072.902733,"logger":"tls.issuance.acme","msg":"waiting on internal rate limiter","identifiers":["www.yoloo.info"]}
Mar  9 13:37:52 PSP caddy[270320]: {"level":"info","ts":1615297072.9027567,"logger":"tls.obtain","msg":"releasing lock","identifier":"www.yoloo.info"}
Mar  9 13:37:52 PSP caddy[270320]: {"level":"info","ts":1615297072.913821,"logger":"tls.on_demand","msg":"obtaining new certificate","server_name":"hosting-status.codeinwp.com"}
Mar  9 13:37:52 PSP caddy[270320]: {"level":"info","ts":1615297072.9141517,"logger":"tls.obtain","msg":"acquiring lock","identifier":"hosting-status.codeinwp.com"}
Mar  9 13:37:52 PSP caddy[270320]: {"level":"info","ts":1615297072.9142845,"logger":"tls.obtain","msg":"lock acquired","identifier":"hosting-status.codeinwp.com"}
Mar  9 13:37:52 PSP caddy[270320]: {"level":"info","ts":1615297072.9156165,"logger":"tls.issuance.acme","msg":"waiting on internal rate limiter","identifiers":["hosting-status.codeinwp.com"]}
Mar  9 13:37:52 PSP caddy[270320]: {"level":"error","ts":1615297072.9371827,"logger":"tls.on_demand","msg":"renewing certificate on-demand failed","subjects":["status.hosttocdo.com"],"not_after":1606092236,"error":"unable to acquire lock 'issue_cert_status.hosttocdo.com': context deadline exceeded"}
Mar  9 13:37:52 PSP caddy[270320]: {"level":"info","ts":1615297072.9655807,"logger":"tls.issuance.acme","msg":"waiting on internal rate limiter","identifiers":["madina.spiru.la"]}
Mar  9 13:37:52 PSP caddy[270320]: {"level":"info","ts":1615297072.965612,"logger":"tls.obtain","msg":"releasing lock","identifier":"madina.spiru.la"}
Mar  9 13:37:52 PSP caddy[270320]: {"level":"error","ts":1615297072.9798586,"logger":"tls.on_demand","msg":"renewing certificate on-demand failed","subjects":["status.shower.im"],"not_after":1597274968,"error":"unable to acquire lock 'issue_cert_status.shower.im': context deadline exceeded"}
Mar  9 13:37:52 PSP caddy[270320]: {"level":"info","ts":1615297072.9812086,"logger":"tls.issuance.acme","msg":"waiting on internal rate limiter","identifiers":["status.hexawp.com"]}
Mar  9 13:37:52 PSP caddy[270320]: {"level":"info","ts":1615297072.9812434,"logger":"tls.obtain","msg":"releasing lock","identifier":"status.hexawp.com"}
Mar  9 13:37:54 PSP caddy[270320]: {"level":"info","ts":1615297074.4657636,"logger":"tls.issuance.acme","msg":"waiting on internal rate limiter","identifiers":["status.typhoontech.app"]}
Mar  9 13:37:54 PSP caddy[270320]: {"level":"info","ts":1615297074.4658148,"logger":"tls.obtain","msg":"releasing lock","identifier":"status.typhoontech.app"}
Mar  9 13:37:54 PSP caddy[270320]: {"level":"info","ts":1615297074.7071617,"logger":"tls.on_demand","msg":"obtaining new certificate","server_name":"status.typhoontech.app"}
Mar  9 13:37:54 PSP caddy[270320]: {"level":"info","ts":1615297074.7076058,"logger":"tls.obtain","msg":"acquiring lock","identifier":"status.typhoontech.app"}
Mar  9 13:37:54 PSP caddy[270320]: {"level":"info","ts":1615297074.7077622,"logger":"tls.obtain","msg":"lock acquired","identifier":"status.typhoontech.app"}
Mar  9 13:37:54 PSP caddy[270320]: {"level":"info","ts":1615297074.7089229,"logger":"tls.issuance.acme","msg":"waiting on internal rate limiter","identifiers":["status.typhoontech.app"]}
Mar  9 13:37:56 PSP caddy[270320]: {"level":"info","ts":1615297076.004046,"logger":"tls.issuance.acme","msg":"waiting on internal rate limiter","identifiers":["status.dnhps.com"]}
Mar  9 13:37:56 PSP caddy[270320]: {"level":"info","ts":1615297076.0041149,"logger":"tls.obtain","msg":"releasing lock","identifier":"status.dnhps.com"}
Mar  9 13:37:58 PSP caddy[270320]: {"level":"info","ts":1615297078.5688443,"logger":"tls.issuance.acme","msg":"waiting on internal rate limiter","identifiers":["apitravel.spiru.la"]}
Mar  9 13:37:58 PSP caddy[270320]: {"level":"info","ts":1615297078.5688987,"logger":"tls.issuance.acme","msg":"done waiting on internal rate limiter","identifiers":["apitravel.spiru.la"]}
Mar  9 13:37:58 PSP caddy[270320]: {"level":"warn","ts":1615297078.5689652,"logger":"tls.issuance.acme.acme_client","msg":"HTTP request failed; retrying","url":"https://acme.zerossl.com/v2/DV90/newNonce","error":"performing request: Head \"https://acme.zerossl.com/v2/DV90/newNonce\": context deadline exceeded"}
Mar  9 13:37:58 PSP caddy[270320]: {"level":"error","ts":1615297078.5690038,"logger":"tls.obtain","msg":"will retry","error":"[apitravel.spiru.la] Obtain: [apitravel.spiru.la] creating new order: fetching new nonce from server: context deadline exceeded (ca=https://acme.zerossl.com/v2/DV90)","attempt":1,"retrying_in":60,"elapsed":90.000223868,"max_duration":2592000}
Mar  9 13:37:58 PSP caddy[270320]: {"level":"info","ts":1615297078.569022,"logger":"tls.obtain","msg":"releasing lock","identifier":"apitravel.spiru.la"}
Mar  9 13:37:58 PSP caddy[270320]: {"level":"info","ts":1615297078.8312547,"logger":"tls.issuance.acme","msg":"waiting on internal rate limiter","identifiers":["status.twistedxmodz.xyz"]}
Mar  9 13:37:58 PSP caddy[270320]: {"level":"info","ts":1615297078.8314202,"logger":"tls.obtain","msg":"releasing lock","identifier":"status.twistedxmodz.xyz"}
Mar  9 13:38:01 PSP caddy[270320]: {"level":"info","ts":1615297081.0060153,"logger":"tls.on_demand","msg":"obtaining new certificate","server_name":"yaiko.status.yaiko.dev"}
Mar  9 13:38:01 PSP caddy[270320]: {"level":"info","ts":1615297081.0065687,"logger":"tls.obtain","msg":"acquiring lock","identifier":"yaiko.status.yaiko.dev"}
Mar  9 13:38:01 PSP caddy[270320]: {"level":"info","ts":1615297081.006781,"logger":"tls.obtain","msg":"lock acquired","identifier":"yaiko.status.yaiko.dev"}
Mar  9 13:38:01 PSP caddy[270320]: {"level":"info","ts":1615297081.008445,"logger":"tls.issuance.acme","msg":"waiting on internal rate limiter","identifiers":["yaiko.status.yaiko.dev"]}
Mar  9 13:38:02 PSP caddy[270320]: {"level":"info","ts":1615297082.5257046,"logger":"tls.issuance.acme","msg":"waiting on internal rate limiter","identifiers":["status.miniserv.pl"]}
Mar  9 13:38:02 PSP caddy[270320]: {"level":"info","ts":1615297082.5257502,"logger":"tls.obtain","msg":"releasing lock","identifier":"status.miniserv.pl"}
Mar  9 13:38:02 PSP caddy[270320]: {"level":"info","ts":1615297082.544717,"logger":"tls.issuance.acme","msg":"waiting on internal rate limiter","identifiers":["stats.webdance.com.au"]}
Mar  9 13:38:02 PSP caddy[270320]: {"level":"info","ts":1615297082.54476,"logger":"tls.obtain","msg":"releasing lock","identifier":"stats.webdance.com.au"}
Mar  9 13:38:02 PSP caddy[270320]: {"level":"info","ts":1615297082.674262,"logger":"tls.issuance.acme","msg":"waiting on internal rate limiter","identifiers":["status.webhoster.pl"]}
Mar  9 13:38:02 PSP caddy[270320]: {"level":"info","ts":1615297082.6743057,"logger":"tls.obtain","msg":"releasing lock","identifier":"status.webhoster.pl"}
Mar  9 13:38:02 PSP caddy[270320]: {"level":"info","ts":1615297082.8393939,"logger":"tls.issuance.acme","msg":"waiting on internal rate limiter","identifiers":["status.ghostrecordz.eu"]}
Mar  9 13:38:02 PSP caddy[270320]: {"level":"info","ts":1615297082.83945,"logger":"tls.obtain","msg":"releasing lock","identifier":"status.ghostrecordz.eu"}
Mar  9 13:38:07 PSP caddy[270320]: {"level":"info","ts":1615297087.2086945,"logger":"tls.on_demand","msg":"attempting certificate renewal","server_name":"status.theddie.edu.pl","identifiers":["status.theddie.edu.pl"],"expiration":1610042637,"remaining":-5254450.208689045}
Mar  9 13:38:07 PSP caddy[270320]: {"level":"info","ts":1615297087.2093134,"logger":"tls.renew","msg":"acquiring lock","identifier":"status.theddie.edu.pl"}
Mar  9 13:38:07 PSP caddy[270320]: {"level":"info","ts":1615297087.2122083,"logger":"tls.on_demand","msg":"attempting certificate renewal","server_name":"status.hugmanrique.me","identifiers":["status.hugmanrique.me"],"expiration":1609013263,"remaining":-6283824.212203651}
Mar  9 13:38:07 PSP caddy[270320]: {"level":"info","ts":1615297087.212539,"logger":"tls.renew","msg":"acquiring lock","identifier":"status.hugmanrique.me"}
Mar  9 13:38:07 PSP caddy[270320]: {"level":"info","ts":1615297087.2128868,"logger":"tls.on_demand","msg":"obtaining new certificate","server_name":"status.shodan.io"}
Mar  9 13:38:07 PSP caddy[270320]: {"level":"info","ts":1615297087.2132835,"logger":"tls.obtain","msg":"acquiring lock","identifier":"status.shodan.io"}
Mar  9 13:38:07 PSP caddy[270320]: {"level":"info","ts":1615297087.2134612,"logger":"tls.obtain","msg":"lock acquired","identifier":"status.shodan.io"}
Mar  9 13:38:07 PSP caddy[270320]: {"level":"info","ts":1615297087.2149608,"logger":"tls.issuance.acme","msg":"waiting on internal rate limiter","identifiers":["status.shodan.io"]}
Mar  9 13:38:07 PSP caddy[270320]: {"level":"info","ts":1615297087.5138009,"logger":"tls.on_demand","msg":"obtaining new certificate","server_name":"uptime.techwithjake.com"}
Mar  9 13:38:07 PSP caddy[270320]: {"level":"info","ts":1615297087.5141878,"logger":"tls.obtain","msg":"acquiring lock","identifier":"uptime.techwithjake.com"}
Mar  9 13:38:07 PSP caddy[270320]: {"level":"info","ts":1615297087.5143194,"logger":"tls.obtain","msg":"lock acquired","identifier":"uptime.techwithjake.com"}
Mar  9 13:38:07 PSP caddy[270320]: {"level":"info","ts":1615297087.5157826,"logger":"tls.issuance.acme","msg":"waiting on internal rate limiter","identifiers":["uptime.techwithjake.com"]}
Mar  9 13:38:07 PSP caddy[270320]: {"level":"info","ts":1615297087.6635063,"logger":"tls.on_demand","msg":"obtaining new certificate","server_name":"stats.colada365.app"}
Mar  9 13:38:07 PSP caddy[270320]: {"level":"info","ts":1615297087.6638849,"logger":"tls.obtain","msg":"acquiring lock","identifier":"stats.colada365.app"}
Mar  9 13:38:08 PSP caddy[270320]: {"level":"info","ts":1615297088.3042514,"logger":"tls.on_demand","msg":"obtaining new certificate","server_name":"stats.cloudcoin.global"}
Mar  9 13:38:08 PSP caddy[270320]: {"level":"info","ts":1615297088.3046052,"logger":"tls.obtain","msg":"acquiring lock","identifier":"stats.cloudcoin.global"}
Mar  9 13:38:08 PSP caddy[270320]: {"level":"error","ts":1615297088.6252325,"logger":"tls.on_demand","msg":"renewing certificate on-demand failed","subjects":["status.rivmedia.co.uk"],"not_after":1609131058,"error":"unable to acquire lock 'issue_cert_status.rivmedia.co.uk': context deadline exceeded"}
Mar  9 13:38:08 PSP caddy[270320]: {"level":"info","ts":1615297088.696749,"logger":"tls.on_demand","msg":"attempting certificate renewal","server_name":"status.rivmedia.co.uk","identifiers":["status.rivmedia.co.uk"],"expiration":1609131058,"remaining":-6166030.696743915}
Mar  9 13:38:08 PSP caddy[270320]: {"level":"info","ts":1615297088.697153,"logger":"tls.renew","msg":"acquiring lock","identifier":"status.rivmedia.co.uk"}
Mar  9 13:38:10 PSP caddy[270320]: {"level":"info","ts":1615297090.7793872,"logger":"tls.on_demand","msg":"obtaining new certificate","server_name":"uptime.87host.net"}
Mar  9 13:38:10 PSP caddy[270320]: {"level":"info","ts":1615297090.7797647,"logger":"tls.obtain","msg":"acquiring lock","identifier":"uptime.87host.net"}
Mar  9 13:38:10 PSP caddy[270320]: {"level":"info","ts":1615297090.7798836,"logger":"tls.obtain","msg":"lock acquired","identifier":"uptime.87host.net"}
Mar  9 13:38:10 PSP caddy[270320]: {"level":"info","ts":1615297090.7810757,"logger":"tls.issuance.acme","msg":"waiting on internal rate limiter","identifiers":["uptime.87host.net"]}
Mar  9 13:38:11 PSP caddy[270320]: {"level":"info","ts":1615297091.139444,"logger":"tls.on_demand","msg":"obtaining new certificate","server_name":"appstatus.ownlab.in"}
Mar  9 13:38:11 PSP caddy[270320]: {"level":"info","ts":1615297091.1398427,"logger":"tls.obtain","msg":"acquiring lock","identifier":"appstatus.ownlab.in"}
Mar  9 13:38:11 PSP caddy[270320]: {"level":"info","ts":1615297091.1400092,"logger":"tls.obtain","msg":"lock acquired","identifier":"appstatus.ownlab.in"}
Mar  9 13:38:11 PSP caddy[270320]: {"level":"info","ts":1615297091.1410894,"logger":"tls.issuance.acme","msg":"waiting on internal rate limiter","identifiers":["appstatus.ownlab.in"]}
Mar  9 13:38:15 PSP caddy[270320]: {"level":"info","ts":1615297095.1038663,"logger":"tls.on_demand","msg":"obtaining new certificate","server_name":"my1drv.github.io"}
Mar  9 13:38:15 PSP caddy[270320]: {"level":"info","ts":1615297095.1042616,"logger":"tls.obtain","msg":"acquiring lock","identifier":"my1drv.github.io"}
Mar  9 13:38:15 PSP caddy[270320]: {"level":"info","ts":1615297095.1044188,"logger":"tls.obtain","msg":"lock acquired","identifier":"my1drv.github.io"}
Mar  9 13:38:15 PSP caddy[270320]: {"level":"info","ts":1615297095.1057913,"logger":"tls.issuance.acme","msg":"waiting on internal rate limiter","identifiers":["my1drv.github.io"]}
Mar  9 13:38:15 PSP caddy[270320]: {"level":"info","ts":1615297095.2099547,"logger":"tls.issuance.acme","msg":"waiting on internal rate limiter","identifiers":["buescher-uptime.concept-visions.de"]}
Mar  9 13:38:15 PSP caddy[270320]: {"level":"info","ts":1615297095.2100086,"logger":"tls.issuance.acme","msg":"done waiting on internal rate limiter","identifiers":["buescher-uptime.concept-visions.de"]}
Mar  9 13:38:15 PSP caddy[270320]: {"level":"warn","ts":1615297095.210066,"logger":"tls.issuance.acme.acme_client","msg":"HTTP request failed; retrying","url":"https://acme.zerossl.com/v2/DV90/newNonce","error":"performing request: Head \"https://acme.zerossl.com/v2/DV90/newNonce\": context deadline exceeded"}
Mar  9 13:38:15 PSP caddy[270320]: {"level":"error","ts":1615297095.2100968,"logger":"tls.obtain","msg":"will retry","error":"[buescher-uptime.concept-visions.de] Obtain: [buescher-uptime.concept-visions.de] creating new order: fetching new nonce from server: context deadline exceeded (ca=https://acme.zerossl.com/v2/DV90)","attempt":1,"retrying_in":60,"elapsed":90.000087448,"max_duration":2592000}
Mar  9 13:38:15 PSP caddy[270320]: {"level":"info","ts":1615297095.2101128,"logger":"tls.obtain","msg":"releasing lock","identifier":"buescher-uptime.concept-visions.de"}
Mar  9 13:38:15 PSP caddy[270320]: {"level":"info","ts":1615297095.2752817,"logger":"tls.on_demand","msg":"attempting certificate renewal","server_name":"status.ormistonpark.org.uk","identifiers":["status.ormistonpark.org.uk"],"expiration":1590011411,"remaining":-25285684.27527808}

5. What I already tried:

For internal rate limiter;

  • We added “ask” directive to avoid internal rate limiter, did not affected it at all
  • Tried to change “burst” and “interval” directive values

For HTTP error

  • Tried connect CA via traceroute, telnet
  • Checked firewall settings
2

Please always use ask and/or burst/interval. Not doing so opens you to abuse and DDOS attacks from bad actors. If you don’t have them on, then anyone could make a request to your server with any hostname and have your server issue a certificate. Someone could spin up a script that makes requests like foo1.evil.com then foo2.evil.com then foo3.evil.com and so on until you run out of storage space or hit rate limits, causing other legitimate requests to get blocked.

@matt will need to follow-up on the rate limiter, he can comment on that better than I can. But I strongly recommend you consider signing up for a business support plan since you’re running at such scale:

https://caddyserver.com/business

3

@francislavoie We added ask directive. Thanks for advice.

@matt Could you please clearify that what’s the caddy’s internal rate limit count. I found in an old post you said that there is a limit after 10 certs in 1 minute, is it still the same ? And we would like to use a load balancer with Caddy, would it have any impact on internal limits (for ex. if we have 2 servers with same acme ID, would it double the internal limits) ?

4

Yes

This throttling is internal and is not distributed; however, cert management is, and the internal throttling happens inside the distributed lock. So an instance will acquire a lock for a specific cert operation (reserving its spot), then block on the internal throttle until it is ready, then release the lock when it is finished.

This is necessary because we’ve had concerns/complaints from CAs when we don’t do enough throttling.

5

@matt We have special rate limits from CA’s assigned to our Acme ID as I mentioned before. Can we increase the internal rate limits somehow ?

6

All you have to do is change these values:

Then recompile with those changes. :+1:

7

This topic was automatically closed after 30 days. New replies are no longer allowed.