ZeroSSL errors, config with API key

Hi everyone! :wave: I’ve been using Caddy for a couple years, hoping to get some guidance on proper config for ZeroSSL (or anything else that looks wrong).

1. The problem I’m having:

Before now, we’ve been using Caddy with Let’s Encrypt. We have a large number (thousands) of subdomains and other custom domains, so we often hit Let’s Encrypt rate limits. Falling back to ZeroSSL never seemed to help. I’m now trying again to switch to ZeroSSL exclusively to work around these limits.

After updating the Caddyfile to only use ZeroSSL, we still get errors in the log when renewing domains. I’ve also updated Caddy to the latest version.

I have a paid ($50/month) ZeroSSL account. No certificates from Caddy have ever shown up in the dashboard. This leads me to think the API key is not wired up correctly. I have tried many different variations of the issuer config, including email, etc.

Thanks!

2. Error messages and/or full log output:

Recent errors look like this (I’ve replaced the domain with just “domain” here):

2023/12/30 18:36:41.300	ERROR	tls.renew	could not get certificate from issuer	{"identifier": "domain", "issuer": "acme.zerossl.com-v2-DV90", "error": "[domain] solving challenges: authz https://acme.zerossl.com/v2/DV90/authz/gvsJBCc4dVPkjCy1897U4Q has unexpected status; order will fail: invalid (order=https://acme.zerossl.com/v2/DV90/order/cLkJ-yQWBhftsARl1KGNmw) (ca=https://acme.zerossl.com/v2/DV90)"}
2023/12/30 18:36:41.300	ERROR	tls.renew	will retry	{"error": "[domain] Renew: [domain] solving challenges: authz https://acme.zerossl.com/v2/DV90/authz/gvsJBCc4dVPkjCy1897U4Q has unexpected status; order will fail: invalid (order=https://acme.zerossl.com/v2/DV90/order/cLkJ-yQWBhftsARl1KGNmw) (ca=https://acme.zerossl.com/v2/DV90)", "attempt": 3, "retrying_in": 120, "elapsed": 191.884578831, "max_duration": 2592000}

Before upgrading Caddy, I was also getting HTTP 429 rate limit errors.

3. Caddy version:

v2.7.6 h1:w0NymbG2m9PcvKWsrXO6EEkY9Ru4FJK8uQbYcev1p3A=

4. How I installed and ran Caddy:

a. System environment:

Debian 4.19.0-17-amd64

b. Command:

sudo xcaddy build --with github.com/gamalan/caddy-tlsredis
sudo ./caddy start

c. Service/unit/compose file:

d. My complete Caddy config:

I’ve replaced API keys and Redis info. The Redis module is used to track cert info across multiple web servers.

{
	on_demand_tls {
		ask https://micro.blog/pages/ssl/check # will pass ?domain=, return 200 OK
		interval 2m
		burst 100 # 100 every 2 minutes
	}

	storage redis {
		host ""
		port 1234
		password "ABCDEFGHIJKL"
		db 1
	}

	log {
		output file /var/log/caddy-main.log {
			roll_keep_for 5d
		}
		format console
	}
}

:443 {
	tls {
		on_demand
		issuer zerossl ABCDEFGHIJKL
	}

	root * /home/web/sites/{host}
	file_server

	header Access-Control-Allow-Origin "*"

	log {
		output file /var/log/caddy-access.log {
			roll_keep_for 5d
		}
		format console
	}

	handle_errors {
		@502 {
			expression {http.error.status_code} == 502
		}
		handle @502 {
			rewrite * 502.html
			file_server
		}
		handle {
			rewrite * /pages/migration/redirect{uri}
			reverse_proxy micro.blog {
				header_up Host micro.blog
				header_up X-Migration-Host {host}
			}
		}
	}
}

And here’s an example from the logs for when the HTTP 429 error pops up:

2023/12/30 20:00:23.832	ERROR	tls.renew	could not get certificate from issuer	{"identifier": "domain", "issuer": "acme.zerossl.com-v2-DV90", "error": "[domain] solving challenges: getting authorization at https://acme.zerossl.com/v2/DV90/authz/xmpMRjNpoil1Mqxzz0zRqg: attempt 1: https://acme.zerossl.com/v2/DV90/authz/xmpMRjNpoil1Mqxzz0zRqg: HTTP 429: <html>\r\n<head><title>429 Too Many Requests</title></head>\r\n<body>\r\n<center><h1>429 Too Many Requests</h1></center>\r\n<hr><center>nginx</center>\r\n</body>\r\n</html>\r\n (order=https://acme.zerossl.com/v2/DV90/order/zXC9gKDa8kAT5fJwO2mAvQ) (ca=https://acme.zerossl.com/v2/DV90)"}
2023/12/30 20:00:23.832	ERROR	tls.renew	will retry	{"error": "[domain] Renew: [domain] solving challenges: getting authorization at https://acme.zerossl.com/v2/DV90/authz/xmpMRjNpoil1Mqxzz0zRqg: attempt 1: https://acme.zerossl.com/v2/DV90/authz/xmpMRjNpoil1Mqxzz0zRqg: HTTP 429: <html>\r\n<head><title>429 Too Many Requests</title></head>\r\n<body>\r\n<center><h1>429 Too Many Requests</h1></center>\r\n<hr><center>nginx</center>\r\n</body>\r\n</html>\r\n (order=https://acme.zerossl.com/v2/DV90/order/zXC9gKDa8kAT5fJwO2mAvQ) (ca=https://acme.zerossl.com/v2/DV90)", "attempt": 2, "retrying_in": 120, "elapsed": 82.523814932, "max_duration": 2592000}

Hm, I saw another report recently about an error like that (“unexpected status”), which I haven’t seen before.

I suspect it’s an error on the ZeroSSL side, something with their CA software. Have you tried reaching out to their support?

Thanks for the reply! Yes, I reached out to ZeroSSL support yesterday too. I’ll update this issue when I hear back.

I’ve tried a couple other things after posting, including the EAB credentials. No luck. Seems like a bug somewhere, but can’t tell if it’s on the ZeroSSL side or a problem sending API key / credentials to them from Caddy.

1 Like

hi @mantonr, any updates on this issue from ZeroSSL? We are encountering the same issue with them and support is taking there time to answer

hi, please let us know if anyone has found a way around this. we’re running into the same issues, neither rotating our API key nor playing with other parameters (such as timeout) are resulting in success.

Funny thing is, the exact same config worked for us for over an year, and it just suddenly stopped working.

Caddyfile globals -

	email my@email.com
	acme_ca https://acme.zerossl.com/v2/DV90
	cert_issuer zerossl <API_KEY> {
		email my@email.com
		timeout 5m
	}

(email and API_KEY redacted)

Am finally having a chance to look into this with ZeroSSL, so I’ll let you know what comes of it :+1:

So it sounds like what I need to do is implement the ZeroSSL API in the ZeroSSL issuer. Right now we only use the API to get ACME EAB credentials, then we use ACME. But the user should be able to provide their API key and then opt-in to API usage, that will gain the full benefits of their paid account.

1 Like

do let us know when an issue is open, we might be able to contribute to get this done if required

1 Like

I’d be open to that, for a faster fix. The API looks like a pretty standard HTTP API:

Would you be willing to elaborate on this? Is there something about a paid account that’s causing ACME requests to be treated differently? I’m curious if they stated anything outright about restrictions being put on ACME requests. I’m mostly hoping to know if there are any workarounds until Caddy supports API-based issuing.

From my naive investigation of extracting the authz response, it appears that if a challenge can’t resolve the DNS entry back to Caddy, ZeroSSL is marking the cert as invalid until it expires (seems to be 30-31 days after attempted issuance) and there’s no means of having it retry or invalidate the certificate to start over. Further authz attempts continue to refer back to this “invalid” cert or challenge even when attempted from completely different ACME clients.

I’m not well-versed in the ACME protocol, but it seems like if a challenge fails, the related challenge and cert is simply “stuck” until it expires. Is that expected behavior according to any protocol specs?

My understanding is incomplete, but it sounds like ZeroSSL has more control over their API services than their ACME endpoints which use a different software stack (that isn’t Boulder, unfortunately). That includes the ability to offer better quality response times and higher limits.

I don’t know of any RFC 8555 requirement or suggestion that invalid authzs be cached and reused, indeed that seems like it could be problematic. I’ll see what I can find out…

In the meantime, track Implemented the zerossl API to issue ssl certificates by armadi1809 · Pull Request #6068 · caddyserver/caddy · GitHub

My understanding is incomplete, but it sounds like ZeroSSL has more control over their API services than their ACME endpoints which use a different software stack (that isn’t Boulder, unfortunately). That includes the ability to offer better quality response times and higher limits.

Thanks, @matt. That is good context to have. Definitely not trying to use this convo as a means for ZeroSSL support – they’d just been a little unresponsive about it and thought your conversation might’ve illuminated something. Interestingly, a while after opening a support ticket with them, providing the stuck authz request/responses that I extracted, whatever was blocking further retries was cleared for that cert and the other certs I had in a similar situation and I was able to retry successfully with correct DNS records.

It smelled a bit like maybe a limit was being applied on a given domain’s authz retry count (or something?) and then it tripped a trigger to back off on any further retries for a month. That’s pure supposition, though, of course since they never replied.

I don’t know of any RFC 8555 requirement or suggestion that invalid authzs be cached and reused, indeed that seems like it could be problematic. I’ll see what I can find out…

If it’d be helpful to have some extracts of the authz responses, I do have the ones I provided to ZeroSSL. But if none of that is helpful for Caddy, then certainly no need to track further. I’ve sub’d to that PR and the issue you linked. I appreciate the outreach; Caddy is a wonderful piece of software for exactly this reason. Thank you.

1 Like

Always happy to help. We have great collaborators that make it possible. Thanks for your participation!