HTTP 404 urn:ietf:params:acme:error:malformed - No such authorization

1. The problem I’m having:

We are seeing an HTTP 404 urn:ietf:params:acme:error:malformed - No such authorization when trying to renew a LetsEncrypt certificate. The certificate was created and renewed once successfully already.

Could this be a race condition at Lets Encrypt like mentioned here and here as the HTTP response to the order is an HTTP 200 when I checked https://acme-v02.api.letsencrypt.org/acme/order/2051124377/378333410467

2. Error messages and/or full log output:

{
  "level": "error",
  "ts": 1745737875.7638295,
  "logger": "tls.renew",
  "msg": "could not get certificate from issuer",
  "identifier": "cache.systems",
  "issuer": "acme-v02.api.letsencrypt.org-directory",
  "error": "HTTP 404 urn:ietf:params:acme:error:malformed - No such authorization"
}

{"level":"error","ts":1745737875.7612014,"msg":"validating authorization","problem":{"type":"urn:ietf:params:acme:error:malformed","title":"","detail":"No such authorization","instance":"","subproblems":null},"order":"https://acme-v02.api.letsencrypt.org/acme/order/2051124377/378333410467","attempt":1,"max_attempts":3,"stacktrace":"github.com/mholt/acmez/v3.(*Client).ObtainCertificate
	github.com/mholt/acmez/v3@v3.0.0/client.go:152
github.com/caddyserver/certmagic.(*ACMEIssuer).doIssue
	github.com/caddyserver/certmagic@v0.21.6/acmeissuer.go:477
github.com/caddyserver/certmagic.(*ACMEIssuer).Issue
	github.com/caddyserver/certmagic@v0.21.6/acmeissuer.go:371
github.com/caddyserver/caddy/v2/modules/caddytls.(*ACMEIssuer).Issue
	github.com/caddyserver/caddy/v2@v2.9.1/modules/caddytls/acmeissuer.go:249
github.com/caddyserver/certmagic.(*Config).renewCert.func2
	github.com/caddyserver/certmagic@v0.21.6/config.go:906
github.com/caddyserver/certmagic.doWithRetry
	github.com/caddyserver/certmagic@v0.21.6/async.go:104
github.com/caddyserver/certmagic.(*Config).renewCert
	github.com/caddyserver/certmagic@v0.21.6/config.go:982
github.com/caddyserver/certmagic.(*Config).RenewCertAsync
	github.com/caddyserver/certmagic@v0.21.6/config.go:768
github.com/caddyserver/certmagic.(*Config).renewDynamicCertificate.func2
	github.com/caddyserver/certmagic@v0.21.6/handshake.go:751"}

The interesting thing is that the authorization returns an HTTP 200.

curl -v https://acme-v02.api.letsencrypt.org/acme/order/2051124377/378333410467
* Host acme-v02.api.letsencrypt.org:443 was resolved.
* IPv6: (none)
* IPv4: 172.65.32.248
*   Trying 172.65.32.248:443...
* ALPN: curl offers h2,http/1.1
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* TLSv1.3 (IN), TLS handshake, Server hello (2):
* TLSv1.3 (IN), TLS change cipher, Change cipher spec (1):
* TLSv1.3 (IN), TLS handshake, Encrypted Extensions (8):
* TLSv1.3 (IN), TLS handshake, Certificate (11):
* TLSv1.3 (IN), TLS handshake, CERT verify (15):
* TLSv1.3 (IN), TLS handshake, Finished (20):
* TLSv1.3 (OUT), TLS change cipher, Change cipher spec (1):
* TLSv1.3 (OUT), TLS handshake, Finished (20):
* SSL connection using TLSv1.3 / TLS_AES_256_GCM_SHA384 / x25519 / RSASSA-PSS
* ALPN: server accepted h2
* Server certificate:
*  subject: CN=acme-v02.api.letsencrypt.org
*  start date: Mar 21 18:03:47 2025 GMT
*  expire date: Jun 19 18:03:46 2025 GMT
*  subjectAltName: host "acme-v02.api.letsencrypt.org" matched cert's "acme-v02.api.letsencrypt.org"
*  issuer: C=US; O=Let's Encrypt; CN=R11
*  SSL certificate verify ok.
*   Certificate level 0: Public key type RSA (2048/112 Bits/secBits), signed using sha256WithRSAEncryption
*   Certificate level 1: Public key type RSA (2048/112 Bits/secBits), signed using sha256WithRSAEncryption
*   Certificate level 2: Public key type RSA (4096/152 Bits/secBits), signed using sha256WithRSAEncryption
* Connected to acme-v02.api.letsencrypt.org (172.65.32.248) port 443
* using HTTP/2
* [HTTP/2] [1] OPENED stream for https://acme-v02.api.letsencrypt.org/acme/order/2051124377/378333410467
* [HTTP/2] [1] [:method: GET]
* [HTTP/2] [1] [:scheme: https]
* [HTTP/2] [1] [:authority: acme-v02.api.letsencrypt.org]
* [HTTP/2] [1] [:path: /acme/order/2051124377/378333410467]
* [HTTP/2] [1] [user-agent: curl/8.13.0]
* [HTTP/2] [1] [accept: */*]
> GET /acme/order/2051124377/378333410467 HTTP/2
> Host: acme-v02.api.letsencrypt.org
> User-Agent: curl/8.13.0
> Accept: */*
>
* Request completely sent off
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
< HTTP/2 200
< server: nginx
< date: Tue, 29 Apr 2025 20:05:47 GMT
< content-type: application/json
< content-length: 347
< cache-control: public, max-age=0, no-cache
< link: <https://acme-v02.api.letsencrypt.org/directory>;rel="index"
< x-frame-options: DENY
< strict-transport-security: max-age=604800
<
{
  "status": "pending",
  "expires": "2025-05-04T07:11:15Z",
  "identifiers": [
    {
      "type": "dns",
      "value": "cache.systems"
    }
  ],
  "authorizations": [
    "https://acme-v02.api.letsencrypt.org/acme/authz/2051124377/511793574767"
  ],
  "finalize": "https://acme-v02.api.letsencrypt.org/acme/finalize/2051124377/378333410467"
* Connection #0 to host acme-v02.api.letsencrypt.org left intact

3. Caddy version:

The issue occurred on Caddy 2.9.1. I’ve since upgraded to 2.10.0

4. How I installed and ran Caddy:

a. System environment:

Docker

b. Command:

CMD ["caddy", "run", "--config", "/etc/caddy/Caddyfile", "--adapter", "caddyfile"]

c. Service/unit/compose file:

d. My complete Caddy config:

{
	auto_https off # Disables both certificate automation and HTTP-to-HTTPS redirects.
	http_port 80
	https_port 443
	on_demand_tls {
		ask http://{env.REDIRECT_SERVICE_HOST}/check
	}

	{$CADDY_DEBUG}

	email {env.EMAIL}
	# All values are optional, below are the defaults
	storage redis cluster {
		address {
			{env.REDIS_HOST}:{env.REDIS_PORT}
		}
		username {env.REDIS_USERNAME}
		password {env.REDIS_PASSWORD}
		db 0
		timeout 5
		key_prefix "caddy"
		compression false 
		tls_enabled true
		tls_insecure false
	}

	order cache before rewrite
	cache {
		otter
	}

	metrics
	grace_period 10s
	shutdown_delay 30s
}

(dev) {
	tls internal {
		on_demand
	}
}

(prod) {
	tls {
		issuer acme
		issuer zerossl {$ZEROSSL_API_KEY}
		on_demand
	}
}

:80, :443 {
	cache
	reverse_proxy {
		to {env.REDIRECT_SERVICE_HOST}:80
		transport http {
			dial_timeout 2s
			response_header_timeout 30s
		}

		header_down X-Powered-By "{$CI_COMMIT_SHORT_SHA:local}"
	}
	log {
		output stdout
	}

	#see (dev) and (prod) above for TLS configs
	import {$APP_ENV:dev}

	header {
		-Content-Length
	}
}

http://:2020 {
	metrics /metrics
	handle /healthz {
		@goingDown vars {http.shutting_down} true
		respond @goingDown "Bye-bye in {http.time_until_shutdown} seconds" 503
		respond {$CI_COMMIT_SHORT_SHA:local} 200
	}
}

5. Links to relevant resources: