Unexpected Let's Encrypt expiration notice

[basil (Basil Hendroff)]

1. Caddy version (caddy version):

v2.4.0-beta.1.built using the Cloudflare module to address an issue identified with the map directive. See Migrate to using a wildcard certificate post #16. If required, I can upgrade to beta 2.

2. How I run Caddy:

I use Caddyfile rather than JSON.

a. System environment:

TrueNAS 12.0-U3 (a FreeBSD derivative)

b. Command:

service caddy start

c. Service/unit/compose file:

n/a

d. My complete Caddyfile or JSON config:

This is an extract from the relevant section of my Caddyfile

www.caffigoalkeeping.com, caffigoalkeeping.com,
www.caffigoalkeeping.com.au, caffigoalkeeping.com.au {
  ...
  reverse_proxy 10.1.1.51
}

3. The problem I’m having:

To my surprise, I recently received a Let’s Encrypt expiration notice for www.caffigoalkeeping.com.au.

4. Error messages and/or full log output:

n/a

5. What I already tried:

I used a Let’s Encrypt debug toolkit to confirm the issue.

New certificates have been generated for caffigoalkeeping.com and www.caffigoalkeeping.com.

A new certificate has been generated for caffigoalkeeping.com.au, but not for www.caffigoalkeeping.com.au.

I also had an in-depth discussion with @francislavoie on the merits of redirection versus multiple site definition Www handling - Use the same multiple site definition or use redirection?. Not sure if there is a tie in here?

6. Links to relevant resources:

1. Www handling - Use the same multiple site definition or use redirection?
2. Let’s Encrypt debug toolkit

[francislavoie]

If I visit www.caffigoalkeeping.com.au and look at the certificate (click the lock to the left of the address bar, click view certificate), it’s issued by Cloudflare, and not Let’s Encrypt or ZeroSSL.

What’s in your logs? Any indication of attempts to renew? Check Caddy’s storage, look at the certificate files, you can use an online tool to check the cert expiry (the certificates are public info, so it’s okay to paste into online tools, but your private keys (the .key files) should stay secret).

[basil (Basil Hendroff)]

@francislavoie

If I visit www.caffigoalkeeping.com.au and look at the certificate (click the lock to the left of the address bar, click view certificate), it’s issued by Cloudflare, and not Let’s Encrypt or ZeroSSL.

Curious, that’s different to what I’m seeing.

Any bearing on this issue Wildcard Domains aren’t being made or I am an idiot post #2? I am using Caddy 2.4.0-beta 1.

[francislavoie]

I’m seeing a multi-SAN certificate from Cloudflare:

@basil

Any bearing on this issue Wildcard Domains aren’t being made or I am an idiot post #2? I am using Caddy 2.4.0-beta 1.

Yeah, maybe try a fresh build with xcaddy.

[basil (Basil Hendroff)]

@francislavoie

Yeah, maybe try a fresh build with xcaddy.

Right, I’ll crack on with that, monitor and report back later.

[basil (Basil Hendroff)]

Hang on… I’m thinking now, maybe there isn’t an issue at all. It appears ZeroSSL rather than Let’s Encrypt has issued the certificate for www.caffigoalkeeping.com.au. LE has still reissued the certs for the other three domains/subdomains. Could that be why LE has sent the expiration notice for www.caffigoalkeeping.com.au? If so, several questions spring to mind:

1. Is a ZeoSSL tool available to view issued certs?
2. To avoid receiving bogus expiration notices, is there a way to have certs issued by ZeroSSL or LE, but not both at the same time?

@francislavoie

look at the certificate files,

Yup, I suspect that’s what’s happened…

@francislavoie

If I visit www.caffigoalkeeping.com.au and look at the certificate (click the lock to the left of the address bar, click view certificate), it’s issued by Cloudflare, and not Let’s Encrypt or ZeroSSL.

I wonder if what you’re seeing is a side effect of me using the Cloudflare CDN?

@francislavoie

Yeah, maybe try a fresh build with xcaddy.

I’ve gone ahead and updated to the latest beta anyway.

root@caddy:/var/log # caddy version
v2.4.0-beta.2 h1:DUaK4qtL3T0/gAm0fVVkHgcMN04r4zGpfPUZWHRR8QU=

[francislavoie]

@basil

I wonder if what you’re seeing is a side effect of me using the Cloudflare CDN?

Probably. Maybe requests from Canada, where I am, are getting proxied by Cloudflare for me, but not for you, for some reason. Depends how DNS is being resolved globally I guess.

@basil

It appears ZeroSSL rather than Let’s Encrypt has issued the certificate for www.caffigoalkeeping.com.au.

Makes sense.

@basil

To avoid receiving bogus expiration notices, is there a way to have certs issued by ZeroSSL or LE, but not both at the same time?

Not really. The emails are opt-in when you provide an email in your config (which is recommended so CAs can contact you if there’s concern of abuse or whatever, but is optional with Caddy). The two CAs aren’t aware if Caddy decides to use the other for whatever reason. You could configure Caddy to only use Let’s Encrypt or only use ZeroSSL but then you lose the redundancy benefits of having both enabled in case either go out of order.

/cc @matt something to keep in mind re multi-issuer is stuff like this, if you have any ideas on the topic.

@basil

Is a ZeoSSL tool available to view issued certs?

ZeroSSL allows you to sign up with the email in your config to see a personalized dashboard. I’ve not had the chance to play around with that yet, but give it a shot.

Both Let’s Encrypt and ZeroSSL push to the Certificate Transparency logs that Let’s Encrypt runs, so you can see it here:

crt.sh | www.caffigoalkeeping.com.au

[basil (Basil Hendroff)]

@francislavoie

Probably. Maybe requests from Canada, where I am, are getting proxied by Cloudflare for me, but not for you, for some reason. Depends how DNS is being resolved globally I guess

Explanation in the references, but essentially a Cloudflare cert is used between the browser and Cloudflare, and a ZeroSSL cert between Cloudflare and Caddy. Interestingly, Cloudflare certs are valid for a year.

Cloudflare references:

1. Understanding Universal SSL
2. End-to-end HTTPS with Cloudflare - Part 3: SSL options - Full(strict)

@francislavoie

ZeroSSL allows you to sign up with the email in your config to see a personalized dashboard. I’ve not had the chance to play around with that yet, but give it a shot.

Unfortunately, the ZeroSSL certs issued to Caddy against the email address are not displayed in the ZeroSSL dashboard.

[francislavoie]

@basil

Unfortunately, the ZeroSSL certs issued to Caddy against the email address are not displayed in the ZeroSSL dashboard.

Might only show for certs issued after you’ve created the account. But I dunno.

[matt (Matt Holt)]

@francislavoie

/cc @matt something to keep in mind re multi-issuer is stuff like this, if you have any ideas on the topic.

I suppose this is expected; each CA will do their own thing in this regard. One of the reasons we don’t randomize which CA is used (first) is because the consistency can be important, or at least less annoying like in this case. The secondary CA is only used if the primary has an error.

@basil

Hang on… I’m thinking now, maybe there isn’t an issue at all. It appears ZeroSSL rather than Let’s Encrypt has issued the certificate for www.caffigoalkeeping.com.au. LE has still reissued the certs for the other three domains/subdomains. Could that be why LE has sent the expiration notice for www.caffigoalkeeping.com.au? If so, several questions spring to mind:

1. Is a ZeoSSL tool available to view issued certs?
2. To avoid receiving bogus expiration notices, is there a way to have certs issued by ZeroSSL or LE, but not both at the same time?

I’m guessing that getting a certificate from LE failed for that domain once upon a time, but succeeded with ZeroSSL, so it used that one. Caddy will reuse the same CA the next time it goes to renew the certificate. Putting a CDN in front of a site can definitely cause challenges to fail, if the CDN is misconfigured (the TLS-ALPN challenge will almost always fail behind a CDN, but the HTTP challenge will likely succeed unless the CDN intercepts that too).

[basil (Basil Hendroff)]

@matt

The secondary CA is only used if the primary has an error.

On the one hand, having a backup is useful, particularly if renewing on the day of certificate expiry. OTOH, if the first renewal attempt is occurring X days before cert expiry, then X daily retries to obtain a cert with the same CA are possible before it becomes necessary to attempt a failover to a second CA. This would virtually eliminate switching CAs due to one-off renewal errors, which I suspect has happened in this case. The scenario that played out here might have been avoided.

[basil (Basil Hendroff)]

@basil

Unfortunately, the ZeroSSL certs issued to Caddy against the email address are not displayed in the ZeroSSL dashboard

This is what I have in the global section of my Caddyfile:

{
  email basil.hendroff@udance.com.au
  acme_dns cloudflare [REDACTED]
#  debug
}

At present, I get regular email notifications from LE regarding certificates issued to me, but based on the discussion within this thread, it’s ambiguous whether I’ll get similar notifications from ZeroSSL as well.

It appears to me I’ve got two options:

1. Use the cert_issuer global option to have only one CA issue certs. The hints are in this thread
   Question about the Multi-Issuer Support, or
2. Persevere with multi-CA issued certs, but try to get notifications working and find better cert tools for each CA.

My preference is for the latter. @matt presents some options in this wiki article Using ZeroSSL’s ACME endpoint , but it’s not clear to me whether the global options specified mean:

1. ZeroSSL will usurp LE and be the only CA issuing certs;
2. I’ll begin to receive email from ZeroSSL…the first global option seems to suggest this as it links an email with the ZeroSSL endpoint;
3. Using EAB credentials will allow issued certs to appear in the ZeroSSL dashboard.

It’s all a bit puzzling atm. In an ideal world, what I’m hoping for is:

1. Advance notification of events such as upcoming cert expiry from all CAs issuing certs (linked to an email address); CA maintenance, etc.; and
2. Useful CA cert tools apart from crt.sh. For example, this LE debug toolkit allows me to see what LE certs are issued against domains of interest. The ZeroSSL dashboard holds some promise here if it can be made to work with ZeroSSL certs issued through Caddy.

[francislavoie]

You could use a service like https://ohdear.app/ (they use Caddy!) to monitor the certificate status of your sites (will warn you if it gets too close to expiry without being renewed). Ultimately you can ignore emails from LE generally.

[basil (Basil Hendroff)]

Now, why didn’t I think about looking for a tool like that? I had a look around. Let’s Monitor provide a free SSL monitoring service.