*.stagings.mydomain.com? Please note that the Help template does say:
DO NOT REDACT anything except credentials
or helpers will be sad
Can you confirm you’re having a similar problem with *.internal.mydomain.com, but *.mydomain.com is okay?
1 Like
Sorry about that, I didn’t know if it would be helpful.
I am having the same problem with *.internal.mydomain.com, however since *.mydomain.com was added a while ago and still gets renewed with a new letsencrypt wildcard certificate every 3 months I assume it does not share the same issue.
I am not sure if this is a big problem but I am wondering why the *.mydomain.com grabs certs fine from LE but the others fail and fallback to ZeroSSL (which does succeed).
I believe what’s happening is that Let’s Encrypt support single-level wildcard certificates, but not multi-level wildcard certificates, so *.mydomain.com is supported, but *.*.mydomain.com is not. More info here. It’s possible that ZeroSSL do support multi-level wildcard domains (on a paid plan?) and that’s why it worked for you.
To work around this problem with Let’s Encrypt, you could define three domains in Cloudflare internal.mydomain.com, stagings.mydomain.com and mydomain.com. That would result in three wildcard certificates being issued, however, as ZeroSSL seems to address the problem, I wonder if the extra effort is worth it?
Caddy only issues wildcard certificates if a domain with a * is specified as a site address. It’ll get individual ones otherwise.
@basil @francislavoie using crt.sh I was able to see that in the past my pfsense firewall with the acme plugin was able to successfully request a certificate for *.internal.mydomain.com, whereas caddy was not able to. I have ensured that the API token permissions are the same. Is it possible maybe there is a timing issue because LE is tried first, and ZeroSSL is tried second (as shown in the logs). Maybe if caddy tried ZeroSSL first and then secondly LE then it would obtain the certificate from LE?
The *.internal.mydomain.com would not be a multi level wildcard certificate so I don’t think LE should have a problem with issuance.
The only thing I have made before I experienced this behavior is to upgrade to caddy 2.4.0-beta, this behavior could have existed in the past but I only tested it after upgrading so I am not sure.
The log in the OP seems to suggest this.
Mar 23 22:32:55 intranet caddy[603]: {"level":"info","ts":1616538775.0949454,"logger":"http","msg":"enabling automatic TLS certificate management","domains":["*.internal.mydomain.com","*.stagings.mydomain.com","*.mydomain.com"]}
While not exactly the same. I have a wildcard certificate issued for *.udance.com.au and that seems to be fine. @francislavoie something else is going on here.
@keshav It’s dawned on me now that’s what you’ve done. Ignore everything I’ve said about multi-level wildcard certificates. What you have here is three single-level wildcard domains. If it’s worked with ZeroSSL, it probably should have worked with Let’s Encrypt. Can you confirm you have the three domains set up in Cloudflare?
That is right, I do not have any DNS entries setup in cloudflare for these domains as I am using DNS challenge anyway to obtain certificates. This allows me to obtain certificates for internal services without having external DNS pointing to them.
@matt @francislavoie @basil So I think I am on to something, I switched to the staging endpoint of LE by adding the option in my global options. I noticed that while caddy was attempting to obtain certificates using LE there was never any TXT record created in cloudflare, which then gives us the error:
HTTP 403 urn:ietf:params:acme:error:unauthorized - No TXT record found at _acme-challenge.stagings.mydomain.com","order":"https://acme-v02.api.letsencrypt.org/acme/order/REDACTED","attempt":1,"max_attempts":3}
When caddy then switches to zerossl, I can see the TXT record successfully created.
Therefore, I think there must be something that has changed with how caddy 2.4.0 handles LE acme, as compared to ZeroSSL.
1 Like
How exactly did you build Caddy? What command did you use (if you didn’t use the download page)?
You can read through this thread for context, you may be running into the same problem:
@francislavoie I used the main caddy download page to get the executable for the build. I looked at the post you mentioned, our logs are similar as he gets a 400, and I get a 403.
I have a not dissimilar situation. In order for the DNS challenge to work, it needs to write to a TXT record under the domain name. Do you have
stagings.mydomain.com and internal.mydomain.com defined as separate domains under Cloudflare?
I have only defined mydomain.com in cloudflare. I don’t think that should be an issue right? If zerossl is able to pass the dns challenge then LE should be able to as well from my understanding.
I’m reasonably confident now, this is where the issue lies. I can’t speak for ZeroSSL, but it’s clear from this reference that Let’s Encrypt only support single-level wildcard domains i.e. *.mydomain.com, not *.*.mydomain.com. For Let’s Encrypt to support *.internal.mydomain.com and *.stagings.mydomain.com, both internal.mydomain.com and stagings.mydomain.com need to be treated as if they are separate domains.
.
I am not totally sure if I understand, I have been able to obtain a certificate for *.internal.mydomain.com in the past. I have even logs on crt.sh to show that, I have never had any DNS entries in cloudflare for the *.internal.mydomain.com because I didn’t want it to be resolvable externally. The *.internal.mydomain.com domain should still be treated as a single level wildcard and there shouldn’t really be a problem whether it be LE or ZeroSSL, there is no functional difference between the two that would allow one the ability to issue certs that the other cannot. From refreshing the cloudflare dashboard repeatedly while caddy tried to solve the DNS challenge the problem seems to lie in the Let’s Encrypt stage when creating the TXT records for the domain. When this fails and later falls back to ZeroSSL I’m able to see the relevant TXT records show up in cloudflare and then a certificate is issued.
I wonder if starting a thread (that references this thread) in the Let’s Encrypt forum might be the next step? It may shed some light on what’s happening here, like why it’s worked for you in the past and why it’s not working now, and why it works for ZeroSSL, but not Let’s Encrypt?
Good idea! Is it possible that there may have been any changes to caddy’s Lets Encrypt component in the recent beta build?
Have you tried either of the solutions in the thread I linked above?
Either try a build of Caddy v2.3.0 with an older commit of the cloudflare plugin, or try a newer build of Caddy (the commit I mentioned near the bottom of that thread) with the latest version of the cloudflare plugin.
2 Likes
That did it!!! Thank You! Is it best to stay on 2.3.0 until a new version of 2.4 comes out?
Yeah, like I said, either v2.3.0 with an older cloudflare commit, or a specific commit after v2.4.0-beta.1 which has the changes needed for the latest version of cloudflare to work properly. We’ll probably release a v2.4.0-beta.2 soon, so you could use that when that’s out.
1 Like
This topic was automatically closed after 30 days. New replies are no longer allowed.