Er, what is up with <del> in your config? Can you please post your full, unmodified config file. I need to be able to reproduce it…
Please post your full logs and steps as well so I can reproduce them.
I’m sorry for the shoddy paste. I was kind of tired and worn out. I knew you’d be out as well. As I went to bed, I felt that the paste would’ve been confusing. I wanted to edit the post after I woke up; but again, clearly, you beat me to it.
What I posted was the diff from my old config. The <del> was supposed to mean ‘deleted’ from old config.
Configuration
Here’s the full configuration :
{
"apps" : {
"http" : {
"servers" : {
"caddy.test.shinenelson.xyz" : {
"listen" : [
":80",
":443"
],
"automatic_https" : {
"disable_redirects" : true
},
"routes" : [
{
"group" : "shine",
"match" : [
{
"host" : [
"shine.caddy.test.shinenelson.xyz"
]
}
],
"handle" : [
{
"handler" : "static_response",
"body": "Hi there, love from shine and Caddy!"
}
],
"terminal" : true
},
{
"group" : "shine",
"match" : [
{
"host" : [
"*.shine.caddy.test.shinenelson.xyz"
]
}
],
"handle": [
{
"@id" : "proxy_shine",
"handler" : "reverse_proxy",
"upstreams" : [
{
"dial" : "localhost:8080"
},
{
"dial" : "localhost:8081"
}
]
},
{
"handler": "static_response",
"body" : "hey, you were not supposed to get here. - shine"
}
],
"terminal" : true
}
],
"tls_connection_policies" : [
{
"match" : {
"sni" : [
"shine.caddy.test.shinenelson.xyz",
"shine.shine.caddy.test.shinenelson.xyz",
"something.shine.caddy.test.shinenelson.xyz",
"*.shine.caddy.test.shinenelson.xyz"
]
}
}
]
}
}
},
"tls" : {
"automation" : {
"policies" : [
{
"subjects" : [
"shine.caddy.test.shinenelson.xyz",
"*.shine.caddy.test.shinenelson.xyz"
],
"issuer" : {
"module" : "acme",
"ca" : "https://acme-staging-v02.api.letsencrypt.org/directory"
},
"on_demand" : true
}
],
"on_demand" : {
"rate_limit" : {
"interval" : "2160h",
"burst" : 3
}
}
}
}
}
}
I got the automatic on-demand ( and it happens on-demand, alright ) TLS certificate provisioned for shine.caddy.test.shinenelson.xyz and anymore subjects that is manually added to tls.automation.policies[0].subjects , but wildcard sub-domains are not being provisioned.
Error
It’s a single line error :
http: TLS handshake error from 183.82.170.125:47540: no certificate available for 'a.shine.caddy.test.shinenelson.xyz'
What I already tried
- Increasing the
on_demand.rate_limits - Removing the
on_demand.rate_limitblock entirely ( I didn’t see it when I adapted a dummy Caddyfile ) - Adding more
subjectstotls.automation.policies[0].subjectsprovisions the TLS certificates as expected.
=== Edit ===
Caddy version
v2.0.0-beta.20 h1:oUNG1uh0UV8LWLlAVDZolFzk112++V/pxY+fF0HLmlY=
Thanks for the info. You don’t have to apologize for that, please make sure to take care. 
I’ll see if I can reproduce it, but I already see a problem: To get wildcard certificates from Let’s Encrypt, you need to use the DNS challenge: Automatic HTTPS — Caddy Documentation
oh, so, it can’t be like I put a wildcard matcher and then get a TLS certificate on-demand per sub-domain that I use?
That’s how I understood the on-demand thing to work. Otherwise, it’s just a static list right?
So, if I got it right, on-demand is useful when you don’t know all the hostnames up front, or if domain names you know of may not be properly configured right away (e.g. DNS records not yet set correctly), would work only if I also set up a wildcard DNS challenge and get a wildcard SNI TLS certificate, is that how it is?
No, no, on-demand has nothing to do with it. Let’s Encrypt requires using the DNS challenge for any wildcard certificates.
I understand the Let’s Encrypt part of it but probably my assumption of the on-demand feature is not clear.
Let me clarify my intention of use here : I have a sub-domain that I want to give away. People get their own sub-domains on the first-level sub-domain ( caddy.test.shinenelson.xyz ). I have no clue as to what those sub-domains that people are going to pick. So, I want to generate a TLS certificate when someone tries one of the sub-domains on my first-level sub-domain.
I know the obvious idea of getting a wildcard certificate from Let’s Encrypt for the first-level sub-domain, but I was hoping that the on-demand provisioning would give me only the used sub-domains and not the whole sub-domain ( via the wildcard ). Is that too much of an ask from the on-demand feature?
I recommend against that, because Let’s Encrypt rate limits the number of certificates you can get for subdomains per week. Use a wildcard certificate when you have an undefined number of subdomains.
Again, this doesn’t have anything to do with on-demand mode; all that does is change when certificates are managed (the kind of certificate is orthogonal).
When using a wildcard certificate, you don’t need to (and probably shouldn’t) use on-demand mode. For 99% of sites, on-demand is not the right solution, it’s basically just for SaaS that need to manage lots of arbitrary domain names they are not in control of.
Thank you for the clarification. I think I have a better understanding now.
Oh, I think I get it now. It’s just about when ( emphasis on the when right? ) the certificate management lifecycle starts. While in the normal mode, the management lifecycle starts during server initialization which could block / delay the startup if there were too many domains. I hope I got it right this time.
I don’t intend on being a pain, but I think this is what confused me in the first place. If you don’t mind ( and have the time, no pressure to answer at all ), can you please explain this concept to me?
On one hand, you need to provide a static set of subjects to match in the configuration. And I think I get the when the lifecycle is being triggered argument as well. Then where’s this ‘arbitrary domains’ concept coming from? If they’re arbitrary and they have no control over the domain, how are they going to put them in the subjects list? How does that work? An example would be great. ( Again, no pressure to answer if you have other pressing matters to look into ). Anyone from the community who understands the concept could help as well.
Close! Certificate management never blocks startup, it’s always in the background of the main goroutine (unless you set manage_sync to true, but don’t do that unless you have a very good reason). The only difference is that Caddy will obtain “on-demand” certificates, well, on-demand… i.e. at the first handshake that requires it, rather than in the background at config load.
Sure. Say you run a SaaS / web service, and your customers bring their own domain names. You tell them “point your domains to our server” but you have no idea when or if that will ever actually happen. If you started getting a cert for that domain right away, it would fail, since you don’t know when the customer has updated their DNS. Sure, you could poll, but DNS is all a matter of perspective anyway, so it’s unreliable and expensive to do that.
All you can do is whitelist their domain when they sign up for your service, and whenever your site starts seeing handshakes for that server name, then you know it’s time to get a certificate.
1 Like
To clarify the “whitelist” is with using the "ask" option JSON Config Structure - Caddy Documentation
Caddy would “ask” the SaaS whether a domain is okay to use. You’d probably have a DB table that has all the domains you allow your customers to use to point to your service, and reply with HTTP 200 if it’s in the list, 4xx otherwise.
1 Like
Oh man, that’s the innovation right there! That’s amazing stuff! Kudos to you for coming up with that idea!
One more reason why you deserve more love in terms of sponsorship. You’ll be the first person that I sponsor when I can actually afford it.
And you’re really nice and welcoming which is another thing ( other than fast response times ) that’s not seen in other open source projects that I’ve put my hands in.
2 Likes
No problem. And thanks.
I’ll spend a little time to make sure the policy matching works correctly.
What is happening in your case is with the automation policies. Those matchers are taken literally, since you can actually have a certificate with a subject of *.shine.caddy.test.shinenelson.xyz (a wildcard certificate), vs. "all certificates for subdomains of shine.caddy.test.shinenelson.xyz".
I suppose we could try lumping them together, i.e. doing an exact match first, and if not, then trying a wildcard match… not sure if that would break anything though… probably not?
@shine I think this commit will do what you want, however, where an automation policy can match a non-wildcard name with a wildcard subject in its list: caddytls: Match automation policies by wildcard subjects too · caddyserver/caddy@c87f82f · GitHub
If you could download the build artifact from that commit and try it out, that’d be much appreciated!
Hey, I’m sorry I missed this notification, I was away from the keyboard.
I’m not sure what this change is supposed to address. Is it the thing about generating “on-demand” TLS certificates even for wildcard domains like my actual intention?
I thought that that was a confusion on my part. Did you just make that a feature? 
Should I chuck the wildcard automation stuff out of the config again?
To clarify, – and summarize, – this config:
"tls" : {
"automation" : {
"policies" : [
{
"subjects" : [
"shine.caddy.test.shinenelson.xyz",
"*.shine.caddy.test.shinenelson.xyz"
],
...
used to do exact matching only. Now it matches on wildcards like TLS connection policies, even for non-wildcard certificates.
I originally assumed you were trying to do wildcard certs on-demand because that’s how the automation policy selection worked. I’ve changed it though so that now wildcards allow selecting on non-exact matches.
Wow man, this is precisely what I was asking for! This was the functionality that I had in my head when I read about the on-demand feature and the “arbitrary domains” use-case.
I love you what you did there. You just converted an out-of-scope thing into a feature.
However, I thought you said that this would have rate limit implications on Let’s Encrypt’s side right? So, would that be a good idea putting it out there?
Also, what happens to domains that would actually want a proper wildcard domain like you said?
Now, this would be good if it was limited to the internal ( SmallStep ) CA or some other provider without a rate limitation ( or something that is very, very lenient ).
My change had nothing to do with the Let’s Encrypt rate limits. Any use of on-demand is subject to abuse unless you put in protections; that’s why it’s not on by default. And if enabled, make sure to make wise use of it.
They should still be matched to the correct automation policy.
The issuer that’s used is also irrelevant, Caddy’s cert management will behave the same regardless of where it gets a cert from.
How? I don’t get it.
My scenario was that I specify *.shine.caddy.test.shinenelson.xyz in my automation policy and then when I request a.shine.caddy.test.shinenelson.xyz or b.shine.caddy.test.shinenelson.xyz, I’d get 2 separate TLS certificates per sub-domain that was being requested. And with this change, that’s exactly what it did as well.
Now, my question is - what if someone else wanted a single TLS certificate with *.shine.caddy.test.shinenelson.xyz as the SNI?
Oh, I think I get it now. My scenario would work only if on_demand is enabled right?
Otherwise, it would provision the wildcard TLS certificate during the server initialization itself ( of course, in the background! ) so then any subsequent wildcard-matching requests would pick the wildcard TLS certificate.
Good - that’s my intention as well.
But before the change, an automation policy for *.shine.caddy.test.shinenelson.xyz was literal, and would only match for those wildcard certificates.
Then they would configure Caddy’s tls app to manage a certificate for that name (or do that implicitly through auto-HTTPS in the http app).
Yes, exactly. If an exact match is not available to answer a TLS handshake, then a matching wildcard will be used instead.