Host matchers and TLS certificates

This is more of a question than a support request. I couldn’t find a better suited Category, hence posting here. ( Community moderators, feel free to move this topic elsewhere if deemed fit )

There are host matchers all over the place. Almost to the point of confusion.

In my very little usage so far, I’ve come across at least 3 places :

  • apps.https.servers[*].routes[*].match[*].host[*]; then there’s the same thing inside of the subroute handler ( obviously, because nesting is cool :metal: )
  • apps.https.servers[*].tls_connection_policies[*].match.sni[*]
  • apps.tls.automation.policies[*].subjects[*]

And the best part is that all of these trigger automatic HTTPS ( if enabled ).

Now, here’s my question : what if I didn’t want automatic HTTPS to trigger on one of these? viz, on apps.https.servers[*].routes[*].match[*].host[*]

I get the part that we need to trigger automatic HTTPS from anywhere where there is a host match ( because that is the USP of caddy, right? :sunglasses: ). However, I think that triggering it on apps.https.servers[*].routes[*].match[*].host[*] is kind of over-kill if apps.https.servers[*].tls_connection_policies[*].match.sni[*] and apps.tls.automation.policies[*].subjects[*] are already present in the configuration. Here’s an example configuration :

{
  "apps" : {
    "http" : {
      "servers" : {
        "caddy.test.shinenelson.xyz" : {
          "listen" : [
            ":80",
            ":443"
          ],
          "automatic_https" : {
            "disable_redirects" : true
          },
          "routes" : [
            {
              "group" : "shine",
              "match" : [
                {
                  "host" : [
                    "shine.caddy.test.shinenelson.xyz"
                  ]
                }
              ],
              "handle" : [
                {
                  "handler" : "static_response",
                  "body": "Hi there, love from shine and Caddy!"
                }
              ],
              "terminal" : true
            },
            {
              "group" : "shine",
              "match" : [
                {
                  "host" : [
                    "*.shine.caddy.test.shinenelson.xyz"
                  ]
                }
              ],
              "handle": [
                {
                  "@id" : "proxy_shine",
                  "handler" : "reverse_proxy",
                  "upstreams" : [
                    {
                      "dial" : "localhost:8080"
                    },
                    {
                      "dial" : "localhost:8081"
                    }
                  ]
                },
                {
                  "handler": "static_response",
                  "body" : "hey, you were not supposed to get here. - shine"
                }
              ],
              "terminal" : true
            },
            {
              "group" : "shine",
              "@id" : "static_shine",
              "terminal" : true
            }
          ],
          "tls_connection_policies" : [
            {
              "match" : {
                "sni" : [
                  "shine.caddy.test.shinenelson.xyz",
                  "*.shine.caddy.test.shinenelson.xyz",
                ]
              }
            }
          ]
        }
      }
    },
    "tls" : {
      "automation" : {
        "policies" : [
          {
            "subjects" : [
              "shine.caddy.test.shinenelson.xyz",
              "*.shine.caddy.test.shinenelson.xyz"
            ],
            "issuer" : {
              "module" : "acme",
              "ca" : "https://acme-staging-v02.api.letsencrypt.org/directory",
              "challenges" : {
                "dns" : {
                  "provider" : "digitalocean",
                  "auth_token" : "[REDACTED]"
                }
              }
            }
          }
        ]
      }
    }
  }
}

( PS : I’ve posted different iterations of the same configuration in another thread )

With the above configuration, on caddy run, the automatic HTTPS trigger kicks in and fetches the TLS certificates for the SNIs - shine.caddy.test.shinenelson.xyz and *.shine.caddy.test.shinenelson.xyz per the apps.tls.automation.policies[*].subjects[*] ( and caches the certificates in memory as well? :thinking: ) which is all well and good. That’s how it is supposed to be. :+1:

The last route with "@id" : "static_shine" is put in dynamically via the admin API. The curl request for that is :

curl -sSL -X PATCH -H 'Content-Type: application/json' -d "{ \"group\" : \"shine\", \"@id\" : \"shine\", \"match\" : [ { \"host\" : [ \"static.shine.caddy.test.shinenelson.xyz\" ] } ], \"handle\": [ { \"handler\" : \"static_response\", \"body\" : \"static_reponse. much wow. - shine\", \"terminal\" : true }" localhost:2019/load

PS : I’ve unfurled the data into the curl command by hand. I’m actually passing it from a shell script that generates the JSON payload. The shell script itself doesn’t do anything fancy except take some user environment parameters and puts them in the configuration ( everywhere you find shine above came from whoami, etc ).

As soon as I load this new configuration in, caddy server validates the configuration and reloads. This time however, it sees there’s a new host ( static.shine.caddy.test.shinenelson.xyz ) in the matchers and tries to fetch a new certificate for the new host.

Ideally, it should look for a TLS certificate that’s already available and see whether the new host matches any of the existing TLS certificates before attempting to fetch a new certificate for the host.
Especially, in this case, static.shine.caddy.test.shinenelson.xyz matches the wildcard certificate for *.shine.caddy.test.shinenelson.xyz. The reason I got a wildcard certificate in the first place was to avoid the overhead of requesting new certificates and hitting Let’s Encrypt’s rate limits ( contrary to my earlier request for the on-demand wildcard TLS certificates. I was unaware of the abuse vector there at that point ).

Is this a valid question? Or am I missing some configuration magic that I fail to understand here?

Your question seems related to this discussion from earlier (and GH issue). You basically need a way to tell Caddy to manage an equivalent wildcard certificate for a domain, rather than a certificate for that domain itself. Caddy can’t tell the difference otherwise. If you give it a domain a.b.tld and *.b.tld, it doesn’t know that it’s not supposed to manage a separate certificate for a.b.tld – since that is a matter of key management, which we don’t go into the business of presuming without explicit user consent.

Thing is, the other guy wants example.com to get a cert for *.example.com, but you want sub.example.com to get a cert for *.example.com, which is different logic.

Actually, Caddy should already be seeing that wildcard certificate and not activating auto-HTTPS for the new host in the matcher, but it can only do that if the wildcard cert has been loaded already. Since the wildcard is also in a host matcher, no certificate for it has been loaded… so it doesn’t know to skip it.

I guess in the meantime you could use the skip property: JSON Config Structure - Caddy Documentation - to skip the non-wildcard subdomain… maybe?

The apps.tls.automation.policies[*].subjects[*] has the wildcard domain which does fetch the wildcard certificate correctly too.

Also, this new domain comes much later after the first server initialization ( its loaded by the API ). By then, the background ACME runner already has the TLS certificate for the wildcard domain with it. ( I’m not sure whether the caching in the memory is there or not but I’m guessing it is there )

I too was hoping what you said, that the new host matcher should see the wildcard certificate already and not provision the new certificate, but that was not the case for me with at least 2 tries before I gave up.

Unfortunately, no, these sub-domains are dynamic and randomly chosen. It isn’t recorded anywhere.
( Reference : my use-case from the other thread, wrongly for on-demand, about random people picking their own sub-domains ).

Or maybe, I could do 2 curl requests to the API to add them dynamically to the skip list along with the route. First, add the domain to the skip list and then enable the route. That way automatic HTTPS won’t get triggered.

That sounds like a feasible workaround.

I’ll need to a add a new @id within apps.https.servers[*].automatic_https for that to work though.
This is adding complexity to my teardown function though. I thought it would be as easy as just removing the whole route and be done with it. Apparently that wouldn’t work now, would it?

I know what you mean, but what happens is first all the names are scanned and organized and filtered and <bunch of other logic here>, then they go to the certificate manager for processing as a batch. In other words, it doesn’t know that a matching wildcard cert will be loaded for that name.

We could add some logic that examines the batch before sending it off for processing and see if any names in the batch match with any wildcards in the batch, and if so, discard the more specific name, however, maybe that’s not what the user wants? Maybe they do want both certs? I guess this could be configurable… but then the question is… just at the auto-HTTPS level? i.e. is this logic that should be built into the HTTP server, or the core of the certificate manager instead? If it’s in the core, how does the user force it to obtain both certs?

Ah, that’s the trick: every API request is a new server initialization. There is virtually no global state, so an API request to change the config is just as good as starting the server anew (with a few nuances related to network sockets, but that’s unrelated).

Oh, I get it now. This is more complicated than I initially thought. Though, in theory, it’s all very easy ( isn’t everything very easy, in theory? )

Wouldn’t that be a straight-forward solution to implement? Or would it require twisting someone’s arm for it?

I think it should be in CertMagic, in my humble opinion. I mean, why would a user who didn’t bother to leverage the auto HTTPS benefit from this? Also, I don’t see a use-case where they’d want it either. They bring their own certificates and are responsible to load it correctly.

Now, I can think of one use-case where we’d need to put this in the HTTP server - a domain had a wildcard TLS certificate loaded manually, but another sub-domain was enabled for auto HTTPS, then we’d have to pull the logic into the HTTP server.

Oh no, this is getting even more trickier the more I think about it. I’m sure you might have many more permutations and combinations done in your head, than me, already.

Like @smtalk suggested on github, I guess we just let the user decide on the SANs to be on the certificate. Or maybe another flag that says apps.tls.automation.policies[*].subjects[*] == apps.tls.automation.policies[*].sans[*]? ( just making a suggestion, here )

@matt question : how do you specify the wildcard certificate without being in a host matcher? Other than loading it manually from a file or PEMLoader?

What do you mean by that? Specify it for what?

you said that the wildcard certificate would be seen only if it was loaded already right?

I was asking how to achieve that.

The only way I know to specify the wildcard is through the host matchers. So, with your statement, I thought there was another way to load the wildcard certificates separately. Did I get that right? Or were you indicating that it was a platform limitation?

Another way I could think of having the wildcard certificate loaded already is to specify static external certificates from the file system or PEMLoader which route I went down once but went back to automatic HTTPS with caddy because the static system was an overhead to maintain when caddy already had the system built-in.

You could use the automate certificate loader, as mentioned here: Automatic HTTPS — Caddy Documentation - but I have not tested it for your use case, you will have to try and see.

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.