Dynamic TLS selection

Tomasz_Trebski · July 20, 2023, 2:09pm

1. The problem I’m having:

Trying to implement a use case where I should decide how to provide certificate for domains which depends on the domain. Assume I have 4 categories of domains:

a) those I control, wildcard certificates where DNS happens via Route53
b) those I control, non wildcard certs, domains like app.foo.com, app.foo.com where DNS happens via Route53
c) those I cannot control, customer’s controlled, but customers expects certs to be generated on my end plan on using http-01 challange
d) those I cannot control and where certificates are provided externally

AFAIR cases a), b) I can easily resolve like this:

:443,
*.{$WILDCARD_DOMAIN}:443, {$WILDCARD_DOMAIN}:443 {
  import route53_tls
}

route53_tls snipped configures DNS challange via Route53 and on_demand_tls. Both confirmed working.

b) could also be handled via http-01 challange to support use c), couldn’t it?

which only leaves use-case d) which either means operating on CaddyAPI to provide configuration next to uploading certs to disk Caddy can read from or changing Caddyfile and reloading the service on the fly neither which is making me happy.

Anyway. Ideally I would be super happy if I could simply ask some server (i.e. similar to on_demand_tls) of mine that it would not only respond if domain is known but also provide details as to TLS configuration so that I can change caddy internally based on that. For example said service tells me that domain is use-case c) I do modify state by setting the domain to http-01. Service says use-case d) and in response sends me certs I already change caddy based on that response.

Example responses

{ "domain": "x.wildcard.com", "ssl": "auto", "wildcard": 1, "mine": "yes"}
{ "domain": "custom.domain", "ssl": "auto", "mine": "yes" }
{ "domain": "another.domain", "ssl": "auto", "mine": "no" }}
{ "domain": "and.another.domain", "ssl": "manual", "mine": "no" , "certKey": "...", "certCrt": "..."}

2. Error messages and/or full log output:

N/A

3. Caddy version:

v2.6.4 h1:2hwYqiRwk1tf3VruhMpLcYTg+11fCdr8S3jhNAdnPy8=

4. How I installed and ran Caddy:

Docker / ECS

FROM caddy:builder-alpine AS builder

RUN apk add curl
RUN xcaddy build \
  --with github.com/caddy-dns/route53 \
  --with github.com/silinternational/certmagic-storage-dynamodb/v3

FROM caddy:alpine

COPY --from=builder /usr/bin/caddy /usr/bin/caddy

WORKDIR /etc/caddy
COPY ./Caddyfile ./

EXPOSE 80
EXPOSE 443
EXPOSE 2019
EXPOSE 8888

HEALTHCHECK CMD [ "curl", "-f", "http://localhost:8888" ]

a. System environment:

ECS service is primary runtime

b. Command:

N/A, I do not override how caddy is launched in container

c. Service/unit/compose file:

N/A, container is launched in ECS and there is no problem with this and that is not relevant to the question

d. My complete Caddy config:

There is none, I am at lost in resolving the use thus the question.

5. Links to relevant resources:

N/A

francislavoie · July 20, 2023, 7:04pm

Reloading shouldn’t be a problem. In v2.7.0 we’re significantly improving the situation for config reloads in that it will avoid reloading certs from storage which was a big part % of the time taken during reloads.

We kinda have support for that, actually. We have the concept of “certificate managers” that you can configure with tls > get_certificate. See tls (Caddyfile directive) — Caddy Documentation

You’d still have to insert in your config the list of domains for which certs should be fetched though, because otherwise Caddy won’t know whether to use on_demand or not.

/cc @matt I wonder if a get_certificate HTTP endpoint could respond with a different response that would indicate that Caddy should fetch a cert using On-Demand instead. But… that sounds really complex to implement because of the layering at play. Since that would be quite a niche need, likely the kind of feature that would need to have its work sponsored (paid for) I figure.

Tomasz_Trebski · July 20, 2023, 8:02pm

Reloading on its own perhaps not but changing the file will require a bit of implementation unless I really put a sidecar-ish container that will receive event Bruh, I got new shiny domain and will POST relevant piece of configuration using admin endpoint? I mean that is certainly preferably vs changing Caddyfile as in changing the physical file. Not only that means having something working inside the Caddy container but also means reworking Caddy’s parsing logic. Not to mention that I still will be forced to reload the server via CLI.

Can’t I simply write some Go extension? I am building stuff on AWS so assume using AWS GO + SQS to receive an event informing about domain change to later use some internal Caddy’s GO API to inject configuration?

On different side? Would following be possible

{domain:443} {
   # assume cert files are named after domain
   # template away
   tls  /certs/{domain}.crt /certs/{domain}.key {
     on_demand            # keep checking if domain is known
   }
}

Then I guess it would only be a matter of precedence of one configuration block against another as in:

1 block matches against known wildcard domain → wildcard certs
2nd block matches against the example from above → certificates are explicit albeit Caddy needs to try and see if they are there
3rd block starts with :443 and fallbacks to http-01 for any domain that is not wildcard-ed not with custom certs and it does not matter if I control domain or this is customer’s domain because caddy simply puts relevant http-01 endpoint to domain and can resolve challange.

matt · July 20, 2023, 10:02pm

Caddy is designed to be reloaded frequently; and as Francis mentioned the reloads will become very lightweight in 2.7.

I’m not really sure I follow the problem here; is there a problem with simply running caddy reload or making an HTTP request to update its config?

Sure, but that sounds more complicated than necessary. (Just from the info provided so far.)

Francis is correct, but let me offer my take:

Put these domains in your Caddy config. Use the tls directive (if Caddyfile; or an automation policy if JSON) to customize how the certificates are obtained.

Use on-demand TLS for this.

Use a certificate manager module for this. Whether it’s an existing one (there is an “HTTP” one for general purpose use) or a custom one (maybe based on the HTTP module to start with?), this can get the certificate to use during handshake-time from an entity that is obtaining and keeping it renewed.

Your cert manager module can determine whether it should obtain a certificate or not; and if not, Caddy can fall back to its own on-demand TLS.

francislavoie · July 21, 2023, 2:12am

Does it already do that? That was my question above, I thought it wouldn’t fallback and just fail the handshake.

matt · July 21, 2023, 2:53am

I just checked the code, and it logs a debug message if there’s no certificate and no error:

github.com

caddyserver/certmagic/blob/51b31907aaa07f21e1bf2974193d11cdbf32802b/handshake.go#L767-L768


      
          		logger.Debug("all external certificate managers yielded no certificates and no errors", zap.String("sni", hello.ServerName))
          		return Certificate{}, nil

and then it returns an empty value and nil error, and then it will try on-demand issuance if enabled.

francislavoie · July 21, 2023, 3:00am

Oh okay so you could configure it like this?

https://
	tls {
		get_certificate http http://some-endpoint
		on_demand
	}
}

If that works, nice!

Then yeah @Tomasz_Trebski you should have everything you need.

Tomasz_Trebski · July 21, 2023, 5:53am

Thx @francislavoie @matt , that clears some doubts.

The problem is not either of those but the time which my team does not have in abundance. Naturally I would want something that just works but as you can imagine that’s rarely the case and as @francislavoie rightfully pointed out I might be hitting a niche here which is hardly a justification for you to implement that.

So your idea is to have them explicit (not wildcards but could also be done for them) in here? Have this sort of queue and use API to let Caddy know there is new domain as in (pseudocode):

POST
example.com {
  import route53_tls
}

where route53_tls does the on_demand double-check and ask for cert via dns-01?

On the other hand you would leave domains I cannot control but where certs are to be generated fallback to http-01 via

:443 {
  tls {
    on_demand
  }
}

and couple that with @francislavoie i.e. get_certificate http http://some-endpoint which actually already gives me use-case d) i.e. first Caddy asks my endpoint if there’s a certificate in store and my endpoint returns it but in case of 404 Caddy generates new certificate.

To wrap this up and circle back to original post and use cases:

a) … => explicit in Caddy config, this gonna work for me just fine
b) … => post them via caddy API
c) + d) use explicit code snippet from @francislavoie to first ask for cert then fallback to on_demand

Let me cook something up to see if we can make it happen.

Tomasz_Trebski · July 21, 2023, 11:39am

@francislavoie @matt

It is more or less working but what I think stopped working is wildcard case.
Here’s my config to dicuss:

{
  debug

  on_demand_tls {
    ask       {$DOMAINS_MGR}/check
    interval  1m
    burst     5
  }

  storage dynamodb {$CERTS_TABLE} {
    aws_region {$AWS_REGION}
  }
}

(logs) {
  log {
    output stderr
    format json
  }
}

(logic) {
  import logs

  bind {$ADDRESS}
  encode gzip

  header /* {
      -Server
  }

  reverse_proxy /* {
    to                                                   {$AGW_URL}
    header_up Host                           {http.reverse_proxy.upstream.host}
    header_up X-Real-IP                   {http.reverse-proxy.upstream.address}
    header_up X-Forwarded-Proto {scheme}
  }
}

(http_01_tls) {
  import logic
  tls {
    get_certificate http {$DOMAINS_MGR}/certificate
    on_demand
  }
}

(route53_tls) {
  import logic
  tls {
    dns route53
    on_demand
  }
}

:8888 {
  import logs
  respond / 204
}

# wildcard, for default case
*.wild.domains.net:443 {
  import route53_tls
}

# domains from customers, with out without cert
# cdoc - CustomerDomainOwnCert
# cdcd - CustomerDomainCustomerCert
cdoc-0.example.net,
cdoc-1.example.net,
cdcc-0.example.net,
cdcc-1.example.net {
  import http_01_tls
}

# rest of domains, assumed to be owned
# those are not wildcard domains i.e.
# something like super.com or another.net
:443 {
  import route53_tls
}

All the time I try to get through to the page for which wildcard certificate ought to be used I keep getting a cert that belongs to the very particular domain.
AFAIR, Caddy docs mention, that precedence is given to the best matching block which in case of my configuration means:

check block for domains that fully qualified (cases c) and d)
check the block for wildcards
fallback to :443

Am I missing something? I am pretty sure I got this working before but it got lost once I added the code to support other cases. Naughty ( ) of me to not store wildcard configuration in git somewhere, I should have done that.

log example that shows that specific cert is being generated:

2023-07-21T12:01:07.834000+00:00 ui/ui/1ae09fcf5804429ea7751661f9a873b0 {"level":"info","ts":1689940867.834722,"logger":"tls.obtain","msg":"obtaining certificate","identifier":"wild-8.wild.domains.net"}

Tomasz_Trebski · July 24, 2023, 7:02am

@matt @francislavoie would you mind helping out just a tiny bit ?:>

I have seen this thread: Mixing wildcard certificate with on_demand feature - #4 by pdnz which is exactly what I am trying to do but I already have the route53 in place.

Also in logs I can see entries like:

{"level":"info","ts":1690183666.7853959,"logger":"http.acme_client","msg":"trying to solve challenge","identifier":"test.ays-dev-tt.net","challenge_type":"dns-01","ca":"https://acme-v02.api.letsencrypt.org/directory"}

francislavoie · July 24, 2023, 7:42am

What are your full debug logs from when the first request came in for that domain?

Tomasz_Trebski · July 24, 2023, 7:52am

addywildcarddomains-caddy-1  | {"level":"info","ts":1690185077.961121,"msg":"using provided configuration","config_file":"/etc/caddy/Caddyfile","config_adapter":"caddyfile"}
caddywildcarddomains-caddy-1  | {"level":"warn","ts":1690185077.9626849,"logger":"caddyfile","msg":"Unnecessary header_up X-Forwarded-Host: the reverse proxy's default behavior is to pass headers to the upstream"}
caddywildcarddomains-caddy-1  | {"level":"warn","ts":1690185077.9627624,"logger":"caddyfile","msg":"Unnecessary header_up X-Forwarded-Proto: the reverse proxy's default behavior is to pass headers to the upstream"}
caddywildcarddomains-caddy-1  | {"level":"warn","ts":1690185077.9671943,"msg":"Caddyfile input is not formatted; run the 'caddy fmt' command to fix inconsistencies","adapter":"caddyfile","file":"/etc/caddy/Caddyfile","line":2}
caddywildcarddomains-caddy-1  | {"level":"info","ts":1690185077.9700787,"logger":"admin","msg":"admin endpoint started","address":"localhost:2019","enforce_origin":false,"origins":["//127.0.0.1:2019","//localhost:2019","//[::1]:2019"]}
caddywildcarddomains-caddy-1  | {"level":"info","ts":1690185077.9706311,"logger":"http","msg":"server is listening only on the HTTPS port but has no TLS connection policies; adding one to enable TLS","server_name":"srv0","https_port":443}
caddywildcarddomains-caddy-1  | {"level":"info","ts":1690185077.9711363,"logger":"http","msg":"enabling automatic HTTP->HTTPS redirects","server_name":"srv0"}
caddywildcarddomains-caddy-1  | {"level":"info","ts":1690185077.9715438,"logger":"tls.cache.maintenance","msg":"started background certificate maintenance","cache":"0xc00029a0e0"}
caddywildcarddomains-caddy-1  | {"level":"info","ts":1690185077.9786193,"logger":"http","msg":"enabling HTTP/3 listener","addr":":443"}
caddywildcarddomains-caddy-1  | {"level":"info","ts":1690185077.978765,"msg":"failed to sufficiently increase receive buffer size (was: 208 kiB, wanted: 2048 kiB, got: 416 kiB). See https://github.com/quic-go/quic-go/wiki/UDP-Receive-Buffer-Size for details."}
caddywildcarddomains-caddy-1  | {"level":"debug","ts":1690185077.9788616,"logger":"http","msg":"starting server loop","address":"[::]:443","tls":true,"http3":true}
caddywildcarddomains-caddy-1  | {"level":"info","ts":1690185077.9788744,"logger":"http.log","msg":"server running","name":"srv0","protocols":["h1","h2","h3"]}
caddywildcarddomains-caddy-1  | {"level":"debug","ts":1690185077.9801319,"logger":"http","msg":"starting server loop","address":"[::]:8888","tls":false,"http3":false}
caddywildcarddomains-caddy-1  | {"level":"info","ts":1690185077.9802086,"logger":"http.log","msg":"server running","name":"srv1","protocols":["h1","h2","h3"]}
caddywildcarddomains-caddy-1  | {"level":"debug","ts":1690185077.9803035,"logger":"http","msg":"starting server loop","address":"[::]:80","tls":false,"http3":false}
caddywildcarddomains-caddy-1  | {"level":"info","ts":1690185077.9803162,"logger":"http.log","msg":"server running","name":"remaining_auto_https_redirects","protocols":["h1","h2","h3"]}
caddywildcarddomains-caddy-1  | {"level":"info","ts":1690185077.9803226,"logger":"http","msg":"enabling automatic TLS certificate management","domains":["*.ays-dev-tt.net","ays-dev-tt.net"]}
caddywildcarddomains-caddy-1  | {"level":"info","ts":1690185077.9816577,"logger":"tls","msg":"cleaning storage unit","description":"FileStorage:/data/caddy"}
caddywildcarddomains-caddy-1  | {"level":"info","ts":1690185077.9829724,"msg":"autosaved config (load with --resume flag)","file":"/config/caddy/autosave.json"}
caddywildcarddomains-caddy-1  | {"level":"info","ts":1690185077.9830227,"msg":"serving initial configuration"}
caddywildcarddomains-caddy-1  | {"level":"info","ts":1690185077.9842074,"logger":"tls","msg":"finished cleaning storage units"}
caddywildcarddomains-caddy-1  | {"level":"debug","ts":1690185096.7524934,"logger":"events","msg":"event","name":"tls_get_certificate","id":"acec861b-40e2-48cb-88ad-2aa60cb7b7b4","origin":"tls","data":{"client_hello":{"CipherSuites":[4866,4867,4865,49200,49196,49192,49188,49172,49162,159,107,57,52393,52392,52394,65413,196,136,129,157,61,53,192,132,49199,49195,49191,49187,49171,49161,158,103,51,190,69,156,60,47,186,65,49169,49159,5,4,49170,49160,22,10,255],"ServerName":"test2.ays-dev-tt.net","SupportedCurves":[29,23,24,25],"SupportedPoints":"AA==","SignatureSchemes":[2054,1537,1539,2053,1281,1283,2052,1025,1027,513,515],"SupportedProtos":["h2","http/1.1"],"SupportedVersions":[772,771,770,769],"Conn":{}}}}
caddywildcarddomains-caddy-1  | {"level":"debug","ts":1690185096.752702,"logger":"tls.handshake","msg":"no matching certificates and no custom selection logic","identifier":"test2.ays-dev-tt.net"}
caddywildcarddomains-caddy-1  | {"level":"debug","ts":1690185096.7527103,"logger":"tls.handshake","msg":"no matching certificates and no custom selection logic","identifier":"*.ays-dev-tt.net"}
caddywildcarddomains-caddy-1  | {"level":"debug","ts":1690185096.7527137,"logger":"tls.handshake","msg":"no matching certificates and no custom selection logic","identifier":"*.*.net"}
caddywildcarddomains-caddy-1  | {"level":"debug","ts":1690185096.7527168,"logger":"tls.handshake","msg":"no matching certificates and no custom selection logic","identifier":"*.*.*"}
caddywildcarddomains-caddy-1  | {"level":"debug","ts":1690185096.752722,"logger":"tls.handshake","msg":"all external certificate managers yielded no certificates and no errors","remote_ip":"172.19.0.1","remote_port":"56270","sni":"test2.ays-dev-tt.net"}
caddywildcarddomains-caddy-1  | {"level":"debug","ts":1690185096.7624805,"logger":"tls","msg":"response from ask endpoint","domain":"test2.ays-dev-tt.net","url":"http://domains:5000/check?domain=test2.ays-dev-tt.net","status":200}
caddywildcarddomains-caddy-1  | {"level":"info","ts":1690185096.7625916,"logger":"tls.on_demand","msg":"obtaining new certificate","remote_ip":"172.19.0.1","remote_port":"56270","server_name":"test2.ays-dev-tt.net"}
caddywildcarddomains-caddy-1  | {"level":"info","ts":1690185096.766266,"logger":"tls.obtain","msg":"acquiring lock","identifier":"test2.ays-dev-tt.net"}
caddywildcarddomains-caddy-1  | {"level":"info","ts":1690185096.7765398,"logger":"tls.obtain","msg":"lock acquired","identifier":"test2.ays-dev-tt.net"}
caddywildcarddomains-caddy-1  | {"level":"info","ts":1690185096.7769156,"logger":"tls.obtain","msg":"obtaining certificate","identifier":"test2.ays-dev-tt.net"}
caddywildcarddomains-caddy-1  | {"level":"debug","ts":1690185096.7770662,"logger":"events","msg":"event","name":"cert_obtaining","id":"d76628a1-0ed9-4fd2-8507-275c2833545d","origin":"tls","data":{"identifier":"test2.ays-dev-tt.net"}}
caddywildcarddomains-caddy-1  | {"level":"debug","ts":1690185096.7774956,"logger":"tls.obtain","msg":"trying issuer 1/2","issuer":"acme-v02.api.letsencrypt.org-directory"}
caddywildcarddomains-caddy-1  | {"level":"debug","ts":1690185097.7298977,"logger":"http.acme_client","msg":"http request","method":"GET","url":"https://acme-v02.api.letsencrypt.org/directory","headers":{"User-Agent":["Caddy/2.6.4 CertMagic acmez (linux; amd64)"]},"response_headers":{"Cache-Control":["public, max-age=0, no-cache"],"Content-Length":["752"],"Content-Type":["application/json"],"Date":["Mon, 24 Jul 2023 07:51:37 GMT"],"Server":["nginx"],"Strict-Transport-Security":["max-age=604800"],"X-Frame-Options":["DENY"]},"status_code":200}
caddywildcarddomains-caddy-1  | {"level":"debug","ts":1690185097.9304242,"logger":"http.acme_client","msg":"http request","method":"HEAD","url":"https://acme-v02.api.letsencrypt.org/acme/new-nonce","headers":{"User-Agent":["Caddy/2.6.4 CertMagic acmez (linux; amd64)"]},"response_headers":{"Cache-Control":["public, max-age=0, no-cache"],"Date":["Mon, 24 Jul 2023 07:51:38 GMT"],"Link":["<https://acme-v02.api.letsencrypt.org/directory>;rel=\"index\""],"Replay-Nonce":["853FbhLwrQtxN_nvAiIHs7MJl_uEq9wc1j4FFJI_nbwATbc"],"Server":["nginx"],"Strict-Transport-Security":["max-age=604800"],"X-Frame-Options":["DENY"]},"status_code":200}
caddywildcarddomains-caddy-1  | {"level":"debug","ts":1690185098.1485722,"logger":"http.acme_client","msg":"http request","method":"POST","url":"https://acme-v02.api.letsencrypt.org/acme/new-acct","headers":{"Content-Type":["application/jose+json"],"User-Agent":["Caddy/2.6.4 CertMagic acmez (linux; amd64)"]},"response_headers":{"Boulder-Requester":["1222163617"],"Cache-Control":["public, max-age=0, no-cache"],"Content-Length":["332"],"Content-Type":["application/json"],"Date":["Mon, 24 Jul 2023 07:51:38 GMT"],"Link":["<https://acme-v02.api.letsencrypt.org/directory>;rel=\"index\"","<https://letsencrypt.org/documents/LE-SA-v1.3-September-21-2022.pdf>;rel=\"terms-of-service\""],"Location":["https://acme-v02.api.letsencrypt.org/acme/acct/1222163617"],"Replay-Nonce":["27126TZcJd0xMkdFwtfPgcjLyeTmlbbs99drMCgZHkIkK8M"],"Server":["nginx"],"Strict-Transport-Security":["max-age=604800"],"X-Frame-Options":["DENY"]},"status_code":201}
caddywildcarddomains-caddy-1  | {"level":"info","ts":1690185098.1603208,"logger":"http","msg":"waiting on internal rate limiter","identifiers":["test2.ays-dev-tt.net"],"ca":"https://acme-v02.api.letsencrypt.org/directory","account":"tomasz.trebski@appyourself.net"}
caddywildcarddomains-caddy-1  | {"level":"info","ts":1690185098.1605787,"logger":"http","msg":"done waiting on internal rate limiter","identifiers":["test2.ays-dev-tt.net"],"ca":"https://acme-v02.api.letsencrypt.org/directory","account":"tomasz.trebski@appyourself.net"}
caddywildcarddomains-caddy-1  | {"level":"debug","ts":1690185098.4053288,"logger":"http.acme_client","msg":"http request","method":"POST","url":"https://acme-v02.api.letsencrypt.org/acme/new-order","headers":{"Content-Type":["application/jose+json"],"User-Agent":["Caddy/2.6.4 CertMagic acmez (linux; amd64)"]},"response_headers":{"Boulder-Requester":["1222163617"],"Cache-Control":["public, max-age=0, no-cache"],"Content-Length":["346"],"Content-Type":["application/json"],"Date":["Mon, 24 Jul 2023 07:51:38 GMT"],"Link":["<https://acme-v02.api.letsencrypt.org/directory>;rel=\"index\""],"Location":["https://acme-v02.api.letsencrypt.org/acme/order/1222163617/196962663357"],"Replay-Nonce":["2712HVWlK6gcHx0mjfzrQh2z7W-bVjITx2cAqjj6_UZHhUE"],"Server":["nginx"],"Strict-Transport-Security":["max-age=604800"],"X-Frame-Options":["DENY"]},"status_code":201}
caddywildcarddomains-caddy-1  | {"level":"debug","ts":1690185098.6101499,"logger":"http.acme_client","msg":"http request","method":"POST","url":"https://acme-v02.api.letsencrypt.org/acme/authz-v3/248489224017","headers":{"Content-Type":["application/jose+json"],"User-Agent":["Caddy/2.6.4 CertMagic acmez (linux; amd64)"]},"response_headers":{"Boulder-Requester":["1222163617"],"Cache-Control":["public, max-age=0, no-cache"],"Content-Length":["804"],"Content-Type":["application/json"],"Date":["Mon, 24 Jul 2023 07:51:38 GMT"],"Link":["<https://acme-v02.api.letsencrypt.org/directory>;rel=\"index\""],"Replay-Nonce":["853FE9LOaPvMOUOPR3hPFQFu9HOSkPmt--bg_1Rsc4TL5JU"],"Server":["nginx"],"Strict-Transport-Security":["max-age=604800"],"X-Frame-Options":["DENY"]},"status_code":200}
caddywildcarddomains-caddy-1  | {"level":"debug","ts":1690185098.6107538,"logger":"http.acme_client","msg":"no solver configured","challenge_type":"tls-alpn-01"}
caddywildcarddomains-caddy-1  | {"level":"info","ts":1690185098.610776,"logger":"http.acme_client","msg":"trying to solve challenge","identifier":"test2.ays-dev-tt.net","challenge_type":"dns-01","ca":"https://acme-v02.api.letsencrypt.org/directory"}
caddywildcarddomains-caddy-1  | {"level":"debug","ts":1690185100.393418,"logger":"http.acme_client","msg":"waiting for solver before continuing","identifier":"test2.ays-dev-tt.net","challenge_type":"dns-01"}
caddywildcarddomains-caddy-1  | {"level":"debug","ts":1690185116.113534,"logger":"http.acme_client","msg":"done waiting for solver","identifier":"test2.ays-dev-tt.net","challenge_type":"dns-01"}
caddywildcarddomains-caddy-1  | {"level":"debug","ts":1690185116.3140285,"logger":"http.acme_client","msg":"http request","method":"POST","url":"https://acme-v02.api.letsencrypt.org/acme/chall-v3/248489224017/ze5YJg","headers":{"Content-Type":["application/jose+json"],"User-Agent":["Caddy/2.6.4 CertMagic acmez (linux; amd64)"]},"response_headers":{"Boulder-Requester":["1222163617"],"Cache-Control":["public, max-age=0, no-cache"],"Content-Length":["186"],"Content-Type":["application/json"],"Date":["Mon, 24 Jul 2023 07:51:56 GMT"],"Link":["<https://acme-v02.api.letsencrypt.org/directory>;rel=\"index\"","<https://acme-v02.api.letsencrypt.org/acme/authz-v3/248489224017>;rel=\"up\""],"Location":["https://acme-v02.api.letsencrypt.org/acme/chall-v3/248489224017/ze5YJg"],"Replay-Nonce":["2712_6H5qNJEyKfhCW-sfzlqlRV0hnQQCTNoTG8RiihaNU4"],"Server":["nginx"],"Strict-Transport-Security":["max-age=604800"],"X-Frame-Options":["DENY"]},"status_code":200}
caddywildcarddomains-caddy-1  | {"level":"debug","ts":1690185116.3141494,"logger":"http.acme_client","msg":"challenge accepted","identifier":"test2.ays-dev-tt.net","challenge_type":"dns-01"}
caddywildcarddomains-caddy-1  | {"level":"debug","ts":1690185116.7788312,"logger":"http.acme_client","msg":"http request","method":"POST","url":"https://acme-v02.api.letsencrypt.org/acme/authz-v3/248489224017","headers":{"Content-Type":["application/jose+json"],"User-Agent":["Caddy/2.6.4 CertMagic acmez (linux; amd64)"]},"response_headers":{"Boulder-Requester":["1222163617"],"Cache-Control":["public, max-age=0, no-cache"],"Content-Length":["515"],"Content-Type":["application/json"],"Date":["Mon, 24 Jul 2023 07:51:56 GMT"],"Link":["<https://acme-v02.api.letsencrypt.org/directory>;rel=\"index\""],"Replay-Nonce":["2712PYDO7sQZL_dGzGhsxKfGnNcpVUS_1wLnJ41VAbRWoC8"],"Server":["nginx"],"Strict-Transport-Security":["max-age=604800"],"X-Frame-Options":["DENY"]},"status_code":200}
caddywildcarddomains-caddy-1  | {"level":"info","ts":1690185118.1503334,"logger":"http.acme_client","msg":"authorization finalized","identifier":"test2.ays-dev-tt.net","authz_status":"valid"}
caddywildcarddomains-caddy-1  | {"level":"info","ts":1690185118.150518,"logger":"http.acme_client","msg":"validations succeeded; finalizing order","order":"https://acme-v02.api.letsencrypt.org/acme/order/1222163617/196962663357"}
caddywildcarddomains-caddy-1  | {"level":"debug","ts":1690185118.5872414,"logger":"http.acme_client","msg":"http request","method":"POST","url":"https://acme-v02.api.letsencrypt.org/acme/finalize/1222163617/196962663357","headers":{"Content-Type":["application/jose+json"],"User-Agent":["Caddy/2.6.4 CertMagic acmez (linux; amd64)"]},"response_headers":{"Boulder-Requester":["1222163617"],"Cache-Control":["public, max-age=0, no-cache"],"Content-Length":["448"],"Content-Type":["application/json"],"Date":["Mon, 24 Jul 2023 07:51:58 GMT"],"Link":["<https://acme-v02.api.letsencrypt.org/directory>;rel=\"index\""],"Location":["https://acme-v02.api.letsencrypt.org/acme/order/1222163617/196962663357"],"Replay-Nonce":["853F7yYPhUOEflYfxSehPRM4N-ZCrvHkOWzgk6_b5JNm1YY"],"Server":["nginx"],"Strict-Transport-Security":["max-age=604800"],"X-Frame-Options":["DENY"]},"status_code":200}
caddywildcarddomains-caddy-1  | {"level":"debug","ts":1690185118.8078496,"logger":"http.acme_client","msg":"http request","method":"POST","url":"https://acme-v02.api.letsencrypt.org/acme/cert/0371e10bab4b599d8de74fec03fbb775b16a","headers":{"Content-Type":["application/jose+json"],"User-Agent":["Caddy/2.6.4 CertMagic acmez (linux; amd64)"]},"response_headers":{"Cache-Control":["public, max-age=0, no-cache"],"Content-Length":["5256"],"Content-Type":["application/pem-certificate-chain"],"Date":["Mon, 24 Jul 2023 07:51:58 GMT"],"Link":["<https://acme-v02.api.letsencrypt.org/directory>;rel=\"index\"","<https://acme-v02.api.letsencrypt.org/acme/cert/0371e10bab4b599d8de74fec03fbb775b16a/1>;rel=\"alternate\""],"Replay-Nonce":["853FdJL3kD0XLjOrHtjXZAEDV8AFiNgLHDdrBLLyXi1tJ0s"],"Server":["nginx"],"Strict-Transport-Security":["max-age=604800"],"X-Frame-Options":["DENY"]},"status_code":200}
caddywildcarddomains-caddy-1  | {"level":"debug","ts":1690185119.0296867,"logger":"http.acme_client","msg":"http request","method":"POST","url":"https://acme-v02.api.letsencrypt.org/acme/cert/0371e10bab4b599d8de74fec03fbb775b16a/1","headers":{"Content-Type":["application/jose+json"],"User-Agent":["Caddy/2.6.4 CertMagic acmez (linux; amd64)"]},"response_headers":{"Cache-Control":["public, max-age=0, no-cache"],"Content-Length":["3332"],"Content-Type":["application/pem-certificate-chain"],"Date":["Mon, 24 Jul 2023 07:51:59 GMT"],"Link":["<https://acme-v02.api.letsencrypt.org/directory>;rel=\"index\"","<https://acme-v02.api.letsencrypt.org/acme/cert/0371e10bab4b599d8de74fec03fbb775b16a/0>;rel=\"alternate\""],"Replay-Nonce":["853FNSiIYCmlvreUHkrGtw9MREJI_W7Vzq9P8VJha2C5UCk"],"Server":["nginx"],"Strict-Transport-Security":["max-age=604800"],"X-Frame-Options":["DENY"]},"status_code":200}
caddywildcarddomains-caddy-1  | {"level":"info","ts":1690185119.0298977,"logger":"http.acme_client","msg":"successfully downloaded available certificate chains","count":2,"first_url":"https://acme-v02.api.letsencrypt.org/acme/cert/0371e10bab4b599d8de74fec03fbb775b16a"}
caddywildcarddomains-caddy-1  | {"level":"info","ts":1690185119.0324125,"logger":"tls.obtain","msg":"certificate obtained successfully","identifier":"test2.ays-dev-tt.net"}
caddywildcarddomains-caddy-1  | {"level":"debug","ts":1690185119.0325215,"logger":"events","msg":"event","name":"cert_obtained","id":"63efe594-4465-4a26-8e75-96b32616d73a","origin":"tls","data":{"identifier":"test2.ays-dev-tt.net","issuers":"acme-v02.api.letsencrypt.org-directory","renewal":false,"storage_key":"test2.ays-dev-tt.net"}}
caddywildcarddomains-caddy-1  | {"level":"info","ts":1690185119.0325613,"logger":"tls.obtain","msg":"releasing lock","identifier":"test2.ays-dev-tt.net"}
caddywildcarddomains-caddy-1  | {"level":"debug","ts":1690185119.0327036,"logger":"tls.handshake","msg":"no matching certificates and no custom selection logic","identifier":"test2.ays-dev-tt.net"}
caddywildcarddomains-caddy-1  | {"level":"debug","ts":1690185119.032719,"logger":"tls.handshake","msg":"no matching certificates and no custom selection logic","identifier":"*.ays-dev-tt.net"}
caddywildcarddomains-caddy-1  | {"level":"debug","ts":1690185119.0327244,"logger":"tls.handshake","msg":"no matching certificates and no custom selection logic","identifier":"*.*.net"}
caddywildcarddomains-caddy-1  | {"level":"debug","ts":1690185119.0327284,"logger":"tls.handshake","msg":"no matching certificates and no custom selection logic","identifier":"*.*.*"}
caddywildcarddomains-caddy-1  | {"level":"debug","ts":1690185119.0327346,"logger":"tls.handshake","msg":"all external certificate managers yielded no certificates and no errors","remote_ip":"172.19.0.1","remote_port":"56270","sni":"test2.ays-dev-tt.net"}
caddywildcarddomains-caddy-1  | {"level":"debug","ts":1690185119.0333095,"logger":"tls","msg":"loading managed certificate","domain":"test2.ays-dev-tt.net","expiration":1697957518,"issuer_key":"acme-v02.api.letsencrypt.org-directory","storage":"FileStorage:/data/caddy"}
caddywildcarddomains-caddy-1  | {"level":"debug","ts":1690185119.4500487,"logger":"tls.cache","msg":"added certificate to cache","subjects":["test2.ays-dev-tt.net"],"expiration":1697957518,"managed":true,"issuer_key":"acme-v02.api.letsencrypt.org-directory","hash":"ab25fa03417a4d247e8150ebe102b4eaf07bca6f71722e011a0fc4036ed9037a","cache_size":1,"cache_capacity":10000}
caddywildcarddomains-caddy-1  | {"level":"debug","ts":1690185119.4502397,"logger":"events","msg":"event","name":"cached_managed_cert","id":"51a2523a-693e-46f9-818b-74ed59d95c2d","origin":"tls","data":{"sans":["test2.ays-dev-tt.net"]}}
caddywildcarddomains-caddy-1  | {"level":"debug","ts":1690185119.4502912,"logger":"tls.handshake","msg":"loaded certificate from storage","remote_ip":"172.19.0.1","remote_port":"56270","subjects":["test2.ays-dev-tt.net"],"managed":true,"expiration":1697957518,"hash":"ab25fa03417a4d247e8150ebe102b4eaf07bca6f71722e011a0fc4036ed9037a"}
caddywildcarddomains-caddy-1  | {"level":"debug","ts":1690185119.459658,"logger":"http.handlers.reverse_proxy","msg":"selected upstream","dial":"my.appit.website:443","total_upstreams":1}
caddywildcarddomains-caddy-1  | {"level":"debug","ts":1690185119.9077697,"logger":"http.handlers.reverse_proxy","msg":"upstream roundtrip","upstream":"my.appit.website:443","duration":0.447842226,"request":{"remote_ip":"172.19.0.1","remote_port":"56270","proto":"HTTP/2.0","method":"GET","host":"my.appit.website","uri":"/","headers":{"Accept":["*/*"],"X-Forwarded-For":["172.19.0.1"],"X-Forwarded-Proto":["https"],"X-Forwarded-Host":["test2.ays-dev-tt.net"],"X-Forwarded-Port":[""],"X-Real-Ip":["{http.reverse-proxy.upstream.address}"],"User-Agent":["curl/7.88.1"]},"tls":{"resumed":false,"version":772,"cipher_suite":4865,"proto":"h2","server_name":"test2.ays-dev-tt.net"}},"headers":{"Date":["Mon, 24 Jul 2023 07:52:00 GMT"],"Server":["Apache/2.4.34 (Red Hat) OpenSSL/1.0.2k-fips"],"Cache-Control":["public, max-age=0"],"Content-Length":["23828"],"X-Robots-Tag":["noindex, nofollow, noarchive, nosnippet"],"Accept-Ranges":["bytes"],"Last-Modified":["Fri, 21 Jul 2023 10:15:55 GMT"],"Etag":["W/\"5d14-18977f2e778\""],"Content-Type":["text/html; charset=UTF-8"]},"status":200}

that’s the complete output

Tomasz_Trebski · July 24, 2023, 7:54am

And here’s the curl

curl -v https://test2.ays-dev-tt.net
*   Trying 127.0.0.1:443...
* Connected to test2.ays-dev-tt.net (127.0.0.1) port 443 (#0)
* ALPN: offers h2,http/1.1
* (304) (OUT), TLS handshake, Client hello (1):
*  CAfile: /etc/ssl/cert.pem
*  CApath: none
* (304) (IN), TLS handshake, Server hello (2):
* (304) (IN), TLS handshake, Unknown (8):
* (304) (IN), TLS handshake, Certificate (11):
* (304) (IN), TLS handshake, CERT verify (15):
* (304) (IN), TLS handshake, Finished (20):
* (304) (OUT), TLS handshake, Finished (20):
* SSL connection using TLSv1.3 / AEAD-AES128-GCM-SHA256
* ALPN: server accepted h2
* Server certificate:
*  subject: CN=test2.ays-dev-tt.net
*  start date: Jul 24 06:51:58 2023 GMT
*  expire date: Oct 22 06:51:57 2023 GMT
*  subjectAltName: host "test2.ays-dev-tt.net" matched cert's "test2.ays-dev-tt.net"
*  issuer: C=US; O=Let's Encrypt; CN=R3
*  SSL certificate verify ok.
* using HTTP/2
* h2h3 [:method: GET]
* h2h3 [:path: /]
* h2h3 [:scheme: https]
* h2h3 [:authority: test2.ays-dev-tt.net]
* h2h3 [user-agent: curl/7.88.1]
* h2h3 [accept: */*]
* Using Stream ID: 1 (easy handle 0x7fe493011400)
> GET / HTTP/2
> Host: test2.ays-dev-tt.net
> user-agent: curl/7.88.1
> accept: */*
>
< HTTP/2 200
< accept-ranges: bytes
< alt-svc: h3=":443"; ma=2592000
< cache-control: public, max-age=0
< content-type: text/html; charset=UTF-8
< date: Mon, 24 Jul 2023 07:53:31 GMT
< etag: W/"5d14-18977f2e778"
< last-modified: Fri, 21 Jul 2023 10:15:55 GMT
< x-robots-tag: noindex, nofollow, noarchive, nosnippet
< content-length: 23828
<

francislavoie · July 24, 2023, 8:03am

Ah I see what’s going on.

You enabled on_demand for your *.wild.domains.net. This means Caddy will not attempt to fetch a wildcard cert at server-start because it was marked as On-Demand.

Your wildcard site blocks should not have on_demand, so that Caddy manages certs for those up-front.

Tomasz_Trebski · July 24, 2023, 8:04am

So I cannot mix the case where I both ensure that domain is known and wildcard cert is generated?
As in the it is implied if domain is known because of being part of wildcard?

@francislavoie ?

francislavoie · July 24, 2023, 8:17am

The way you do that is by not using on_demand.

on_demand should only be used for domains you don’t know ahead of time.

Tomasz_Trebski · July 24, 2023, 8:30am

Well, I kinda don’t.
We have this case we can have thousands of domains targetting the user apps. Those domains are always provided to customers by default. But such domain should only be available the moment such app is created in system.
Later if customer wants it can either request custom domain or tell us it has its own domain. Regardless such domain should CNAME the default domain that was provided.

So the wildcard certificate is to not have 4000 individual certs for those apps because this is the system default and one is quite enough. In the same time an app domain should not work if app is not created.

On the other hand, if that is the custom certificate, it is always available etc and something deeper in the system will simply respond an app is unknown. So I guess we’re good.

PS. now the setup works. At least wildcard case.
I will now deploy stuff to AWS to verify full coverage and get back to you if everything is fine.

PS2. @francislavoie @matt final question. Given external domains (autocert of external cert) and the fact that they need to be injected explicitly into Caddy can I still use Caddyfile and simply let rest of configuration go over the API or should I eject into JSON already and carry on with it?

francislavoie · July 24, 2023, 9:05am

You can’t mix Caddyfile and JSON. And if you use the API, then you’re making in-memory changes to config, and those changes are not reflected back to your Caddyfile, so if you restart the server from your Caddyfile, the changes from the API are lost to the wind.

So it’s up to you. If you want to primarily use the API, then do that. If you want to primarily use the Caddyfile, then you’ll need to do what you need to manipulate your Caddyfile as needed.

You can push an updated Caddyfile via the API (just set the Content-Type header), so you could maintain the Caddyfile in some database or w/e and use a mutex to update it as needed, and push the updated config to the server(s) to trigger a reload.

Tomasz_Trebski · July 24, 2023, 10:07am

I see that those are my options:

Store Caddy configuration in some database, use mutex / queue system to push changes to server
Use API only but that also means storing configuration elsewhere next to Caddy?

Tomasz_Trebski · July 24, 2023, 10:19am

Just to reiterate on that.
Is there really no option to enforce/limit wildcard case only to domains that are actually known i.e. to use on_demand or something similar?