Dynamic TLS selection

matt · July 20, 2023, 10:02pm

Caddy is designed to be reloaded frequently; and as Francis mentioned the reloads will become very lightweight in 2.7.

I’m not really sure I follow the problem here; is there a problem with simply running caddy reload or making an HTTP request to update its config?

Sure, but that sounds more complicated than necessary. (Just from the info provided so far.)

Francis is correct, but let me offer my take:

Put these domains in your Caddy config. Use the tls directive (if Caddyfile; or an automation policy if JSON) to customize how the certificates are obtained.

Use on-demand TLS for this.

Use a certificate manager module for this. Whether it’s an existing one (there is an “HTTP” one for general purpose use) or a custom one (maybe based on the HTTP module to start with?), this can get the certificate to use during handshake-time from an entity that is obtaining and keeping it renewed.

Your cert manager module can determine whether it should obtain a certificate or not; and if not, Caddy can fall back to its own on-demand TLS.

francislavoie · July 21, 2023, 2:12am

Does it already do that? That was my question above, I thought it wouldn’t fallback and just fail the handshake.

matt · July 21, 2023, 2:53am

I just checked the code, and it logs a debug message if there’s no certificate and no error:

github.com

caddyserver/certmagic/blob/51b31907aaa07f21e1bf2974193d11cdbf32802b/handshake.go#L767-L768


      
          		logger.Debug("all external certificate managers yielded no certificates and no errors", zap.String("sni", hello.ServerName))
          		return Certificate{}, nil

and then it returns an empty value and nil error, and then it will try on-demand issuance if enabled.

francislavoie · July 21, 2023, 3:00am

Oh okay so you could configure it like this?

https://
	tls {
		get_certificate http http://some-endpoint
		on_demand
	}
}

If that works, nice!

Then yeah @Tomasz_Trebski you should have everything you need.

Tomasz_Trebski · July 21, 2023, 5:53am

Thx @francislavoie @matt , that clears some doubts.

The problem is not either of those but the time which my team does not have in abundance. Naturally I would want something that just works but as you can imagine that’s rarely the case and as @francislavoie rightfully pointed out I might be hitting a niche here which is hardly a justification for you to implement that.

So your idea is to have them explicit (not wildcards but could also be done for them) in here? Have this sort of queue and use API to let Caddy know there is new domain as in (pseudocode):

POST
example.com {
  import route53_tls
}

where route53_tls does the on_demand double-check and ask for cert via dns-01?

On the other hand you would leave domains I cannot control but where certs are to be generated fallback to http-01 via

:443 {
  tls {
    on_demand
  }
}

and couple that with @francislavoie i.e. get_certificate http http://some-endpoint which actually already gives me use-case d) i.e. first Caddy asks my endpoint if there’s a certificate in store and my endpoint returns it but in case of 404 Caddy generates new certificate.

To wrap this up and circle back to original post and use cases:

a) … => explicit in Caddy config, this gonna work for me just fine
b) … => post them via caddy API
c) + d) use explicit code snippet from @francislavoie to first ask for cert then fallback to on_demand

Let me cook something up to see if we can make it happen.

Tomasz_Trebski · July 21, 2023, 11:39am

@francislavoie @matt

It is more or less working but what I think stopped working is wildcard case.
Here’s my config to dicuss:

{
  debug

  on_demand_tls {
    ask       {$DOMAINS_MGR}/check
    interval  1m
    burst     5
  }

  storage dynamodb {$CERTS_TABLE} {
    aws_region {$AWS_REGION}
  }
}

(logs) {
  log {
    output stderr
    format json
  }
}

(logic) {
  import logs

  bind {$ADDRESS}
  encode gzip

  header /* {
      -Server
  }

  reverse_proxy /* {
    to                                                   {$AGW_URL}
    header_up Host                           {http.reverse_proxy.upstream.host}
    header_up X-Real-IP                   {http.reverse-proxy.upstream.address}
    header_up X-Forwarded-Proto {scheme}
  }
}

(http_01_tls) {
  import logic
  tls {
    get_certificate http {$DOMAINS_MGR}/certificate
    on_demand
  }
}

(route53_tls) {
  import logic
  tls {
    dns route53
    on_demand
  }
}

:8888 {
  import logs
  respond / 204
}

# wildcard, for default case
*.wild.domains.net:443 {
  import route53_tls
}

# domains from customers, with out without cert
# cdoc - CustomerDomainOwnCert
# cdcd - CustomerDomainCustomerCert
cdoc-0.example.net,
cdoc-1.example.net,
cdcc-0.example.net,
cdcc-1.example.net {
  import http_01_tls
}

# rest of domains, assumed to be owned
# those are not wildcard domains i.e.
# something like super.com or another.net
:443 {
  import route53_tls
}

All the time I try to get through to the page for which wildcard certificate ought to be used I keep getting a cert that belongs to the very particular domain.
AFAIR, Caddy docs mention, that precedence is given to the best matching block which in case of my configuration means:

check block for domains that fully qualified (cases c) and d)
check the block for wildcards
fallback to :443

Am I missing something? I am pretty sure I got this working before but it got lost once I added the code to support other cases. Naughty ( ) of me to not store wildcard configuration in git somewhere, I should have done that.

log example that shows that specific cert is being generated:

2023-07-21T12:01:07.834000+00:00 ui/ui/1ae09fcf5804429ea7751661f9a873b0 {"level":"info","ts":1689940867.834722,"logger":"tls.obtain","msg":"obtaining certificate","identifier":"wild-8.wild.domains.net"}

Tomasz_Trebski · July 24, 2023, 7:02am

@matt @francislavoie would you mind helping out just a tiny bit ?:>

I have seen this thread: Mixing wildcard certificate with on_demand feature - #4 by pdnz which is exactly what I am trying to do but I already have the route53 in place.

Also in logs I can see entries like:

{"level":"info","ts":1690183666.7853959,"logger":"http.acme_client","msg":"trying to solve challenge","identifier":"test.ays-dev-tt.net","challenge_type":"dns-01","ca":"https://acme-v02.api.letsencrypt.org/directory"}

francislavoie · July 24, 2023, 7:42am

What are your full debug logs from when the first request came in for that domain?

Tomasz_Trebski · July 24, 2023, 7:52am

addywildcarddomains-caddy-1  | {"level":"info","ts":1690185077.961121,"msg":"using provided configuration","config_file":"/etc/caddy/Caddyfile","config_adapter":"caddyfile"}
caddywildcarddomains-caddy-1  | {"level":"warn","ts":1690185077.9626849,"logger":"caddyfile","msg":"Unnecessary header_up X-Forwarded-Host: the reverse proxy's default behavior is to pass headers to the upstream"}
caddywildcarddomains-caddy-1  | {"level":"warn","ts":1690185077.9627624,"logger":"caddyfile","msg":"Unnecessary header_up X-Forwarded-Proto: the reverse proxy's default behavior is to pass headers to the upstream"}
caddywildcarddomains-caddy-1  | {"level":"warn","ts":1690185077.9671943,"msg":"Caddyfile input is not formatted; run the 'caddy fmt' command to fix inconsistencies","adapter":"caddyfile","file":"/etc/caddy/Caddyfile","line":2}
caddywildcarddomains-caddy-1  | {"level":"info","ts":1690185077.9700787,"logger":"admin","msg":"admin endpoint started","address":"localhost:2019","enforce_origin":false,"origins":["//127.0.0.1:2019","//localhost:2019","//[::1]:2019"]}
caddywildcarddomains-caddy-1  | {"level":"info","ts":1690185077.9706311,"logger":"http","msg":"server is listening only on the HTTPS port but has no TLS connection policies; adding one to enable TLS","server_name":"srv0","https_port":443}
caddywildcarddomains-caddy-1  | {"level":"info","ts":1690185077.9711363,"logger":"http","msg":"enabling automatic HTTP->HTTPS redirects","server_name":"srv0"}
caddywildcarddomains-caddy-1  | {"level":"info","ts":1690185077.9715438,"logger":"tls.cache.maintenance","msg":"started background certificate maintenance","cache":"0xc00029a0e0"}
caddywildcarddomains-caddy-1  | {"level":"info","ts":1690185077.9786193,"logger":"http","msg":"enabling HTTP/3 listener","addr":":443"}
caddywildcarddomains-caddy-1  | {"level":"info","ts":1690185077.978765,"msg":"failed to sufficiently increase receive buffer size (was: 208 kiB, wanted: 2048 kiB, got: 416 kiB). See https://github.com/quic-go/quic-go/wiki/UDP-Receive-Buffer-Size for details."}
caddywildcarddomains-caddy-1  | {"level":"debug","ts":1690185077.9788616,"logger":"http","msg":"starting server loop","address":"[::]:443","tls":true,"http3":true}
caddywildcarddomains-caddy-1  | {"level":"info","ts":1690185077.9788744,"logger":"http.log","msg":"server running","name":"srv0","protocols":["h1","h2","h3"]}
caddywildcarddomains-caddy-1  | {"level":"debug","ts":1690185077.9801319,"logger":"http","msg":"starting server loop","address":"[::]:8888","tls":false,"http3":false}
caddywildcarddomains-caddy-1  | {"level":"info","ts":1690185077.9802086,"logger":"http.log","msg":"server running","name":"srv1","protocols":["h1","h2","h3"]}
caddywildcarddomains-caddy-1  | {"level":"debug","ts":1690185077.9803035,"logger":"http","msg":"starting server loop","address":"[::]:80","tls":false,"http3":false}
caddywildcarddomains-caddy-1  | {"level":"info","ts":1690185077.9803162,"logger":"http.log","msg":"server running","name":"remaining_auto_https_redirects","protocols":["h1","h2","h3"]}
caddywildcarddomains-caddy-1  | {"level":"info","ts":1690185077.9803226,"logger":"http","msg":"enabling automatic TLS certificate management","domains":["*.ays-dev-tt.net","ays-dev-tt.net"]}
caddywildcarddomains-caddy-1  | {"level":"info","ts":1690185077.9816577,"logger":"tls","msg":"cleaning storage unit","description":"FileStorage:/data/caddy"}
caddywildcarddomains-caddy-1  | {"level":"info","ts":1690185077.9829724,"msg":"autosaved config (load with --resume flag)","file":"/config/caddy/autosave.json"}
caddywildcarddomains-caddy-1  | {"level":"info","ts":1690185077.9830227,"msg":"serving initial configuration"}
caddywildcarddomains-caddy-1  | {"level":"info","ts":1690185077.9842074,"logger":"tls","msg":"finished cleaning storage units"}
caddywildcarddomains-caddy-1  | {"level":"debug","ts":1690185096.7524934,"logger":"events","msg":"event","name":"tls_get_certificate","id":"acec861b-40e2-48cb-88ad-2aa60cb7b7b4","origin":"tls","data":{"client_hello":{"CipherSuites":[4866,4867,4865,49200,49196,49192,49188,49172,49162,159,107,57,52393,52392,52394,65413,196,136,129,157,61,53,192,132,49199,49195,49191,49187,49171,49161,158,103,51,190,69,156,60,47,186,65,49169,49159,5,4,49170,49160,22,10,255],"ServerName":"test2.ays-dev-tt.net","SupportedCurves":[29,23,24,25],"SupportedPoints":"AA==","SignatureSchemes":[2054,1537,1539,2053,1281,1283,2052,1025,1027,513,515],"SupportedProtos":["h2","http/1.1"],"SupportedVersions":[772,771,770,769],"Conn":{}}}}
caddywildcarddomains-caddy-1  | {"level":"debug","ts":1690185096.752702,"logger":"tls.handshake","msg":"no matching certificates and no custom selection logic","identifier":"test2.ays-dev-tt.net"}
caddywildcarddomains-caddy-1  | {"level":"debug","ts":1690185096.7527103,"logger":"tls.handshake","msg":"no matching certificates and no custom selection logic","identifier":"*.ays-dev-tt.net"}
caddywildcarddomains-caddy-1  | {"level":"debug","ts":1690185096.7527137,"logger":"tls.handshake","msg":"no matching certificates and no custom selection logic","identifier":"*.*.net"}
caddywildcarddomains-caddy-1  | {"level":"debug","ts":1690185096.7527168,"logger":"tls.handshake","msg":"no matching certificates and no custom selection logic","identifier":"*.*.*"}
caddywildcarddomains-caddy-1  | {"level":"debug","ts":1690185096.752722,"logger":"tls.handshake","msg":"all external certificate managers yielded no certificates and no errors","remote_ip":"172.19.0.1","remote_port":"56270","sni":"test2.ays-dev-tt.net"}
caddywildcarddomains-caddy-1  | {"level":"debug","ts":1690185096.7624805,"logger":"tls","msg":"response from ask endpoint","domain":"test2.ays-dev-tt.net","url":"http://domains:5000/check?domain=test2.ays-dev-tt.net","status":200}
caddywildcarddomains-caddy-1  | {"level":"info","ts":1690185096.7625916,"logger":"tls.on_demand","msg":"obtaining new certificate","remote_ip":"172.19.0.1","remote_port":"56270","server_name":"test2.ays-dev-tt.net"}
caddywildcarddomains-caddy-1  | {"level":"info","ts":1690185096.766266,"logger":"tls.obtain","msg":"acquiring lock","identifier":"test2.ays-dev-tt.net"}
caddywildcarddomains-caddy-1  | {"level":"info","ts":1690185096.7765398,"logger":"tls.obtain","msg":"lock acquired","identifier":"test2.ays-dev-tt.net"}
caddywildcarddomains-caddy-1  | {"level":"info","ts":1690185096.7769156,"logger":"tls.obtain","msg":"obtaining certificate","identifier":"test2.ays-dev-tt.net"}
caddywildcarddomains-caddy-1  | {"level":"debug","ts":1690185096.7770662,"logger":"events","msg":"event","name":"cert_obtaining","id":"d76628a1-0ed9-4fd2-8507-275c2833545d","origin":"tls","data":{"identifier":"test2.ays-dev-tt.net"}}
caddywildcarddomains-caddy-1  | {"level":"debug","ts":1690185096.7774956,"logger":"tls.obtain","msg":"trying issuer 1/2","issuer":"acme-v02.api.letsencrypt.org-directory"}
caddywildcarddomains-caddy-1  | {"level":"debug","ts":1690185097.7298977,"logger":"http.acme_client","msg":"http request","method":"GET","url":"https://acme-v02.api.letsencrypt.org/directory","headers":{"User-Agent":["Caddy/2.6.4 CertMagic acmez (linux; amd64)"]},"response_headers":{"Cache-Control":["public, max-age=0, no-cache"],"Content-Length":["752"],"Content-Type":["application/json"],"Date":["Mon, 24 Jul 2023 07:51:37 GMT"],"Server":["nginx"],"Strict-Transport-Security":["max-age=604800"],"X-Frame-Options":["DENY"]},"status_code":200}
caddywildcarddomains-caddy-1  | {"level":"debug","ts":1690185097.9304242,"logger":"http.acme_client","msg":"http request","method":"HEAD","url":"https://acme-v02.api.letsencrypt.org/acme/new-nonce","headers":{"User-Agent":["Caddy/2.6.4 CertMagic acmez (linux; amd64)"]},"response_headers":{"Cache-Control":["public, max-age=0, no-cache"],"Date":["Mon, 24 Jul 2023 07:51:38 GMT"],"Link":["<https://acme-v02.api.letsencrypt.org/directory>;rel=\"index\""],"Replay-Nonce":["853FbhLwrQtxN_nvAiIHs7MJl_uEq9wc1j4FFJI_nbwATbc"],"Server":["nginx"],"Strict-Transport-Security":["max-age=604800"],"X-Frame-Options":["DENY"]},"status_code":200}
caddywildcarddomains-caddy-1  | {"level":"debug","ts":1690185098.1485722,"logger":"http.acme_client","msg":"http request","method":"POST","url":"https://acme-v02.api.letsencrypt.org/acme/new-acct","headers":{"Content-Type":["application/jose+json"],"User-Agent":["Caddy/2.6.4 CertMagic acmez (linux; amd64)"]},"response_headers":{"Boulder-Requester":["1222163617"],"Cache-Control":["public, max-age=0, no-cache"],"Content-Length":["332"],"Content-Type":["application/json"],"Date":["Mon, 24 Jul 2023 07:51:38 GMT"],"Link":["<https://acme-v02.api.letsencrypt.org/directory>;rel=\"index\"","<https://letsencrypt.org/documents/LE-SA-v1.3-September-21-2022.pdf>;rel=\"terms-of-service\""],"Location":["https://acme-v02.api.letsencrypt.org/acme/acct/1222163617"],"Replay-Nonce":["27126TZcJd0xMkdFwtfPgcjLyeTmlbbs99drMCgZHkIkK8M"],"Server":["nginx"],"Strict-Transport-Security":["max-age=604800"],"X-Frame-Options":["DENY"]},"status_code":201}
caddywildcarddomains-caddy-1  | {"level":"info","ts":1690185098.1603208,"logger":"http","msg":"waiting on internal rate limiter","identifiers":["test2.ays-dev-tt.net"],"ca":"https://acme-v02.api.letsencrypt.org/directory","account":"tomasz.trebski@appyourself.net"}
caddywildcarddomains-caddy-1  | {"level":"info","ts":1690185098.1605787,"logger":"http","msg":"done waiting on internal rate limiter","identifiers":["test2.ays-dev-tt.net"],"ca":"https://acme-v02.api.letsencrypt.org/directory","account":"tomasz.trebski@appyourself.net"}
caddywildcarddomains-caddy-1  | {"level":"debug","ts":1690185098.4053288,"logger":"http.acme_client","msg":"http request","method":"POST","url":"https://acme-v02.api.letsencrypt.org/acme/new-order","headers":{"Content-Type":["application/jose+json"],"User-Agent":["Caddy/2.6.4 CertMagic acmez (linux; amd64)"]},"response_headers":{"Boulder-Requester":["1222163617"],"Cache-Control":["public, max-age=0, no-cache"],"Content-Length":["346"],"Content-Type":["application/json"],"Date":["Mon, 24 Jul 2023 07:51:38 GMT"],"Link":["<https://acme-v02.api.letsencrypt.org/directory>;rel=\"index\""],"Location":["https://acme-v02.api.letsencrypt.org/acme/order/1222163617/196962663357"],"Replay-Nonce":["2712HVWlK6gcHx0mjfzrQh2z7W-bVjITx2cAqjj6_UZHhUE"],"Server":["nginx"],"Strict-Transport-Security":["max-age=604800"],"X-Frame-Options":["DENY"]},"status_code":201}
caddywildcarddomains-caddy-1  | {"level":"debug","ts":1690185098.6101499,"logger":"http.acme_client","msg":"http request","method":"POST","url":"https://acme-v02.api.letsencrypt.org/acme/authz-v3/248489224017","headers":{"Content-Type":["application/jose+json"],"User-Agent":["Caddy/2.6.4 CertMagic acmez (linux; amd64)"]},"response_headers":{"Boulder-Requester":["1222163617"],"Cache-Control":["public, max-age=0, no-cache"],"Content-Length":["804"],"Content-Type":["application/json"],"Date":["Mon, 24 Jul 2023 07:51:38 GMT"],"Link":["<https://acme-v02.api.letsencrypt.org/directory>;rel=\"index\""],"Replay-Nonce":["853FE9LOaPvMOUOPR3hPFQFu9HOSkPmt--bg_1Rsc4TL5JU"],"Server":["nginx"],"Strict-Transport-Security":["max-age=604800"],"X-Frame-Options":["DENY"]},"status_code":200}
caddywildcarddomains-caddy-1  | {"level":"debug","ts":1690185098.6107538,"logger":"http.acme_client","msg":"no solver configured","challenge_type":"tls-alpn-01"}
caddywildcarddomains-caddy-1  | {"level":"info","ts":1690185098.610776,"logger":"http.acme_client","msg":"trying to solve challenge","identifier":"test2.ays-dev-tt.net","challenge_type":"dns-01","ca":"https://acme-v02.api.letsencrypt.org/directory"}
caddywildcarddomains-caddy-1  | {"level":"debug","ts":1690185100.393418,"logger":"http.acme_client","msg":"waiting for solver before continuing","identifier":"test2.ays-dev-tt.net","challenge_type":"dns-01"}
caddywildcarddomains-caddy-1  | {"level":"debug","ts":1690185116.113534,"logger":"http.acme_client","msg":"done waiting for solver","identifier":"test2.ays-dev-tt.net","challenge_type":"dns-01"}
caddywildcarddomains-caddy-1  | {"level":"debug","ts":1690185116.3140285,"logger":"http.acme_client","msg":"http request","method":"POST","url":"https://acme-v02.api.letsencrypt.org/acme/chall-v3/248489224017/ze5YJg","headers":{"Content-Type":["application/jose+json"],"User-Agent":["Caddy/2.6.4 CertMagic acmez (linux; amd64)"]},"response_headers":{"Boulder-Requester":["1222163617"],"Cache-Control":["public, max-age=0, no-cache"],"Content-Length":["186"],"Content-Type":["application/json"],"Date":["Mon, 24 Jul 2023 07:51:56 GMT"],"Link":["<https://acme-v02.api.letsencrypt.org/directory>;rel=\"index\"","<https://acme-v02.api.letsencrypt.org/acme/authz-v3/248489224017>;rel=\"up\""],"Location":["https://acme-v02.api.letsencrypt.org/acme/chall-v3/248489224017/ze5YJg"],"Replay-Nonce":["2712_6H5qNJEyKfhCW-sfzlqlRV0hnQQCTNoTG8RiihaNU4"],"Server":["nginx"],"Strict-Transport-Security":["max-age=604800"],"X-Frame-Options":["DENY"]},"status_code":200}
caddywildcarddomains-caddy-1  | {"level":"debug","ts":1690185116.3141494,"logger":"http.acme_client","msg":"challenge accepted","identifier":"test2.ays-dev-tt.net","challenge_type":"dns-01"}
caddywildcarddomains-caddy-1  | {"level":"debug","ts":1690185116.7788312,"logger":"http.acme_client","msg":"http request","method":"POST","url":"https://acme-v02.api.letsencrypt.org/acme/authz-v3/248489224017","headers":{"Content-Type":["application/jose+json"],"User-Agent":["Caddy/2.6.4 CertMagic acmez (linux; amd64)"]},"response_headers":{"Boulder-Requester":["1222163617"],"Cache-Control":["public, max-age=0, no-cache"],"Content-Length":["515"],"Content-Type":["application/json"],"Date":["Mon, 24 Jul 2023 07:51:56 GMT"],"Link":["<https://acme-v02.api.letsencrypt.org/directory>;rel=\"index\""],"Replay-Nonce":["2712PYDO7sQZL_dGzGhsxKfGnNcpVUS_1wLnJ41VAbRWoC8"],"Server":["nginx"],"Strict-Transport-Security":["max-age=604800"],"X-Frame-Options":["DENY"]},"status_code":200}
caddywildcarddomains-caddy-1  | {"level":"info","ts":1690185118.1503334,"logger":"http.acme_client","msg":"authorization finalized","identifier":"test2.ays-dev-tt.net","authz_status":"valid"}
caddywildcarddomains-caddy-1  | {"level":"info","ts":1690185118.150518,"logger":"http.acme_client","msg":"validations succeeded; finalizing order","order":"https://acme-v02.api.letsencrypt.org/acme/order/1222163617/196962663357"}
caddywildcarddomains-caddy-1  | {"level":"debug","ts":1690185118.5872414,"logger":"http.acme_client","msg":"http request","method":"POST","url":"https://acme-v02.api.letsencrypt.org/acme/finalize/1222163617/196962663357","headers":{"Content-Type":["application/jose+json"],"User-Agent":["Caddy/2.6.4 CertMagic acmez (linux; amd64)"]},"response_headers":{"Boulder-Requester":["1222163617"],"Cache-Control":["public, max-age=0, no-cache"],"Content-Length":["448"],"Content-Type":["application/json"],"Date":["Mon, 24 Jul 2023 07:51:58 GMT"],"Link":["<https://acme-v02.api.letsencrypt.org/directory>;rel=\"index\""],"Location":["https://acme-v02.api.letsencrypt.org/acme/order/1222163617/196962663357"],"Replay-Nonce":["853F7yYPhUOEflYfxSehPRM4N-ZCrvHkOWzgk6_b5JNm1YY"],"Server":["nginx"],"Strict-Transport-Security":["max-age=604800"],"X-Frame-Options":["DENY"]},"status_code":200}
caddywildcarddomains-caddy-1  | {"level":"debug","ts":1690185118.8078496,"logger":"http.acme_client","msg":"http request","method":"POST","url":"https://acme-v02.api.letsencrypt.org/acme/cert/0371e10bab4b599d8de74fec03fbb775b16a","headers":{"Content-Type":["application/jose+json"],"User-Agent":["Caddy/2.6.4 CertMagic acmez (linux; amd64)"]},"response_headers":{"Cache-Control":["public, max-age=0, no-cache"],"Content-Length":["5256"],"Content-Type":["application/pem-certificate-chain"],"Date":["Mon, 24 Jul 2023 07:51:58 GMT"],"Link":["<https://acme-v02.api.letsencrypt.org/directory>;rel=\"index\"","<https://acme-v02.api.letsencrypt.org/acme/cert/0371e10bab4b599d8de74fec03fbb775b16a/1>;rel=\"alternate\""],"Replay-Nonce":["853FdJL3kD0XLjOrHtjXZAEDV8AFiNgLHDdrBLLyXi1tJ0s"],"Server":["nginx"],"Strict-Transport-Security":["max-age=604800"],"X-Frame-Options":["DENY"]},"status_code":200}
caddywildcarddomains-caddy-1  | {"level":"debug","ts":1690185119.0296867,"logger":"http.acme_client","msg":"http request","method":"POST","url":"https://acme-v02.api.letsencrypt.org/acme/cert/0371e10bab4b599d8de74fec03fbb775b16a/1","headers":{"Content-Type":["application/jose+json"],"User-Agent":["Caddy/2.6.4 CertMagic acmez (linux; amd64)"]},"response_headers":{"Cache-Control":["public, max-age=0, no-cache"],"Content-Length":["3332"],"Content-Type":["application/pem-certificate-chain"],"Date":["Mon, 24 Jul 2023 07:51:59 GMT"],"Link":["<https://acme-v02.api.letsencrypt.org/directory>;rel=\"index\"","<https://acme-v02.api.letsencrypt.org/acme/cert/0371e10bab4b599d8de74fec03fbb775b16a/0>;rel=\"alternate\""],"Replay-Nonce":["853FNSiIYCmlvreUHkrGtw9MREJI_W7Vzq9P8VJha2C5UCk"],"Server":["nginx"],"Strict-Transport-Security":["max-age=604800"],"X-Frame-Options":["DENY"]},"status_code":200}
caddywildcarddomains-caddy-1  | {"level":"info","ts":1690185119.0298977,"logger":"http.acme_client","msg":"successfully downloaded available certificate chains","count":2,"first_url":"https://acme-v02.api.letsencrypt.org/acme/cert/0371e10bab4b599d8de74fec03fbb775b16a"}
caddywildcarddomains-caddy-1  | {"level":"info","ts":1690185119.0324125,"logger":"tls.obtain","msg":"certificate obtained successfully","identifier":"test2.ays-dev-tt.net"}
caddywildcarddomains-caddy-1  | {"level":"debug","ts":1690185119.0325215,"logger":"events","msg":"event","name":"cert_obtained","id":"63efe594-4465-4a26-8e75-96b32616d73a","origin":"tls","data":{"identifier":"test2.ays-dev-tt.net","issuers":"acme-v02.api.letsencrypt.org-directory","renewal":false,"storage_key":"test2.ays-dev-tt.net"}}
caddywildcarddomains-caddy-1  | {"level":"info","ts":1690185119.0325613,"logger":"tls.obtain","msg":"releasing lock","identifier":"test2.ays-dev-tt.net"}
caddywildcarddomains-caddy-1  | {"level":"debug","ts":1690185119.0327036,"logger":"tls.handshake","msg":"no matching certificates and no custom selection logic","identifier":"test2.ays-dev-tt.net"}
caddywildcarddomains-caddy-1  | {"level":"debug","ts":1690185119.032719,"logger":"tls.handshake","msg":"no matching certificates and no custom selection logic","identifier":"*.ays-dev-tt.net"}
caddywildcarddomains-caddy-1  | {"level":"debug","ts":1690185119.0327244,"logger":"tls.handshake","msg":"no matching certificates and no custom selection logic","identifier":"*.*.net"}
caddywildcarddomains-caddy-1  | {"level":"debug","ts":1690185119.0327284,"logger":"tls.handshake","msg":"no matching certificates and no custom selection logic","identifier":"*.*.*"}
caddywildcarddomains-caddy-1  | {"level":"debug","ts":1690185119.0327346,"logger":"tls.handshake","msg":"all external certificate managers yielded no certificates and no errors","remote_ip":"172.19.0.1","remote_port":"56270","sni":"test2.ays-dev-tt.net"}
caddywildcarddomains-caddy-1  | {"level":"debug","ts":1690185119.0333095,"logger":"tls","msg":"loading managed certificate","domain":"test2.ays-dev-tt.net","expiration":1697957518,"issuer_key":"acme-v02.api.letsencrypt.org-directory","storage":"FileStorage:/data/caddy"}
caddywildcarddomains-caddy-1  | {"level":"debug","ts":1690185119.4500487,"logger":"tls.cache","msg":"added certificate to cache","subjects":["test2.ays-dev-tt.net"],"expiration":1697957518,"managed":true,"issuer_key":"acme-v02.api.letsencrypt.org-directory","hash":"ab25fa03417a4d247e8150ebe102b4eaf07bca6f71722e011a0fc4036ed9037a","cache_size":1,"cache_capacity":10000}
caddywildcarddomains-caddy-1  | {"level":"debug","ts":1690185119.4502397,"logger":"events","msg":"event","name":"cached_managed_cert","id":"51a2523a-693e-46f9-818b-74ed59d95c2d","origin":"tls","data":{"sans":["test2.ays-dev-tt.net"]}}
caddywildcarddomains-caddy-1  | {"level":"debug","ts":1690185119.4502912,"logger":"tls.handshake","msg":"loaded certificate from storage","remote_ip":"172.19.0.1","remote_port":"56270","subjects":["test2.ays-dev-tt.net"],"managed":true,"expiration":1697957518,"hash":"ab25fa03417a4d247e8150ebe102b4eaf07bca6f71722e011a0fc4036ed9037a"}
caddywildcarddomains-caddy-1  | {"level":"debug","ts":1690185119.459658,"logger":"http.handlers.reverse_proxy","msg":"selected upstream","dial":"my.appit.website:443","total_upstreams":1}
caddywildcarddomains-caddy-1  | {"level":"debug","ts":1690185119.9077697,"logger":"http.handlers.reverse_proxy","msg":"upstream roundtrip","upstream":"my.appit.website:443","duration":0.447842226,"request":{"remote_ip":"172.19.0.1","remote_port":"56270","proto":"HTTP/2.0","method":"GET","host":"my.appit.website","uri":"/","headers":{"Accept":["*/*"],"X-Forwarded-For":["172.19.0.1"],"X-Forwarded-Proto":["https"],"X-Forwarded-Host":["test2.ays-dev-tt.net"],"X-Forwarded-Port":[""],"X-Real-Ip":["{http.reverse-proxy.upstream.address}"],"User-Agent":["curl/7.88.1"]},"tls":{"resumed":false,"version":772,"cipher_suite":4865,"proto":"h2","server_name":"test2.ays-dev-tt.net"}},"headers":{"Date":["Mon, 24 Jul 2023 07:52:00 GMT"],"Server":["Apache/2.4.34 (Red Hat) OpenSSL/1.0.2k-fips"],"Cache-Control":["public, max-age=0"],"Content-Length":["23828"],"X-Robots-Tag":["noindex, nofollow, noarchive, nosnippet"],"Accept-Ranges":["bytes"],"Last-Modified":["Fri, 21 Jul 2023 10:15:55 GMT"],"Etag":["W/\"5d14-18977f2e778\""],"Content-Type":["text/html; charset=UTF-8"]},"status":200}

that’s the complete output

Tomasz_Trebski · July 24, 2023, 7:54am

And here’s the curl

curl -v https://test2.ays-dev-tt.net
*   Trying 127.0.0.1:443...
* Connected to test2.ays-dev-tt.net (127.0.0.1) port 443 (#0)
* ALPN: offers h2,http/1.1
* (304) (OUT), TLS handshake, Client hello (1):
*  CAfile: /etc/ssl/cert.pem
*  CApath: none
* (304) (IN), TLS handshake, Server hello (2):
* (304) (IN), TLS handshake, Unknown (8):
* (304) (IN), TLS handshake, Certificate (11):
* (304) (IN), TLS handshake, CERT verify (15):
* (304) (IN), TLS handshake, Finished (20):
* (304) (OUT), TLS handshake, Finished (20):
* SSL connection using TLSv1.3 / AEAD-AES128-GCM-SHA256
* ALPN: server accepted h2
* Server certificate:
*  subject: CN=test2.ays-dev-tt.net
*  start date: Jul 24 06:51:58 2023 GMT
*  expire date: Oct 22 06:51:57 2023 GMT
*  subjectAltName: host "test2.ays-dev-tt.net" matched cert's "test2.ays-dev-tt.net"
*  issuer: C=US; O=Let's Encrypt; CN=R3
*  SSL certificate verify ok.
* using HTTP/2
* h2h3 [:method: GET]
* h2h3 [:path: /]
* h2h3 [:scheme: https]
* h2h3 [:authority: test2.ays-dev-tt.net]
* h2h3 [user-agent: curl/7.88.1]
* h2h3 [accept: */*]
* Using Stream ID: 1 (easy handle 0x7fe493011400)
> GET / HTTP/2
> Host: test2.ays-dev-tt.net
> user-agent: curl/7.88.1
> accept: */*
>
< HTTP/2 200
< accept-ranges: bytes
< alt-svc: h3=":443"; ma=2592000
< cache-control: public, max-age=0
< content-type: text/html; charset=UTF-8
< date: Mon, 24 Jul 2023 07:53:31 GMT
< etag: W/"5d14-18977f2e778"
< last-modified: Fri, 21 Jul 2023 10:15:55 GMT
< x-robots-tag: noindex, nofollow, noarchive, nosnippet
< content-length: 23828
<

francislavoie · July 24, 2023, 8:03am

Ah I see what’s going on.

You enabled on_demand for your *.wild.domains.net. This means Caddy will not attempt to fetch a wildcard cert at server-start because it was marked as On-Demand.

Your wildcard site blocks should not have on_demand, so that Caddy manages certs for those up-front.

Tomasz_Trebski · July 24, 2023, 8:04am

So I cannot mix the case where I both ensure that domain is known and wildcard cert is generated?
As in the it is implied if domain is known because of being part of wildcard?

@francislavoie ?

francislavoie · July 24, 2023, 8:17am

The way you do that is by not using on_demand.

on_demand should only be used for domains you don’t know ahead of time.

Tomasz_Trebski · July 24, 2023, 8:30am

Well, I kinda don’t.
We have this case we can have thousands of domains targetting the user apps. Those domains are always provided to customers by default. But such domain should only be available the moment such app is created in system.
Later if customer wants it can either request custom domain or tell us it has its own domain. Regardless such domain should CNAME the default domain that was provided.

So the wildcard certificate is to not have 4000 individual certs for those apps because this is the system default and one is quite enough. In the same time an app domain should not work if app is not created.

On the other hand, if that is the custom certificate, it is always available etc and something deeper in the system will simply respond an app is unknown. So I guess we’re good.

PS. now the setup works. At least wildcard case.
I will now deploy stuff to AWS to verify full coverage and get back to you if everything is fine.

PS2. @francislavoie @matt final question. Given external domains (autocert of external cert) and the fact that they need to be injected explicitly into Caddy can I still use Caddyfile and simply let rest of configuration go over the API or should I eject into JSON already and carry on with it?

francislavoie · July 24, 2023, 9:05am

You can’t mix Caddyfile and JSON. And if you use the API, then you’re making in-memory changes to config, and those changes are not reflected back to your Caddyfile, so if you restart the server from your Caddyfile, the changes from the API are lost to the wind.

So it’s up to you. If you want to primarily use the API, then do that. If you want to primarily use the Caddyfile, then you’ll need to do what you need to manipulate your Caddyfile as needed.

You can push an updated Caddyfile via the API (just set the Content-Type header), so you could maintain the Caddyfile in some database or w/e and use a mutex to update it as needed, and push the updated config to the server(s) to trigger a reload.

Tomasz_Trebski · July 24, 2023, 10:07am

I see that those are my options:

Store Caddy configuration in some database, use mutex / queue system to push changes to server
Use API only but that also means storing configuration elsewhere next to Caddy?

Tomasz_Trebski · July 24, 2023, 10:19am

Just to reiterate on that.
Is there really no option to enforce/limit wildcard case only to domains that are actually known i.e. to use on_demand or something similar?

francislavoie · July 24, 2023, 10:28am

Yeah, by not setting on_demand for site blocks with a wildcard.

Tomasz_Trebski · July 24, 2023, 11:16am

Ok, I think that solution works quite nice.
Everything that’s left for me is figure out complete process to automate handling the domains that are external i.e. how to represent and update the configuration and persist it. Those are, however, questions to be answered elsewhere.

Thx a lot for help.

system · August 23, 2023, 11:17am

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.