I’ve learned that AWS ALB is not passing the domain being used via ClientHello and this was causing Caddy to fail serving a proper certificate. What I’m trying to figure out is the best way to just utilize self signed certificates. My thought is this:
This is setup appears to be working fine; but I don’t know if this would be considered the best practice for sitting behind a load balancer that does not pass the domain being used during the TLS negotiation.
I know that the self signed certificates are only good for 7 days; but I didn’t see any mention that new ones are auto reissued. Does anyone know if this happens or should I transition to just using my own self signed cert?
Seconding what Francis said, use Caddy 2. Caddy 1’s tls self_signed was never meant for production use (hence the lifetime of 7 days) – just development.
In Caddy 2, its “self-signed” certificates are fully managed (renewed) with short lifetimes, with proper PKI, and can be trusted easily on the local machine.
Generally, using “self-signed” (or more accurately, locally-signed) certificates in production is a bad idea. Why do you need self-signed certs?
If you need to set up a server that doesn’t support SNI, you have to choose a certificate to serve in that case, if you don’t have one for the server’s IP address. Caddy’s default_sni option can help with that.
To give a bit more detail in our setup. We have over 20 AWS Organizations. In each of those organizations there are between 2 and 15 web servers which all run multiple tomcat instances. Normally we run between 3 to 5 tomcat instances per server but it can be upwards to 10 during peak seasons. We have things pretty well automated to where we can say “this customer runs on these set of servers” and the AWS/ALB auto adds or removes instances to the target groups and the web servers will auto bring online or take offline the tomcat instances and IIS sites. The domains for each “customer” is also unique.
I really want to replace IIS and Caddy I feel is the right fit and it should help strip out some complexity. Since the AWS/ALB doesn’t pass the domain name in the ClientHello during TLS negotiation we would never end up using the issued Let’s Encrypt certificates so I kind of feel as though this should be turned off since I’m not actually using the service. I believe the other issue I’m going to run into is that when I have 2 or more servers running and Caddy goes out to get a certificate; it will probably fail because the incoming traffic could end up going to a different server. HTTP validation also only seems to work but that is with a single server in the mix.
I “think” the way to make Caddy work behind the AWS/ALB is to have a single global certificate and have all sites use that. This is basically how our IIS is setup. It would be nice to not use self-signed but its feeling like the only option I have based on how we have things setup.
How do you know which certificate to use without SNI? Is there only one per backend, then?
Apparently the AWS/ALB is dumb and Caddy will never know which certificate to use since the AWS/ALB isn’t passing the domain name in the ClientHello during TLS negotiation. I’ve yet to find an answer to this on the Amazon side.
What is “this” that should be turned off, exactly?
“this” is the Let’s Encrypt feature Caddy provides.
You know that Caddy can solve ACME challenges in a fleet, right?
Sure didn’t and that’s pretty awesome! I’m exactly 2 days into learning caddy so I got major gaps in my knowledge =)
Gotcha, thanks for clarifying. Yeah, if you need to choose from multiple certs on the backend, then you’ll either need SNI or set a default SNI value that is equal to that customer’s domain (assuming one domain per backend) – if so, then you can still use the managed LE certs.