I’m learning to Anycast using BGP and my own public ASN and PA IPv6 space. I’ve got caddy setup on one machine and have read a few posts here and on Github on multi hosting the same site with TLS.
The sites have the same content in different locations, but the same IP address announced by my ASN in that region.
What’s the right way to serve the same site on two/multiple hosts with the same domain name with TLS?
How do I get each Caddy install to return a custom tag (PoP location) so I can differentiate between nodes in Catchpoint?
That might get iffy if you have a large number of POPs, in terms of LetsEncrypt’s rate limit. There’s some early stage TLS storage plugins floating around these forums, though, I think…
If you want to do it by header, you could set it manually in each POP’s Caddyfile:
Interesting question. Do you have any specific requirements for how this information must be transmitted? Response headers are pretty much the gold standard for this kind of information.
Theoretically, you could have Caddy insert this information directly into a HTML response. Using filter, you could, as one example, find the <head> tag and insert a <meta> tag with the POP-specific data immediately after. Meta tags are usually document-specific information, rather than server-specific, though - but you don’t have to use them, you could put that info anywhere you like.
This method would introduce a non-insignificant processing burden on all requests, though - if you’re trying to implement a speedy CDN-like infrastructure, it might be counterproductive.
Ok, got DNS validation working via Vultr!
I’m going to give reverse proxying a shot, but I wished to know if the setup I had in mind is sensible:
20 PoPs
Each PoP has 1 Caddy instance as a RP/LB, load-balancing a five Caddy server farm.
My Q is, should the RP/LB Caddy instances terminate TLS? As in, should I not enable HTTPS on the Caddy instances in each farm at each PoP?
I can definitely see hitting Let’s Encrypt’s limits with 100 Caddy farm-nodes behind 20 RP/LB, each running TLS.
Is it okay to serve HTTP on the farm nodes, and HTTPS on the Caddy RP/LB at each PoP?
This is all for testing, I don’t have real dynamic content I’m planning to serve through this setup.
I think the answer is a bit of a trade off that depends on a few things.
Like your networking - are the farm-nodes privately networked to their local load balancer? If so, HTTP is a clear winner - you shave off a tiny bit of overhead per request from TLS, and you save yourself the headache of replicating or requisitioning certificates for all the farm nodes.
Otherwise, it depends on what kind of threat model you expect and what data is being transferred as to whether it’s worth the difficulty of not terminating TLS at the local load balancers.
Thanks, HTTP seems to be the way to go. If I have opsec issues in my server farm, I have bigger issues than lack of TLS within the same PoP. I setup 5 Caddy servers behind a Caddy RP/LB and I get a 502 Bad Gateway error.
The 5 node farm config:
# cat ./Caddyfile.nodes
localhost:81 {
root /home/strykar
tls off
}
localhost:82 {
root /home/strykar
tls off
}
localhost:83 {
root /home/strykar
tls off
}
localhost:84 {
root /home/strykar
tls off
}
localhost:85 {
root /home/strykar
tls off
}
The Caddy RP/LB is launched as: /usr/local/bin/caddy -agree -email s@mail.com -host anycast.slackware.in -conf /home/strykar/Caddyfile -pidfile /var/run/caddy.pid -root /home/strykar/
The Caddy 5 node farm is launched as: /usr/local/bin/caddy -disable-http-challenge -disable-tls-sni-challenge -conf ./Caddyfile.nodes
I’ve tried disabling things and moving to a really small config, removing MIME types but I always get a 502 Bad Gateway The nodes are indeed serving HTTP as confirmed by curl localhost:81 which serves/shows the static index.html I read a dozen posts of similar issues with 502’s and I can’t figure out what I’m doing wrong. Where do I start to troubleshoot this?
Never mind, I see changing the proxy directive to proxy / localhost:81 localhost:82 localhost:83 localhost:84 localhost:85 { }
and the RP/LB launch option to /usr/local/bin/caddy -agree -email s@mail.com -host anycast.slackware.in -conf /home/strykar/Caddyfile -pidfile /var/run/caddy.pid fixes it.
The issue was with one or some of the policy least_conn health_check /home/strykar transparent websocket except /static /robots.txt insecure_skip_verify options I had set.
What’s the correct way of determining which of the farm nodes responded from that PoP?
Or is X-POP-Tag the most one can get?
Gonna set up logging and try to replicate this to another PoP this week.
Thanks for all your help.
You can easily have one header to indicate the load balancer used and one header to indicate the farm node. X-POP-Tag and X-Node-Tag maybe, but again, you can call it whatever you like (the ‘X-’ prefixture is just a common practice, but unnecessary).
A simple header / X-Node-Tag "abcxyz123" on the farm nodes will do the trick. It will get picked up with all the other headers by the load balancer and passed back to the client as normal, so the client will see both headers.