Multi-Region Storage

[davidtucker]

I am creating a cloud architecture (AWS) for Caddy that will span multiple regions. I am trying to determine the best approach for storage that enables the multi-region architecture to respond with minimal latency at each region.

I am planning on using MemoryDB (a durable Redis-compatible database service on AWS) within a single region. This service does provide a multi-region active-active configuration, but that would have a sync time between 1 and 2 seconds, which could result in the CertMagic locks not working as intended. This could have some of the same issues with using S3 as a storage mechanism.

Instead, I am planning to use VPC peering between each region and have the other regions request back to the “core” region through this private connection. There will obviously be some latency in this process, even though the storage mechanism itself is relatively fast. This, however, should preserve the atomic operations in storage for the global stack.

Here are my questions:

1. Will Caddy use available memory to store certificates on each instance? Is there any configuration for this cache?
2. How often would each instance need to fetch the certificate from Storage?
3. Has there been any thought to limiting which server can actually manage the certificate process? In this way, there could be one writer and many readers, which would open up some different storage approaches for a global deployment.

Thanks in advance.

[matt (Matt Holt)]

@davidtucker

Will Caddy use available memory to store certificates on each instance? Is there any configuration for this cache?

Yep, once a cert is loaded it stays in memory until it expires (and can’t be renewed) or the cache is full and it gets evicted.

Here are the cache options in JSON: JSON Config Structure - Caddy Documentation

@davidtucker

How often would each instance need to fetch the certificate from Storage?

Just once, unless it gets evicted from the cache.

@davidtucker

Has there been any thought to limiting which server can actually manage the certificate process? In this way, there could be one writer and many readers, which would open up some different storage approaches for a global deployment.

Hmm, not really; I think that would actually be pretty complicated… I think one thing that could work is a custom storage module that takes care of this. It could use the same underlying storage (wrap any other storage module), but could be configured with read/write access – and some way to communicate between the other Caddy instances.

---

As long as the atomicity guarantees are preserved, a little latency won’t hurt too much.