Redis or S3 storage with Caddy in Primary/Secondary configuration

Let’s say I have two Caddy instances running, in primary/secondary configuration using IP failover. On-demand TLS will be in use.

Only one server will receive traffic at a time, but I need to keep TLS data in sync, to avoid rate-limiting issues on failover.

I’m trying to figure out the simplest and most reliable approach. Consensus seems to favour the Redis plugin. But I’m not thrilled at the prospect of adding a Redis cluster to our fleet, purely for this purpose.

If I have a single Redis server - potentially even without persistence - could my two Caddy servers recover from a Redis failure using their local cache? Would they manage to get back in sync when Redis comes online again?

Ultimately, I’d like to better understand Caddy’s failure and recovery behaviour when it comes shared storage.

And given that contention isn’t an issue with this setup, would even one of the S3 plugins suffice?

Any feedback or discussion is appreciated.

Unfortunately, no; someone could write a storage plugin that could provide this functionality though, i.e. a storage plugin that wraps another with a fallback or something.

But the key is that the storage mechanism needs reliable locking so the Caddy instances don’t step on eachother’s toes. Something doing a fallback like this would make the locking difficult to be reliable. “Eventual consistency” is fundamentally incompatible with that requirement.

I wouldn’t recommend any of the S3 options, because they don’t implement locking properly.

If you’re on Amazon, you could consider using DynamoDB; problem is that DynamoDB has cost issues with the storage cleanup scanning that Caddy does, i.e. enumerate all the keys, check their expiry time and delete them if too old. Enumerating all the keys is super expensive with DynamoDB for some reason.

I think Redis will ultimately be the cheapest option. AWS has MemoryDB, you could try that – I don’t use AWS myself so I couldn’t really say how well that works, but it’s supposedly Redis compatible so it should be fine.

1 Like

For writes yes, but if writes go to all the storage backends (including the redundant/failover ones), it should be fine I’d think. Depends on performance requirements though. Obviously writing to multiple storage devices will likely be slower than writing to just one.

The reads could still happen in a redundant fashion.

But yeah, seconding everything Francis said, :heavy_plus_sign: :100:

1 Like

Thanks guys, you’ve clarified things a lot. It’s clear Redis is the way to go.

That said, I’ve just seen there’s a Postgres plugin, which would be ideal since we already have a cluster. It seems a little less mature than the Redis plugin. Are any of your clients using it?

This topic was automatically closed after 30 days. New replies are no longer allowed.