Guidance on contributing to or creating a new plugin for distributed shared storage

[Carter_Bryden]

Hey folks,

I’ve been using Caddy v2 in a clustered mode (globally distributed) and sharing storage using the redis plugin. I make heavy use of the TLS on demand feature as well.

The main issue I’m having with using Caddy this way is that the time it takes to negotiate TLS for each request can be a lot longer if the Caddy instance is far from the redis storage instance (for example, Japan to UK).

So first, does Caddy load the TLS information for every request from storage, or does it keep it in memory? If it keeps it in memory and just uses the storage as persistence, it might be something else I’m doing wrong.

If it does load it from storage every time, I’m wondering if anyone has any thoughts on the best way I can help solve this the latency issue for Caddy. I can probably find a solution for my own use-case outside of Caddy, though it might be hacky, but I’d like to see if I can come up with something useful for everyone.

Some ideas:

• Use some kind of replicated, distributed filesystem. Seems like this could have holes though, and not sure how locks would work.
• Contribute to the redis plugin and get it working with redis clusters (maybe using a read-only replica when loading, but saving to main when creating/managing a cert?)
• A new plugin, potentially one that uses local (file?) storage and only checks a remote storage if the cert doesn’t exist locally. With on demand TLS it would be even slower on first request but would be fast after that. Not sure what the renewal flow would look like though.
• A contribution to Caddy core if you have any ideas that you think would be better in that direction

Let me know if any of you have thoughts or ideas on what would be the best way to solve this for all Caddy users. I’d love to contribute to and document whatever we think makes the most sense

[francislavoie]

@Carter_Bryden

So first, does Caddy load the TLS information for every request from storage, or does it keep it in memory? If it keeps it in memory and just uses the storage as persistence, it might be something else I’m doing wrong.

Once certs are issued, Caddy keeps up to 10,000 certs in memory. If you have more than that, then it may trigger calls to the storage if the request is to an unknown certificate (and this may also trigger on-demand issuance if none is found). This capacity is configurable.

caddyserver/caddy/blob/b4cef492cc2b4c6ee0e43b083488823cbf5e3a48/modules/caddytls/tls.go#L103-L108

@Carter_Bryden

Use some kind of replicated, distributed filesystem. Seems like this could have holes though, and not sure how locks would work.

Yeah, that’s an option; you can use something like ceph or glusterfs. Locks are just files that exist on the filesystem while operations are active.

@Carter_Bryden

Contribute to the redis plugin and get it working with redis clusters (maybe using a read-only replica when loading, but saving to main when creating/managing a cert?)

That’s also viable.

[Carter_Bryden]

Ah, I’d suspected that it cached the certs in memory, but the results I was seeing made me doubt it. Turns out it was an error on my part and some of my instances weren’t actually active so of course those were slower.

That’s excellent new though, since that means that it inherently already works pretty much exactly the way I was hoping to accomplish! Thanks for taking the time.

[Carter_Bryden]

Actually one more quick question, if you know off-hand: any idea how much memory each cert takes up in memory? Trying to figure out how roughly many reverse proxy sites I can have per GB of memory.

[francislavoie]

Not a lot. I’d guesstimate like 4kb per cert, which would be like 40MB for 10k certs?