With a Caddy Cluster, do Acme Challenges need to hit the same instance?

1. Caddy version (caddy version):


2. How I run Caddy:

We’re investigating running caddy on Fly.io with their TCP passthrough setting, so unterminated TLS can hit many containers at the edge. We’re using DynamoDB as cluster storage.

a. System environment:


b. Command:

We’re using the base Caddy image and adding a custom Caddyfile.

c. Service/unit/compose file:

FROM caddy:2.2.0
COPY Caddyfile /etc/caddy/Caddyfile

d. My complete Caddyfile or JSON config:

    http_port 80
    https_port 443
reverse_proxy {
    to https://ourorigin.dev:443
    header_up Host {http.reverse_proxy.upstream.host}
    transport http {
        versions allow_h2c

3. The problem I’m having:

To be clear, we haven’t actually had a problem. Rather, we’ve investigated the source code and we’re not sure if we’ve just gotten lucky or if Caddy does indeed support our use case.

When we host with Fly.io, several instances of Caddy are made available in different locations with an Anycast IP for global load balancing.

When we made a request our Anycast IP, one of the Caddy instances received the request and successfully generated a certificate. Subsequent requests served the certificate.

So far so good, but after investigating further, we’re not sure if this is guaranteed to happen or if we got was lucky.

Our concern is that:

  1. A request will come in on a host that does not have a certificate.
  2. The Caddy instance that receives the request will initiate automatic HTTPs.
  3. In the case of the HTTP challenge, that Caddy instance will serve the .well-known file, but maybe(?) other Caddy instances will not. Because we’re using an anycast IP, Let’s Encrypt may end up requesting the .well-known file from the wrong server.

Is this a realistic concern? Our investigation of the CertMagic Storage Interface led us to believe the clustering solution will work for us after certificates are issued, but there might be an issue during the issuing process.

We’re a bit new to Acme and Caddy and Anycast so apologies in advance if this is a dumb question! So far everything is going as planned and we’ve been really impressed with the simplicity, so many many thanks for making this available.

Nope! Any Caddy instance can initiate the ACME flow and any other can complete it by solving the challenge, as long as they all share the same storage. Caddy writes to the storage the challenge data, and when another receives a request for the challenge, it will read from the storage to see if it can solve it.

Mentioned briefly here:

Any Caddy instances that are configured to use the same storage will automatically share those resources and coordinate certificate management as a cluster.

1 Like

Awesome! That’s what we weren’t sure of. Very cool that you’re doubling up the storage interface for certificates and challenges.

1 Like

This topic was automatically closed after 30 days. New replies are no longer allowed.