Latency in a clustered Caddy setup with remote storage?

chrjoergensen · January 15, 2021, 9:03am

We are moving from a single Caddy node setup to a cluster setup with multiple nodes.

Our nodes are not going to be in the same private network, they are actually not even in the same datacenter. To be able to complete ACME challenges from Let’s Encrypt the nodes need to use a shared storage facility such as S3 or Redis. I have read up on similar posts in this forum, and from what I read sharing storage through a plugin seems pretty straight forward.

But I have one concern that I can not find answers to, so I thought I would ask it here: Are the obtained certificates solely stored in the shared storage facility, or are they also being cached locally on the individual node? If they are not cached, does this mean that the shared storage facility is being hit every time a request comes though? This would introduce a single point of failure, and in our use case lot’s of latency.

If the answer to my question is simply “of course they are cached, stupid” I apologize in advance

francislavoie · January 15, 2021, 9:36am

Yep, they are cached:

github.com

caddyserver/certmagic/blob/725b69d53d57cb1b8f43df814a2c11052eda0f4e/cache.go

// Copyright 2015 Matthew Holt
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
//     http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.

package certmagic

import (
	"fmt"
	weakrand "math/rand" // seeded elsewhere
	"strings"

This file has been truncated. show original

Caddy sets a default capacity of 10000 certificates:

github.com

caddyserver/caddy/blob/bd17eb205d6ac464c64eb888a6f4b57445b6c59c/modules/caddytls/tls.go#L100

    
      
          		Logger: t.logger.Named("cache"),
          	}
          	if t.Automation != nil {
          		cacheOpts.OCSPCheckInterval = time.Duration(t.Automation.OCSPCheckInterval)
          		cacheOpts.RenewCheckInterval = time.Duration(t.Automation.RenewCheckInterval)
          	}
          	if t.Cache != nil {
          		cacheOpts.Capacity = t.Cache.Capacity
          	}
          	if cacheOpts.Capacity <= 0 {
          		cacheOpts.Capacity = 10000
          	}
          	t.certCache = certmagic.NewCache(cacheOpts)
          
          
	// certificate loaders
          	val, err := ctx.LoadModule(t, "CertificatesRaw")
          	if err != nil {
          		return fmt.Errorf("loading certificate loader modules: %s", err)
          	}
          	for modName, modIface := range val.(map[string]interface{}) {
          		if modName == "automate" {

In the docs:

chrjoergensen · January 15, 2021, 9:45am

Great to hear that. Thanks a lot!

system · February 14, 2021, 9:45am

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.