Question about List API for Caddy/CertMagic Storage

Hello,

I am looking at creating a Caddy (CertMagic?) Storage backend, and I had a question about the List operation.

According to the interface spec, certmagic/storage.go at v0.17.2 · caddyserver/certmagic · GitHub, List takes in a prefix of some sort, and if recursive is true, we should “walk” the directory structure and return any matches of prefix.

	// List returns all keys that match prefix.
	// If recursive is true, non-terminal keys
	// will be enumerated (i.e. "directories"
	// should be walked); otherwise, only keys
	// prefixed exactly by prefix will be listed.
	List(ctx context.Context, prefix string, recursive bool) ([]string, error)

If we are using a strict key-value store (where there’s no hierarchy), what would it mean to have “recursive==true” and return a list of results? Would we instead list everything in the KV store, and return matches based on “prefix” value, regardless of whether or not recursive is true?

Where would I find examples of what the prefix value will be, so I can add tests to make sure I am doing the right thing in my Storage plugin code?

Thank you for your time.

1 Like

Probably, yes. If your KV store has support for searching by prefix, that would be ideal, but if not then yeah you probably need to filter yourself against all the keys.

This is a problem that’s come up before with implementations like dynamodb because “scanning” is a pretty expensive operation there so it made usage costs kinda high.

Recursive=true is usually only called when trying to look for expired items in the store. The storage_clean_interval config option can be used to tune how often that happens. By default it’s just once a day (plus on server startup) but it could be made like 2 weeks or something pretty safely if needed.

1 Like

Thank you for the info.

Should I base my testing off of the storage_test.go certmagic/storage_test.go at v0.17.2 · caddyserver/certmagic · GitHub file from the repo, since it looks like it has some paths and prefixes in it I could use? Or should I use something else to base my tests off of?

I don’t know why you should enumerate the entire KV store; if recursive is true, you should return only keys that start with the given prefix.

As an example, think of a Unix-style directory path: /a/b/c/ – anything “inside” that folder should be returned, e.g. /a/b/c/1, /a/b/c/2 etc.

As an example, think of a Unix-style directory path: /a/b/c/ – anything “inside” that folder should be returned, e.g. /a/b/c/1 , /a/b/c/2 etc.

Ahh, my bad. I guess I think of recursive listing as walking down the directory path and enumerating each node as the list. So, if we have the following:

/a/b/c/1
/a/b/c/2
/a/b/c/d/e/3
/a/b/c/d/f/4

And the prefix was /a/b/c/ and recursive==true, I would expect to get:

1
2
d/e/3
d/f/4

But based on what you mentioned above, it should only return this?

1
2
d/   # Implying there's more here?

Sorry, I may be over-thinking it. :frowning:

1 Like

(Just now saw the other replies after the topic post. Huh…)

To be clear, you should return the entire key – don’t strip the prefix or modify it in any way.

Correct, but the entire keys, not just the suffixes.

No, because /a/b/c/d/e/3 is also in /a/b/c/ – and only return /a/b/c/d if it’s actually a value.

1 Like

Hello,

I have another related question. After working through the implemenation of the Storage backend, I can Store & Load (among the other operations) my certificates, which is great.

However, when I start up Caddy, it doesn’t seem to see my certificate from my backend Storage implementation… I assumed (probably incorrectly) that when Caddy sees a TLS connection with an SNI hostname, it would see if a cert exists in the Cache, and if not, it would attempt to find it in Storage (and if that fails, then we would see “no certificate found for hostname”).

In my case, it seems Caddy starts up, inspects the Storage (finding my cert), but then nothing is reported for the incoming TLS connection:

2023/05/15 23:52:13.766 INFO    using provided configuration    {"config_file": "config.json", "config_adapter": ""}
2023/05/15 23:52:13.767 INFO    [INFO] Redirecting sink to: stderr
[INFO] Redirected sink to here (stderr)
2023/05/15 23:52:13.767 INFO    redirected default logger       {"from": "stderr", "to": "stdout"}
2023/05/15 23:52:13.767 INFO    admin   admin endpoint started  {"address": "127.0.0.1:7443", "enforce_origin": false, "origins": ["https://caddy.example.com:443"]}
2023/05/15 23:52:13.768 INFO    tls.cache.maintenance   started background certificate maintenance      {"cache": "0xc00025a8c0"}
2023/05/15 23:52:13.769 DEBUG   caddy.storage.vault     Load() from url {"url": "https://vault.localdev.klmh.co:8443/v1/klm/secrets/data/production/api/certificates/ocsp/localhost-843a3daf"}
2023/05/15 23:52:13.769 INFO    caddy.storage.vault     Logging in to vault using approle credentials
2023/05/15 23:52:13.787 DEBUG   caddy.storage.vault     Using newly created approle token for auth
2023/05/15 23:52:13.790 WARN    tls     stapling OCSP   {"error": "no OCSP stapling for [localhost *.localdev.klmh.co *.localdev.us-west.mywordpress.io 127.0.0.1 ::1]: no OCSP server specified in certificate"}
2023/05/15 23:52:13.790 DEBUG   events  event   {"name": "cached_unmanaged_cert", "id": "0fde1713-b53e-4843-a6cf-3b9cd709626c", "origin": "tls", "data": {"sans":["localhost","*.localdev.klmh.co","*.localdev.us-west.mywordpress.io","127.0.0.1","::1"]}}
2023/05/15 23:52:13.790 DEBUG   tls.cache       added certificate to cache      {"subjects": ["localhost", "*.localdev.klmh.co", "*.localdev.us-west.mywordpress.io", "127.0.0.1", "::1"], "expiration": "2025/08/15 20:43:05.000", "managed": false, "issuer_key": "", "hash": "74424cf96176266af94f548c9d80c0d3541f2ae1c10d54da5f545ff3cfcac114", "cache_size": 1, "cache_capacity": 10000}
2023/05/15 23:52:13.790 DEBUG   http    starting server loop    {"address": "[::]:10443", "tls": false, "http3": false}
2023/05/15 23:52:13.790 INFO    tls     cleaning storage unit   {"description": "&{0xc0004406c0}"}
2023/05/15 23:52:13.790 DEBUG   http    starting server loop    {"address": "[::]:10080", "tls": false, "http3": false}
2023/05/15 23:52:13.791 DEBUG   caddy.storage.vault     List() at url   {"url": "https://vault.localdev.klmh.co:8443/v1/klm/secrets/metadata/production/api/certificates/ocsp", "recursive": false}
2023/05/15 23:52:13.791 INFO    http.log        server running  {"name": "default", "protocols": ["h1", "h2", "h3"]}
2023/05/15 23:52:13.791 INFO    serving initial configuration
2023/05/15 23:52:13.791 DEBUG   caddy.storage.vault     Using approle client token for auth
2023/05/15 23:52:13.795 DEBUG   caddy.storage.vault     List() at url   {"url": "https://vault.localdev.klmh.co:8443/v1/klm/secrets/metadata/production/api/certificates/certificates", "recursive": false}
2023/05/15 23:52:13.795 DEBUG   caddy.storage.vault     Using approle client token for auth
2023/05/15 23:52:13.798 DEBUG   caddy.storage.vault     List() at url   {"url": "https://vault.localdev.klmh.co:8443/v1/klm/secrets/metadata/production/api/certificates/certificates/acme-staging-v02.api.letsencrypt.org-directory/", "recursive": false}
2023/05/15 23:52:13.798 DEBUG   caddy.storage.vault     Using approle client token for auth
2023/05/15 23:52:13.801 DEBUG   caddy.storage.vault     List() at url   {"url": "https://vault.localdev.klmh.co:8443/v1/klm/secrets/metadata/production/api/certificates/certificates/acme-staging-v02.api.letsencrypt.org-directory/abc123.localdev.mywordpress.io/", "recursive": false}
2023/05/15 23:52:13.801 DEBUG   caddy.storage.vault     Using approle client token for auth
2023/05/15 23:52:13.805 DEBUG   caddy.storage.vault     Load() from url {"url": "https://vault.localdev.klmh.co:8443/v1/klm/secrets/data/production/api/certificates/certificates/acme-staging-v02.api.letsencrypt.org-directory/abc123.localdev.mywordpress.io/abc123.localdev.mywordpress.io.crt"}
2023/05/15 23:52:13.805 DEBUG   caddy.storage.vault     Using approle client token for auth
2023/05/15 23:52:13.808 DEBUG   caddy.storage.vault     List() at url   {"url": "https://vault.localdev.klmh.co:8443/v1/klm/secrets/metadata/production/api/certificates/certificates/acme-staging-v02.api.letsencrypt.org-directory/abc123.localdev.mywordpress.io/", "recursive": false}
2023/05/15 23:52:13.808 DEBUG   caddy.storage.vault     Using approle client token for auth
2023/05/15 23:52:13.812 INFO    tls     finished cleaning storage units

From curl when I try to connect to the running Caddy instance using the TLS cert from Storage above:

$ curl -i https://abc123.localdev.mywordpress.io:8443 --connect-to abc123.localdev.mywordpress.io:8443:0.0.0.0:10443
curl: (35) error:1408F10B:SSL routines:ssl3_get_record:wrong version number

And lastly, my config.json:

{
  "admin": {
    "disabled": false,
    "listen": "127.0.0.1:7443",
    "enforce_origin": false,
    "origins": [
      "https://caddy.example.com:443"
    ],
    "config": {
      "persist": false
    }
  },
  "logging": {
    "logs": {
      "default": {
        "encoder": {
          "format": "console"
        },
        "level": "debug",
        "writer": {
          "output": "stdout"
        }
      }
    },
    "sink": {
      "writer": {
        "output": "stderr"
      }
    }
  },
  "storage": {
    "address": "https://vault.localdev.klmh.co:8443",
    "approle_role_id": "dead-beef",
    "approle_secret_id": "ea7-beef",
    "log_level": "debug",
    "module": "vault",
    "insecure_skip_verify": true,
    "path_prefix": "production/api/certificates",
    "secrets_path": "klm/secrets"
  },
  "apps": {
    "tls": {
      "certificates": {
        "load_folders": [
          "/tmp/certificates"
        ]
      }
    },
    "http": {
      "grace_period": "30s",
      "servers": {
        "default": {
          "listen": [
            "0.0.0.0:10443",
            "0.0.0.0:10080"
          ],
          "listener_wrappers": [
            {
              "wrapper": "http_redirect"
            },
            {
              "wrapper": "tls"
            }
          ],
          "metrics": {},
          "routes": [
            {
              "group": "default",
              "handle": [
                {
                  "handler": "reverse_proxy",
                  "load_balancing": {
                    "selection_policy": {
                      "policy": "least_conn"
                    }
                  },
                  "transport": {
                    "protocol": "http",
                    "tls": {
                      "insecure_skip_verify": true
                    }
                  },
                  "upstreams": [
                    {
                      "dial": "127.0.0.1:9443",
                      "max_requests": 1024
                    }
                  ]
                }
              ]
            }
          ]
        }
      }
    }
  }
}

I am wondering if my List() is incorrect (though, according to the logs, it does seem like it’s trying to Load() it from Storage).

Is there something else I need to do to load the cert into Cache if I am using a custom Storage provider?

EDIT: Interestingly, if I make a curl connection for a TLS certificate that does exist (which was manually loaded for *.localdev.klmh.co–see the logs above), it reports the same curl error? I must be doing something wrong, but I’m stumped.

$ curl -i https://foobar.localdev.klmh.co:8443 --connect-to foobar.localdev.klmh.co:8443:0.0.0.0:10443
curl: (35) error:1408F10B:SSL routines:ssl3_get_record:wrong version number

Update: On a hunch, I did a tcpdump filtering on port 10443, and I noticed the server was sending a plaintext response of 400 Bad Request, which is confusing.

So, I did the same curl, but with http instead of https, and I got this:

$ curl -i http://abc123.localdev.mywordpress.io:8443 --connect-to abc123.localdev.mywordpress.io:8443:0.0.0.0:10443
HTTP/1.0 308 Permanent Redirect
Location: https://abc123.localdev.mywordpress.io:8443/
Content-Length: 0

It doesn’t make sense to listen to both an HTTP and HTTPS port on the same server; a server can only do one protocol (HTTP or HTTPS) at a time. Also using 0.0.0.0 means IPv6 would not work. So I suggest changing this to only :10443. Caddy will enable HTTP->HTTPS redirects automatically by creating a new server on port 80 (you can change that by setting the http_port on the http app).

I think you might not be hitting Caddy at all. You would see Server: Caddy if you were.

1 Like

Thank you for the response francislavoie.

It doesn’t make sense to listen to both an HTTP and HTTPS port on the same server; a server can only do one protocol (HTTP or HTTPS) at a time.

Yes, great point, I will fix that.

I think you might not be hitting Caddy at all. You would see Server: Caddy if you were.

I think that is only true if it doesn’t hit some other handler/module first (like httpredirect).

It feels like Caddy is getting confused about the incoming connection (maybe my curl statement is incorrect?) at this location: caddy/modules/caddyhttp/httpredirectlistener.go at e8352aef38642c20ff528836b6581094f087eb99 · caddyserver/caddy · GitHub

That func does have a 308 permanent redirect in place (see lines 115-124), but it’s a part of the connection sniffer. So, for some reason, it thinks my curl command is plaintext (non-TLS) incoming on a TLS port, which I think means either my curl is incorrect, or my server Caddy config is incorrect?

In the debug log, it shows this too, implying it’s a TLS listener, I beleive?

2023/05/16 01:50:55.567 DEBUG   http    starting server loop    {"address": "[::]:10443", "tls": true, "http3": true}

What do you think?

maybe my curl statement is incorrect?

Yes, that is the problem, my bad. I will start a new thread if I encounter other issues. Thank you.

What was wrong with your curl command?

Also why are you enabling http_redirect in this case anyway? Do you expect clients to make HTTP requests to your HTTPS port?

Caddy always emits the Server header for automatic redirects, for example:

$ curl -v "http://caddyserver.com"
*   Trying 165.227.20.207:80...
* Connected to caddyserver.com (165.227.20.207) port 80 (#0)
> GET / HTTP/1.1
> Host: caddyserver.com
> User-Agent: curl/8.0.1
> Accept: */*
> 
< HTTP/1.1 308 Permanent Redirect
< Connection: close
< Location: https://caddyserver.com/
< Server: Caddy
< Date: Tue, 16 May 2023 15:22:11 GMT
< Content-Length: 0
< 
* Closing connection 0

I changed my curl command, but I am still getting the same behaviour, presumably because Caddy thinks my listener is not configured for TLS? I’m sure I’m doing something wrong, but can’t tell what. Here’s the updated info.

  • starting server loop {"address": "[::]:10443", "tls": false, "http3": false}

Config

{
  "admin": {
    "disabled": false,
    "listen": "127.0.0.1:7443",
    "enforce_origin": false,
    "origins": [
      "https://caddy.example.com:443"
    ],
    "config": {
      "persist": false
    }
  },
  "logging": {
    "logs": {
      "default": {
        "encoder": {
          "format": "console"
        },
        "level": "debug",
        "writer": {
          "output": "stdout"
        }
      }
    },
    "sink": {
      "writer": {
        "output": "stderr"
      }
    }
  },
  "storage": {
    "address": "https://vault.localdev.klmh.co:8443",
    "approle_role_id": "dead-beef",
    "approle_secret_id": "ea7-beef",
    "log_level": "debug",
    "module": "vault",
    "insecure_skip_verify": true,
    "path_prefix": "production/api/certificates",
    "secrets_path": "klm/secrets"
  },
  "apps": {
    "tls": {
      "certificates": {
        "load_folders": [
          "/tmp/certificates"
        ]
      }
    },
    "http": {
      "grace_period": "30s",
      "servers": {
        "default": {
          "listen": [
            "0.0.0.0:10443"
          ],
          "metrics": {},
          "routes": [
            {
              "group": "default",
              "handle": [
                {
                  "handler": "static_response",
                  "status_code": 200,
                  "body": "SUCCESS",
                  "close": true
                }
              ]
            }
          ]
        }
      }
    }
  }
}

Debug Logs

2023/05/16 15:43:52.102 INFO    using provided configuration    {"config_file": "config.json", "config_adapter": ""}
2023/05/16 15:43:52.102 INFO    [INFO] Redirecting sink to: stderr
[INFO] Redirected sink to here (stderr)
2023/05/16 15:43:52.103 INFO    redirected default logger       {"from": "stderr", "to": "stdout"}
2023/05/16 15:43:52.103 INFO    admin   admin endpoint started  {"address": "127.0.0.1:7443", "enforce_origin": false, "origins": ["https://caddy.example.com:443"]}
2023/05/16 15:43:52.104 INFO    tls.cache.maintenance   started background certificate maintenance      {"cache": "0xc00028acb0"}
2023/05/16 15:43:52.104 DEBUG   caddy.storage.vault     Load() from url {"url": "https://vault.localdev.klmh.co:8443/v1/klm/secrets/data/production/api/certificates/ocsp/localhost-843a3daf"}
2023/05/16 15:43:52.104 INFO    caddy.storage.vault     Logging in to vault using approle credentials
2023/05/16 15:43:52.122 DEBUG   caddy.storage.vault     Using newly created approle token for auth
2023/05/16 15:43:52.125 WARN    tls     stapling OCSP   {"error": "no OCSP stapling for [localhost *.localdev.klmh.co *.localdev.us-west.mywordpress.io 127.0.0.1 ::1]: no OCSP server specified in certificate"}
2023/05/16 15:43:52.125 DEBUG   events  event   {"name": "cached_unmanaged_cert", "id": "c476f824-1523-42f1-ae01-b14f63b7968e", "origin": "tls", "data": {"sans":["localhost","*.localdev.klmh.co","*.localdev.us-west.mywordpress.io","127.0.0.1","::1"]}}
2023/05/16 15:43:52.125 DEBUG   tls.cache       added certificate to cache      {"subjects": ["localhost", "*.localdev.klmh.co", "*.localdev.us-west.mywordpress.io", "127.0.0.1", "::1"], "expiration": "2025/08/15 20:43:05.000", "managed": false, "issuer_key": "", "hash": "74424cf96176266af94f548c9d80c0d3541f2ae1c10d54da5f545ff3cfcac114", "cache_size": 1, "cache_capacity": 10000}
2023/05/16 15:43:52.125 DEBUG   http    starting server loop    {"address": "[::]:10443", "tls": false, "http3": false}
2023/05/16 15:43:52.125 INFO    tls     cleaning storage unit   {"description": "&{0xc0004ca840}"}
2023/05/16 15:43:52.125 INFO    http.log        server running  {"name": "default", "protocols": ["h1", "h2", "h3"]}
2023/05/16 15:43:52.125 INFO    serving initial configuration
2023/05/16 15:43:52.125 DEBUG   caddy.storage.vault     List() at url   {"url": "https://vault.localdev.klmh.co:8443/v1/klm/secrets/metadata/production/api/certificates/ocsp", "recursive": false}
2023/05/16 15:43:52.125 DEBUG   caddy.storage.vault     Using approle client token for auth
2023/05/16 15:43:52.128 DEBUG   caddy.storage.vault     List() at url   {"url": "https://vault.localdev.klmh.co:8443/v1/klm/secrets/metadata/production/api/certificates/certificates", "recursive": false}
2023/05/16 15:43:52.129 DEBUG   caddy.storage.vault     Using approle client token for auth
2023/05/16 15:43:52.131 DEBUG   caddy.storage.vault     List() at url   {"url": "https://vault.localdev.klmh.co:8443/v1/klm/secrets/metadata/production/api/certificates/certificates/acme-staging-v02.api.letsencrypt.org-directory/", "recursive": false}
2023/05/16 15:43:52.131 DEBUG   caddy.storage.vault     Using approle client token for auth
2023/05/16 15:43:52.135 DEBUG   caddy.storage.vault     List() at url   {"url": "https://vault.localdev.klmh.co:8443/v1/klm/secrets/metadata/production/api/certificates/certificates/acme-staging-v02.api.letsencrypt.org-directory/abc123.localdev.mywordpress.io/", "recursive": false}
2023/05/16 15:43:52.135 DEBUG   caddy.storage.vault     Using approle client token for auth
2023/05/16 15:43:52.139 DEBUG   caddy.storage.vault     Load() from url {"url": "https://vault.localdev.klmh.co:8443/v1/klm/secrets/data/production/api/certificates/certificates/acme-staging-v02.api.letsencrypt.org-directory/abc123.localdev.mywordpress.io/abc123.localdev.mywordpress.io.crt"}
2023/05/16 15:43:52.139 DEBUG   caddy.storage.vault     Using approle client token for auth
2023/05/16 15:43:52.143 DEBUG   caddy.storage.vault     List() at url   {"url": "https://vault.localdev.klmh.co:8443/v1/klm/secrets/metadata/production/api/certificates/certificates/acme-staging-v02.api.letsencrypt.org-directory/abc123.localdev.mywordpress.io/", "recursive": false}
2023/05/16 15:43:52.143 DEBUG   caddy.storage.vault     Using approle client token for auth
2023/05/16 15:43:52.147 INFO    tls     finished cleaning storage units

Updated curl Command (https)

$ curl -vi https://abc123.localdev.mywordpress.io:8443 --resolve abc123.localdev.mywordpress.io:8443:127.0.0.1 --connect-to abc123.localdev.mywordpress.io:8443:0.0.0.0:10443
* Added abc123.localdev.mywordpress.io:8443:127.0.0.1 to DNS cache
* Connecting to hostname: 0.0.0.0
* Connecting to port: 10443
*   Trying 0.0.0.0:10443...
* Connected to 0.0.0.0 (127.0.0.1) port 10443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* successfully set certificate verify locations:
*  CAfile: /etc/ssl/certs/ca-certificates.crt
*  CApath: /etc/ssl/certs
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* error:1408F10B:SSL routines:ssl3_get_record:wrong version number
* Closing connection 0
curl: (35) error:1408F10B:SSL routines:ssl3_get_record:wrong version number

Updated curl Command (http)

Note: Don’t need --resolve flag since there’s no TLS involved

$ curl -vi http://abc123.localdev.mywordpress.io:8443 --connect-to abc123.localdev.mywordpress.io:8443:0.0.0.0:10443
* Connecting to hostname: 0.0.0.0
* Connecting to port: 10443
*   Trying 0.0.0.0:10443...
* Connected to 0.0.0.0 (127.0.0.1) port 10443 (#0)
> GET / HTTP/1.1
> Host: abc123.localdev.mywordpress.io:8443
> User-Agent: curl/7.74.0
> Accept: */*
> 
* Mark bundle as not supporting multiuse
< HTTP/1.1 200 OK
HTTP/1.1 200 OK
< Connection: close
Connection: close
< Content-Type: text/plain; charset=utf-8
Content-Type: text/plain; charset=utf-8
< Server: Caddy
Server: Caddy
< Date: Tue, 16 May 2023 15:49:22 GMT
Date: Tue, 16 May 2023 15:49:22 GMT
< Content-Length: 7
Content-Length: 7

< 
* Closing connection 0
SUCCESS

tcpdump Capture

The tcpdump capture does show the SNI being set correctly in the ClientHello, so I think my curl command is correct now:

Ah yeah, if you don’t have a subroute with a host matcher then TLS won’t be enabled by Automatic HTTPS. You can turn it on by turning on tls_connection_policies (it can just be an empty object so one exists at all)

Ah yeah, if you don’t have a subroute with a host matcher then TLS won’t be enabled by Automatic HTTPS. You can turn it on by turning on tls_connection_policies (it can just be an empty object so one exists at all)

I tried that previously, but it didn’t seem to work. My config has this:

[ ... ]
          "routes": [
            {
              "group": "static",
              "handle": [
                {
                  "handler": "static_response",
                  "status_code": 200,
                  "body": "SUCCESS",
                  "close": true
                }
              ]
            }
          ],
          "tls_connection_policies": []

But, when Caddy starts, it still says:

2023/05/16 17:18:26.143 DEBUG   http    starting server loop    {"address": "[::]:10443", "tls": false, "http3": false}
2023/05/16 17:18:26.143 INFO    http.log        server running  {"name": "default", "protocols": ["h1", "h2", "h3"]}

Ignore the above message @francislavoie … I changed my config to this:

Config

{
  "admin": {
    "disabled": false,
    "listen": "127.0.0.1:7443",
    "enforce_origin": false,
    "origins": [
      "https://caddy.example.com:443"
    ],
    "config": {
      "persist": false
    }
  },
  "logging": {
    "logs": {
      "default": {
        "encoder": {
          "format": "console"
        },
        "level": "debug",
        "writer": {
          "output": "stdout"
        }
      }
    },
    "sink": {
      "writer": {
        "output": "stderr"
      }
    }
  },
  "storage": {
    "address": "https://vault.localdev.klmh.co:8443",
    "approle_role_id": "dead-beef",
    "approle_secret_id": "ea7-beef",
    "log_level": "debug",
    "module": "vault",
    "insecure_skip_verify": true,
    "path_prefix": "production/api/certificates",
    "secrets_path": "klm/secrets"
  },
  "apps": {
    "tls": {
      "certificates": {
        "load_folders": [
          "/tmp/certificates"
        ]
      }
    },
    "http": {
      "grace_period": "30s",
      "servers": {
        "default": {
          "automatic_https": {
            "disable_redirects": true
          },
          "listen": [
            "0.0.0.0:10443"
          ],
          "metrics": {},
          "routes": [
            {
              "group": "static",
              "handle": [
                {
                  "handler": "static_response",
                  "status_code": 200,
                  "body": "SUCCESS",
                  "close": true
                }
              ]
            }
          ],
          "tls_connection_policies": [{}]
        }
      }
    }
  }
}

And when I make a request, I see this in the logs now (still not able to use my TLS cert from Storage):

Log

2023/05/16 17:22:13.252 INFO    using provided configuration    {"config_file": "config.json", "config_adapter": ""}
2023/05/16 17:22:13.253 INFO    [INFO] Redirecting sink to: stderr
[INFO] Redirected sink to here (stderr)
2023/05/16 17:22:13.253 INFO    redirected default logger       {"from": "stderr", "to": "stdout"}
2023/05/16 17:22:13.254 INFO    admin   admin endpoint started  {"address": "127.0.0.1:7443", "enforce_origin": false, "origins": ["https://caddy.example.com:443"]}
2023/05/16 17:22:13.255 INFO    tls.cache.maintenance   started background certificate maintenance      {"cache": "0xc00024c230"}
2023/05/16 17:22:13.255 DEBUG   caddy.storage.vault     Load() from url {"url": "https://vault.localdev.klmh.co:8443/v1/klm/secrets/data/production/api/certificates/ocsp/localhost-843a3daf"}
2023/05/16 17:22:13.255 INFO    caddy.storage.vault     Logging in to vault using approle credentials
2023/05/16 17:22:13.271 DEBUG   caddy.storage.vault     Using newly created approle token for auth
2023/05/16 17:22:13.274 WARN    tls     stapling OCSP   {"error": "no OCSP stapling for [localhost *.localdev.klmh.co *.localdev.us-west.mywordpress.io 127.0.0.1 ::1]: no OCSP server specified in certificate"}
2023/05/16 17:22:13.274 DEBUG   events  event   {"name": "cached_unmanaged_cert", "id": "a9561d40-cbd5-4af4-b166-18fd4a7f9947", "origin": "tls", "data": {"sans":["localhost","*.localdev.klmh.co","*.localdev.us-west.mywordpress.io","127.0.0.1","::1"]}}
2023/05/16 17:22:13.274 DEBUG   tls.cache       added certificate to cache      {"subjects": ["localhost", "*.localdev.klmh.co", "*.localdev.us-west.mywordpress.io", "127.0.0.1", "::1"], "expiration": "2025/08/15 20:43:05.000", "managed": false, "issuer_key": "", "hash": "74424cf96176266af94f548c9d80c0d3541f2ae1c10d54da5f545ff3cfcac114", "cache_size": 1, "cache_capacity": 10000}
2023/05/16 17:22:13.274 WARN    http    automatic HTTP->HTTPS redirects are disabled    {"server_name": "default"}
2023/05/16 17:22:13.274 INFO    http    enabling HTTP/3 listener        {"addr": "0.0.0.0:10443"}
2023/05/16 17:22:13.274 INFO    tls     cleaning storage unit   {"description": "&{0xc0005d86c0}"}
failed to sufficiently increase receive buffer size (was: 208 kiB, wanted: 2048 kiB, got: 416 kiB). See https://github.com/quic-go/quic-go/wiki/UDP-Receive-Buffer-Size for details.
2023/05/16 17:22:13.274 DEBUG   caddy.storage.vault     List() at url   {"url": "https://vault.localdev.klmh.co:8443/v1/klm/secrets/metadata/production/api/certificates/ocsp", "recursive": false}
2023/05/16 17:22:13.274 DEBUG   caddy.storage.vault     Using approle client token for auth
2023/05/16 17:22:13.274 DEBUG   http    starting server loop    {"address": "[::]:10443", "tls": true, "http3": true}
2023/05/16 17:22:13.274 INFO    http.log        server running  {"name": "default", "protocols": ["h1", "h2", "h3"]}
2023/05/16 17:22:13.274 INFO    serving initial configuration
2023/05/16 17:22:13.277 DEBUG   caddy.storage.vault     List() at url   {"url": "https://vault.localdev.klmh.co:8443/v1/klm/secrets/metadata/production/api/certificates/certificates", "recursive": false}
2023/05/16 17:22:13.277 DEBUG   caddy.storage.vault     Using approle client token for auth
2023/05/16 17:22:13.281 DEBUG   caddy.storage.vault     List() at url   {"url": "https://vault.localdev.klmh.co:8443/v1/klm/secrets/metadata/production/api/certificates/certificates/acme-staging-v02.api.letsencrypt.org-directory/", "recursive": false}
2023/05/16 17:22:13.281 DEBUG   caddy.storage.vault     Using approle client token for auth
2023/05/16 17:22:13.284 DEBUG   caddy.storage.vault     List() at url   {"url": "https://vault.localdev.klmh.co:8443/v1/klm/secrets/metadata/production/api/certificates/certificates/acme-staging-v02.api.letsencrypt.org-directory/abc123.localdev.mywordpress.io/", "recursive": false}
2023/05/16 17:22:13.284 DEBUG   caddy.storage.vault     Using approle client token for auth
2023/05/16 17:22:13.288 DEBUG   caddy.storage.vault     Load() from url {"url": "https://vault.localdev.klmh.co:8443/v1/klm/secrets/data/production/api/certificates/certificates/acme-staging-v02.api.letsencrypt.org-directory/abc123.localdev.mywordpress.io/abc123.localdev.mywordpress.io.crt"}
2023/05/16 17:22:13.288 DEBUG   caddy.storage.vault     Using approle client token for auth
2023/05/16 17:22:13.291 DEBUG   caddy.storage.vault     List() at url   {"url": "https://vault.localdev.klmh.co:8443/v1/klm/secrets/metadata/production/api/certificates/certificates/acme-staging-v02.api.letsencrypt.org-directory/abc123.localdev.mywordpress.io/", "recursive": false}
2023/05/16 17:22:13.291 DEBUG   caddy.storage.vault     Using approle client token for auth
2023/05/16 17:22:13.296 INFO    tls     finished cleaning storage units
2023/05/16 17:22:20.991 DEBUG   events  event   {"name": "tls_get_certificate", "id": "1b1effab-9d6f-44b3-9451-d44cb505b62b", "origin": "tls", "data": {"client_hello":{"CipherSuites":[4866,4867,4865,49196,49200,159,52393,52392,52394,49195,49199,158,49188,49192,107,49187,49191,103,49162,49172,57,49161,49171,51,157,156,61,60,53,47,255],"ServerName":"abc123.localdev.mywordpress.io","SupportedCurves":[29,23,30,25,24],"SupportedPoints":"AAEC","SignatureSchemes":[1027,1283,1539,2055,2056,2057,2058,2059,2052,2053,2054,1025,1281,1537,771,769,770,1026,1282,1538],"SupportedProtos":["h2","http/1.1"],"SupportedVersions":[772,771],"Conn":{}}}}
2023/05/16 17:22:20.992 DEBUG   tls.handshake   no matching certificates and no custom selection logic  {"identifier": "abc123.localdev.mywordpress.io"}
2023/05/16 17:22:20.992 DEBUG   tls.handshake   no matching certificates and no custom selection logic  {"identifier": "*.localdev.mywordpress.io"}
2023/05/16 17:22:20.992 DEBUG   tls.handshake   no matching certificates and no custom selection logic  {"identifier": "*.*.mywordpress.io"}
2023/05/16 17:22:20.992 DEBUG   tls.handshake   no matching certificates and no custom selection logic  {"identifier": "*.*.*.io"}
2023/05/16 17:22:20.992 DEBUG   tls.handshake   no matching certificates and no custom selection logic  {"identifier": "*.*.*.*"}
2023/05/16 17:22:20.992 DEBUG   tls.handshake   all external certificate managers yielded no certificates and no errors {"remote_ip": "127.0.0.1", "remote_port": "48750", "sni": "abc123.localdev.mywordpress.io"}
2023/05/16 17:22:20.992 DEBUG   tls.handshake   no certificate matching TLS ClientHello {"remote_ip": "127.0.0.1", "remote_port": "48750", "server_name": "abc123.localdev.mywordpress.io", "remote": "127.0.0.1:48750", "identifier": "abc123.localdev.mywordpress.io", "cipher_suites": [4866, 4867, 4865, 49196, 49200, 159, 52393, 52392, 52394, 49195, 49199, 158, 49188, 49192, 107, 49187, 49191, 103, 49162, 49172, 57, 49161, 49171, 51, 157, 156, 61, 60, 53, 47, 255], "cert_cache_fill": 0.0001, "load_if_necessary": true, "obtain_if_necessary": true, "on_demand": false}
2023/05/16 17:22:20.992 DEBUG   http.stdlib     http: TLS handshake error from 127.0.0.1:48750: no certificate available for 'abc123.localdev.mywordpress.io'

@kmott When I run your exact config (I only removed the storage portion because I don’t have that module, and changed the load_folders path to one that exists on my machine), I get this output in my logs:

2023/05/16 17:28:20.674 DEBUG   http    starting server loop    {"address": "[::]:10443", "tls": true, "http3": true}

Make sure you are running the config you think you are.

Oh, you replied while I was replying, I guess…

Anyway… it looks like Caddy can’t find the right certificate in its cache. I would expect the .key file to also be loaded…

Does it work with the default storage? Strip your config down to the basic working form and then make one change at a time until it doesn’t work.

Anyway… it looks like Caddy can’t find the right certificate in its cache. I would expect the .key file to also be loaded…

Yeah, that’s sort of what I was originally thinking… basically that I needed to do something else special in my Caddy config to tell it to load my certs from external Storage in to the Cache so they can be used.

Do I need to add something like this, maybe (app.tls.certificates.load_storage)?

EDIT: I don’t know if it matters, but an external process placed the files in Storage (Caddy did not do it itself), using CertMagic Storage.