1. The problem I’m having:
Howdy! I need to run Caddy in AWS with auto-scaling and that requires shared storage. I have gone down the path of building Caddy with GitHub - gamalan/caddy-tlsredis: Redis Storage using for Caddy TLS Data and have managed to get caddy to start with the following settings:
“storage”: {
“Client”: null,
“ClientLocker”: null,
“Logger”: null,
“address”: “”,
“aes_key”: “a key”,
“db”: 0,
“host”: “a host”,
“key_prefix”: “caddytls”,
“module”: “redis”,
“password”: “”,
“port”: “6379”,
“timeout”: 5,
“tls_enabled”: false,
“tls_insecure”: true,
“username”: “”,
“value_prefix”: “redis”
I noticed after running caddy adapt --pretty that the Client and ClientLocker lines were present. I did not add these to my Caddyfile, so the system must need or want them? IDK
2. Error messages and/or full log output:
tail -f /logs/caddy-shared.log
{"level":"debug","ts":1689223460.606051,"logger":"tls.handshake","msg":"no certificate matching TLS ClientHello","remote_ip":"24.17.48.38","remote_port":"36340","server_name":"bleem-api.peoplebrowser.com","remote":"24.17.48.38:36340","identifier":"bleem-api.peoplebrowser.com","cipher_suites":[51914,4865,4866,4867,49195,49199,49196,49200,52393,52392,49171,49172,156,157,47,53],"cert_cache_fill":0,"load_if_necessary":true,"obtain_if_necessary":true,"on_demand":false}
{"level":"debug","ts":1689223460.6061308,"logger":"http.stdlib","msg":"http: TLS handshake error from 24.17.48.38:36340: no certificate available for 'bleem-api.peoplebrowser.com'"}
{"level":"debug","ts":1689223460.6861198,"logger":"events","msg":"event","name":"tls_get_certificate","id":"48ac316a-2b50-4407-aad8-78bfe9387c5f","origin":"tls","data":{"client_hello":{"CipherSuites":[51914,4865,4866,4867,49195,49199,49196,49200,52393,52392,49171,49172,156,157,47,53],"ServerName":"bleem-api.peoplebrowser.com","SupportedCurves":[35466,29,23,24],"SupportedPoints":"AA==","SignatureSchemes":[1027,2052,1025,1283,2053,1281,2054,1537,513],"SupportedProtos":["h2","http/1.1"],"SupportedVersions":[51914,772,771],"Conn":{}}}}
{"level":"debug","ts":1689223460.6861966,"logger":"tls.handshake","msg":"no matching certificates and no custom selection logic","identifier":"bleem-api.peoplebrowser.com"}
{"level":"debug","ts":1689223460.6862082,"logger":"tls.handshake","msg":"no matching certificates and no custom selection logic","identifier":"*.peoplebrowser.com"}
{"level":"debug","ts":1689223460.6862137,"logger":"tls.handshake","msg":"no matching certificates and no custom selection logic","identifier":"*.*.com"}
{"level":"debug","ts":1689223460.6862192,"logger":"tls.handshake","msg":"no matching certificates and no custom selection logic","identifier":"*.*.*"}
{"level":"debug","ts":1689223460.6862261,"logger":"tls.handshake","msg":"all external certificate managers yielded no certificates and no errors","remote_ip":"24.17.48.38","remote_port":"36350","sni":"bleem-api.peoplebrowser.com"}
{"level":"debug","ts":1689223460.6862345,"logger":"tls.handshake","msg":"no certificate matching TLS ClientHello","remote_ip":"24.17.48.38","remote_port":"36350","server_name":"bleem-api.peoplebrowser.com","remote":"24.17.48.38:36350","identifier":"bleem-api.peoplebrowser.com","cipher_suites":[51914,4865,4866,4867,49195,49199,49196,49200,52393,52392,49171,49172,156,157,47,53],"cert_cache_fill":0,"load_if_necessary":true,"obtain_if_necessary":true,"on_demand":false}
{"level":"debug","ts":1689223460.6863856,"logger":"http.stdlib","msg":"http: TLS handshake error from 24.17.48.38:36350: no certificate available for 'bleem-api.peoplebrowser.com'"}
3. Caddy version:
caddy version
v2.6.4 h1:2hwYqiRwk1tf3VruhMpLcYTg+11fCdr8S3jhNAdnPy8=
4. How I installed and ran Caddy:
308 wget https://github.com/caddyserver/xcaddy/releases/download/v0.3.4/xcaddy_0.3.4_linux_amd64.tar.gz
309 tar xvzf xcaddy_0.3.4_linux_amd64.tar.gz
310 install -v xcaddy /usr/local/bin
313 rm -rvf /tmp/xcaddy
317 xcaddy build --with GitHub - gamalan/caddy-tlsredis: Redis Storage using for Caddy TLS Data
a. System environment:
Ubuntu Server 22.04 running in EC2
$echo $CADDY_CLUSTERING
redis
caddy list-modules
admin.api.load
admin.api.metrics
admin.api.pki
admin.api.reverse_proxy
caddy.adapters.caddyfile
caddy.config_loaders.http
caddy.listeners.http_redirect
caddy.listeners.tls
caddy.logging.encoders.console
caddy.logging.encoders.filter
caddy.logging.encoders.filter.cookie
caddy.logging.encoders.filter.delete
caddy.logging.encoders.filter.hash
caddy.logging.encoders.filter.ip_mask
caddy.logging.encoders.filter.query
caddy.logging.encoders.filter.regexp
caddy.logging.encoders.filter.rename
caddy.logging.encoders.filter.replace
caddy.logging.encoders.json
caddy.logging.writers.discard
caddy.logging.writers.file
caddy.logging.writers.net
caddy.logging.writers.stderr
caddy.logging.writers.stdout
caddy.storage.file_system
events
http
http.authentication.hashes.bcrypt
http.authentication.hashes.scrypt
http.authentication.providers.http_basic
http.encoders.gzip
http.encoders.zstd
http.handlers.acme_server
http.handlers.authentication
http.handlers.copy_response
http.handlers.copy_response_headers
http.handlers.encode
http.handlers.error
http.handlers.file_server
http.handlers.headers
http.handlers.map
http.handlers.metrics
http.handlers.push
http.handlers.request_body
http.handlers.reverse_proxy
http.handlers.rewrite
http.handlers.static_response
http.handlers.subroute
http.handlers.templates
http.handlers.tracing
http.handlers.vars
http.ip_sources.static
http.matchers.expression
http.matchers.file
http.matchers.header
http.matchers.header_regexp
http.matchers.host
http.matchers.method
http.matchers.not
http.matchers.path
http.matchers.path_regexp
http.matchers.protocol
http.matchers.query
http.matchers.remote_ip
http.matchers.vars
http.matchers.vars_regexp
http.precompressed.br
http.precompressed.gzip
http.precompressed.zstd
http.reverse_proxy.selection_policies.cookie
http.reverse_proxy.selection_policies.first
http.reverse_proxy.selection_policies.header
http.reverse_proxy.selection_policies.ip_hash
http.reverse_proxy.selection_policies.least_conn
http.reverse_proxy.selection_policies.random
http.reverse_proxy.selection_policies.random_choose
http.reverse_proxy.selection_policies.round_robin
http.reverse_proxy.selection_policies.uri_hash
http.reverse_proxy.transport.fastcgi
http.reverse_proxy.transport.http
http.reverse_proxy.upstreams.a
http.reverse_proxy.upstreams.multi
http.reverse_proxy.upstreams.srv
pki
tls
tls.certificates.automate
tls.certificates.load_files
tls.certificates.load_folders
tls.certificates.load_pem
tls.certificates.load_storage
tls.client_auth.leaf
tls.get_certificate.http
tls.get_certificate.tailscale
tls.handshake_match.remote_ip
tls.handshake_match.sni
tls.issuance.acme
tls.issuance.internal
tls.issuance.zerossl
tls.stek.distributed
tls.stek.standard
Standard modules: 100
caddy.storage.redis
Non-standard modules: 1
Unknown modules: 0
b. Command:
To start - systemctl start caddy
To reload - systemctl reload caddy
$systemctl edit caddy
GNU nano 4.8 /etc/systemd/system/caddy.service.d/.#override.confd81dc227b1553c05
[Service]
ExecStart=
ExecReload=
ExecStart=/usr/local/bin/caddy run --environ --config /etc/caddy/Caddyfile
ExecReload=/usr/local/bin/caddy reload --config /etc/caddy/Caddyfile
c. Service/unit/compose file:
d. My complete Caddy config:
{
debug
admin 0.0.0.0:2019
storage redis {
host caddy-ssl-redis-cluster-0001-001.ab000000001.usw1.cache.amazonaws.com
port 6379
aes_key 21111111-911b-4114-a111-311111111117
module redis
password ""
db 0
key_prefix caddytls
value_prefix redis
timeout 5
tls_enabled false
tls_insecure true
}
log {
output file /logs/caddy-shared.log {
roll_size 20M
roll_keep 5
}
}
on_demand_tls {
ask http://api1:9005/
interval 1m
burst 10
}
}
bleem-api.peoplebrowser.com {
header Strict-Transport-Security "max-age=31536000"
encode zstd gzip
tls {
issuer zerossl akeywithnumbers {
email contact@example.com
}
}
reverse_proxy 172.31.43.231:1210
}
5. Links to relevant resources:
Howdy! I am hoping to have a central Elasticache Redis database store domain certificates for my 80k+ domains. My plan is to utilize auto-scaling with multiple caddy servers running as any given time. They will all be serving the same domains and therefore I need shared storage. I think I’ve exhausted my Google Fu and sure would appreciate some help.
Prior to attempting to change storage to Redis, Caddy did work and certificates were generated as needed with storage set like this:
storage file_system {
root /mnt
}
I