I’m sorry, I have to redact the domain as it’s a client and I don’t have permission to share it. In this case I don’t think it will matter - it’s a subdomain on a domain for which I have many other subdomains running successfully through the same caddy reverse proxy. If that’s an issue though, I understand.
1. Caddy version (caddy version
):
2.4.2
2. How I run Caddy:
Caddy API as a systemd service
a. System environment:
Ubuntu 20.04
b. Command:
service caddy-api start
c. Service/unit/compose file:
N/A
d. My complete Caddyfile or JSON config:
{
"admin": {
"disabled": false,
"listen": "localhost:2019"
},
"apps": {
"http": {
"servers": {
"srv0": {
"listen": [
":80",
":443"
],
"routes": [
{
"handle": [
{
"encodings": {
"gzip": {}
},
"handler": "encode"
},
{
"handler": "reverse_proxy",
"headers": {
"request": {
"set": {
"Host": [
"clientsubdomain.clientdomain.com"
],
"X-Forwarded-Proto": [
"https"
],
"apx-incoming-host": [
"clientsubdomain.clientdomain.com"
]
}
},
"response": {
"set": {
"apx-hit": [
"true"
]
}
}
},
"transport": {
"protocol": "http",
"tls": {}
},
"upstreams": [
{
"dial": "helloworld.vercel.app:443"
}
]
}
],
"match": [
{
"host": [
"clientsubdomain.clientdomain.com"
]
}
],
"terminal": true,
"@id": "vhost_1012"
}
]
}
}
},
"tls": {
"automation": {
"on_demand": {
"ask": "https://approximated.app/check-domain"
}
}
}
},
"logging": {
"logs": {
"default": {
"level": "DEBUG"
}
}
}
}
3. The problem I’m having:
A domain using on-demand TLS is failing to acquire a lock from the redis storage. This is for a cluster of caddy servers distributed globally, sharing storage through the redis plugin. It seems stuck with the lock being in use by “other”, but I’m not sure why or how it got into that situation. This might be specific to the redis storage plugin, so I understand if this isn’t the right place. I’m hoping someone has just run across this before.
4. Error messages and/or full log output:
I’m sorry, I have to redact the domain name here, I don’t have permission to share it. I can say that the target address works okay if connected to directly, and that it’s a subdomain on a domain which has many other subdomains being served successfully through caddy’s reverse proxy. I understand if you can’t help without that info.
{"level":"debug","ts":1642369706.8329415,"logger":"http.stdlib","msg":"http: TLS handshake error from <ip_address>:<port>: unable to acquire lock 'issue_cert_clientsubdomain.clientdomain.com: can't obtain lock, it still being held by other, redislock: not obtained"}
5. What I already tried:
I’ve tried restarting the caddy server, restarting the redis server, and deleting and re-adding the domain to the caddy configuration. My guess is that I need to remove something in redis make it re-acquire or just get a fresh cert, but I’m not confident in what entry that might be.