TLS handshake error: Can't obtain lock, it still held by other

I’m sorry, I have to redact the domain as it’s a client and I don’t have permission to share it. In this case I don’t think it will matter - it’s a subdomain on a domain for which I have many other subdomains running successfully through the same caddy reverse proxy. If that’s an issue though, I understand.

1. Caddy version (caddy version):

2.4.2

2. How I run Caddy:

Caddy API as a systemd service

a. System environment:

Ubuntu 20.04

b. Command:

service caddy-api start

c. Service/unit/compose file:

N/A

d. My complete Caddyfile or JSON config:

{
   "admin": {
      "disabled": false,
      "listen": "localhost:2019"
   },
   "apps": {
      "http": {
         "servers": {
            "srv0": {
               "listen": [
                  ":80",
                  ":443"
               ],
               "routes": [
                  {
                     "handle": [
                        {
                           "encodings": {
                              "gzip": {}
                           },
                           "handler": "encode"
                        },
                        {
                           "handler": "reverse_proxy",
                           "headers": {
                              "request": {
                                 "set": {
                                    "Host": [
                                       "clientsubdomain.clientdomain.com"
                                    ],
                                    "X-Forwarded-Proto": [
                                       "https"
                                    ],
                                    "apx-incoming-host": [
                                       "clientsubdomain.clientdomain.com"
                                    ]
                                 }
                              },
                              "response": {
                                 "set": {
                                    "apx-hit": [
                                       "true"
                                    ]
                                 }
                              }
                           },
                           "transport": {
                              "protocol": "http",
                              "tls": {}
                           },
                           "upstreams": [
                              {
                                 "dial": "helloworld.vercel.app:443"
                              }
                           ]
                        }
                     ],
                     "match": [
                        {
                           "host": [
                              "clientsubdomain.clientdomain.com"
                           ]
                        }
                     ],
                     "terminal": true,
                     "@id": "vhost_1012"
                  }
               ]
            }
         }
      },
      "tls": {
         "automation": {
            "on_demand": {
               "ask": "https://approximated.app/check-domain"
            }
         }
      }
   },
   "logging": {
      "logs": {
         "default": {
            "level": "DEBUG"
         }
      }
   }
}

3. The problem I’m having:

A domain using on-demand TLS is failing to acquire a lock from the redis storage. This is for a cluster of caddy servers distributed globally, sharing storage through the redis plugin. It seems stuck with the lock being in use by “other”, but I’m not sure why or how it got into that situation. This might be specific to the redis storage plugin, so I understand if this isn’t the right place. I’m hoping someone has just run across this before.

4. Error messages and/or full log output:

I’m sorry, I have to redact the domain name here, I don’t have permission to share it. I can say that the target address works okay if connected to directly, and that it’s a subdomain on a domain which has many other subdomains being served successfully through caddy’s reverse proxy. I understand if you can’t help without that info.

{"level":"debug","ts":1642369706.8329415,"logger":"http.stdlib","msg":"http: TLS handshake error from <ip_address>:<port>: unable to acquire lock 'issue_cert_clientsubdomain.clientdomain.com: can't obtain lock, it still being held by other, redislock: not obtained"}

5. What I already tried:

I’ve tried restarting the caddy server, restarting the redis server, and deleting and re-adding the domain to the caddy configuration. My guess is that I need to remove something in redis make it re-acquire or just get a fresh cert, but I’m not confident in what entry that might be.

6. Links to relevant resources:

That config doesn’t seem right. Where is the redis storage configured?

In any case, it might be a bug in the redis plugin.

Please upgrade to v2.4.6.

You probably don’t need these two header sets for your proxy, btw. Caddy sets the Host and X-Forwarded-Proto automatically already (documented in the Caddyfile docs):

Hmm. The lock in redis is meant to only last 10 seconds as far as I can tell, so it shouldn’t continue to be stuck

:thinking:

This topic was automatically closed after 30 days. New replies are no longer allowed.