Caddy tries to renew deleted hosts / certificates

We have Caddy with consul storage for certificates.

I have noticed in the logs that Caddy is trying to renew certificates which i have deleted from the Consul storage and from the Caddy config (updated via /load API).

{"level":"info","ts":1623858901.8527253,"logger":"tls.cache.maintenance","msg":"attempting certificate renewal","identifiers":["img.domain.eu"],"remaining":1440843.147277408}
{"level":"error","ts":1623861330.5041435,"logger":"tls.renew","msg":"will retry","error":"key caddytls/certificates/acme.zerossl.com-v2-dv90/img.domain.eu/img.domain.eu.crt does not exist","attempt":7,"retrying_in":1200,"elapsed":2428.2326516,"max_duration":2592000}

When caddy was started the domain WAS listed in tls.automation.policies.subjects.

After i have deleted the certificate from Consul storage i have updated the CADDY config via /load API and also removed the domain.

Even so that there is no reference to the deleted domain + certificate, caddy still tries to renew it.

Restarting caddy fixed it but it’s not something we want to do.
Any other suggested fix ?

Should i open an issue on Github ?

Could you please fill out the help topic template? We need to know what version you’re using, what platform, what your config looks like, etc.

1 Like

Yeah, it would be helpful if you could make it minimally reproducible, as I haven’t heard of this before. The steps to reproduce the bug would be greatly appreciated, or it’s mainly just guesswork. (A lot of times the problem ends up being in the user’s process or system, rather than in Caddy. For example, did all API requests complete with a 2xx status? Anyway, it’s not always user error, but we need to be sure it’s a bug in Caddy first.)

1. Caddy version (caddy version):

v2.4.2 h1:chB106RlsIaY4mVEyq9OQM5g/9lHYVputo/LAX2ndFg=

2. How I run Caddy:

I am running caddy on 2 separate Debian machines which both connect to the same Consul cluster for storage.

a. System environment:

Debian

b. Command:

caddy run -config /etc/caddyserver/caddy.json -pidfile /var/run/caddyserver.pid

d. My complete Caddyfile or JSON config:

{
  "admin": {
    "config": {
      "persist": true
    },
    "listen": "127.0.0.1:2019"
  },
  "apps": {
    "http": {
      "http_port": 80,
      "servers": {
        "nzm": {
          "listen": [
            "*:443"
          ],
          "logs": {
            "default_logger_name": "log0"
          },
          "routes": [
            {
              "handle": [
                {
                  "handler": "subroute",
                  "routes": [
                    {
                      "handle": [
                        {
                          "handler": "subroute",
                          "routes": [
                            {
                              "handle": [
                                {
                                  "error": "Not found",
                                  "handler": "error",
                                  "status_code": 404
                                }
                              ]
                            }
                          ]
                        }
                      ],
                      "match": [
                        {
                          "path": [
                            "/favicon.ico",
                            "/admin/*"
                          ]
                        }
                      ]
                    },
                    {
                      "handle": [
                        {
                          "handler": "reverse_proxy",
                          "headers": {
                            "request": {
                              "set": {
                                "Host": [
                                  "cdn.site.com"
                                ],
                                "X-Real-Host": [
                                  "{http.request.host}"
                                ],
                                "X-Real-Ip": [
                                  "{http.request.remote.host}"
                                ]
                              }
                            },
                            "response": {
                              "delete": [
                                "Server"
                              ]
                            }
                          },
                          "load_balancing": {
                            "selection_policy": {
                              "policy": "least_conn"
                            }
                          },
                          "upstreams": [
                            {
                              "dial": "127.0.0.1:90"
                            }
                          ]
                        }
                      ]
                    }
                  ]
                }
              ],
              "match": [
                {
                  "host": [
                    "cdn.domain.eu",
                    "cdn.domain.io",
                    "cdn.domain.ro"
                  ]
                }
              ],
              "terminal": true
            }
          ]
        }
      }
    },
    "tls": {
      "automation": {
        "policies": [
          {
            "disable_ocsp_stapling": true,
            "issuers": [
              {
                "challenges": {
                  "http": {
                    "disabled": true
                  }
                },
                "email": "catalin-caddy@domain.ro",
                "module": "acme"
              },
              {
                "challenges": {
                  "http": {
                    "disabled": true
                  }
                },
                "email": "catalin-caddy@domain.ro",
                "module": "zerossl"
              }
            ],
            "subjects": [
              "cdn.domain.io",
              "cdn.domain.ro",
              "cdn.domain.eu"
            ]
          }
        ]
      }
    }
  },
  "logging": {
    "logs": {
      "default": {
        "exclude": [
          "http.log.access.log0"
        ]
      },
      "log0": {
        "encoder": {
          "format": "console",
          "time_format": "iso8601"
        },
        "include": [
          "http.log.access.log0"
        ],
        "level": "DEBUG",
        "writer": {
          "filename": "/var/log/caddy.acces.log",
          "output": "file",
          "roll_keep": 5,
          "roll_keep_days": 30,
          "roll_size_mb": 954
        }
      }
    }
  },
  "storage": {
    "address": "localhost:8501",
    "aes_key": "",
    "module": "consul",
    "prefix": "caddytls",
    "tls_enabled": true
  }
}

API update goes through just fine most of the time. Sometimes it times out (i will open another topic for this too).

The API update just adds or removes hosts for which we need to get SSL certificates or remove SSL certificates and it’s ran every 5 minutes. (that is the only changing part). Adding entries in tls.automation.policies.subjects and in match.host route.

We have about 1000 sites for which we use caddy (just switched from single node traefik cause of the consul integration caddy has).

1 Like

That’s my first concern, if the API request doesn’t complete, I’m not sure there’s any guarantee the config change finishes. You can set grace_period to get this under control if you have lots of long-lived connections.

It does not look to be timeout related (it’s rare and subsequent config updates work fine). I have reduced the grace_period to allow for faster updates but the same happens.

I think the fow is the following:

  • server is started first time
  • checks the certificates to be renewed and adds them to a queue for SSL renewals
  • the config gets updated and some of the hosts in the renew cert queue are not there anymore (in the main config nor policies)
  • the tls.maintenance queue does not check the updated config to find out that it’s not required anymore for some hosts to renew certificates
{"level":"error","ts":1623861330.5041435,"logger":"tls.renew","msg":"will retry","error":"key caddytls/certificates/acme.zerossl.com-v2-dv90/img.domain.eu/img.domain.eu.crt does not exist","attempt":7,"retrying_in":1200,"elapsed":2428.2326516,"max_duration":2592000}

Here in tls.renew caddy notices that the cert is gone too (it was there on start since it detected that the certificate expired).

This topic was automatically closed after 30 days. New replies are no longer allowed.