ACME account is not regenerated when ACME server gets reinstalled

1. The problem I’m having:

I have two caddy instances configured as ACME server and ACME client. I discovered that the client renews his certificates by sending a ACME request containing an AccountID to the server. This AccountID can be found on the client unter “caddy/acme/localhost-7742-acme-device_ca-directory/users/default/default.json”.
For context:

  • the client possesses a local reverse proxy on localhost:7742 which adds mTLS encryption to the communication between the client and the server
  • the root CA on the server is called “device_ca”

On the server side, this AccountID could be found under “caddy/acme_server/device_ca/db”.

If the ACME server is now reinstalled and loses the database, the ACME certificate renewal requests from the client are no longer accepted by the server, which is what happened to me.
The result is a continuous error loop of “Account does not exist” on the client.

Is there a possibility to restart the ACME process again and create a new AccountID?

2. Error messages and/or full log output:

2024/01/26 09:22:34.336 ERROR   tls.obtain      could not get certificate from issuer   {"identifier": "10.33.41.102", "issuer": "localhost:7742-acme-device_ca-directory", "error": "HTTP 0 urn:ietf:params:acme:error:accountDoesNotExist - Account does not exist"}
2024/01/26 09:22:34.337 ERROR   tls.obtain      will retry      {"error": "[10.33.41.102] Obtain: [10.33.41.102] creating new order: attempt 1: https://localhost:7742/acme/device_ca/new-order: HTTP 0 urn:ietf:params:acme:error:accountDoesNotExist - Account does not exist (ca=https://localhost:7742/acme/device_ca/directory)", "attempt": 1, "retrying_in": 60, "elapsed": 2.38484625, "max_duration": 2592000} 

3. Caddy version:

On ACME server: v2.7.4 h1:J8nisjdOxnYHXlorUKXY75Gr6iBfudfoGhrJ8t7/flI=
On ACME client: v2.6.2 h1:wKoFIxpmOJLGl3QXoo6PNbYvGW4xLEgo32GPBEjWL8o=

4. How I installed and ran Caddy:

I have the binary installed on the device

a. System environment:

Debain 10 without systemd, arm64 architecture

b. Command:

/usr/bin/caddy run --pidfile /run/caddy-standard.pid --resume --environ --envfile /etc/caddy/caddy-standard.env```

d. My complete Caddy config:

ACME server:

{
  "admin": {
    "listen": "unix//run/caddy-standard.sock",
    "origins": [
      "localhost"
    ]
  },
  "apps": {
    "http": {
      "servers": {
        "acme_server": {
          "automatic_https": {
            "disable_redirects": true
          },
          "listen": [
            "localhost:7741"
          ],
          "routes": [
            {
              "handle": [
                {
                  "handler": "subroute",
                  "routes": [
                    {
                      "handle": [
                        {
                          "ca": "device_ca",
                          "handler": "acme_server"
                        }
                      ]
                    }
                  ]
                }
              ],
              "match": [
                {
                  "host": [
                    "localhost"
                  ]
                }
              ]
            }
          ]
        },
        "external-interfaces": {
          "automatic_https": {
            "disable_redirects": true
          },
          "listen": [
            "0.0.0.0:80",
            "0.0.0.0:443"
          ],
          "routes": [
            {
              "@id": "test",
              "handle": [
                {
                  "handler": "subroute",
                  "routes": [
                    {
                      "handle": [
                        {
                          "handler": "rate_limit",
                          "key": "{remote.ip}",
                          "rate": "1000r/m"
                        },
                        {
                          "handler": "reverse_proxy",
                          "upstreams": [
                            {
                              "dial": "localhost:3000"
                            }
                          ]
                        }
                      ],
                      "match": [
                        {
                          "path": [
                            "/test",
                            "/test/*"
                          ]
                        }
                      ]
                    },
                    {
                      "handle": [
                        {
                          "handler": "rate_limit",
                          "key": "{remote.ip}",
                          "rate": "20r/m"
                        },
                        {
                          "handler": "subroute",
                          "routes": [
                            {
                              "handle": [
                                {
                                  "handler": "reverse_proxy",
                                  "headers": {
                                    "request": {
                                      "set": {
                                        "Host": [
                                          "localhost:7742"
                                        ]
                                      }
                                    }
                                  },
                                  "transport": {
                                    "protocol": "http",
                                    "tls": {
                                      "insecure_skip_verify": true
                                    }
                                  },
                                  "upstreams": [
                                    {
                                      "dial": "localhost:7741"
                                    }
                                  ]
                                }
                              ],
                              "match": [
                                {
                                  "vars_regexp": {
                                    "{http.request.tls.client.subject}": {
                                      "pattern": "(.*CN=TEST.*)"
                                    }
                                  }
                                }
                              ]
                            }
                          ]
                        }
                      ],
                      "match": [
                        {
                          "path": [
                            "/acme/*"
                          ]
                        }
                      ]
                    }
                  ]
                }
              ]
            },
            {
              "@id": "askendpoint",
              "handle": [
                {
                  "handler": "static_response",
                  "status_code": "200"
                }
              ],
              "match": [
                {
                  "path": [
                    "/ask*"
                  ]
                }
              ]
            },
            {
              "@id": "device-certificates",
              "handle": [
                {
                  "handler": "subroute",
                  "routes": [
                    {
                      "handle": [
                        {
                          "handler": "static_response",
                          "headers": {
                            "Location": [
                              "/device-certificates/root-ca/"
                            ]
                          },
                          "status_code": "302"
                        }
                      ],
                      "match": [
                        {
                          "path": [
                            "/device-certificates/root-ca"
                          ]
                        }
                      ]
                    },
                    {
                      "handle": [
                        {
                          "handler": "subroute",
                          "routes": [
                            {
                              "handle": [
                                {
                                  "handler": "rewrite",
                                  "strip_path_prefix": "/device-certificates/root-ca"
                                }
                              ]
                            },
                            {
                              "handle": [
                                {
                                  "handler": "file_server",
                                  "hide": [
                                    "*.key"
                                  ],
                                  "index_names": [
                                    "root.crt"
                                  ],
                                  "root": "/home/test/apps/standard/config/caddy/pki/authorities/device_ca"
                                }
                              ]
                            }
                          ]
                        }
                      ],
                      "match": [
                        {
                          "path": [
                            "/device-certificates/root-ca/*"
                          ]
                        }
                      ]
                    }
                  ]
                }
              ]
            }
          ],
          "strict_sni_host": false,
          "tls_connection_policies": [
            {
              "client_authentication": {
                "mode": "verify_if_given",
                "trusted_ca_certs_pem_files": [
                  "/datafs/test/device-certificates/client.cert.pem"
                ]
              },
              "protocol_min": "tls1.2"
            }
          ]
        }
      }
    },
    "pki": {
      "certificate_authorities": {
        "device_ca": {
          "intermediate_common_name": "Standard Intermediate CA",
          "root_common_name": "Standard Root CA"
        }
      }
    },
    "tls": {
      "automation": {
        "on_demand": {
          "ask": "http://localhost/ask",
          "rate_limit": {
            "burst": 5,
            "interval": "1m"
          }
        },
        "policies": [
          {
            "issuers": [
              {
                "ca": "device_ca",
                "module": "internal"
              }
            ],
            "key_type": "rsa2048",
            "on_demand": true
          }
        ]
      }
    }
  },
  "logging": {
    "logs": {
      "default": {
        "encoder": {
          "format": "console"
        },
        "level": "info",
        "writer": {
          "filename": "/home/test/var/caddy/caddy-standard.log",
          "output": "file",
          "roll_keep": 3,
          "roll_size_mb": 5
        }
      }
    }
  }
}

ACME client:

{
  "admin": {
    "listen": "unix//run/caddy-standard.sock",
    "origins": [
      "localhost"
    ]
  },
  "apps": {
    "http": {
      "servers": {
        "acme-client-reverse-proxy": {
          "automatic_https": {
            "disable_redirects": true
          },
          "listen": [
            "localhost:7742"
          ],
          "routes": [
            {
              "handle": [
                {
                  "handler": "subroute",
                  "routes": [
                    {
                      "handle": [
                        {
                          "handler": "subroute",
                          "routes": [
                            {
                              "handle": [
                                {
                                  "handler": "reverse_proxy",
                                  "transport": {
                                    "protocol": "http",
                                    "tls": {
                                      "client_certificate_file": "/datafs/test/device-certificates/client.cert.pem",
                                      "client_certificate_key_file": "/datafs/test/device-certificates/client.key.pem",
                                      "root_ca_pem_files": [
                                        "/datafs/test/device-certificates/acme_root.crt"
                                      ]
                                    }
                                  },
                                  "upstreams": [
                                    {
                                      "dial": "192.168.0.100:443"
                                    }
                                  ]
                                }
                              ]
                            }
                          ]
                        }
                      ],
                      "match": [
                        {
                          "path": [
                            "/acme/*"
                          ]
                        }
                      ]
                    }
                  ]
                }
              ],
              "match": [
                {
                  "host": [
                    "localhost"
                  ]
                }
              ]
            }
          ],
          "tls_connection_policies": [
            {
              "certificate_selection": {
                "any_tag": [
                  "cert0"
                ]
              },
              "match": {
                "sni": [
                  "localhost"
                ]
              }
            }
          ]
        },
        "external-interfaces": {
          "automatic_https": {
            "disable_redirects": true
          },
          "listen": [
            "0.0.0.0:80",
            "192.168.0.101:443"
          ],
          "routes": [
            {
              "@id": "test",
              "handle": [
                {
                  "handler": "subroute",
                  "routes": [
                    {
                      "handle": [
                        {
                          "handler": "reverse_proxy",
                          "upstreams": [
                            {
                              "dial": "localhost:3000"
                            }
                          ]
                        }
                      ],
                      "match": [
                        {
                          "path": [
                            "/test",
                            "/test/*"
                          ]
                        }
                      ]
                    }
                  ]
                }
              ]
            }
          ],
          "tls_connection_policies": [
            {
              "cipher_suites": [
                "TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256",
                "TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256",
                "TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384",
                "TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384",
                "TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305_SHA256",
                "TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305_SHA256"
              ],
              "protocol_max": "tls1.3",
              "protocol_min": "tls1.2"
            }
          ]
        }
      }
    },
    "tls": {
      "automation": {
        "on_demand": {
          "rate_limit": {
            "burst": 5,
            "interval": "1m"
          }
        },
        "policies": [
          {
            "issuers": [
              {
                "ca": "https://localhost:7742/acme/device_ca/directory",
                "module": "acme",
                "trusted_roots_pem_files": [
                  "/datafs/test/device-certificates/client.cert.pem",
                  "/datafs/test/device-certificates/acme_root.crt"
                ]
              }
            ],
            "key_type": "rsa2048",
            "on_demand": true
          }
        ]
      },
      "certificates": {
        "load_files": [
          {
            "certificate": "/datafs/test/device-certificates/client.cert.pem",
            "key": "/datafs/test/device-certificates/client.key.pem",
            "tags": [
              "cert0"
            ]
          }
        ]
      }
    }
  },
  "logging": {
    "logs": {
      "default": {
        "encoder": {
          "format": "console"
        },
        "level": "info",
        "writer": {
          "filename": "/home/test/var/caddy/caddy-standard.log",
          "output": "file",
          "roll_keep": 3,
          "roll_size_mb": 5
        }
      }
    }
  }
}

5. Links to relevant resources:

1 Like

Pretty sure if you delete this folder and reload the client-caddy process, it’ll create a new account and ask for new certificates

As for this, I’ll have to double check the code-path. My understanding renewal is similar to first-request, but I have to check.

The question I have is, should the client not recover from this problem by itself? So if the server gives negative feedback for an account, try again to create a new account?

1 Like

Yeah, hence :slight_smile:

2 Likes

Trying to decide whether a new key should be used or reuse the same key.

In general, reusing keys is bad practice, but I can see where it’d be helpful in this case if a DB restoration is occurring, and you don’t want to duplicate accounts in the meantime. And the server losing the account doesn’t really say anything about the private key (whether it was compromised or whatever).

@marcsal @ypnos Would you be able to try the patch at

? I haven’t had a chance to test locally (requires a bit of set up).

2 Likes

Thank you @matt! We will test this and provide feedback as soon as possible.

1 Like

Awesome! Thank you!

I tried to recreate a scenario where the accountID did not match any of the accountIDs in the ACME server db, but unfortunateley caddy did not print out the error message or renew the accountID on the client.

1 Like

Hmm, I should probably at least add a log when this happens. I’ll do that today and that should give us a clue. :thinking:

@marcsal Ok, I’ve pushed a commit that adds a log when it happens, so now if you test it and there’s no log, we know that CertMagic isn’t picking up the missing account properly. Which would be very weird… but if we do see the log, then something about our recreation logic might be wrong.