Using existing LetsEncrypt certificates generated by Certbot for startup and then automatically generating new ones

[pk044 (Paweł Kiersznowski)]

1. The problem I’m having:

I’m migrating from nginx to Caddy and I noticed that Caddy has a few seconds of downtime on startup when it’s trying to generate the certificates. I want to avoid the downtime by loading existing certificates and letting Caddy generate new ones while still distributing the HTTP traffic to backend servers, but I’m unsure how to approach this. My first idea was to:

1. Add auto_https ignore_loaded_certs to global config

2. add the following line to $API_DOMAINS block:

   tls /etc/caddy/certs/cert.pem /etc/caddy/certs/privkey.pem

3. Build a Docker image with existing certificates

4. Run Caddy and let the certificates generate and upload it to S3 so it can be reused on the next run.

So far it managed to load the certificates and it runs well, but the certification renewal was not triggered - I guess this would make sense if there’s no need to do it yet, but I’d rather make 100% sure the process goes well once it’s deployed to production. Is there any way to force the certification renewal anyhow?

2. Error messages and/or full log output:

3. Caddy version:

v2.10.0

4. How I installed and ran Caddy:

I’m running Caddy inside Docker.

a. System environment:

Linux, Caddy in Docker

b. Command:

c. Service/unit/compose file:

d. My complete Caddy config:

{
	storage s3 {
		host "REDACTED"
		bucket "REDACTED"
		access_id "REDACTED"
		secret_key "REDACTED"
		prefix "ssl"
		insecure false #disables SSL if true
	}
    email {$EMAIL}
}

{$API_DOMAINS} {
    tls /etc/caddy/certs/cert.pem /etc/caddy/certs/privkey.pem
	handle_path /api/* {
		@allowed_origin {
			header Origin REDACTED
		}

		@preflight {
			method OPTIONS
			header Origin REDACTED
		}

		header @allowed_origin Access-Control-Allow-Origin "REDACTED"
		header @allowed_origin Access-Control-Allow-Methods "GET, POST, PUT, DELETE, OPTIONS"
		header @allowed_origin Access-Control-Allow-Headers "Content-Type,Accept,X-API-KEY,X-API-SECRET,Origin"

		header @preflight Access-Control-Max-Age "1728000"
		header @preflight Content-Type "text/plain; charset=UTF-8"
		header @preflight Content-Length "0"

		respond @preflight 204

		reverse_proxy http://REDACTED:9001 {
			header_up Host {http.request.host}
		}
	}

	handle /* {
		reverse_proxy http://REDACTED {
			header_up Host {http.request.host}
		}
	}
	handle /taxis/ws {
		reverse_proxy http://REDACTED {
			header_up Connection Upgrade
			header_up Upgrade websocket
		}
	}
}

{$ADMIN_DOMAIN} {
	handle_path /api/* {
		reverse_proxy http://REDACTED {
			header_up Host {http.request.host}
			transport http {
				response_header_timeout 20s
			}
		}
	}

	handle /* {
		reverse_proxy http://admin-dashboard:5000 {
		}
	}
}

5. Links to relevant resources:

[timelordx]

I can’t help with what you’re trying to do, but how about persisting the certificates between container restarts? You could mount persistent storage for the data folder or use something like Redis to store the certificates. That way the very first start might be a bit delayed, but subsequent restarts wouldn’t be.

[Mohammed90 (Mohammed Al Sahaf)]

If you’re using filesystem as the backing storage for Caddy, you’ll have to replicate the directory structure that Caddy generates to track the certificates. It also includes a metadata JSON file. Successful migration of the form you’re describing requires replicating both directory structure and the metadata JSON file.

[pk044 (Paweł Kiersznowski)]

Is there any guide how to do that? there is some data in that JSON that I’m not sure how should be generated -e.g. _uniqueIdentifier, url, the LetsEncrypt account, etc.

[Mohammed90 (Mohammed Al Sahaf)]

From my quick search, certbot doesn’t seem to store or expose the email address of the LE account on disk. I could be wrong. Nonetheless, Caddy directory structure is like this:

caddy
    ├── acme
    │   ├── acme-v02.api.letsencrypt.org-directory
    │   │   ├── challenge_tokens
    │   │   └── users
    │   │       └── default
    │   │           ├── default.json
    │   │           └── default.key
    ├── certificates
    │   ├── acme-v02.api.letsencrypt.org-directory
    │   │   ├── www.example.com
    │   │   │   ├── www.example.com.crt
    │   │   │   ├── www.example.com.json
    │   │   │   └── www.example.com.key

The default is used when no email address is used, which is convenient. Certbot stores the private key in JWK format, while Caddy stores it as PEM, so you’ll have to convert that. The default.json contains this structure:

{
	"status": "valid",
	"contact": [],
	"termsOfServiceAgreed": true,
	"orders": "",
	"location": "https://acme-v02.api.letsencrypt.org/acme/acct/<account-id>"
}

The www.example.com.json contains a structure of this format:

{
	"sans": [
		"www.example.com"
	],
	"issuer_data": {
		"url": "https://acme-v02.api.letsencrypt.org/acme/cert/<ID>",
		"ca": "https://acme-v02.api.letsencrypt.org/directory"
	}
}

[pk044 (Paweł Kiersznowski)]

How do you get the certificate ID though? I tried to use the ID from the Certbot directory structure and paste it at the end of the url from issuer_data, but I get a 404 - while the one from Caddy works fine.

[Mohammed90 (Mohammed Al Sahaf)]

You’ll have to ask certbot about this. We get it from Let’s Encrypt and store it in the JSON. They could be storing somewhere else.