Seeking Assistance to Recover a Private Key

1. The problem I’m having:

I accidentally pinned my device’s root certificate to the Let’s Encrypt R3 certificate, which has now been retired. My Caddy server automatically updates my service certificates 30 days in advance, resulting in the intermediate certificate changing from R3 to E6. I also noticed that it updated the certificate’s private key. Is there any way I can revert to the old certificate (R3)?

2. Error messages and/or full log output:

Jun 25 12:09:16 VM-0-5-ubuntu caddy[1157318]: {"level":"info","ts":1719288556.2157938,"logger":"tls.cache.maintenance","msg":"certificate expires soon; queuing for renewal","identifiers":["broker.opoiot.com"],"remaining":2591401.784206366}
Jun 25 12:09:16 VM-0-5-ubuntu caddy[1157318]: {"level":"info","ts":1719288556.2158365,"logger":"tls.cache.maintenance","msg":"attempting certificate renewal","identifiers":["broker.opoiot.com"],"remaining":2591401.784167772}
Jun 25 12:09:16 VM-0-5-ubuntu caddy[1157318]: {"level":"info","ts":1719288556.2160573,"logger":"tls.renew","msg":"acquiring lock","identifier":"broker.opoiot.com"}
Jun 25 12:09:16 VM-0-5-ubuntu caddy[1157318]: {"level":"info","ts":1719288556.2195597,"logger":"tls.renew","msg":"lock acquired","identifier":"broker.opoiot.com"}
Jun 25 12:09:16 VM-0-5-ubuntu caddy[1157318]: {"level":"info","ts":1719288556.219896,"logger":"tls.renew","msg":"renewing certificate","identifier":"broker.opoiot.com","remaining":2591401.78010481}
Jun 25 12:09:16 VM-0-5-ubuntu caddy[1157318]: {"level":"info","ts":1719288556.220277,"logger":"tls.issuance.acme","msg":"waiting on internal rate limiter","identifiers":["broker.opoiot.com"],"ca":"https://acme-v02.api.letsencrypt.org/directory","account":"fake@email.com"}
Jun 25 12:09:16 VM-0-5-ubuntu caddy[1157318]: {"level":"info","ts":1719288556.2203114,"logger":"tls.issuance.acme","msg":"done waiting on internal rate limiter","identifiers":["broker.opoiot.com"],"ca":"https://acme-v02.api.letsencrypt.org/directory","account":"fake@email.com"}
Jun 25 12:09:16 VM-0-5-ubuntu caddy[1157318]: {"level":"info","ts":1719288556.6542556,"logger":"tls.issuance.acme.acme_client","msg":"trying to solve challenge","identifier":"broker.opoiot.com","challenge_type":"tls-alpn-01","ca":"https://acme-v02.api.letsencrypt.org/directory"}
Jun 25 12:09:17 VM-0-5-ubuntu caddy[1157318]: {"level":"info","ts":1719288557.3126352,"logger":"tls","msg":"served key authentication certificate","server_name":"broker.opoiot.com","challenge":"tls-alpn-01","remote":"23.178.112.201:45339","distributed":false}
Jun 25 12:09:18 VM-0-5-ubuntu caddy[1157318]: {"level":"info","ts":1719288558.0345423,"logger":"tls","msg":"served key authentication certificate","server_name":"broker.opoiot.com","challenge":"tls-alpn-01","remote":"54.184.134.91:24844","distributed":false}
Jun 25 12:09:18 VM-0-5-ubuntu caddy[1157318]: {"level":"info","ts":1719288558.042857,"logger":"tls","msg":"served key authentication certificate","server_name":"broker.opoiot.com","challenge":"tls-alpn-01","remote":"3.22.216.236:11036","distributed":false}
Jun 25 12:09:18 VM-0-5-ubuntu caddy[1157318]: {"level":"info","ts":1719288558.281167,"logger":"tls","msg":"served key authentication certificate","server_name":"broker.opoiot.com","challenge":"tls-alpn-01","remote":"16.16.24.15:16186","distributed":false}
Jun 25 12:09:18 VM-0-5-ubuntu caddy[1157318]: {"level":"info","ts":1719288558.5890107,"logger":"tls","msg":"served key authentication certificate","server_name":"broker.opoiot.com","challenge":"tls-alpn-01","remote":"47.129.61.162:28932","distributed":false}
Jun 25 12:09:18 VM-0-5-ubuntu caddy[1157318]: {"level":"info","ts":1719288558.8514256,"logger":"tls.issuance.acme.acme_client","msg":"authorization finalized","identifier":"broker.opoiot.com","authz_status":"valid"}
Jun 25 12:09:19 VM-0-5-ubuntu caddy[1157318]: {"level":"info","ts":1719288559.7634702,"logger":"tls.renew","msg":"certificate renewed successfully","identifier":"broker.opoiot.com"}
Jun 25 12:09:19 VM-0-5-ubuntu caddy[1157318]: {"level":"info","ts":1719288559.763582,"logger":"tls.renew","msg":"releasing lock","identifier":"broker.opoiot.com"}
Jun 25 12:09:19 VM-0-5-ubuntu caddy[1157318]: {"level":"info","ts":1719288559.763656,"logger":"tls","msg":"reloading managed certificate","identifiers":["broker.opoiot.com"]}
Jun 25 12:09:19 VM-0-5-ubuntu caddy[1157318]: {"level":"info","ts":1719288559.8399413,"logger":"tls.cache","msg":"replaced certificate in cache","subjects":["broker.opoiot.com"],"new_expiration":1727060959}

3. Caddy version:

v2.7.6

4. How I installed and ran Caddy:

apt and systemd

a. System environment:

Ubuntu 22.04.4 LTS

b. Command:

sudo systemctl start caddy

c. Service/unit/compose file:

# caddy.service
#
# For using Caddy with a config file.
#
# Make sure the ExecStart and ExecReload commands are correct
# for your installation.
#
# See https://caddyserver.com/docs/install for instructions.
#
# WARNING: This service does not use the --resume flag, so if you
# use the API to make changes, they will be overwritten by the
# Caddyfile next time the service is restarted. If you intend to
# use Caddy's API to configure it, add the --resume flag to the
# `caddy run` command or use the caddy-api.service file instead.

[Unit]
Description=Caddy
Documentation=https://caddyserver.com/docs/
After=network.target network-online.target
Requires=network-online.target

[Service]
Type=notify
User=caddy
Group=caddy
ExecStart=/usr/bin/caddy run --environ --config /etc/caddy/Caddyfile
ExecReload=/usr/bin/caddy reload --config /etc/caddy/Caddyfile --force
TimeoutStopSec=5s
LimitNOFILE=1048576
LimitNPROC=512
PrivateTmp=true
ProtectSystem=full
AmbientCapabilities=CAP_NET_ADMIN CAP_NET_BIND_SERVICE

[Install]
WantedBy=multi-user.target

d. My complete Caddy config:

broker.opoiot.com, broker.opoiot.com:18888 {
  @broker_wss {
    path /mqttwss
    header Connection *Upgrade*
    header Upgrade websocket
    header Sec-Websocket-Protocol mqtt
  }
  route {
    reverse_proxy @broker_wss localhost:8083 {
      rewrite /mqtt
    }
    respond "Sorry, we don't provide services in this interface" 404
  }
  log {
    output file /var/log/caddy/emqx_broker.log
  }
}

5. Links to relevant resources:

1 Like

Sorry to hear about that. Caddy doesn’t store old keys, it overwrites the files… so you’d have to use some file-system-level recovery (like a snapshot, if your FS supports it) or a backup if you have one.

I’ve been considering implementing a way to store old keys but this is the only valid use case I’ve run across so far that I can remember.

In general though, I strongly advise against key pinning. :sweat_smile:

1 Like

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.