Swapping certs between two servers?

Help
1

Say I have two boxes each running caddy, www.example.com at 1.1.1.1 and backup.example.com at 1.1.1.2.

I want to swap them so that the public server is on the other box. I can swap the IP addresses of the boxes to avoid DNS issues.

What’s the best way to avoid any Let’s Encrypt cert problems when I bring them back up? Can I just swap the current .caddy directories between the two boxes?

2

If you’re worried about certificate issues, there won’t be any as long as Caddy is accessible at the IP address specified by each of its managed domains. You don’t need to copy the .caddy directory at all (unless you’ve got more domains than LetsEncrypt’s rate limit).

Requisitioning a new set of certificates is pretty quick, and you avoid transmitting your private keys (even if you would have been transmitting them securely).

With that said, you can indeed copy the entire .caddy directory, and if it’s placed in the correct location with the correct permissions, Caddy will start using them instead of requesting new ones.

If it’s absolute zero downtime you require, though, here’s how I’d do it:

  1. sudo rsync -az [old-host]:/root/.caddy /root/.caddy on [new-host]
    (or other file path as appropriate)
  2. sudo rsync -az [old-host]:/etc/Caddyfile /etc/Caddyfile on [new-host]
    (or other file path as appropriate)
  3. Replace the caddyfile on [old-host] with this:
http:// {
  proxy / http://[new-host] {
    transparent
    websocket
  }
}
https:// {
  proxy / https://[new-host] {
    transparent
    websocket
  }
  tls {
    max_certs 1
  }
}
  1. Start Caddy on [new-host]
  2. pkill -SIGUSR1 caddy on [old-host], now all new requests to either host are served by [new-host]
  3. Change DNS records, propagation time is now irrelevant
  4. Take down [old-host] after a few days, or once its access logs dry up
6 Likes
3

Thank you so much, this is really helpful.

4

Run caddy on new host looks OK, all sites listed, no errors. Run caddy on old host looks OK, http:// and https:// running, no errors.
But when I access a hostname with Firefox, it says: “Secure Connection Failed” and “An error occurred during a connection to passchier.net. SSL received a record that exceeded the maximum permissible length. Error code: SSL_ERROR_RX_RECORD_TOO_LONG”. Nothing appears in the logs both on old and new VPS.

5

Aaargh! Now all my sites are down, because I had to switch back to my old VPS and didn’t shut the new one down in time, and caddy refuses to run due to letsencrypt rate limit exceeded…

How long does that last?? Is there anything that can be done?

EDIT: Apparently, that doesn’t last very long, less than an hour for sure.

6

Did you copy the .caddy folder across? How did you get rate limited?

That error looks like Caddy didn’t pick up the certificates on the old VM. That might be my oversight; I think there might be no reason for old-host Caddy to load them from disk.

Try with

:443 {
  proxy / https://[new-host] {
    transparent
    websocket
  }
  tls {
    max_certs 1
  }
}

perhaps Caddy will check on disk for each request and load the required cert.

7

I copied the .caddy folder and the Caddyfile. The issue was that the proxy was not working. (And is still not working.)

When both instances were running with the old/full Caddyfile is when letsencrypt started to complain.

It seems like this proxy setup is not working… On the old VPS I now have:

:80 {
  proxy / http://45.58.49.179 {
    transparent
    websocket
  }
}
:443 {
  proxy / https://45.58.49.179 {
    transparent
    websocket
  }
  tls {
    max_certs 1
  }
}
``
8

LetsEncrypt should not complain unless you’re spinning up sites you don’t have certificates for. Which host are you getting errors on?

9

I switched the old host back to the proxy config, and within an hour letsencrypt started working again and I could run caddy on the new host again.

Do you mean I should be able to run the exact same config on 2 totally different hosts (different location etc.)?

I am more concerned with finding a way to run a successful proxy. When I go to passchier.net it doesn’t work (here).

10

No. One host should have the normal configuration, with all the applicable sites; the other should proxy all requests to the host with the normal configuration.

Both sites should have the complete set of ACME files before attempting this, so LetsEncrypt shouldn’t be contacted at any point (unless those certificates are under 30 days).

11

Spitballing, try again with this:

http:// {
  proxy / http://[new-host] {
    transparent
    websocket
  }
}
https:// {
  proxy / https://[new-host] {
    transparent
    websocket
  }
  tls {
    max_certs 1
  }
}
12

OK, I get a 502 now, nothing logged on the new host or the old host. Added logging to the old host, and it logs: “[ERROR 502 /] x509: cannot validate certificate for 45.58.49.179 because it doesn’t contain any IP SANs”.

EDIT: Solved it. It needs a hostname and not an IP address. As soon as I put the hostname in, it started to work.

13

Good to hear! Does it work for all sites you’re running?

14

Yes, it all seems to work; I’ve changed the A pointers of all the domains too now, but still getting traffic on the proxy. But that works, the key was using a (any?) domainname that resolves to the intended IP address. So no IP…! (I did that because I didn’t have a domain name resolving to the new IP yet…)

Thanks for your faithful help as always, Matthew!

1 Like