Caddy cert expired yet still being served. Now caddy service (systemd) won't restart

Hi everyone.

So short version: I went to my site today (https://www.calhoun.io) and saw the ominous “Your connection is not private” message in Chrome. It looks like my certain expires March 12 (today) and was never updated. I am trying to figure out:

A. Why my SSL cert was allowed to expire in the first place
B. How to get my server back up and running with a new SSL cert - I managed to break my systemd service for caddy (it wont start up again) and I’m unsure what is going on, so any ideas are welcome as I dig into it.
C. Any issues that might be in my systemd service file or my caddyfile that might be causing me issues.

To attempt to debug the issue, I have tried:

  1. Restarting the caddy systemd service (shown below). This didn’t work and caddy failed to restart at this point taking my site down.
  2. Updating caddy (I believe I was on 0.9.4, I am now on 0.9.5). No help so far.
  3. Manually running caddy from within an ssh session and updating the Caddyfile to point to the old SSL certs. This will get things running, but the certs are expired so still problematic. I mostly wanted to get this working to verify that the certs did indeed expire today (they did).

I’m happy to provide any info that might help. My Caddyfile and systemd service file are below, as well as the status of my caddy service now in case it helps

I am using systemd on Ubuntu 16.04 with a caddy.service file that looks like…

[Unit]
Description=Caddy HTTP/2 web server
Documentation=https://caddyserver.com/docs
After=network.target

[Service]
User=root

Restart=on-failure
StartLimitInterval=86400
StartLimitBurst=5

; Limit the number of file descriptors; see `man systemd.exec` for more limit settings.
LimitNOFILE=1048576
; Unmodified caddy is not expected to use more than that.
LimitNPROC=64

; Environment=CADDYPATH=/etc/caddy
WorkingDirectory=/var/www
ExecStart=/usr/local/bin/caddy -agree -email my@email.com
; StartLimitInterval=600

[Install]
WantedBy=multi-user.target

My caddyfile looks like…

www.calhoun.io/samples {
        root /var/www/samples
        gzip
        log /var/www/caddy.log
}

www.calhoun.io/beta {
        root /var/www/beta
        gzip
        log /var/www/caddy.log
}

www.calhoun.io/static {
        root /var/www/static
        gzip
        log /var/www/caddy.log
}

www.calhoun.io {
  proxy / localhost:2368 {
    transparent
  }
}

Results from systemctl status caddy.service are:

● caddy.service - Caddy HTTP/2 web server
   Loaded: loaded (/etc/systemd/system/caddy.service; enabled; vendor preset: enabled)
   Active: inactive (dead) (Result: exit-code) since Sun 2017-03-12 21:28:40 UTC; 1h 19min ago
     Docs: https://caddyserver.com/docs
  Process: 7363 ExecStart=/usr/local/bin/caddy -agree -email jon@calhoun.io (code=exited, status=1/FAILURE)
 Main PID: 7363 (code=exited, status=1/FAILURE)

Mar 12 22:30:42 temp2 systemd[1]: caddy.service: Start request repeated too quickly.
Mar 12 22:30:42 temp2 systemd[1]: Failed to start Caddy HTTP/2 web server.
Mar 12 22:41:47 temp2 systemd[1]: caddy.service: Start request repeated too quickly.
Mar 12 22:41:47 temp2 systemd[1]: Failed to start Caddy HTTP/2 web server.
Mar 12 22:45:08 temp2 systemd[1]: caddy.service: Start request repeated too quickly.
Mar 12 22:45:08 temp2 systemd[1]: Failed to start Caddy HTTP/2 web server.
Mar 12 22:45:19 temp2 systemd[1]: caddy.service: Start request repeated too quickly.
Mar 12 22:45:19 temp2 systemd[1]: Failed to start Caddy HTTP/2 web server.
Mar 12 22:46:47 temp2 systemd[1]: caddy.service: Start request repeated too quickly.
Mar 12 22:46:47 temp2 systemd[1]: Failed to start Caddy HTTP/2 web server.

Thanks so much for any help!

What’s in Caddy’s logs though? (You’re not running with the -log option which you’ll have to do to tell Caddy where to log to.)

When I attempt to restart caddy manually I get this:

Activating privacy features...2017/03/12 22:56:58 get directory at 'https://acme-v01.api.letsencrypt.org/directory': failed to get "https://acme-v01.api.letsencrypt.org/directory": Get https://acme-v01.api.letsencrypt.org/directory: x509: failed to load system roots and no roots provided

When I attempt to curl that URL I see:

curl: (77) Problem with the SSL CA cert (path? access rights?)

So I’m digging into that first.

:thinking: So your system doesn’t have a root store or your process don’t have permission to access it… The good news is that isn’t a Caddy problem. :wink: Make sure your system has a root certificate store.

Yeah, I really shouldn’t have posted here but it wasn’t until I typed things out (and forced myself to stop and think about them) that it started to click. Site is back and now I just need to fix up the systemd stuff.

For anyone in the future - I am 99% sure what happened is when I migrated my server things worked at first because I copied over my old SSL certs and caddy was using them. Once they expired, caddy wasn’t able to get a new ssl cert because it didn’t have a root cert store. To fix this I ran update-ca-certificates and I was good to go.

2 Likes

This topic was automatically closed 90 days after the last reply. New replies are no longer allowed.