Expired certificate not renewing (solved)

Help
1

G’day guys,

I use manager.io as my accounting package, it is an awesome cross-platform package that I’ve been using for about 2 years. At some point in that time, I set up Caddy to facilitate SSL and get rid of the ugly “You connection is not secure” messages from my browser.

The Caddy setup was provided by the developers of manager and has operated without problem until I recently made some (momentary, but extended) changes to my firewall (port forwardings) for testing other services.

During the change/outage the certificate expired. It was a number of days before I realised this as I don’t use the software every day.

Upon the next time I used the software I was greeted with the insecure page message. I realised my error and re-forwarded the ports back to Caddy and manager. I mostly use the software internally and so port forwarding is not really necessary. However, I want a clean approach to using it without browser warnings.

In researching this over the last few days, I have learned about ACME and custom setups for internal domains and that will be my next adventure (ie, getting that to work), but for the moment, I’m trying to learn what it is that I’ve done wrong, what it is I need to do to fix it, and remedy the situation.

This has been for several days now (I thought it may be an issue that needed time to resolve itself, but that’s not the case). I’m now reaching out for any advice that can help me fix this.

I’m thinking I’m obviously missing something pretty simple, because I’m not finding any references to anyone else having problems with expired certificates not renewing as caddy takes care of this well in advance.

explain what you are trying to do,

The certificate for my site has expired and upon reverting everything to normal (ie re-opening the ports) the certificate will not renew.

show what you have already tried,

  • I have stopped and restarted Caddy (I learned later that passing SIGUSR1 is the better way to reload the configuration, and I have done this as well to n avail). Caddy starts without an issue and reports no errors (that I can see)
  • Caddy is run as a service through systemctl and checking journalctl reveals nothing out of the ordinary (below).
  • I have stopped the service and run it from the command line, there is no errors appearing. I have logged errors to an external file, and it remains empty.
  • I have tried removing (renaming) the certificate file(s). I ended up pruning the whole .caddy subdirectory on the server.
  • The ports are definitely open as I can access the site from the internet

include error messages and log output,

There just aren’t any (that I can find)

~$ journalctl -xef -u caddy.service
Nov 02 05:25:09 mubology systemd[1]: Started caddy.service.
-- Subject: Unit caddy.service has finished start-up
-- Defined-By: systemd
-- Support: http://www.ubuntu.com/support
-- 
-- Unit caddy.service has finished starting up.
-- 
-- The start-up result is RESULT.
Nov 02 05:25:09 mubology caddy[6275]: Activating privacy features... done.
Nov 02 05:25:09 mubology caddy[6275]: https://
Nov 02 05:25:09 mubology caddy[6275]: http://
Nov 02 05:30:51 mubology systemd[1]: Stopping caddy.service...
-- Subject: Unit caddy.service has begun shutting down
-- Defined-By: systemd
-- Support: http://www.ubuntu.com/support
-- 
-- Unit caddy.service has begun shutting down.
Nov 02 05:30:51 mubology systemd[1]: Stopped caddy.service.
-- Subject: Unit caddy.service has finished shutting down
-- Defined-By: systemd
-- Support: http://www.ubuntu.com/support
-- 
-- Unit caddy.service has finished shutting down.
Nov 02 05:30:57 mubology systemd[1]: Started caddy.service.
-- Subject: Unit caddy.service has finished start-up
-- Defined-By: systemd
-- Support: http://www.ubuntu.com/support
-- 
-- Unit caddy.service has finished starting up.
-- 
-- The start-up result is RESULT.
Nov 02 05:30:57 mubology caddy[6355]: Activating privacy features... done.
Nov 02 05:30:57 mubology caddy[6355]: https://
Nov 02 05:30:57 mubology caddy[6355]: http://
^C

and link to any relevant resources.

The service is running with the following parameters:

~$ cat /etc/systemd/system/caddy.service 
[Unit]
After=network.target

[Service]
LimitNOFILE=1048576
ExecStart=/usr/local/bin/caddy -agree=true -conf=/usr/share/manager-server/caddy.conf -agree
Restart=on-failure
StartLimitInterval=600

[Install]
WantedBy=multi-user.target

a check via running processes:

~$ ps aux | grep caddy
...  /usr/local/bin/caddy -agree=true -conf=/usr/share/manager-server/caddy.conf -agree

caddy.conf contains:

:~$ cat /usr/share/manager-server/caddy.conf 
:443 {
  tls { max_certs 100 }
  proxy / localhost:8082 
  errors /var/log/caddy-err.log
  log /var/log/caddy-log.log
}

Downloading the page directly to the command line reveals the following (nothing really, except that the certificate expired a week ago):

~$ curl -Ikv https://<mywebsite>
* Rebuilt URL to: https://<mywebsite>
*   Trying 192.168.5.2...
* TCP_NODELAY set
* Connected to <mywebsite> (192.168.5.2) port 443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* successfully set certificate verify locations:
*   CAfile: /etc/ssl/certs/ca-certificates.crt
  CApath: /etc/ssl/certs
* TLSv1.2 (OUT), TLS handshake, Client hello (1):
* TLSv1.2 (IN), TLS handshake, Server hello (2):
* TLSv1.2 (IN), TLS handshake, Certificate (11):
* TLSv1.2 (IN), TLS handshake, Server key exchange (12):
* TLSv1.2 (IN), TLS handshake, Server finished (14):
* TLSv1.2 (OUT), TLS handshake, Client key exchange (16):
* TLSv1.2 (OUT), TLS change cipher, Client hello (1):
* TLSv1.2 (OUT), TLS handshake, Finished (20):
* TLSv1.2 (IN), TLS handshake, Finished (20):
* SSL connection using TLSv1.2 / ECDHE-RSA-AES256-GCM-SHA384
* ALPN, server accepted to use h2
* Server certificate:
*  subject: CN=<mywebsite>
*  start date: Jul 28 19:51:29 2018 GMT
*  expire date: Oct 26 19:51:29 2018 GMT
*  issuer: C=US; O=Let's Encrypt; CN=Let's Encrypt Authority X3
*  SSL certificate verify result: certificate has expired (10), continuing anyway.
* Using HTTP2, server supports multi-use
* Connection state changed (HTTP/2 confirmed)
* Copying HTTP/2 data in stream buffer to connection buffer after upgrade: len=0
* Using Stream ID: 1 (easy handle 0x55f5a29df8e0)
> HEAD / HTTP/2
> Host: <mywebsite>
> User-Agent: curl/7.58.0
> Accept: */*
> 
* Connection state changed (MAX_CONCURRENT_STREAMS updated)!
< HTTP/2 200 
HTTP/2 200 
< cache-control: no-store
cache-control: no-store
< date: Fri, 02 Nov 2018 23:22:44 GMT
date: Fri, 02 Nov 2018 23:22:44 GMT
< server: Caddy
server: Caddy
< server: Mono-HTTPAPI/1.0
server: Mono-HTTPAPI/1.0
< content-length: 0
content-length: 0

If there’s anything else I can post to help diagnose this, just let me know.

TIA

2

It could be running into the internal rate limit for On-Demand TLS acquisition.

Try restarting Caddy to reset the timer, then request your site immediately after.

3

I’ve done that a number of times, to no avail. I just restarted the service then and immediately tried refreshing the browser (the browser computer had just rebooted) and now I’m getting the infamous “The site can’t provide a secure connection” ERR_SSL_PROTOCOL_ERROR.

I haven’t changed anything (else) in the last 24 hours since posting here, I don’t know why I’m not seeing it at all now.

The caddy logs are empty, the site works on the port it’s running on (8082), but caddy is not forwarding anything now.

:~$ curl -Ikv https://<mywebsite>
* Rebuilt URL to: https://<mywebsite>/
*   Trying 192.168.5.2...
* TCP_NODELAY set
* Connected to <mywebsite> (192.168.5.2) port 443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* successfully set certificate verify locations:
*   CAfile: /etc/ssl/certs/ca-certificates.crt
  CApath: /etc/ssl/certs
* TLSv1.2 (OUT), TLS handshake, Client hello (1):
* TLSv1.2 (IN), TLS alert, Server hello (2):
* error:14094438:SSL routines:ssl3_read_bytes:tlsv1 alert internal error
* stopped the pause stream!
* Closing connection 0
curl: (35) error:14094438:SSL routines:ssl3_read_bytes:tlsv1 alert internal error

I have now even tried removing all instances of caddy, all folders .caddy and caddy from /usr/local/bin, root and user’s home directory, and did a reinstall from https://getcaddy.com via curl and bash.

I’m getting the same ERR_SSL_PROTOCOL_ERROR and the error logs and journalctl are empty.

I’ve done a portscan and https/443 is definitely open as I’m getting the same error both locally and externally.

I’m really at a loss.

4

Ok, clearing out everything has obviously helped. In creating my post yesterday and documenting everything, one thing I additionally did was remove the forwarding of port 80 (I’d forgotten port 80 needed to be forwarded as well).

I assume that’s why i got the protocol error today and once I’d opened it up, the site was available with a nice fresh certificate in tow.

Port 80 was open until yesterday, but it’s priority was lower and there was an older port forward to the old server.

So I see this is my fault through invalid port forwards, I also assume this is why it was happening that way before yesterday. TBH, I don’t recall exactly how it was setup yesterday, I deleted out a whole lot of rules.

Perhaps caddy could include something in the error log if the certificate process is faulting somewhere.

But I’m glad it working now. :smiley:

2 Likes
5

Glad it’s working now!

These kinds of events are entered in the process log (add the CLI flag -log /path/to/caddy.log to your start command).

https://caddyserver.com/docs/cli

6

Looking back to the original post, I mention the error logging that I’d set up there (to /var/log/caddy-err.log), but it has consistently been empty. In fact the only errors that ever appeared there were in relation to pages being served.

Does that (the CLI version) log a different set of errors compared to when setting it up in the configuration file?

If it is, I didn’t realise there would be a difference, and hence that would explain why I haven’t seen anything.

7

That’s the HTTP request error log. The process log is given by the -log flag which has information related to errors out-of-band of HTTP requests.

8

Excellent, at least now I know :smiley:

Thanks guys!

9

This topic was automatically closed 90 days after the last reply. New replies are no longer allowed.