Error verifying certs on upstream server after cert renewal

1. The problem I’m having:

I have a rather simple Caddy setup that takes care of the TLS handshake and then passes on to an upstream server. It uses https. I nenewed the certs on the upstream server, and they are all valid and up to date. But now my Caddy setup errors on verifying the certificate.

2. Error messages and/or full log output:

from Status Caddy which shows the error:
Jun 17 15:50:55 Dig-Ocean04-Caddy caddy[3028975]: {"level":"debug","ts":1718639455.2892745,"logger":"http.handlers.reverse_proxy","msg":"upstream roundtrip","upstream":"si>
Jun 17 15:50:55 Dig-Ocean04-Caddy caddy[3028975]: {"level":"error","ts":1718639455.2898178,"logger":"http.log.error","msg":"tls: failed to verify certificate: x509: certif>
Jun 17 15:50:59 Dig-Ocean04-Caddy caddy[3028975]: {"level":"debug","ts":1718639459.174295,"logger":"events","msg":"event","name":"tls_get_certificate","id":"f86656db-95fd->
Jun 17 15:50:59 Dig-Ocean04-Caddy caddy[3028975]: {"level":"debug","ts":1718639459.1752944,"logger":"tls.handshake","msg":"choosing certificate","identifier":"www.surethin>
Jun 17 15:50:59 Dig-Ocean04-Caddy caddy[3028975]: {"level":"debug","ts":1718639459.1755283,"logger":"tls.handshake","msg":"default certificate selection results","identifi>
Jun 17 15:50:59 Dig-Ocean04-Caddy caddy[3028975]: {"level":"debug","ts":1718639459.1756985,"logger":"tls.handshake","msg":"matched certificate in cache","remote_ip":"205.1>
Jun 17 15:51:00 Dig-Ocean04-Caddy caddy[3028975]: {"level":"debug","ts":1718639460.214579,"logger":"events","msg":"event","name":"tls_get_certificate","id":"0099d9a3-8f98->
Jun 17 15:51:00 Dig-Ocean04-Caddy caddy[3028975]: {"level":"debug","ts":1718639460.2154574,"logger":"tls.handshake","msg":"choosing certificate","identifier":"www.surethin>
Jun 17 15:51:00 Dig-Ocean04-Caddy caddy[3028975]: {"level":"debug","ts":1718639460.2156327,"logger":"tls.handshake","msg":"default certificate selection results","identifi>
Jun 17 15:51:00 Dig-Ocean04-Caddy caddy[3028975]: {"level":"debug","ts":1718639460.2157674,"logger":"tls.handshake","msg":"matched certificate in cache","remote_ip":"23.10>
From  curl -vL "https://surething.com":

*   Trying 159.223.195.50:443...
* Connected to surething.com (159.223.195.50) port 443
* ALPN: curl offers h2,http/1.1
* (304) (OUT), TLS handshake, Client hello (1):
*  CAfile: /etc/ssl/cert.pem
*  CApath: none
* (304) (IN), TLS handshake, Server hello (2):
* (304) (IN), TLS handshake, Unknown (8):
* (304) (IN), TLS handshake, Certificate (11):
* (304) (IN), TLS handshake, CERT verify (15):
* (304) (IN), TLS handshake, Finished (20):
* (304) (OUT), TLS handshake, Finished (20):
* SSL connection using TLSv1.3 / AEAD-CHACHA20-POLY1305-SHA256
* ALPN: server accepted h2
* Server certificate:
*  subject: CN=surething.com
*  start date: May 19 13:10:51 2024 GMT
*  expire date: Aug 17 13:10:50 2024 GMT
*  subjectAltName: host "surething.com" matched cert's "surething.com"
*  issuer: C=US; O=Let's Encrypt; CN=R3
*  SSL certificate verify ok.
* using HTTP/2
* [HTTP/2] [1] OPENED stream for https://surething.com/
* [HTTP/2] [1] [:method: GET]
* [HTTP/2] [1] [:scheme: https]
* [HTTP/2] [1] [:authority: surething.com]
* [HTTP/2] [1] [:path: /]
* [HTTP/2] [1] [user-agent: curl/8.4.0]
* [HTTP/2] [1] [accept: */*]
> GET / HTTP/2
> Host: surething.com
> User-Agent: curl/8.4.0
> Accept: */*
>
< HTTP/2 502
< alt-svc: h3=":443"; ma=2592000
< server: Caddy
< content-length: 0
< date: Mon, 17 Jun 2024 15:58:41 GMT
<
* Connection #0 to host surething.com left intact

3. Caddy version:

  • v2.7.5 h1:HoysvZkLcN2xJExEepaFHK92Qgs7xAiCFydN5x5Hs6Q=

4. How I installed and ran Caddy:

  • I installed Caddy on our Ubuntu server using the instructions for Debian, Ubuntu, Raspbian found on your site. Everything went quite smoothly and Caddy seems to be running as a service.

a. System environment:

  • 1 GB Memory / 25 GB Disk / SFO3 - Ubuntu 23.10 x64. Very plain Ubuntu server with nothing else installed. systemd yes, docker no.

b. Command:

Not currently using any commands, running Caddy as a service. Using a Caddyfile for config. ```


### c. Service/unit/compose file:
  • Using systemd but nothing else


### d. My complete Caddy config:

{
debug
}

https://surething.com, https://www.surething.com {
reverse_proxy https://sites.surething.com {
header_up Host {upstream_hostport}
}
}

https://esd.surething.com {
reverse_proxy https://dl.surething.com {
header_up Host {upstream_hostport}
}
}

https://downloads.surething.com, https://downloads2.surething.com {
reverse_proxy https://downloadz.surething.com {
header_up Host {upstream_hostport}
}
}



### 5. Links to relevant resources:
Do I need to clear any caches? regenerate certs on Caddy? Why fhe failure, I've renewed certs successfully on the upstream server before.

I noticed in my log that the line with the error got cut off at the end. The complete description of the error was:

tls: failed to verify certificate: x509: certificate signed by unknown authority

However, it is a valid LetsEncrypt wildcard certificate. Here is the cert:

![image|160x499](upload://aqdheyiT2e1vTSj3eP93lbWZQq2.jpeg)

Hope this makes the issue more clear, thanks!

Hi @cbad,

Sorry to inform you but the Server: Apache/2.4.10 (Ubuntu), not Caddy. :frowning:

Using the online tool Let’s Debug yields OK results.
https://letsdebug.net/surething.com/2039033?debug=y

However with curl Port 443 is serving HTTP not HTTPS

Try the ACME HTTP-01 Challenge of HTTP on the default Port of 80.
And it redirects us from surething.com to HTTPS www.surething.com

$ curl -Ii http://surething.com/.well-known/acme-challenge/sometestfile
HTTP/1.1 302 Found
Date: Tue, 18 Jun 2024 01:16:08 GMT
Server: Apache/2.4.10 (Ubuntu)
Location: https://www.surething.com/.well-known/acme-challenge/sometestfile
Content-Type: text/html; charset=iso-8859-1

Attempting HTTPS on the default Port of 443; FAILS!
Thus the ACME Challenge

$ curl -k -Ii  https://www.surething.com/.well-known/acme-challenge/sometestfile
curl: (35) error:0A000172:SSL routines::wrong signature type

Trying HTTP on the non-default Port of 443; and we get a response.

$ curl -k -Ii  http://www.surething.com:443/.well-known/acme-challenge/sometestfile
HTTP/1.1 400 Bad Request
Date: Tue, 18 Jun 2024 01:16:32 GMT
Server: Apache/2.4.10 (Ubuntu)
Content-Length: 448
Connection: close
Content-Type: text/html; charset=iso-8859-1

Thus there is an inconsistency that I have yet to delve into.

Edit:
And using the exact curl command you show I get

$ curl -vL "https://surething.com"
*   Trying 208.85.243.52:443...
* Connected to surething.com (208.85.243.52) port 443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
*  CAfile: /etc/ssl/certs/ca-certificates.crt
*  CApath: /etc/ssl/certs
* TLSv1.0 (OUT), TLS header, Certificate Status (22):
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* TLSv1.2 (IN), TLS header, Certificate Status (22):
* TLSv1.3 (IN), TLS handshake, Server hello (2):
* TLSv1.2 (IN), TLS header, Certificate Status (22):
* TLSv1.2 (IN), TLS handshake, Certificate (11):
* TLSv1.2 (OUT), TLS header, Unknown (21):
* TLSv1.2 (OUT), TLS alert, unknown CA (560):
* SSL certificate problem: unable to get local issuer certificate
* Closing connection 0
curl: (60) SSL certificate problem: unable to get local issuer certificate
More details here: https://curl.se/docs/sslcerts.html

curl failed to verify the legitimacy of the server and therefore could not
establish a secure connection to it. To learn more about this situation and
how to fix it, please visit the web page mentioned above.

This online tool https://unboundtest.com/ gets these results:
https://unboundtest.com/m/A/surething.com/QN3D52RU
showing an A Record of 208.85.243.52.

This is what would seem to be the DNS Record,
with an A Record of 208.85.243.52 also.

1 Like

@cbad and there seems to be certificate chain issue

Yet here show OK

1 Like

Thank you for looking at this! I’ll have to re-read it in its entirety, but I believe there is an explanation for the problems. I changed the A record for both surething.com and www.surething.com to point directly at our upstream server. So by the time you ran these test, it had been changed. I regretted doing this but I had to in order to keep our site running on all browsers except for Chrome (and maybe Edge).

Chrome doesn’t like the OpenSSL version on our upstream server and flags connections to it as unable to make a secure connection. Other browsers don’t do this. The weird thing is three domains/subdomain all pointing at that server. surething.com and www.surething.com both get flagged by chrome, yet sites.surething.com works no problem. Mystery to me why this is!

Anyway, Caddy solved this problem and was working perfectly for many months. Up until I renewed certs on the upstream server yesterday. Then I got the “tls: failed to verify certificate: x509: certificate signed by unknown authority” and can no loner connect. I really hope there is a solution, Caddy is the bomb!

1 Like

Hi @cbad,

Take what I’ve posted with a grain of salt since

At this point kindly wait for more knowledgeable Caddy community volunteers to assist.

Edit:
Still chain issue with sites.surething.com as shown with these:

1 Like

Thanks very much for all your efforts. I’ll take a look at those things that are still affecting sites.surething.com. Sorry for some of the confusion, but I couldn’t let the site be completely down for too long,

2 Likes

It now seems clear this is a problem with my newly renewed certs, and not Caddy per se. It is specifically a certificate chain issue that fails on most chain checking sites (but not all). I’m not sure, however, how to fix it.

One report says the chain doesn’t contain any intermediate certificates, another says incomplete, one shows the chain with the offending link, but it appears to be the certificate itself (R10). Caddy seems to be failing on this same step.

Hopefully someone might have some thoughts. Nothing changed on my Caddyfile or my upstream server, and it’s been working perfectly for quite a while. The only thing that changed was a renewed certificate that was renewed in the same automated way I have successfully renewed it in the past.

So I’m a little stuck knowing how to proceed. Should I generate a new cert? Is there something I can do to fix this one? Thanks in advance!

Hi @cbad,

Please read these for recent changes with Let’s Encrypt:

Things changed in June 2024:
see: Chains of Trust - Let's Encrypt
and Deploying Let's Encrypt's New Issuance Chains - #4 by mcpherrinm - API Announcements - Let's Encrypt Community Support

And changes in April 2024:

1 Like

Thanks @ Bruce5051!

These look very promising. Checking into them now!

2 Likes

The new certificate is signed by R10, which is part of Let’s Encrypt’s new intermediate certificates:
https://www.ssllabs.com/ssltest/analyze.html?d=downloadz.surething.com&hideResults=on

LE’s post on the new intermediates:

I wonder if your operating system does not trust the new intermdiates. Check if you’ve installed ca-certificates or if there’s an update for it.

2 Likes

I’m silly and too late to the party. I didn’t read that you figured the chain issue. It appears that the upstream Apache isn’t sending the full chain (intermediate), which is why it’s failing. Caddy cannot trace the leaf certificate to a trusted root. There’s an unknown, untrusted intermediate. How’s Apache receiving those certificates? What’s the content of the PEM files configured in Apache for HTTPS? Why are those upstreams on a different server/not caddy?

2 Likes

@Mohammed90 You are right on almost all accounts. It seems the cert that was generated by letsencrypt wasn’t configured properly for the intermediates. Or maybe with the change I missed something in the files (I install them manually on two different servers).

Anyway, I found a site (https://whatsmychaincert.com/) that tests the chain and fixes the cert if not configured correctly. This solved everything and the new cert is twice as big as the one I received from letsencrypt.

And Caddy is working beautifully as it did before!!!

2 Likes

Thanks again to @Bruce5051 and @Mohammed90 for taking the time to help. Very much appreciated!

1 Like

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.