Failing to solve HTTP challenge for issuance of TLS certificate

1. Caddy version (caddy version):

v2.4.5 h1:P1mRs6V2cMcagSPn+NWpD+OEYUYLIf6ecOa48cFGeUg=

2. How I run Caddy:

Via systemd (having installed Caddy via OS package manager):

sudo systemctl start caddy

a. System environment:

Ubuntu 20.04 LTS

b. Command:

sudo systemctl start caddy

Here’s the service status:

sudo systemctl status caddy
● caddy.service - Caddy
     Loaded: loaded (/lib/systemd/system/caddy.service; enabled; vendor preset: enabled)
     Active: active (running) since Sat 2021-10-23 11:07:19 UTC; 1 day 21h ago
       Docs: https://caddyserver.com/docs/
    Process: 62237 ExecReload=/usr/bin/caddy reload --config /etc/caddy/Caddyfile (code=exited, status=0/SUCCESS)
   Main PID: 43973 (caddy)
      Tasks: 7 (limit: 2279)
     Memory: 12.3M
     CGroup: /system.slice/caddy.service
             └─43973 /usr/bin/caddy run --environ --config /etc/caddy/Caddyfile

c. Service/unit/compose file:

I haven’t modified this file at all:

$ sudo cat /lib/systemd/system/caddy.service
# caddy.service
#
# For using Caddy with a config file.
#
# Make sure the ExecStart and ExecReload commands are correct
# for your installation.
#
# See https://caddyserver.com/docs/install for instructions.
#
# WARNING: This service does not use the --resume flag, so if you
# use the API to make changes, they will be overwritten by the
# Caddyfile next time the service is restarted. If you intend to
# use Caddy's API to configure it, add the --resume flag to the
# `caddy run` command or use the caddy-api.service file instead.

[Unit]
Description=Caddy
Documentation=https://caddyserver.com/docs/
After=network.target network-online.target
Requires=network-online.target

[Service]
Type=notify
User=caddy
Group=caddy
ExecStart=/usr/bin/caddy run --environ --config /etc/caddy/Caddyfile
ExecReload=/usr/bin/caddy reload --config /etc/caddy/Caddyfile
TimeoutStopSec=5s
LimitNOFILE=1048576
LimitNPROC=512
PrivateTmp=true
ProtectSystem=full
AmbientCapabilities=CAP_NET_BIND_SERVICE

[Install]
WantedBy=multi-user.target

d. My complete Caddyfile or JSON config:

$ sudo cat /etc/caddy/Caddyfile
www.colloqu.io {
  redir https://colloqu.io{uri}
}


colloqu.io {
  root * /var/www/colloqu.io/public

  encode gzip

  file_server

  request_body {
    max_size 10MB
  }

  try_files maintenance.html

  @notStatic {
    not file
  }

  reverse_proxy @notStatic unix//var/www/colloqu.io/tmp/sockets/puma.sock {
    header_up X-Real-IP {remote_host}
  }

  log {
    output file /var/log/caddy/access.log
  }
}

3. The problem I’m having:

I have just moved a webapp from one VPS to another. Everything looked fine in a browser and the browser claims the TLS certificate is valid until 21 Jan 2022.

When I use curl instead of a browser, curl reports that the certificate has expired (see below).

However I noticed in Caddy’s logs that it is unable to complete the certificate process. Furthermore in the logs of the webapp to which Caddy reverse-proxies, I see that is trying to (but naturally unable to) handle a GET to /.well-known-acme-challenge/<long alphanumeric string>.

Presumably this means that Caddy isn’t handling the certificate challenge request but is instead passing the request to my webapp. What am I doing wrong?

4. Error messages and/or full log output:

$ curl -v https://colloqu.io
* Rebuilt URL to: https://colloqu.io/
*   Trying 45.79.181.170...
* TCP_NODELAY set
* Connected to colloqu.io (45.79.181.170) port 443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* Cipher selection: ALL:!EXPORT:!EXPORT40:!EXPORT56:!aNULL:!LOW:!RC4:@STRENGTH
* successfully set certificate verify locations:
*   CAfile: /etc/ssl/cert.pem
  CApath: none
* TLSv1.2 (OUT), TLS handshake, Client hello (1):
* TLSv1.2 (IN), TLS handshake, Server hello (2):
* TLSv1.2 (IN), TLS handshake, Certificate (11):
* TLSv1.2 (OUT), TLS alert, Server hello (2):
* SSL certificate problem: certificate has expired
* stopped the pause stream!
* Closing connection 0
curl: (60) SSL certificate problem: certificate has expired
More details here: https://curl.haxx.se/docs/sslcerts.html

curl performs SSL certificate verification by default, using a "bundle"
 of Certificate Authority (CA) public keys (CA certs). If the default
 bundle file isn't adequate, you can specify an alternate file
 using the --cacert option.
If this HTTPS server uses a certificate signed by a CA represented in
 the bundle, the certificate verification probably failed due to a
 problem with the certificate (it might be expired, or the name might
 not match the domain name in the URL).
If you'd like to turn off curl's verification of the certificate, use
 the -k (or --insecure) option.
HTTPS-proxy has similar options --proxy-cacert and --proxy-insecure.

Line from my webapp’s log file:

No route matches [GET] "/.well-known/acme-challenge/gOK3vlIYdPXsw5w8bh9yP1_7JquAZsAZgV7U8jqtTYs"

Caddy log:

{
  "level":"error",
  "ts":1635142078.8743775,
  "logger":"tls.issuance.acme",
  "msg":"looking up info for HTTP challenge",
  "host":"colloqu.io",
  "error":"no information found to solve challenge for identifier: colloqu.io"
}

5. What I already tried:

Restarting Caddy.

Searching this forum.
Searching GitHub issues.
Searching the LetsEncrypt forum.

6. Links to relevant resources:

N/A

Are you sure DNS has finished propagation after your server move?

This means Caddy received a request from LE or ZeroSSL to solve the HTTP challenge, but your Caddy instance doesn’t have the information in its storage to solve that challenge anymore.

Do you still have Caddy running on your old server? If so turn that off (especially if they don’t share the same storage), it might be trying to perform cert issuance but since it’s no longer reachable, it will hit your new server instead which won’t know how to solve it.

According to crt.sh | colloqu.io new certificates were successfully issued 2 days ago.

Are you sure DNS has finished propagation after your server move?

I’m not sure how to tell. I updated the DNS A record two days ago on 23 October at around 10am UTC. A few days before that I had set its TTL to 60s.

Do you still have Caddy running on your old server?

The old server was running Nginx which I stopped after updating the A record. The certificates there were issued on 21 September 2021. The certificate issuance mechanism was a cronjob running at 10 minutes after midnight UTC every day (invoking acme.sh) which I also shut down.

I don’t think the problem could be a pending cert issuance request from the old server.

According to crt.sh | colloqu.io new certificates were successfully issued 2 days ago.

Hmm, I wonder why curl says the certificate is outdated?

:man_shrugging:

Seems to work for me, from my machine though:

$ curl -v https://colloqu.io/
*   Trying 45.79.181.170:443...
* Connected to colloqu.io (45.79.181.170) port 443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* successfully set certificate verify locations:
*  CAfile: /etc/ssl/certs/ca-certificates.crt
*  CApath: /etc/ssl/certs
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* TLSv1.3 (IN), TLS handshake, Server hello (2):
* TLSv1.3 (IN), TLS handshake, Encrypted Extensions (8):
* TLSv1.3 (IN), TLS handshake, Certificate (11):
* TLSv1.3 (IN), TLS handshake, CERT verify (15):
* TLSv1.3 (IN), TLS handshake, Finished (20):
* TLSv1.3 (OUT), TLS change cipher, Change cipher spec (1):
* TLSv1.3 (OUT), TLS handshake, Finished (20):
* SSL connection using TLSv1.3 / TLS_AES_128_GCM_SHA256
* ALPN, server accepted to use h2
* Server certificate:
*  subject: CN=colloqu.io
*  start date: Oct 23 10:04:50 2021 GMT
*  expire date: Jan 21 10:04:49 2022 GMT
*  subjectAltName: host "colloqu.io" matched cert's "colloqu.io"
*  issuer: C=US; O=Let's Encrypt; CN=R3
*  SSL certificate verify ok.
* Using HTTP2, server supports multi-use
* Connection state changed (HTTP/2 confirmed)
* Copying HTTP/2 data in stream buffer to connection buffer after upgrade: len=0
* Using Stream ID: 1 (easy handle 0x5571e09b7580)
> GET / HTTP/2
> Host: colloqu.io
> user-agent: curl/7.74.0
> accept: */*
> 
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
* Connection state changed (MAX_CONCURRENT_STREAMS == 250)!
< HTTP/2 401 
< cache-control: no-cache
< content-type: text/javascript
< server: Caddy
< x-request-id: 1eb09a17-2b1a-448e-8e03-ea079f4acb99
< x-runtime: 0.003641
< date: Mon, 25 Oct 2021 11:32:03 GMT
< 
* Connection #0 to host colloqu.io left intact

Thanks for posting that; it was useful. It turns out my curl problem was due to macOS’s built-in curl not supporting TLSv1.3. When I replaced it with a version that does, curl -v https://colloqu.io worked fine.

I wonder if the error messages in the Caddy logs are from when I was installing everything on the server, i.e. for whatever reason Caddy keeps trying to complete an obsolete certificate issuance request. Could that be possible?

I’m not that familiar with the certificate issuance process, but given that a certificate was successfully issued I’m inclined to ignore the error messages and hope they go away.

1 Like

Those log messages come from Caddy receiving requests from Let’s Encrypt that it doesn’t know how to handle, so it just passes it through to your backend.

It might’ve been an issuance transaction that your old cron initiated but never completed. Hard to say with certainty what initiated it. But I think it’s safe to ignore and it should time out eventually.

The gist of it is

  • ACME client sets up a cert signing request, asks the ACME server “hey I want a cert for this domain”, this is called an “order”
  • ACME server says “okay sure, here’s a challenge to prove that you control that domain”
  • ACME client wires up the /.well-known to be ready to receive the HTTP request from the ACME server
  • ACME server will try check for the challenge at the expected URL until it finds it, with some timeout
  • When the ACME server finds the challenge, it’ll finalize the order and make the cert available for download
  • During this, ACME client also polls to check the status of the order, if it’s done then it downloads the cert

Obviously I’m simplifying here, there’s a bunch of other details under the hood like generating keys, signing the CSR, etc. But that’s the relevant bit here.

1 Like

Ok, that all makes sense.

Thank you very much for taking the time to explain everything and help me out. I appreciate it!

1 Like

Latest version of macOS? :face_with_raised_eyebrow: Huh, that’s good to know.

Not the latest – I’m on 10.14.6 (Mojave).

This article describes the problem. The built-in curl is v7.54.0 which uses LibreSSL v2.6.5 which doesn’t support TLSv1.3. As per the article, I resolved the problem with brew install curl-openssl and then updating my bash profile to put the new curl on the PATH before the existing curl.

1 Like

This topic was automatically closed after 30 days. New replies are no longer allowed.