Journal errors/warnings

1. Caddy version (caddy version):

v2.1.1 h1:X9k1+ehZPYYrSqBvf/ocUgdLSRIuiNiMo7CvyGUQKeA=

2. How I run Caddy:

systemd
and to reload Caddyfile changes:
caddy reload ./Caddyfile

a. System environment:

Ubuntu 18.04

b. Command:

n/a

c. Service/unit/compose file:

n/a

d. My complete Caddyfile or JSON config:

{

  # This is pointing to Let's Encrypt Staging environment (for dev)
  # https://letsencrypt.org/docs/staging-environment/
  # This will allow you to get things right before issuing trusted
  # certificates and reduce the chance of your running up against rate limits.
  # acme_ca https://acme-staging-v02.api.letsencrypt.org/directory
  
  # causes all certificates to be issued internally by default,
  # rather than through a (public) ACME CA such as Let's Encrypt.
  # This is useful in development environments.
  # local_certs
  
  # configure automatic HTTPS. It can either disable it entirely (off)
  # or disable only HTTP-to-HTTPS redirects (disable_redirects).
  # auto_https off
}

# Refer to the Caddy docs for more information:
# https://caddyserver.com/docs/caddyfile

(header_options) {
  header_up X-Real-IP {remote_host}
  header_up X-Forwarded-Proto {scheme}
  header_up Access-Control-Allow-Origin *
  header_up Access-Control-Allow-Credentials true
  header_up Access-Control-Allow-Headers Cache-Control,Content-Type
}

:80, :443 {
  # serve photography folder
  root /files/* /opt/ivt/photography
  
  # Set this path to your site's directory.
  root * /opt/ivt/apps/6.0.0/packages/client/spa

  # Enable the static file server.
  file_server

  route /weather/* {
    uri replace /weather /socket.io
    reverse_proxy * http://localhost:3010 {
      import header_options
    }
  }
  route /ptz/* {
    uri replace /ptz /socket.io
    reverse_proxy * http://localhost:3006 {
      import header_options
    }
  }
  route /liveview/* {
    uri replace /liveview /socket.io
    reverse_proxy * http://localhost:3004 {
      import header_options
    }
  }
  route /archive/* {
    uri replace /archive /socket.io
    reverse_proxy * http://localhost:3003 {
      import header_options
    }
  }
  route /alarms/* {
    uri replace /alarms /socket.io
    reverse_proxy * http://localhost:3002 {
      import header_options
    }
  }
  route /console_socket/* {
    uri replace /console_socket /console/socket.io
    reverse_proxy * http://localhost:3001 {
      import header_options
    }
  }
  route /web_app_socket/* {
    uri replace /web_app_socket /web/socket.io
    reverse_proxy * http://localhost:3001 {
      import header_options
    }
  }
  route /dcam-dev/ivt-hvr-web-app/* {
    uri replace /dcam-dev/ivt-hvr-web-app/ /
    reverse_proxy * http://10.10.2.241:80 {
      import header_options
    }
  }
  route /dcam-dev_socket/* {
    uri replace /dcam-dev_socket/socket.io /socket.io
    reverse_proxy * http://10.10.2.241:80 {
      import header_options
    }
  }
  route /dcam-dev_socket_data/* {
    uri replace /dcam-dev_socket_data/socket.io /socket.io
    reverse_proxy * http://10.10.2.241:8080 {
      import header_options
    }
  }
  route /dcam-dev_files/* {
    uri replace /dcam-dev_files/ /
    reverse_proxy * http://10.10.2.241:8080 {
      import header_options
    }
  }
  route /khalid-sam/* {
    reverse_proxy * http://10.10.3.146:80 {
      import header_options
    }
  }
  route /hikvision/* {
    reverse_proxy * http://10.10.1.70:80 {
      import header_options
    }
  }
  route /rainwise/* {
    reverse_proxy * http://10.10.2.95:80 {
      import header_options
    }
  }
  route /api/* {
    reverse_proxy * http://localhost:3001 {
      import header_options
    }
  }
}

3. The problem I’m having:

I see these errors/warnings in the journal and wondering if I can get help to clean them up

4. Error messages and/or full log output:

Aug 18 13:30:48 FLEX-5 caddy[2010]: {"level":"info","ts":1597779048.2883162,"logger":"http","msg":"server is listening only on the HTTP port, so no automatic HTTPS will be applied to this server","server_name":"srv0","http_port":80}
Aug 18 13:30:48 FLEX-5 caddy[2010]: {"level":"info","ts":1597779048.2884388,"logger":"http","msg":"server is listening only on the HTTPS port but has no TLS connection policies; adding one to enable TLS","server_name":"srv1","https_port":443}
Aug 18 13:30:48 FLEX-5 caddy[2010]: {"level":"info","ts":1597779048.2884488,"logger":"http","msg":"enabling automatic HTTP->HTTPS redirects","server_name":"srv1"}
Aug 18 13:30:48 FLEX-5 caddy[2010]: {"level":"info","ts":1597779048.2888637,"logger":"tls","msg":"setting internal issuer for automation policy that has only internal subjects but no issuer configured","subjects":["localhost"]}
Aug 18 13:30:48 FLEX-5 caddy[2010]: {"level":"warn","ts":1597779048.2889013,"logger":"http","msg":"user server is listening on same interface as automatic HTTP->HTTPS redirects; user-configured routes might override these redirects","server_name":"srv0","interface":"tcp/:80"}
Aug 18 13:30:48 FLEX-5 caddy[2010]: {"level":"info","ts":1597779048.293308,"logger":"http","msg":"enabling automatic TLS certificate management","domains":["localhost"]}
Aug 18 13:30:48 FLEX-5 caddy[2010]: {"level":"warn","ts":1597779048.2935982,"logger":"pki.ca.local","msg":"installing root certificate (you might be prompted for password)","path":"storage:pki/authorities/local/root.crt"}
Aug 18 13:30:48 FLEX-5 caddy[2010]: 2020/08/18 13:30:48 not NSS security databases found
Aug 18 13:30:48 FLEX-5 caddy[2010]: 2020/08/18 13:30:48 define JAVA_HOME environment variable to use the Java trust
Aug 18 13:30:48 FLEX-5 caddy[2010]: 2020/08/18 13:30:48 [INFO][cache:0xc000249860] Started certificate maintenance routine
Aug 18 13:30:48 FLEX-5 caddy[2010]: 2020/08/18 13:30:48 [INFO][localhost] Obtain certificate; acquiring lock...
Aug 18 13:30:48 FLEX-5 caddy[2010]: 2020/08/18 13:30:48 [INFO][localhost] Obtain: Lock acquired; proceeding...
Aug 18 13:30:48 FLEX-5 caddy[2010]: 2020/08/18 13:30:48 [INFO][localhost] Certificate obtained successfully
Aug 18 13:30:48 FLEX-5 caddy[2010]: 2020/08/18 13:30:48 [INFO][localhost] Obtain: Releasing lock
Aug 18 13:30:48 FLEX-5 caddy[2010]: 2020/08/18 13:30:48 [WARNING] Stapling OCSP: no OCSP stapling for [localhost]: no OCSP server specified in certificate
Aug 18 13:30:48 FLEX-5 sudo[22382]: pam_unix(sudo:auth): conversation failed
Aug 18 13:30:48 FLEX-5 sudo[22382]: pam_unix(sudo:auth): auth could not identify password for [caddy]
Aug 18 13:30:48 FLEX-5 caddy[2010]: {"level":"error","ts":1597779048.3020384,"logger":"pki.ca.local","msg":"failed to install root certificate","error":"failed to execute sudo: exit status 1","certificate_file":"storage:pki/authorities/local/root.crt"}
Aug 18 13:30:48 FLEX-5 caddy[2010]: 2020/08/18 13:30:48 [INFO][cache:0xc000913680] Stopped certificate maintenance routine

5. What I already tried:

I thought it may be a folder missing, tried this:

sudo mkdir /etc/ssl/caddy
sudo chown -R root:caddy /etc/ssl/caddy
sudo chmod 0770 /etc/ssl/caddy

6. Links to relevant resources:

I’ve never seen this error before. Is caddy in sudoers?

You can probably run caddy trust (make sure as the same user or $HOME directory at least) to debug that further.

Caddy is in the sudoers group (temporary, for now)

Your suggestion did something:

$ caddy trust
2020/08/18 21:11:51.577	WARN	ca.local	installing root certificate (you might be prompted for password)	{"path": "storage:pki/authorities/local/root.crt"}
2020/08/18 15:11:51 certificate installed properly in NSS security databases
2020/08/18 15:11:51 define JAVA_HOME environment variable to use the Java trust
[sudo] password for intelliview: 
2020/08/18 15:12:06 certificate installed properly in linux trusts

But, getting a lot of these right now:

Aug 18 15:14:13 FLEX-5 caddy[2010]: 2020/08/18 15:14:13 http: TLS handshake error from [::1]:32840: no certificate available for 'localhost'
Aug 18 15:14:13 FLEX-5 caddy[2010]: 2020/08/18 15:14:13 http: TLS handshake error from [::1]:32842: no certificate available for 'localhost'
Aug 18 15:14:13 FLEX-5 caddy[2010]: 2020/08/18 15:14:13 http: TLS handshake error from [::1]:32844: no certificate available for 'localhost'
Aug 18 15:14:13 FLEX-5 caddy[2010]: 2020/08/18 15:14:13 http: TLS handshake error from [::1]:32846: no certificate available for 'localhost'

Also getting this certificate that Chrome thinks is invalid:

and this:

Aug 18 15:27:07 FLEX-5 caddy[2010]: {"level":"info","ts":1597786027.7086875,"logger":"admin.api","msg":"received request","method":"POST","host":"localhost:2019","uri":"/load","remote_addr":"127.0.0.1:57644","headers":{"Accept-Encoding":["gzip"],"Content-Length":["7899"],"Content-Type":["application/json"],"Origin":["localhost:2019"],"User-Agent":["Go-http-client/1.1"]}}
Aug 18 15:27:07 FLEX-5 caddy[2010]: {"level":"info","ts":1597786027.7117238,"logger":"admin","msg":"admin endpoint started","address":"tcp/localhost:2019","enforce_origin":false,"origins":["[::1]:2019","127.0.0.1:2019","localhost:2019"]}
Aug 18 15:27:07 FLEX-5 caddy[2010]: {"level":"info","ts":1597786027.7122407,"logger":"http","msg":"server is listening only on the HTTPS port but has no TLS connection policies; adding one to enable TLS","server_name":"srv0","https_port":443}
Aug 18 15:27:07 FLEX-5 caddy[2010]: {"level":"info","ts":1597786027.712261,"logger":"http","msg":"enabling automatic HTTP->HTTPS redirects","server_name":"srv0"}
Aug 18 15:27:07 FLEX-5 caddy[2010]: 2020/08/18 15:27:07 [INFO][cache:0xc0009930e0] Started certificate maintenance routine
Aug 18 15:27:07 FLEX-5 caddy[2010]: 2020/08/18 15:27:07 [INFO][cache:0xc000a1a000] Stopped certificate maintenance routine
Aug 18 15:27:07 FLEX-5 caddy[2010]: {"level":"info","ts":1597786027.7152011,"msg":"autosaved config","file":"/var/lib/caddy/.config/caddy/autosave.json"}
Aug 18 15:27:07 FLEX-5 caddy[2010]: {"level":"info","ts":1597786027.715221,"logger":"admin.api","msg":"load complete"}
Aug 18 15:27:08 FLEX-5 caddy[2010]: {"level":"info","ts":1597786028.2146056,"logger":"admin","msg":"stopped previous server"}
Aug 18 15:27:35 FLEX-5 caddy[2010]: 2020/08/18 15:27:35 http: TLS handshake error from [::1]:33198: no certificate available for 'localhost'
Aug 18 15:27:35 FLEX-5 caddy[2010]: 2020/08/18 15:27:35 http: TLS handshake error from [::1]:33200: no certificate available for 'localhost'
Aug 18 15:27:35 FLEX-5 caddy[2010]: 2020/08/18 15:27:35 http: TLS handshake error from [::1]:33202: no certificate available for 'localhost'

and how to fix this (please):

server is listening only on the HTTPS port but has no TLS connection policies; adding one to enable TLS","server_name":"srv0","https_port":443

Great, so the root cert is trusted.

The no certificate available for 'localhost' error is caused because your Caddyfile config doesn’t tell Caddy any domain names, which is one of the requirements for auto HTTPS:

The Caddyfile is only listening on :80, :443 so Caddy doesn’t know which hostnames to get certificates for. Simply replace those with localhost and it will serve localhost over HTTPS.

This is not an error, just informational.

right, these computers don’t have domains as technically they are similar to set-top boxes. Any way around that? When I did testing in april, I was able to get it to do https, which is what we want.

In case you missed it:

:arrow_down:

Still getting ERR_CERT_INVALID from Chrome

and in journal

Aug 18 15:40:44 FLEX-5 caddy[2010]: 2020/08/18 15:40:44 not NSS security databases found
Aug 18 15:40:44 FLEX-5 caddy[2010]: 2020/08/18 15:40:44 define JAVA_HOME environment variable to use the Java trust
Aug 18 15:40:44 FLEX-5 sudo[20522]: pam_unix(sudo:auth): conversation failed
Aug 18 15:40:44 FLEX-5 sudo[20522]: pam_unix(sudo:auth): auth could not identify password for [caddy]
Aug 18 15:40:44 FLEX-5 caddy[2010]: {"level":"error","ts":1597786844.196335,"logger":"pki.ca.local","msg":"failed to install root certificate","error":"failed to execute sudo: exit status 1","certificate_file":"storage:pki/authorities/local/root.crt"}
Aug 18 15:40:44 FLEX-5 caddy[2010]: {"level":"info","ts":1597786844.196411,"logger":"http","msg":"enabling automatic TLS certificate management","domains":["localhost"]}
Aug 18 15:40:44 FLEX-5 caddy[2010]: 2020/08/18 15:40:44 [WARNING] Stapling OCSP: no OCSP stapling for [localhost]: no OCSP server specified in certificate
Aug 18 15:40:44 FLEX-5 caddy[2010]: 2020/08/18 15:40:44 [INFO][cache:0xc000287080] Stopped certificate maintenance routine
Aug 18 15:40:44 FLEX-5 caddy[2010]: 2020/08/18 15:40:44 [INFO][flex-5] Obtain: Releasing lock
Aug 18 15:40:44 FLEX-5 caddy[2010]: {"level":"info","ts":1597786844.1973028,"msg":"autosaved config","file":"/var/lib/caddy/.config/caddy/autosave.json"}
Aug 18 15:40:44 FLEX-5 caddy[2010]: {"level":"info","ts":1597786844.1973174,"logger":"admin.api","msg":"load complete"}
Aug 18 15:40:44 FLEX-5 caddy[2010]: {"level":"info","ts":1597786844.688532,"logger":"admin","msg":"stopped previous server"}
Aug 18 15:41:54 FLEX-5 caddy[2010]: 2020/08/18 15:41:54 http: TLS handshake error from [::1]:33688: remote error: tls: unknown certificate
Aug 18 15:41:54 FLEX-5 caddy[2010]: 2020/08/18 15:41:54 http: TLS handshake error from [::1]:33690: remote error: tls: unknown certificate
Aug 18 15:41:54 FLEX-5 caddy[2010]: 2020/08/18 15:41:54 http: TLS handshake error from [::1]:33692: remote error: tls: unknown certificate

also set this to avoid blacklisting:

acme_ca https://acme-staging-v02.api.letsencrypt.org/directory

You’ll have to restart the browser in order for trust changes to take effect.

I did that. Here is the Caddy journal.

Aug 19 05:30:44 FLEX-5 caddy[2010]: 2020/08/19 05:30:44 [INFO] [localhost] Maintenance routine: certificate expires in 3h59m59.810002137s; queueing for renewal
Aug 19 05:30:44 FLEX-5 caddy[2010]: 2020/08/19 05:30:44 [INFO] [localhost] Maintenance routine: attempting renewal with 3h59m59.809958444s remaining
Aug 19 05:30:44 FLEX-5 caddy[2010]: 2020/08/19 05:30:44 [INFO][localhost] Renew certificate; acquiring lock...
Aug 19 05:30:44 FLEX-5 caddy[2010]: 2020/08/19 05:30:44 [INFO][localhost] Renew: Lock acquired; proceeding...
Aug 19 05:30:44 FLEX-5 caddy[2010]: 2020/08/19 05:30:44 [INFO][localhost] Renew: 3h59m59.80916667s remaining
Aug 19 05:30:44 FLEX-5 caddy[2010]: 2020/08/19 05:30:44 [INFO][localhost] Certificate renewed successfully
Aug 19 05:30:44 FLEX-5 caddy[2010]: 2020/08/19 05:30:44 [INFO][localhost] Renew: Releasing lock
Aug 19 05:30:44 FLEX-5 caddy[2010]: 2020/08/19 05:30:44 [INFO] Reloading managed certificate for [localhost]
Aug 19 05:30:44 FLEX-5 caddy[2010]: 2020/08/19 05:30:44 [WARNING] Stapling OCSP: no OCSP stapling for [localhost]: no OCSP server specified in certificate
Aug 19 05:30:44 FLEX-5 caddy[2010]: 2020/08/19 05:30:44 [INFO] Replaced certificate in cache for [localhost] (new expiration date: 2020-08-19 23:30:44)
Aug 19 06:23:37 FLEX-5 caddy[2010]: 2020/08/19 06:23:37 http: TLS handshake error from [::1]:38872: remote error: tls: unknown certificate
Aug 19 06:23:37 FLEX-5 caddy[2010]: 2020/08/19 06:23:37 http: TLS handshake error from [::1]:38874: remote error: tls: unknown certificate
Aug 19 06:23:37 FLEX-5 caddy[2010]: 2020/08/19 06:23:37 http: TLS handshake error from [::1]:38876: remote error: tls: unknown certificate
Aug 19 06:23:37 FLEX-5 caddy[2010]: 2020/08/19 06:23:37 http: TLS handshake error from [::1]:38880: remote error: tls: unknown certificate
Aug 19 06:23:37 FLEX-5 caddy[2010]: 2020/08/19 06:23:37 http: TLS handshake error from [::1]:38882: remote error: tls: unknown certificate
Aug 19 06:23:37 FLEX-5 caddy[2010]: 2020/08/19 06:23:37 http: TLS handshake error from [::1]:38884: remote error: tls: unknown certificate
Aug 19 06:23:42 FLEX-5 caddy[2010]: 2020/08/19 06:23:42 http: TLS handshake error from [::1]:38888: remote error: tls: unknown certificate
Aug 19 06:23:42 FLEX-5 caddy[2010]: 2020/08/19 06:23:42 http: TLS handshake error from [::1]:38890: remote error: tls: unknown certificate
Aug 19 06:23:42 FLEX-5 caddy[2010]: 2020/08/19 06:23:42 http: TLS handshake error from [::1]:38892: remote error: tls: unknown certificate
Aug 19 06:23:43 FLEX-5 caddy[2010]: 2020/08/19 06:23:43 http: TLS handshake error from [::1]:38894: remote error: tls: unknown certificate
Aug 19 06:23:43 FLEX-5 caddy[2010]: 2020/08/19 06:23:43 http: TLS handshake error from [::1]:38896: remote error: tls: unknown certificate
Aug 19 06:23:43 FLEX-5 caddy[2010]: 2020/08/19 06:23:43 http: TLS handshake error from [::1]:38898: remote error: tls: unknown certificate
Aug 19 06:24:16 FLEX-5 caddy[2010]: 2020/08/19 06:24:16 http: TLS handshake error from [::1]:38906: remote error: tls: unknown certificate
Aug 19 06:24:16 FLEX-5 caddy[2010]: 2020/08/19 06:24:16 http: TLS handshake error from [::1]:38908: remote error: tls: unknown certificate
Aug 19 06:24:16 FLEX-5 caddy[2010]: 2020/08/19 06:24:16 http: TLS handshake error from [::1]:38910: remote error: tls: unknown certificate

It appears to have got a new certificate overnight, but still has issues. Chrome is spewing ERR_CERT_INVALID
and Firefox gives this error:

An error occurred during a connection to localhost. Peer’s certificate has an invalid signature.

Error code: SEC_ERROR_BAD_SIGNATURE

I wonder if this might be part of the problem:

$ certutil -L
certutil: function failed: SEC_ERROR_LEGACY_DATABASE: The certificate/key database is in an old, unsupported format.

Nope. Running with different params, shows caddy has actually installed a certificate:

$ certutil -L -d sql:${HOME}/.pki/nssdb

 
Certificate Nickname                                         Trust Attributes
                                                             SSL,S/MIME,JAR/XPI
 
Caddy Local Authority - 2020 ECC Root 282545821501334259807900389275737972246 C,,

Another thing I noticed about the certificate is the “This certificate has been verified for the following usages: SSL Server Certificate”
This come from Github:

While the issued by Caddy has nothing for that:

Also, Github has a “Certificate Hierarchy”:
image

and the Caddy certificate does not:
image

I really don’t know much about certificates, but I would think they should look similar sans domain name (ie: self-signed, etc).

That could be a weird thing with the cert viewer you’re using. I’m actually currently reviewing our self-signed cert code. It might be related. Can you tell me if the Subject Alternate Name (SAN) extension is marked “Critical”? (on the leaf cert)

@matt Doesn’t look like it:

1 Like

Great, the good news is this already being tracked upstream: Subject alternative name extension must be marked critical if the "subject" field is empty · Issue #302 · smallstep/certificates · GitHub

Currently blocking the 2.2 release on it. The Smallstep team is quite good, so I expect we’ll figure it out soon enough!

Very good news! I am happy so many teams are responsive and responsible. Love communities like that.

1 Like

I saw this: Subject alternative name extension must be marked critical if the "subject" field is empty · Issue #302 · smallstep/certificates · GitHub

Meaning it is fixed?

When will the next Caddy release happen?

Soon! Next week most likely. There’s a few rough edges I want to smooth out if possible.