agrajag
(Todd Grayson)
March 20, 2021, 11:34pm
1
1. Caddy version (caddy version
):
root@pi:/# caddy version
v2.4.0-beta.1 h1:Ed/tIaN3p6z8M3pEiXWJL/T8JmCqV62FrSJCHKquW/I=
2. How I run Caddy:
a. System environment:
Raspbian lite OS (debian)
root@elite-pi:/var/lib/caddy/.local/share/caddy/pki/authorities/local# cat /etc/*release*
PRETTY_NAME="Raspbian GNU/Linux 10 (buster)"
NAME="Raspbian GNU/Linux"
VERSION_ID="10"
VERSION="10 (buster)"
VERSION_CODENAME=buster
ID=raspbian
ID_LIKE=debian
HOME_URL="http://www.raspbian.org/"
SUPPORT_URL="http://www.raspbian.org/RaspbianForums"
BUG_REPORT_URL="http://www.raspbian.org/RaspbianBugs"
b. Command:
systemctl start caddy
c. Service/unit/compose file:
[Unit]
Description=Caddy
Documentation=https://caddyserver.com/docs/
After=network.target network-online.target
Requires=network-online.target
[Service]
User=caddy
Group=caddy
#plugin modified caddy build for auth-portal and associated plugins
#/usr/bin/caddy now linked to /opt/caddy/releases/[current build]
ExecStart=/usr/bin/caddy run --environ --config /etc/caddy/Caddyfile
#ExecStart=/opt/caddy/bin/caddy_linux_arm7_custom run --environ --config /etc/caddy/Caddyfile
ExecReload=/usr/bin/caddy reload --config /etc/caddy/Caddyfile
#ExecReload=/opt/caddy/bin/caddy_linux_arm7_custom reload --config /etc/caddy/Caddyfile
TimeoutStopSec=5s
LimitNOFILE=1048576
LimitNPROC=512
PrivateTmp=true
ProtectSystem=full
AmbientCapabilities=CAP_NET_BIND_SERVICE
[Install]
WantedBy=multi-user.target
d. My complete Caddyfile or JSON config:
root@elite-pi:/# caddy fmt /etc/caddy/Caddyfile
{
# storage file_system {
# root /opt/caddy/storage
# }
local_certs
http_port 80
https_port 443
}
https://host.example.com, https://10.1.1.20 {
log {
level INFO
format console
output file /var/log/caddy/caddy.log
}
route /auth* {
auth_portal {
path /auth
backends {
local_backend {
method local
path /opt/caddy/assets/conf/local/auth/user_db.json
realm local
}
}
jwt {
token_name access_token
token_secret AnExampleSecretString123
token_lifetime 43200
}
ui {
theme basic
generic_template "/opt/caddy/templates/em20-tmpl/generic.template"
login_template "/opt/caddy/templates/em20-tmpl/login.template"
portal_template "/opt/caddy/templates/em20-tmpl/portal.template"
register_template "/opt/caddy/templates/em20-tmpl/register.template"
whoami_template "/opt/caddy/templates/em20-tmpl/whoami.template"
settings_template "/opt/caddy/templates/em20-tmpl/settings.template"
custom_css_path "/opt/caddy/css/custom.css"
logo_url "/opt/caddy/templates/logo.png"
links {
"Manager" /
"Auth Portal Settings" /auth/settings
"who am i check" /auth/whoami
"Add MFA Authentication App" /auth/settings/mfa/add/app
}
}
}
}
route /gr/* {
jwt {
enable claim headers
}
reverse_proxy http://localhost:3000
}
route /version* {
respond * "1.5.0-a" 200
}
route /ui/* {
jwt {
enable claim headers
}
reverse_proxy http://localhost:1880
}
route /* {
jwt {
primary yes
trusted_tokens {
static_secret {
token_name access_token
token_secret AnExampleSecretString123
}
}
enable claim headers
}
reverse_proxy http://localhost:1081
}
}
3. The problem I’m having:
I am attempting to clean out all system trust (/etc/ssl/), clear the current local certificates and am wondering what is the correct approach to clearing the caddy server’s certificates located in the /var/lib/caddy/.local path.
Is the best approach to just remove the .local path and then restart caddy?
4. Error messages and/or full log output:
N/A
5. What I already tried:
opened 12:43AM - 18 Mar 21 UTC
closed 12:53AM - 20 Apr 21 UTC
feature
discussion
1) introduce "trust cleanup" for the caddy trust command and caddy server startu… p after a TLS configuration change to cleanup invalid previous CA instances
**(and / or)**
2) use unique issuer/subject distinguished names (DN) when generating the caddy local CA's **<<< most important**
The issue is a divergence between the system trust installed local caddy CA, and what caddy is using in its current runtime as root CA. The "new" caddy CA is not being installed because the caddy trust (and first start trust setup) does not recognize that the two certificates are actually different. This is due to issuer/subject DN being the same in the new and old CA certificates. When you examine the CA certificates, they are completly different CA certificates at the TLS cryptographic level.
Our configuration files are present in the caddy forums, most recient being https://caddy.community/t/new-feature-caddy-trust-with-support-for-config-switch/11606. This issue appears to be related to caddyserver github issue#4058.
I'm not 100% sure what scenario triggered the generation of a new rootCA. Regardless its important to understand that issuer/subject DN's being the same between 2 different CA certificates can break trust path evaluation for TLS. I believe we were seeing issues similar to this on windows in forum post https://caddy.community/t/local-caddy-selfsigned-ca-is-creating-certs-that-google-chrome-will-not-accept/10785 where there were duplicate instances of the CA's installed in the microsoft trust stores.
Currently we are testing on linux, and this is the message seen when running "caddy trust" as root on a host where we have installed a new version of the caddy binary (adding plugins), and switched back to the default local TLS configuration with no storage location. The same kind of message is seen on caddy startup as well.
```
root@elite-pi:/var/lib/caddy# caddy trust
2021/03/17 23:02:55.352 INFO ca.local root certificate is already trusted by system {"path": "storage:pki/authorities/local/root.crt"}
```
This is incorrect, however. You can see this by directly inspecting the new caddy CA certificate and comparing with the CA certificate caddy currently has installed for system wide trust. I show the detail below, but first some context.
The issue for TLS is that when a TLS client is attempting to build a path to trusting a server certificate, having duplicates of the same subject DN being present for the CA trust in the client's local trusted CA's, can cause failure to setup the TLS connection. Simply stated, when tls connection setup has to parse previous instances of its local CA configured in /etc/ssl/pki/certificates depending on the implementation, you can end up the first one found will be used to attempt to build the path of trust for the connection.
When examining what caddy updated the system with for certificate trust, its a little messy (I think due to CA being generated for the new year at some point along the way on a new caddy binary install). There are both 2020 and 2021 instances of the CA, these have different issuer/subject DN's so they are NOT conflicting with each other.
```
root@elite-pi:/etc/ssl/certs# ls -l | grep -i caddy
lrwxrwxrwx 1 root root 80 Feb 7 10:43 1e027192.0 -> Caddy_Local_Authority_-_2021_ECC_Root_69038497948112521170214703261324660100.pem
lrwxrwxrwx 1 root root 80 Oct 23 20:23 a6f460bc.0 -> Caddy_Local_Authority_-_2020_ECC_Root_81229713952639004777245312604640354755.pem
lrwxrwxrwx 1 root root 113 Oct 23 20:23 Caddy_Local_Authority_-_2020_ECC_Root_81229713952639004777245312604640354755.pem -> /usr/local/share/ca-certificates/Caddy_Local_Authority_-_2020_ECC_Root_81229713952639004777245312604640354755.crt
lrwxrwxrwx 1 root root 113 Feb 7 10:43 Caddy_Local_Authority_-_2021_ECC_Root_69038497948112521170214703261324660100.pem -> /usr/local/share/ca-certificates/Caddy_Local_Authority_-_2021_ECC_Root_69038497948112521170214703261324660100.crt
```
When I compare at the local CA file vs what was installed in the system in the /var/lib/caddy/.caddy path I see:
```
root@elite-pi:/var/lib/caddy# ls -l /usr/local/share/ca-certificates/Caddy_Local_Authority_-_2020_ECC_Root_81229713952639004777245312604640354755.crt
-rw-r--r-- 1 root root 627 Oct 23 20:23 /usr/local/share/ca-certificates/Caddy_Local_Authority_-_2020_ECC_Root_81229713952639004777245312604640354755.crt
root@elite-pi:/var/lib/caddy# ls -l /var/lib/caddy/.local/share/caddy/pki/authorities/local/root.crt
-rw------- 1 caddy caddy 627 Oct 23 13:00 /var/lib/caddy/.local/share/caddy/pki/authorities/local/root.crt
```
Given the close timeframes I'm assuming we ended up triggering this due to issue#4058
**WHY IS TLS BREAKING?**
The following shows that indeed, the current certificate that caddy is using is different than what is in the system CA certificate trusts. I've trunicated the full certificate detail (...) to focus on the key elements in the trust evaluation and what qualifies as "uniqueness".
First we examine what the caddy runtime has as a local root CA. Note that Issuer and Subject DN are always the same in a root CA certificate. The serial number and subject key identifyer are unique "fingerprint" like qualities of the certificate, while the Issuer and Subject DN are just names, and can be re-used over unique serial number and fingerprint without users realizing the certificate has changed...
```
root@elite-pi:/var/lib/caddy# openssl x509 -in /var/lib/caddy/.local/share/caddy/pki/authorities/local/root.crt -noout -text
Certificate:
...
Serial Number:
e0:ce:9d:36:2e:25:55:07:1d:dc:9b:68:ec:fc:3c:5b
...
Issuer: CN = Caddy Local Authority - 2020 ECC Root
Validity
Not Before: Oct 23 19:00:27 2020 GMT
Not After : Sep 1 19:00:27 2030 GMT
Subject: CN = Caddy Local Authority - 2020 ECC Root
...
Subject Public Key Info:
X509v3 Subject Key Identifier:
8A:92:A0:82:67:B0:C3:B6:D2:C2:CF:2A:ED:F8:D7:8B:6C:85:62:3D
```
And finally the "proof" of the problem, in that even tho "caddy trust" and caddy startup after config change indicate the CA is already installed, as you see below it is not the same certificate (focus on serial number and subject key identifier). Unfortunately, its the Distinguished Name that is used as the point of selection at the RFC level (https://tools.ietf.org/html/rfc8446#section-4.2.4) so having duplicates or mismatched instances of different certificates with the same distinguished names (issuer/subject DN) is poison for trust evaluation.
```
root@elite-pi:/var/lib/caddy# openssl x509 -noout -text -in /usr/local/share/ca-certificates/Caddy_Local_Authority_-_2020_ECC_Root_81229713952639004777245312604640354755.crt
Certificate:
...
Serial Number:
3d:1c:46:1a:df:53:26:9c:6d:b5:91:f4:0a:87:a1:c3
...
Issuer: CN = Caddy Local Authority - 2020 ECC Root
Validity
Not Before: Oct 24 02:23:05 2020 GMT
Not After : Sep 2 02:23:05 2030 GMT
Subject: CN = Caddy Local Authority - 2020 ECC Root
...
X509v3 Subject Key Identifier:
EB:1E:1E:8F:AD:E1:6D:5F:58:94:3F:0F:58:E0:4F:8C:5F:2D:00:3A
```
**So I would suggest as a fix:**
Use unique subject DN strings for caddy generated local CA's, so something like the following where the CA is absolutely uniqe to that node at the moment of time of generation (hash or uuid seeded by mac address and current system time? user configured unique value?) regardless whatever it is the issuer/subject DN should include a unique-string for that system.
```
Issuer: CN = Caddy Local Authority - 2020 ECC Root [unique-string]
```
This same issue could arise over collisions in clients importing the caddy CA for whatever reason where multiple discrete caddy deployments are being run in an internal scenario, using their local CA's. For example an intranet scenario where a number of services integrated together, need to connect to each over TLS through discrete local caddy CA instances.
6. Links to relevant resources:
system
(system)
Closed
April 19, 2021, 11:34pm
2
This topic was automatically closed after 30 days. New replies are no longer allowed.