Internal TLS intermediate cert lifespan

Help
1

1. Caddy version (caddy version):

V2.4.3

2. How I run Caddy:

As a reverse proxy for a couple of internal and external services.

a. System environment:

Run on Proxmox 7 in an Ubuntu 20.04 Server LXC container.

b. Command:

Caddy is run as a service but I manually restart it this way from /etc/caddy/…

caddy reload

c. Service/unit/compose file:

[Unit]
Description=Caddy
Documentation=https://caddyserver.com/docs/
After=network.target network-online.target
Requires=network-online.target

[Service]
Type=notify
User=caddy
Group=caddy
ExecStart=/usr/bin/caddy run --environ --config /etc/caddy/Caddyfile
ExecReload=/usr/bin/caddy reload --config /etc/caddy/Caddyfile
TimeoutStopSec=5s
LimitNOFILE=1048576
LimitNPROC=512
PrivateTmp=true
ProtectSystem=full
AmbientCapabilities=CAP_NET_BIND_SERVICE

[Install]
WantedBy=multi-user.target

d. My complete Caddyfile or JSON config:

shinobi.domain.net {
        tls internal
        reverse_proxy http://1.2.3.4:8080
}

https://vault.domain.net {
        log {
                level INFO
                output file /var/log/caddylog {
                        roll_size 10MB
                        roll_keep 10
                }
        }
        tls internal
        encode gzip
        reverse_proxy /notifications/hub/negotiate 1.2.3.5:8000
        reverse_proxy /notifications/hub 1.2.3.5:3012
        reverse_proxy 1.2.3.5:8000 {
                header_up X-Real-IP {remote_host}
        }
}

3. The problem I’m having:

I am using the ‘tls internal’ function for experimentation purposes with my shinobi server, and so that the Android client will work with the vaultwarden server. Neither server has internet access or a public domain except temporarily when I update them. I use then exclusively over the local network or a VPN. I added the root certificate to both my laptop browsers (Brave and Firefox) and my Android phone but neither the App or browsers would trust the SSL connection. I had to add the intermediate cert and things functioned fine for a few days, but today nothing worked anymore. I eventually figured out that the intermediate cert has been updated and when I checked I found that the new one expires in about a month.

4. Error messages and/or full log output:

There are no error messages as such, just the “untrusted certificate” browser warnings, plus the sync failure in the Android Bitwarden app. I believe that Caddy is behaving exactly as it has been configured to do, so the issue must be in my configuration.

5. What I already tried:

I imported the new intermediate cert to get things up and running again, but manually doing this every month seems impractical and unlikely to be how this system was designed to work. Based on my limited unterstanding of SSL certificate chains of trust, having the root certificate trusted in the browser should be enough for the intermediate certificate to “inherit” that trust. Firefox just reports that the “Connection is not encrypted” but when I expand the error in Brave I see all three certificates, the root, intermediate, and device, but the Issuer is listed as: Caddy Local Authority - ECC Intermediate

It’s as if the browser is not even associating the intermediate cert with the root cert.

6. Links to relevant resources:

Am I missing something obvious here?
Thanks
Judah

2

Based on more research into SSL chains it appears that I need to have Caddy sign the server certs with the root cert directly and remove the intermediate cert, otherwise I will have to constantly be manually updating the intermediate certs on my devices. The documentation mentions in passing that Caddy can use the root cert like this but I have been unable to find any documentation relating to actually implementing this.

In practice I could just skip HTTPS entirely except that Vaultwarden will not work correctly without it. Since I have no intention of exposing my password vault to the internet the only option remaining if I want to continue using Caddy seems to be use the built in CA and import the cert to the devices I want to access my vault as I am trying to do.

The other option would seem to be extend the lifespan of the intermediate cert to a more reasonable length based on this use case, perhaps 12-24 months.

Is there any documentation available for Caddy that explains how to either sign leaf certs with the root directly or extend the intermediate cert lifespan?

3

Intermediate certs have a short lifetime (leaf certs even shorter) – but the root cert is trusted for 10 years.

You shouldn’t need to sign with root unless your device (or whatever software is performing the certificate verification) doesn’t support chains of trust (sigh). The configuration for doing this is in the internal issuer module:

That’s expected: the intermediate is what signs the leaf.

Make sure to restart the various software where you install the root certificate.

4

Hello Matt, thank you both for your reply and your amazing product!

I have tried this with Brave and Firefox on Manjaro, and with Brave and Edge on Windows 10, on two different laptops, and the issue is present everywhere. I did notice something different with Windows. When I examine the certification path in Windows there is a “Certificate Status” field and with the root certificate imported it says “This certificate is ok” for both the Root node and leaf node, but for the intermediate node it says “This certificate has an invalid digital signature”. This seems to imply that the root CA is not signing the intermediate CA correctly perhaps? Is there a way to force Caddy to refresh the internal Root CA and generate all new certs?

5

I would just make sure you’ve installed the right root certificate then. I haven’t really heard of a problem like that before though so my guess is as good as yours.

6

I was fairly certain I had the correct root cert but just in case I started over from scratch. I started with a brand new Ubuntu 20.04 Server container on Proxmox with ports 80 and 443 allowed inbound and here are the commands I ran:

apt update && apt full-upgrade -y
apt install -y debian-keyring debian-archive-keyring apt-transport-https curl
curl -1sLf ‘https://dl.cloudsmith.io/public/caddy/stable/gpg.key’ | sudo tee /etc/apt/trusted.gpg.d/caddy-stable.asc
curl -1sLf ‘https://dl.cloudsmith.io/public/caddy/stable/debian.deb.txt’ | sudo tee /etc/apt/sources.list.d/caddy-stable.list
apt update
apt install caddy
cd /etc/caddy/
nano Caddyfile

Inserted Caddyfile contents given above

caddy run

I scp’d off this file " /root/.local/share/caddy/pki/authorities/local/root.crt " and imported it into Brave, Firefox, and Edge. In all browsers when I navigate to shinobi.domain.net I get an SSL error. Firefox does not give me the option to view the certs, but in Edge and Brave I can click the error and expand the certificate chain. When I do this the certificate on the bottom matches what I see if I run ‘cat root.crt’ in the folder I pulled the root cert from. However, the middle certificate, which I believe is the intermediate certificate, does NOT match what I see when I run ‘cat intermediate.crt’ in that folder.

Am I correct in assuming that the intermediate.crt in that folder should be the intermediate certificate that Caddy is using to sign the shinobi server certificate? If so, what logs or other things can I check on my Caddy implementation to see why these are not matching?

7

I believe so, yes. What are the differences between that certificate and the one the browser shows?

Instead of using the browser, try curl. Browsers do weird things and have known (and unfixed) bugs related to making TLS connections with short-lived certificates.

8

its-a-trap

Caddy will generate a completely new configuration directory and default CA certificate for each “user” you start the service as. The trap gets set if you are attempting to get system trust for the CA configured by running “caddy trust” or start caddy as root on a system. But then you go back to starting caddy using systemctl and the default unit file for it which defines running as user “caddy”. You are showing us symptoms of falling into this trap.

I’m guessing that are importing the caddy root CA certificate from the wrong location. I’m noticing you are focused on /root/.local/* path for the file in what you are sharing , but your systemd unit file is indicating the service is running as caddy. This means you have divergent root CA’s present on the system. When caddy is running, its most likely using its CA config from the /var/lib/caddy/.local/* path instead.

So go check the /var/lib/caddy/.local/* path to verify your root CA is not represented there instead. You can run into issues because the default certificate subject CN for caddy stays the same, but the fingerprint is different, making it hard to identify which certificate is the right one. And if you are “picking” the wrong one you will not be able to establish trust for a TLS connection.

IF YOU DO FIND MORE THAN ONE root.crt, and are confused as to “which one is being used”. You can verify the chain of trust, comparing the “versions” of the root CA you find with the following command:

openssl s_client -connect your.server.ip.or.fqdn:port -CAfile /path/to/file/you/want/to/test/root.crt

You will see a return code of 0 when you are using the proper root CA for the connection,

Here is an example of what I’m talking about from the command line, remember to use ctrl-d to exit the openssl s_client command’s interactive shell.

BAD TEST CASE: First we look at a connection that is failing to establish trust… Note towards the end of the output for the openssl s_client comment indicates failure to get issuers certificate (with a return code of 20, you want that to be 0 to be a proper TLS configuration. We are mapping the CA certificate file out of root’s path with the -CAfile switch to the command. Openssl will apply the root CA certificate you provide it to establish trust.

$ sudo openssl s_client -connect 192.167.37.122:443 -CAfile /root/.local/share/caddy/pki/authorities/local/root.crt

depth=1 CN = Caddy Local Authority - ECC Intermediate
verify error:num=20:unable to get local issuer certificate
CONNECTED(00000003)
---
Certificate chain
 0 s:
   i:/CN=Caddy Local Authority - ECC Intermediate
 1 s:/CN=Caddy Local Authority - ECC Intermediate
   i:/CN=Caddy Local Authority - 2021 ECC Root
---
Server certificate
-----BEGIN CERTIFICATE-----
MIIBuDCCAV6gAwIBAgIQbVwt7gPBuMi/SExAxHg2yDAKBggqhkjOPQQDAjAzMTEw
LwYDVQQDEyhDYWRkeSBMb2NhbCBBdXRob3JpdHkgLSBFQ0MgSW50ZXJtZWRpYXRl
MB4XDTIxMDkwNDE4MjYzOVoXDTIxMDkwNTA2MjYzOVowADBZMBMGByqGSM49AgEG
CCqGSM49AwEHA0IABH2vwCTWHz+GuujV8jsI5QTrwnAPmMlgnYm/rbV7mRzL1M3T
kYLlh8z79jGiRu/FWSKOq7K5JIkJk5LNrDoaoL2jgYYwgYMwDgYDVR0PAQH/BAQD
AgeAMB0GA1UdJQQWMBQGCCsGAQUFBwMBBggrBgEFBQcDAjAdBgNVHQ4EFgQUdd2z
p/imZXhC3unyF2gp/y4ML1UwHwYDVR0jBBgwFoAUv6yXKX103AhkLraxhQCI9/y3
K24wEgYDVR0RAQH/BAgwBocEwKglejAKBggqhkjOPQQDAgNIADBFAiEAq53etBpH
Uk+FNWSSFOzcp9qK6eT/KsRkaCmRSFy7rV0CIFzU0bpA2GGWQcW8/ACio1OXYLam
XPH9bHskeOAJ+vKp
-----END CERTIFICATE-----
subject=
issuer=/CN=Caddy Local Authority - ECC Intermediate
---
No client certificate CA names sent
Peer signing digest: SHA512
Server Temp Key: ECDH, P-256, 256 bits
---
SSL handshake has read 1342 bytes and written 391 bytes
---
New, TLSv1/SSLv3, Cipher is ECDHE-ECDSA-AES256-GCM-SHA384
Server public key is 256 bit
Secure Renegotiation IS supported
Compression: NONE
Expansion: NONE
No ALPN negotiated
SSL-Session:
    Protocol  : TLSv1.2
    Cipher    : ECDHE-ECDSA-AES256-GCM-SHA384
    Session-ID: C4B799F4BD309BCACE15D817228AD614C14C17E9D2490D75867E5A1133864C85
    Session-ID-ctx: 
    Master-Key: 63E3BFBDE712683D56C8EB2AFB2C4B58FE9A79E9A8A4802CBA0FA60374AAF02BFBEFA0BFCFBC751B12976F86355A866E
    Key-Arg   : None
    PSK identity: None
    PSK identity hint: None
    SRP username: None
    TLS session ticket:
    0000 - 64 70 33 ee ef 99 97 2b-93 11 37 b7 0c b4 60 ba   dp3....+..7...`.
    0010 - a7 1b 03 81 69 93 23 35-97 0e 4a 82 bb e5 ab 66   ....i.#5..J....f
    0020 - 18 45 48 6a 49 ba 4e ff-59 29 d1 fc 26 f9 b4 ed   .EHjI.N.Y)..&...
    0030 - 22 d6 80 45 04 82 50 36-d4 91 8e 53 dd 48 45 7d   "..E..P6...S.HE}
    0040 - 0d e0 f7 da fe 80 ef 67-35 ff 75 c0 4d 37 9e 95   .......g5.u.M7..
    0050 - 22 4d 32 26 f7 a8 3a 37-1f 36 a4 f0 7a 0c 98 e0   "M2&..:7.6..z...
    0060 - b9 b6 7f 90 ab 0d c9 01-37 a5 6d aa ff 46 7e b3   ........7.m..F~.
    0070 - 57 f5 90 a9 a7 aa fa de-d1 ef 2d c1 3f 5e 52 df   W.........-.?^R.
    0080 - 56                                                V

    Start Time: 1630780585
    Timeout   : 300 (sec)
    Verify return code: 20 (unable to get local issuer certificate)
---
DONE

Now for the “GOOD TEST” using the actual root CA certificate that caddy is actually starting with by default from the systemctl command.

$ sudo openssl s_client -connect 192.167.37.122:443 -CAfile /var/lib/caddy/.local/share/caddy/pki/authorities/local/root.crt

depth=2 CN = Caddy Local Authority - 2021 ECC Root
verify return:1
depth=1 CN = Caddy Local Authority - ECC Intermediate
verify return:1
depth=0 
verify return:1
CONNECTED(00000003)
---
Certificate chain
 0 s:
   i:/CN=Caddy Local Authority - ECC Intermediate
 1 s:/CN=Caddy Local Authority - ECC Intermediate
   i:/CN=Caddy Local Authority - 2021 ECC Root
---
Server certificate
-----BEGIN CERTIFICATE-----
MIIBuDCCAV6gAwIBAgIQbVwt7gPBuMi/SExAxHg2yDAKBggqhkjOPQQDAjAzMTEw
LwYDVQQDEyhDYWRkeSBMb2NhbCBBdXRob3JpdHkgLSBFQ0MgSW50ZXJtZWRpYXRl
MB4XDTIxMDkwNDE4MjYzOVoXDTIxMDkwNTA2MjYzOVowADBZMBMGByqGSM49AgEG
CCqGSM49AwEHA0IABH2vwCTWHz+GuujV8jsI5QTrwnAPmMlgnYm/rbV7mRzL1M3T
kYLlh8z79jGiRu/FWSKOq7K5JIkJk5LNrDoaoL2jgYYwgYMwDgYDVR0PAQH/BAQD
AgeAMB0GA1UdJQQWMBQGCCsGAQUFBwMBBggrBgEFBQcDAjAdBgNVHQ4EFgQUdd2z
p/imZXhC3unyF2gp/y4ML1UwHwYDVR0jBBgwFoAUv6yXKX103AhkLraxhQCI9/y3
K24wEgYDVR0RAQH/BAgwBocEwKglejAKBggqhkjOPQQDAgNIADBFAiEAq53etBpH
Uk+FNWSSFOzcp9qK6eT/KsRkaCmRSFy7rV0CIFzU0bpA2GGWQcW8/ACio1OXYLam
XPH9bHskeOAJ+vKp
-----END CERTIFICATE-----
subject=
issuer=/CN=Caddy Local Authority - ECC Intermediate
---
No client certificate CA names sent
Peer signing digest: SHA512
Server Temp Key: ECDH, P-256, 256 bits
---
SSL handshake has read 1343 bytes and written 391 bytes
---
New, TLSv1/SSLv3, Cipher is ECDHE-ECDSA-AES256-GCM-SHA384
Server public key is 256 bit
Secure Renegotiation IS supported
Compression: NONE
Expansion: NONE
No ALPN negotiated
SSL-Session:
    Protocol  : TLSv1.2
    Cipher    : ECDHE-ECDSA-AES256-GCM-SHA384
    Session-ID: BECFE09719FCEB2C9B1A4423AD874E88B82AD78FE1C996147AB535EBA5BC6DD6
    Session-ID-ctx: 
    Master-Key: 3CCB7FFD5A38EC7433B7FFF646FB269C41024A1FF81E7179F8098A6207C34E1BB0587BAA5AB5F710035971B4C251FE71
    Key-Arg   : None
    PSK identity: None
    PSK identity hint: None
    SRP username: None
    TLS session ticket:
    0000 - 64 70 33 ee ef 99 97 2b-93 11 37 b7 0c b4 60 ba   dp3....+..7...`.
    0010 - fb 2d 83 37 a3 2c 94 90-0d d3 7f 85 50 5b 48 1d   .-.7.,......P[H.
    0020 - 72 b5 5d b8 7c be e8 04-be 3c ac 9a 18 a1 26 fe   r.].|....<....&.
    0030 - 4d 67 08 ee 69 25 17 f0-3a a7 cb 9e 94 61 c3 63   Mg..i%..:....a.c
    0040 - ba 7b a4 1b cc 0b dd e6-6c bb a7 9b a6 d4 e5 91   .{......l.......
    0050 - 02 df b4 82 b8 e8 e7 ce-94 bb bb 54 8b 09 20 b2   ...........T.. .
    0060 - 87 c9 37 da 35 4c ea 3d-fa 54 56 79 3f 85 aa 24   ..7.5L.=.TVy?..$
    0070 - 8e 2b e6 e1 ce d4 0f ca-de f9 06 6a 63 81 c2 b5   .+.........jc...
    0080 - f2                                                .

    Start Time: 1630780564
    Timeout   : 300 (sec)
    Verify return code: 0 (ok)
---
DONE
1 Like
9

(sorry for the spammy re-edits, posted the wrong output for the good test)

10

You’re totally right, it’s a trap.

The right way to run caddy trust when running Caddy as a systemd service is like this:

sudo HOME=~caddy caddy trust

This will make sure that it sets the HOME env for caddy trust to the HOME of the caddy user (i.e. ~caddy) which Caddy will actually run as.

It needs to be done this way, because you need root/sudo to actually write the root cert to the system’s trust store, but if you just run sudo caddy trust, then it has HOME set to /root, so it’ll generate a new CA there instead of in /var/lib/caddy which is the caddy user’s HOME.

Yes, this is terrible UX and could be better… but it’s complicated.

Also, when Caddy runs, due to the --environ flag, it will print out the paths it’ll use for certificate storage right at the start. Make sure to use those paths when you’re grabbing the root.crt.

2 Likes
11

Thank you! This was the issue. I pulled the root cert from that location and it works!

In a Proxmox unprivileged container the default user is root, so following the path given in the documentation took me to the location where I pulled the original cert, not /var/lib/.

1 Like
12

One nuanced point/question; In our raspberry pi based deployment, we create the caddy user/group specifically like so:

groupadd caddy
useradd --system --gid caddy --create-home --home-dir /var/lib/caddy -c "Caddy web server" --shell /bin/bash caddy

so when implementing your example from our setup script (which is running via sudo) i’m planning on doing the following

HOME=/var/lib/caddy caddy trust

as opposed to
HOME=~caddy caddy trust

which i’m concerned is based on /var/lib being “home”? Or am I completely misunderstanding ~caddy there?

13

~caddy is the same as /var/lib/caddy, because the HOME of the caddy user is /var/lib/caddy. See the script in the .deb:

Basically ~<user> is a shortcut for giving you the HOME of a particular user. ~ on its own is the HOME of the current user.

You can test it by running echo ~caddy

2 Likes
14

This topic was automatically closed after 30 days. New replies are no longer allowed.