Self-signed certs don't remain trusted?

websurfr · February 6, 2022, 9:19pm

1. Caddy version (`caddy version`):

v2.4.5 h1:P1mRs6V2cMcagSPn+NWpD+OEYUYLIf6ecOa48cFGeUg=

2. How I run Caddy:

Installed from cloudsmith.io
Local root cert created using caddy trust
Started via std systemd service file

a. System environment:

Raspberry Pi OS, 2022-01-28 release, (based on debian bullseye)

c. Service/unit/compose file:

[Unit]
Description=Caddy
Documentation=https://caddyserver.com/docs/
After=network.target network-online.target
Requires=network-online.target

[Service]
Type=notify
User=caddy
Group=caddy
ExecStart=/usr/bin/caddy run --environ --config /etc/caddy/Caddyfile
ExecReload=/usr/bin/caddy reload --config /etc/caddy/Caddyfile
TimeoutStopSec=5s
LimitNOFILE=1048576
LimitNPROC=512
PrivateTmp=true
ProtectSystem=full
AmbientCapabilities=CAP_NET_BIND_SERVICE

[Install]
WantedBy=multi-user.target

d. My complete Caddyfile or JSON config:

{
        admin off
        auto_https disable_redirects
        on_demand_tls {
                ask http://localhost:8081/ip-ok
        }
}

:443 {
        # where to look for files
        root * /opt/sensorgnome/web-portal/public
        # if path is / and file need_init exists then rewrite to config.html
        # so the user gets the initial configuration stuff
        @start_init {
                path /
                file need_init
        }
        rewrite @start_init /init/config
        # handle initial configuration paths
        handle_path /init/* {
                # if the request doesn't match a file, try an action
                @init_action not file {path}
                reverse_proxy @init_action localhost:8081
                # else serve file
                file_server
        }
        # handle captive portal magic
        redir /generate_204 /
        redir /gen_204 /
        redir /blank.html /
        redir /mobile/status.php /
        redir /hotspot-detect.html /
        @captive_ua header User-Agent *CaptiveNetworkSupport*
        redir @captive_ua /init/apple-hotspot.html
        # error if we need init and nothing above handles the request
        @need_init file need_init
        respond @need_init 404
        # normal sg-control app only if init is done
        reverse_proxy localhost:8080
        # ensure we get automatic certs and redirect from HTTP
        tls internal {
                on_demand
        }
}

:80 {
        rewrite * /redirect
        reverse_proxy localhost:8081
}

3. The problem I’m having:

In general, the self-signed certificates work well and as expected, except that they’re not trusted after 24 hours, i.e. after renewal. Any suggestions to avoid this?

Specifically:

Access the site with a specific browser&laptop
Jump through the browser’s hoops to accept the certificate
Enjoy the site, then go away
Come back 24hrs later, after the certificate has expired and access the site again using the same browser&laptop
Observe that you have to jump through the hoops all over again, why? I assume because the browser marks the end certificate as trusted, not the intermediate or root cert?

Background:

I’m rebuilding software for an IoT device to be used for scientific data collection in the field with over 1000 devices deployed
About half will have an internet connection on some random network, the other half will not (only accessed through hotspot provided by the device)
The devices will be installed by their users, many at gov agencies or non-profits, most don’t have any network knowledge
Any proper HTTPS cert (non self-signed) would require accessing devices using something like device-1234.example.com with a domain I host, would require updating that domain as devices get new DHCP leases and thus change IP addresses, and would require the use of ACME-DNS for cert renewal, and all that really only works for the devices that are connected (which can be accessed more easily through a central management site…)

It’s quite appalling that browsers don’t support any user-friendly HTTPS option for disconnected IoT devices…

Is there a set-up that lets users trust a device’s self-signed cert once and not every day? (This would also be more secure 'cause it would flag if someone were to try impersonating a device.)

4. Steps to reproduce

Start caddy with auto-https self-signed certs
Access site with web browser (chrome and firefox on linux are what I tested), click through warnings to trust the certificate
Terminate browser process completely
Restart browser, access the site again, observe that the self-signed cert remains trusted (no warning)
Terminate browser process again
Delete caddy’s certificates (in my case rm -rf /var/lib/caddy/.local/share/caddy/certificates/local/192.168.0.92)
Restart caddy
Restart browser, access the site yet again, observe that the new self-signed cert is no longer trusted

The logs after restarting caddy indicate that the root cert did not get changed by caddy, only the site cert:

Feb 06 16:05:39 SG-1BA7RPI47F3A caddy[19953]: {"level":"info","ts":1644181539.6719332,"logger":"pki.ca.local","msg":"root certificate is already trusted by system","path":"storage:pki/authorities/local/root.crt"}
Feb 06 16:05:39 SG-1BA7RPI47F3A caddy[19953]: {"level":"info","ts":1644181539.6724877,"logger":"tls","msg":"cleaning storage unit","description":"FileStorage:/var/lib/caddy/.local/share/caddy"}
...
Feb 06 16:05:41 SG-1BA7RPI47F3A caddy[19953]: {"level":"info","ts":1644181541.199466,"logger":"tls.on_demand","msg":"obtaining new certificate","server_name":"192.168.0.92"}
Feb 06 16:05:41 SG-1BA7RPI47F3A caddy[19953]: {"level":"info","ts":1644181541.2016525,"logger":"tls.obtain","msg":"acquiring lock","identifier":"192.168.0.92"}
Feb 06 16:05:41 SG-1BA7RPI47F3A caddy[19953]: {"level":"info","ts":1644181541.216271,"logger":"tls.obtain","msg":"lock acquired","identifier":"192.168.0.92"}
Feb 06 16:05:41 SG-1BA7RPI47F3A caddy[19953]: {"level":"info","ts":1644181541.2304127,"logger":"tls.obtain","msg":"certificate obtained successfully","identifier":"192.168.0.92"}
Feb 06 16:05:41 SG-1BA7RPI47F3A caddy[19953]: {"level":"info","ts":1644181541.230971,"logger":"tls.obtain","msg":"releasing lock","identifier":"192.168.0.92"}
Feb 06 16:05:41 SG-1BA7RPI47F3A caddy[19953]: {"level":"warn","ts":1644181541.23515,"logger":"tls","msg":"stapling OCSP","error":"no OCSP stapling for [192.168.0.92]: no OCSP server specified in certificate"}

francislavoie · February 6, 2022, 10:42pm

Correct. You need to add the root CA cert to the client systems’ trust store.

You can find the root CA cert at /var/lib/caddy/.local/share/caddy/pki/authorities/local/root.crt

You can google for instructions on adding CA certs to trust stores, for whatever browser/OS you need it on. Unfortunately, this will always be a manual step, but it only needs to be done once per system. Note that some browsers have their own trust store and may ignore the system’s.

As a point of clarification, these are not self-signed certs. A self-signed cert is one where the same private key is used to sign the certificate as to encrypt the connection.

This is not what’s happening, Caddy generates a certificate authority (CA) which has its own private key, and uses that to sign an intermediate which also has its own private key, and the intermediate is what signs the leaf certs. The leaf certs have a 24h lifetime, and the intermediate has a 7 day lifetime.

That command isn’t meant to create the root cert, but to install the root cert into the same machine’s trust store.

But this currently shouldn’t be used directly, because it’ll use the wrong storage location unless you override the HOME environment variable when running that command. It’s complicated, but it’ll hopefully be fixed in the next release to pull from whatever instance of Caddy you currently have running on that machine.

github.com/caddyserver/caddy

pki: Implement `GET /pki/certificates/<id>` API, rework `caddy trust`

caddyserver:master ← caddyserver:trust-improvements

opened 07:00AM - 25 Nov 21 UTC

francislavoie

+483 -110

Fixes https://github.com/caddyserver/caddy/issues/4058, once this is completed. …This is basically half-way to solving the linked issue. This implements a new admin API, `GET /pki/certificates/<id>`, which returns information about the given PKI app's CA, by ID. For example, this is what a response might look like, for requesting info about the `local` CA (the default CA ID): ```bash $ curl localhost:2019/pki/certificates/local ``` ```json { "id": "local", "name": "Caddy Local Authority", "root": "-----BEGIN CERTIFICATE-----\nMIIBpDCCAUmgAwIBAgIQTS5a+3LUKNxC6qN3ZDR8bDAKBggqhkjOPQQDAjAwMS4w\nLAYDVQQDEyVDYWRkeSBMb2NhbCBBdXRob3JpdHkgLSAyMDIxIEVDQyBSb290MB4X\nDTIxMTEyNTA1NDIxMloXDTMxMTAwNDA1NDIxMlowMDEuMCwGA1UEAxMlQ2FkZHkg\nTG9jYWwgQXV0aG9yaXR5IC0gMjAyMSBFQ0MgUm9vdDBZMBMGByqGSM49AgEGCCqG\nSM49AwEHA0IABIFW7LqyWf7ybSXh6FREAgGL+Bz9TktcmcmiKHAZXhWH8zCqun3k\nIg2mya+GgnA0k39OPrCaVUUMb+96UfldA9ujRTBDMA4GA1UdDwEB/wQEAwIBBjAS\nBgNVHRMBAf8ECDAGAQH/AgEBMB0GA1UdDgQWBBTrVHm2qnZLcRzky7u1NO2n8jUJ\n5DAKBggqhkjOPQQDAgNJADBGAiEA2JTbqaLhEbki73/Eb4k/KBVzSTwSKzqbssKQ\n9M9t0FwCIQCAlUr4ZlFzHE/3K6dARYKusR1ck4A3MtucSSyar6lgRw==\n-----END CERTIFICATE-----\n", "intermediate": "-----BEGIN CERTIFICATE-----\nMIIByDCCAW2gAwIBAgIQGRTBhcgdsiLXv0AA6r8GCTAKBggqhkjOPQQDAjAwMS4w\nLAYDVQQDEyVDYWRkeSBMb2NhbCBBdXRob3JpdHkgLSAyMDIxIEVDQyBSb290MB4X\nDTIxMTEyNTA1NDIxMloXDTIxMTIwMjA1NDIxMlowMzExMC8GA1UEAxMoQ2FkZHkg\nTG9jYWwgQXV0aG9yaXR5IC0gRUNDIEludGVybWVkaWF0ZTBZMBMGByqGSM49AgEG\nCCqGSM49AwEHA0IABNHTWLDx5jxS7x3WYEBIOXfuxQP2yvfitpEPiHJNJsf44XF8\ns31IBrDcd81Xzn63b8S2qn6elSTnZK1DdrCdtnSjZjBkMA4GA1UdDwEB/wQEAwIB\nBjASBgNVHRMBAf8ECDAGAQH/AgEAMB0GA1UdDgQWBBTvu/Kpad1XWLbaTHoqNS4U\nLBSfBTAfBgNVHSMEGDAWgBTrVHm2qnZLcRzky7u1NO2n8jUJ5DAKBggqhkjOPQQD\nAgNJADBGAiEAhUhwv6EQCgS+SGMnaze5FvHk0a9yfhDCdqmAdadhsjsCIQCTTNZF\njFM8CQL3vCfpCVzTiPR8EUbaCqxj8O9sg/70dQ==\n-----END CERTIFICATE-----\n" } ``` To get there though, I had to refactor a bit of the `admin.go` and `caddy.go` internals. The tricky bit is that this is the first `AdminRouter` module that needs a `caddy.Context` to be able to fetch Caddy apps from the current config at runtime. To solve it, I set up a temporary slice in `AdminConfig` to remember the routers that had their routes registered (previously, they were just forgotten immediately after), so that we can later call `Provision(ctx)` on them (if they need to be provisioned). I also had to rework how `replaceLocalAdminServer` handles the default admin config, because I needed to make sure it would write the actual admin config in use back to the `caddy.Config` so that we could later do the provisioning stuff. Previously, it would just read from the config if it was non-nil, or use a default, and spin up the admin server. But that's not really consistent with how the rest of Caddy works, because all the other apps are actually kept as references in `caddy.Config`, I think. It also gets rid of a global var, which is nice I guess. --- All that said, this isn't enough to call the issue done, we still need to implement the changes to the `caddy trust` so that it actually uses this new API, which would make the `caddy trust` process not need to care about its own storage as a side effect, instead it would always reach out to the API to get the correct root CA cert.

websurfr · February 7, 2022, 5:34am

I’m a bit hesitant about doing that. If users add the root CA cert to their trust store it means that anything signed by that cert is trusted, right? An attacker who can get to the root CA cert (e.g. physical access to the IoT device) could sign a cert for google.com and spoof the site (ok, it’s not quite as easy, but still). So if I go down that route I’m recommending users to run around with laptops that have a major vulnerability installed, right? Am I completely off-base?

Yup, sorry for the loose terminology.

francislavoie · February 7, 2022, 5:49am

That’s not true, that would only be possible if the private key of the CA was taken. The private key is the important, secret part. The certificate is just a mechanism for transporting the public key, basically.

Obviously, the private keys should never leave the system on which Caddy is running. As long as you protect access to that machine to only those who absolutely need it, then you’re fine.

websurfr · February 7, 2022, 6:18am

Duh, my brain is fried… I did mean the root CA key. In the case of an IoT rPi it’s pretty trivial to steal for anyone having physical access: pop out the SDcard, mount it, read it…

I’m wondering whether it’s possible to create a root CA cert that is constrained to only allow IP addresses and install that in all the devices. This way if users choose to install that in their trust store they are only exposed to attacks on URLs using an IP address, which I’d rate to be a relatively low risk. I wish there was a good solution to all this madness…

francislavoie · February 7, 2022, 6:34am

Nope, trust stores are just an allow list, there’s no additional rules.

If a physical attack is really a concern, then you should have that device under lock and key (literally).

Like, honestly you probably have bigger things to worry about if someone has physical access in the first place.

system · March 8, 2022, 9:19pm

This topic was automatically closed after 30 days. New replies are no longer allowed.