Local Caddy Selfsigned CA is creating certs that google chrome will not accept

1. Caddy version (caddy version):

$ caddy version
v2.2.2 h1:Ha3bvEvkb/GLGEX648/qI5zTt6uJCnfQhZHmZBxhzDY=

2. How I run Caddy:

Caddy is running as a reverse proxy over a set of applications, and is configured for local_certs to provide TLS (The solution is internal only). Clients are connecting via current chrome browser.

The installation is on a raspberry pi running raspbian lite OS (details below).

a. System environment:

Linux pidev 5.4.72-v7l+ #1356 SMP Thu Oct 22 13:57:51 BST 2020 armv7l GNU/Linux

$ cat /etc/os-release
PRETTY_NAME=“Raspbian GNU/Linux 10 (buster)”
NAME=“Raspbian GNU/Linux”
VERSION_ID=“10”
VERSION=“10 (buster)”
VERSION_CODENAME=buster
ID=raspbian
ID_LIKE=debian

$ openssl version
OpenSSL 1.1.1d 10 Sep 2019

Windows 10 desktop client running google chrome Version 87.0.4280.88 (Official Build) (64-bit)

b. Command:

sudo systemctl start caddy

c. Service/unit/compose file:

# caddy.service

    [Unit]
    Description=Caddy
    Documentation=https://caddyserver.com/docs/
    After=network.target network-online.target
    Requires=network-online.target

    [Service]
    User=caddy
    Group=caddy
    #plugin modified caddy build for auth-portal and associated plugins
    #/usr/bin/caddy now linked to /opt/caddy/releases/[current build]
    ExecStart=/usr/bin/caddy run --environ --config /etc/caddy/Caddyfile
    #ExecStart=/opt/caddy/bin/caddy_linux_arm7_custom run --environ --config /etc/caddy/Caddyfile
    ExecReload=/usr/bin/caddy reload --config /etc/caddy/Caddyfile
    #ExecReload=/opt/caddy/bin/caddy_linux_arm7_custom reload --config /etc/caddy/Caddyfile
    TimeoutStopSec=5s
    LimitNOFILE=1048576
    LimitNPROC=512
    PrivateTmp=true
    ProtectSystem=full
    AmbientCapabilities=CAP_NET_BIND_SERVICE

    [Install]
    WantedBy=multi-user.target

d. My complete Caddyfile or JSON config:

{
        storage file_system {
                root /opt/caddy/storage
        }
        local_certs
        http_port 80
        https_port 443
}
https://demo-mguard.int.example.com, https://10.252.252.20 {
        # caddy-auth-portal with local db config
        log {
                level INFO
                format console
                output file /var/log/caddy/caddy.log
        }
        # caddy auth portal login
        route /auth* {
                auth_portal {
                        path /auth
                        backends {
                                local_backend {
                                        method local
                                        path /opt/caddy/assets/conf/local/auth/user_db.json
                                        realm local
                                }
                        }
                        jwt {
                                token_name access_token
                                token_secret AnExampleSecretString123
                        }
                        ui {
                                links {
                                        "Elite Manager" /
                                }
                        }
                }
        }

        route /login {
                reverse_proxy http://localhost:1081
        }
        route /ui/* {
                jwt {
                        enable claim headers
                }
                reverse_proxy http://localhost:1880
        }
        route /gr/* {
                jwt {
                        enable claim headers
                }
                reverse_proxy http://localhost:3000
        }
        route /version* {
                respond * "2.0.0-a" 200
        }
        # custom webapp running on port 1080 route
        route /* {
                jwt {
                        primary yes
                        trusted_tokens {
                                static_secret {
                                        token_name access_token
                                        token_secret AnExampleSecretString123
                                }
                        }
                        enable claim headers
                }
                reverse_proxy http://localhost:1081
        }
}

3. The problem I’m having:

When using the local certs option of caddy for TLS, the certificate issued appears to be triggering a severe response that is beyond the normal “untrusted certificate” process a user has to go through when dealing with selfsigned/untrusted certs. In this case the broswer does not allow the user to “connect anyways” so the user can never fully connect through the proxy.

These (apparently new?) chrome TLS warnings do not allow a user to bypass the “unsecure” connection with the following warning, there is no visible way in the web page to proceed. Attempting to clear HSTS for the domain has no impact. The same alert takes place attempting to connect via IP or FQDN. Whatever is triggering the response in the browser is making the local certs implementation almost unusable?

4. Error messages and/or full log output:

demo-mguard.int.example.com normally uses encryption to protect your information. When Google Chrome tried to connect to demo-mguard.int.example.com this time, the website sent back unusual and incorrect credentials. This may happen when an attacker is trying to pretend to be demo-mguard.int.example.com, or a Wi-Fi sign-in screen has interrupted the connection. Your information is still secure because Google Chrome stopped the connection before any data was exchanged.

You cannot visit demo-mguard.int.example.com right now because the website sent scrambled credentials that Google Chrome cannot process. Network errors and attacks are usually temporary, so this page will probably work later.

(Note it never works)

5. What I already tried:

I’ve examined the certificate being presented with ‘openssl s_client -connect hostname:port | openssl x509 -noout -txt’ to check the certificate, I see the san and no subject DN being which I understand is fine. I’m going to set up wireshark to examine the TLS session setup as well… but wanted to get this discussion started.

I have read the discussions around IP address based HTTPS URL’s and the issue that the caddy has with providing proper response to SSL_get_servername when IP is used, but in our case we went back and configured FQDN and host file entries on the nodes in question (server, client) and are still getting the same harsh dead end on TLS warnings in latest chrome.

I did identify a way to bypass this in chrome by literally clicking in the middle of the page and typing (without quotes)

thisisunsafe

And then chrome whitelists the site for the current session until chrome is restarted. But obviously that is not a usable solution. (thanks stackoverflow!)

6. Links to relevant resources:

where I found where to get past the warning in chrome (not exact same issue, but still provides bypass that allows this to work)

Further testing:

Firefox 81.0.2 on same windows client - worked as expected (certificate warning page, accepted warning, and was allowed to continue through to connection)

Firefox 83.0 on same windows client - worked as well

Issue seems to be unique to latest chrome?

Microsoft edge produces the exact same warning page as chrome and will not allow the user to proceed. The “thisisunsafe” works the same in edge too.

For windows; I’m going to research how to properly clean out the CA Chain here. I had a caddy dev instance on windows that is the same system that is the client to the proxy that is showing the “scrambled information” error message. I’m thinking there is a collision between the root/intermediary CA of the windows server that is in the cert store, and the new root/intermediary. When connecting locally the first time it installed the caddy CA certificates from the windows system.

I think that chrome (chromium) is flipping out over the fingerprint mismatch of the local Root CA to linux Intermediary CA being passed with the server certificate on the connection setup, and goes into a failsafe warning.

I’m going to clean up these old certs, manually import the linux CA certificates into the windows client.

Here is what makes me think this is CA collision:

  1. there are not 100 other people yammering about chrome losing its mind
  2. when looking at the “user” certificate store for trusted CA’s I see 2 of the same caddy CA’s “CN = Caddy Local Authority - 2020 ECC Root”.

  1. the linux certificate is (obviously) different as well.
agrajag@TOR-RYZEN:~$ openssl x509 -in caddyp1.pem -noout -text
Certificate:
    Data:
        Version: 3 (0x2)
        Serial Number:
            1f:74:c5:94:3d:ab:fd:d9:2d:48:73:41:e5:57:86:99
    Signature Algorithm: ecdsa-with-SHA256
        Issuer: CN=Caddy Local Authority - ECC Intermediate
        Validity
            Not Before: Dec 11 03:26:52 2020 GMT
            Not After : Dec 11 15:26:52 2020 GMT
        Subject:
        Subject Public Key Info:
            Public Key Algorithm: id-ecPublicKey
                Public-Key: (256 bit)
                pub:
                    04:dc:d3:c9:35:f5:30:57:63:23:f1:7f:69:c8:c7:
                    a9:86:cb:da:73:b6:bb:e4:a3:80:dd:19:3c:97:48:
                    d1:d3:8c:c6:a8:9b:cc:a2:c4:e9:21:56:17:84:52:
                    ac:96:3e:14:7c:f4:1b:e9:84:50:95:7f:05:0c:b3:
                    8a:fe:10:a0:15
                ASN1 OID: prime256v1
                NIST CURVE: P-256
        X509v3 extensions:
            X509v3 Key Usage: critical
                Digital Signature
            X509v3 Extended Key Usage:
                TLS Web Server Authentication, TLS Web Client Authentication
            X509v3 Subject Key Identifier:
                63:23:50:C0:99:3D:D4:EF:C6:1D:0E:84:E1:52:CD:5C:38:03:11:4E
            X509v3 Authority Key Identifier:
                keyid:75:5D:2B:AD:1D:E4:B1:08:D2:F1:8B:9B:10:B3:E2:37:08:0C:48:AB

            X509v3 Subject Alternative Name: critical
                DNS:demo-mguard.int.example.com
    Signature Algorithm: ecdsa-with-SHA256
         30:46:02:21:00:c1:b9:a4:36:51:20:2e:75:4a:9c:4a:15:e1:
         96:77:62:9c:e9:8a:34:bf:ad:f9:54:2b:81:d8:8d:8a:93:e5:
         b6:02:21:00:85:53:0f:b9:98:c6:30:79:13:a9:60:b1:77:ad:
         74:7c:3d:1b:96:eb:03:37:00:fa:e5:98:b5:93:5f:44:62:c2

I’ve confirmed the issue seen was related to having conflicting root CA’s present for different instances of the Caddy server and its local root CA.

Take away is that you need to make sure to clean out trusted Root CA’s if you have to redeploy caddy and it generates a new unique root CA.

Possible workaround in production situation would be to take the generated certs private key and certs from the CA between installs, but that a whole other discussion.

1 Like

Thanks for following up with the solution!

Caddy persists its root CA in storage along with other certs and keys, so simply preserving that (as is needful anyway) will avoid the problem.

This topic was automatically closed after 30 days. New replies are no longer allowed.