Implementing mTLS

Mihir_Solanki · August 10, 2023, 4:20pm

1. The problem I’m having:

I have a service that is public facing. I use Caddy as a reverse proxy. I am tring to implement mutual tls for this service so that only authorized users having valid certificates are able to access it over the internet.

The following is my caddy file located at "/etc/caddy/"

(mutual_tls) {
        tls {
                protocols tls1.3
                client_auth {
                        mode require_and_verify
                        trusted_ca_cert_file /etc/caddy/cert_name-CA.crt
                        trusted_leaf_cert_file /etc/caddy/cert_name.crt
                }
        }
}


subdomain.domains.com {
        log {
                output file /var/log/caddy/access.log
        }

        import mutual_tls
        reverse_proxy 10.10.0.22:9000
}

For this, I generated the certificates using the following lines:
source: Reverse proxy client certificates for dummies - Caddy edition -Reddit Link

openssl req -x509 -newkey rsa:4096 -keyout cert_name.key -out cert_name.crt -days 365
openssl req -new -key cert_name.key -out cert_name.csr
openssl x509 -req -days 365 -in cert_name.csr -signkey cert_name.key -out cert_name-CA.crt
cat cert_name.crt cert_name.key > cert_name.pem
openssl pkcs12 -export -out cert_name.p12 -inkey cert_name.key -in cert_name.pem

And then I installed the pkcs12 certificate on my windows machine and visited subdomain.domains.com and I got the error noted in point 2.

2. Error messages and/or full log output:

The connection for this site is not secure
subdomain.domains.com didn’t accept your login certificate, or a login certificate may not have been provided.
Try contacting your organization.

ERR_BAD_SSL_CLIENT_AUTH_CERT

3. Caddy version:

v2.7.3 h1:eMCNjOyMgB5A1KgOzT2dXKR4I0Va+YHCJYC8HHu+DP0=

4. How I installed and ran Caddy:

a. System environment:

Raspberry Pi 4B
Raspian OS

b. Command:

Caddy is installed as systemd service to run on boot.

c. Service/unit/compose file:

# caddy.service
#
# For using Caddy with a config file.
#
# Make sure the ExecStart and ExecReload commands are correct
# for your installation.
#
# See https://caddyserver.com/docs/install for instructions.
#
# WARNING: This service does not use the --resume flag, so if you
# use the API to make changes, they will be overwritten by the
# Caddyfile next time the service is restarted. If you intend to
# use Caddy's API to configure it, add the --resume flag to the
# `caddy run` command or use the caddy-api.service file instead.

[Unit]
Description=Caddy
Documentation=https://caddyserver.com/docs/
After=network.target network-online.target
Requires=network-online.target

[Service]
Type=notify
User=caddy
Group=caddy
ExecStart=/usr/bin/caddy run --environ --config /etc/caddy/Caddyfile
ExecReload=/usr/bin/caddy reload --config /etc/caddy/Caddyfile --force
TimeoutStopSec=5s
LimitNOFILE=1048576
LimitNPROC=512
PrivateTmp=true
ProtectSystem=full
AmbientCapabilities=CAP_NET_ADMIN CAP_NET_BIND_SERVICE

[Install]
WantedBy=multi-user.target

d. My complete Caddy config:

(mutual_tls) {
        tls {
                protocols tls1.3
                client_auth {
                        mode require_and_verify
                        trusted_ca_cert_file /etc/caddy/cert_name-CA.crt
                        trusted_leaf_cert_file /etc/caddy/cert_name.crt
                }
        }
}


subdomain.domains.com {
        log {
                output file /var/log/caddy/access.log
        }

        import mutual_tls
        reverse_proxy 10.10.0.22:9000
}

francislavoie · August 10, 2023, 4:45pm

Enable the debug global option. What do you see in your logs?

Mihir_Solanki · August 10, 2023, 5:25pm

The debug log shows this:

{"level":"debug","ts":1691687767.5200946,"logger":"events","msg":"event","name":"tls_get_certificate","id":"aa53893b-228b-403b-b53a-e368626c5d52","origin":"tls","data":{"client_hello":{"CipherSuites":[19018,4865,4866,4867,49195,49199,49196,49200,52393,52392,49171,49172,156,157,47,53],"ServerName":"subdomain.domains.com","SupportedCurves":[51914,29,23,24],"SupportedPoints":"AA==","SignatureSchemes":[1027,2052,1025,1283,2053,1281,2054,1537],"SupportedProtos":["h2","http/1.1"],"SupportedVersions":[27242,772,771],"Conn":{}}}}
{"level":"debug","ts":1691687767.520493,"logger":"tls.handshake","msg":"choosing certificate","identifier":"subdomain.domains.com","num_choices":1}
{"level":"debug","ts":1691687767.5205908,"logger":"tls.handshake","msg":"default certificate selection results","identifier":"subdomain.domains.com","subjects":["subdomain.domains.com"],"managed":true,"issuer_key":"acme-v02.api.letsencrypt.org-directory","hash":"ab850b9104e556712bad27051f9c60e55b022b0ecba05c5b0ead5b8f8bfe422c"}
{"level":"debug","ts":1691687767.5206614,"logger":"tls.handshake","msg":"matched certificate in cache","remote_ip":"redacted-public-ip","remote_port":"57006","subjects":["subdomain.domains.com"],"managed":true,"expiration":1699352671,"hash":"ab850b9104e556712bad27051f9c60e55b022b0ecba05c5b0ead5b8f8bfe422c"}
{"level":"debug","ts":1691687767.5569863,"logger":"http.stdlib","msg":"http: TLS handshake error from redacted-public-ip:57006: EOF"}
{"level":"debug","ts":1691687767.5685923,"logger":"events","msg":"event","name":"tls_get_certificate","id":"95641033-240f-477a-9e0b-40f2404761d0","origin":"tls","data":{"client_hello":{"CipherSuites":[23130,4865,4866,4867,49195,49199,49196,49200,52393,52392,49171,49172,156,157,47,53],"ServerName":"subdomain.domains.com","SupportedCurves":[39578,29,23,24],"SupportedPoints":"AA==","SignatureSchemes":[1027,2052,1025,1283,2053,1281,2054,1537],"SupportedProtos":["h2","http/1.1"],"SupportedVersions":[31354,772,771],"Conn":{}}}}
{"level":"debug","ts":1691687767.5688937,"logger":"tls.handshake","msg":"choosing certificate","identifier":"subdomain.domains.com","num_choices":1}
{"level":"debug","ts":1691687767.5689907,"logger":"tls.handshake","msg":"default certificate selection results","identifier":"subdomain.domains.com","subjects":["subdomain.domains.com"],"managed":true,"issuer_key":"acme-v02.api.letsencrypt.org-directory","hash":"ab850b9104e556712bad27051f9c60e55b022b0ecba05c5b0ead5b8f8bfe422c"}
{"level":"debug","ts":1691687767.5690577,"logger":"tls.handshake","msg":"matched certificate in cache","remote_ip":"redacted-public-ip","remote_port":"57007","subjects":["subdomain.domains.com"],"managed":true,"expiration":1699352671,"hash":"ab850b9104e556712bad27051f9c60e55b022b0ecba05c5b0ead5b8f8bfe422c"}
{"level":"debug","ts":1691687767.5756052,"logger":"http.stdlib","msg":"http: TLS handshake error from redacted-public-ip:57007: EOF"}
{"level":"debug","ts":1691687770.203036,"logger":"events","msg":"event","name":"tls_get_certificate","id":"58b4bbce-2c4a-4a1f-a6a4-0571409b702d","origin":"tls","data":{"client_hello":{"CipherSuites":[2570,4865,4866,4867,49195,49199,49196,49200,52393,52392,49171,49172,156,157,47,53],"ServerName":"subdomain.domains.com","SupportedCurves":[2570,29,23,24],"SupportedPoints":"AA==","SignatureSchemes":[1027,2052,1025,1283,2053,1281,2054,1537],"SupportedProtos":["h2","http/1.1"],"SupportedVersions":[27242,772,771],"Conn":{}}}}
{"level":"debug","ts":1691687770.20332,"logger":"tls.handshake","msg":"choosing certificate","identifier":"subdomain.domains.com","num_choices":1}
{"level":"debug","ts":1691687770.2033901,"logger":"tls.handshake","msg":"default certificate selection results","identifier":"subdomain.domains.com","subjects":["subdomain.domains.com"],"managed":true,"issuer_key":"acme-v02.api.letsencrypt.org-directory","hash":"ab850b9104e556712bad27051f9c60e55b022b0ecba05c5b0ead5b8f8bfe422c"}
{"level":"debug","ts":1691687770.2034419,"logger":"tls.handshake","msg":"matched certificate in cache","remote_ip":"redacted-public-ip","remote_port":"57011","subjects":["subdomain.domains.com"],"managed":true,"expiration":1699352671,"hash":"ab850b9104e556712bad27051f9c60e55b022b0ecba05c5b0ead5b8f8bfe422c"}
{"level":"debug","ts":1691687770.2344997,"logger":"http.stdlib","msg":"http: TLS handshake error from redacted-public-ip:57011: tls: failed to verify certificate: x509: certificate signed by unknown authority"}
{"level":"debug","ts":1691687770.3183453,"logger":"events","msg":"event","name":"tls_get_certificate","id":"1a3853a2-901b-4ebe-a558-20b81f7acd95","origin":"tls","data":{"client_hello":{"CipherSuites":[6682,4865,4866,4867,49195,49199,49196,49200,52393,52392,49171,49172,156,157,47,53],"ServerName":"subdomain.domains.com","SupportedCurves":[51914,29,23,24],"SupportedPoints":"AA==","SignatureSchemes":[1027,2052,1025,1283,2053,1281,2054,1537],"SupportedProtos":["h2","http/1.1"],"SupportedVersions":[60138,772,771],"Conn":{}}}}
{"level":"debug","ts":1691687770.3187346,"logger":"tls.handshake","msg":"choosing certificate","identifier":"subdomain.domains.com","num_choices":1}
{"level":"debug","ts":1691687770.3188384,"logger":"tls.handshake","msg":"default certificate selection results","identifier":"subdomain.domains.com","subjects":["subdomain.domains.com"],"managed":true,"issuer_key":"acme-v02.api.letsencrypt.org-directory","hash":"ab850b9104e556712bad27051f9c60e55b022b0ecba05c5b0ead5b8f8bfe422c"}
{"level":"debug","ts":1691687770.3189075,"logger":"tls.handshake","msg":"matched certificate in cache","remote_ip":"redacted-public-ip","remote_port":"57014","subjects":["subdomain.domains.com"],"managed":true,"expiration":1699352671,"hash":"ab850b9104e556712bad27051f9c60e55b022b0ecba05c5b0ead5b8f8bfe422c"}
{"level":"debug","ts":1691687770.323436,"logger":"http.stdlib","msg":"http: TLS handshake error from redacted-public-ip:57014: EOF"}
{"level":"debug","ts":1691687772.6976502,"logger":"events","msg":"event","name":"tls_get_certificate","id":"515c85a1-7fcd-4ef2-ba1b-7feee5d3c9dd","origin":"tls","data":{"client_hello":{"CipherSuites":[14906,4865,4866,4867,49195,49199,49196,49200,52393,52392,49171,49172,156,157,47,53],"ServerName":"subdomain.domains.com","SupportedCurves":[35466,29,23,24],"SupportedPoints":"AA==","SignatureSchemes":[1027,2052,1025,1283,2053,1281,2054,1537],"SupportedProtos":["h2","http/1.1"],"SupportedVersions":[14906,772,771],"Conn":{}}}}
{"level":"debug","ts":1691687772.6988897,"logger":"tls.handshake","msg":"choosing certificate","identifier":"subdomain.domains.com","num_choices":1}
{"level":"debug","ts":1691687772.6990042,"logger":"tls.handshake","msg":"default certificate selection results","identifier":"subdomain.domains.com","subjects":["subdomain.domains.com"],"managed":true,"issuer_key":"acme-v02.api.letsencrypt.org-directory","hash":"ab850b9104e556712bad27051f9c60e55b022b0ecba05c5b0ead5b8f8bfe422c"}
{"level":"debug","ts":1691687772.6990736,"logger":"tls.handshake","msg":"matched certificate in cache","remote_ip":"redacted-public-ip","remote_port":"57016","subjects":["subdomain.domains.com"],"managed":true,"expiration":1699352671,"hash":"ab850b9104e556712bad27051f9c60e55b022b0ecba05c5b0ead5b8f8bfe422c"}
{"level":"debug","ts":1691687772.7226567,"logger":"http.stdlib","msg":"http: TLS handshake error from redacted-public-ip:57016: tls: failed to verify certificate: x509: certificate signed by unknown authority"}

Something that I noted is tls: failed to verify certificate: x509: certificate signed by unknown authority
These are self signed certificates. Do I have to configure caddy to accept certificates signed by my self generated CA ?

mixu · August 20, 2023, 6:43pm

I also ran into this issue after following those instructions. After trying a lot of different options I think the problem is that the Reddit post has both trusted_ca_cert_file and trusted_leaf_cert_file .

Perhaps this worked at some point? but for me the fix was to change from:

                client_auth {
                        mode require_and_verify
                        trusted_ca_cert_file /etc/caddy/cert_name-CA.crt
                        trusted_leaf_cert_file /etc/caddy/cert_name.crt
                }

to

                client_auth {
                        mode require_and_verify
                        trusted_ca_cert_file /etc/caddy/cert_name.crt
                }

Let me know if that also works for you.

Here are some easy ways to test/debug whether this works by using curl:

curl --cert-type P12 --cert /path-to/cert_name.p12 -vvv https://yoursite.com
openssl s_client -cert /path-to/cert_name.crt -key /path-to/cert_name.key -connect yoursite:443

I’m not really sure what the difference between trusted_ca_cert_file and trusted_leaf_cert_file is or whether supplying both is supported / expected, but that seemed to work for me.

It looks like docs for both options are the same:

trusted_ca_cert_file is a path to a PEM CA certificate file against which to validate client certificates.
trusted_leaf_cert_file is a path to a PEM CA certificate file against which to validate client certificates.

in tls (Caddyfile directive) — Caddy Documentation

francislavoie · August 20, 2023, 7:16pm

Certs are usually a chain, typically root > intermediate > leaf. Trusting the root means that all leafs signed by intermediates signed by the root, are trusted.

The root has a long expiry, so it has some level of permanence by trusting the root. The leaf certs are usually shorter lifetime (default expiry for leafs signed by Caddy’s CA is 12 hours) so trying to trust the leaf will probably only work right away but then stop working.

I’m not sure why it fails in this case though if the leaf cert expiry is longer. Maybe the intermediate is not passed by the client? Or maybe the root isn’t in your system trust store (Caddy doesn’t use trusted_ca_cert_file for verifying leafs I think)?

I’m not well versed in client auth stuff. I don’t use it personally so I can only make some educated guesses here as to what’s happening. But yeah in general you don’t need to trust the leafs, trusting the root is usually enough and should work longer term.

henryreed · August 21, 2023, 7:36pm

Are you sure your CA file is correct? I use mTLS client authentication in Caddy and my config includes both the root CA and the intermediary CA. The intermediary is what signs the leaf (client) certificates.

If your leafs are signed by an intermediary CA, but your “trusted_ca_cert_file” only points to the root CA, then Caddy won’t be able to authenticate the client, since most browsers only push the leaf certificate without the intermediary.

system · September 20, 2023, 7:37pm

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.