Client Auth works with `trusted_leaf_cert_file` but fails if using `trust_pool`

Help
1

1. The problem I’m having:

I’m trying to get caddy with mTLS to work correctly. When I provide my client certificate with a trusted_leaf_cert_file directive, the process works without a hitch. If I try to use trust_pool and provide either the root or intermediate CA for the client certificate, validation fails.

2. Error messages and/or full log output:

Browser throws ERR_BAD_SSL_CLIENT_AUTH_CERT when trying to connect in a Caddyfile that only has the trust_pool directive (see configs below).

{"level":"debug","ts":1726699656.4571843,"logger":"http.stdlib","msg":"http: TLS handshake error from {my_ip}:46205: client leaf certificate failed validation"}

3. Caddy version:

v2.8.4

4. How I installed and ran Caddy:

a. System environment:

Docker on Fedora

c. Service/unit/compose file:

Using Ansible

- name: Start Caddy Proxy
  docker_container:
    name: "caddy"
    image: caddy:latest
    state: started
    restart: yes
    restart_policy: unless-stopped
    ports:
      - "443:443"
      - "80:80"
    volumes:
      - "/path-to/caddy:/etc/caddy"
    comparisons:
      "*": strict
  become: yes

d. My complete Caddy config:

{
	debug
}

*.mydomain.com {
	log {
		output stderr
	}

	tls {
		client_auth {
			mode require_and_verify
			# trusted_leaf_cert_file /etc/caddy/leaf_cert.crt
			trust_pool file {
				pem_file /etc/caddy/root-ca.crt
				pem_file /etc/caddy/intermediate-ca.crt
			}
		}
	}
	reverse_proxy * {
		to {host}:443
		transport http {
			resolvers 10.10.0.1
			tls
		}
	}
}

5. Links to relevant resources:

Took a look at Client cert auth using trust_pool and I’ve verified my files are correct (leafs are leafs, trust pool certs are Root CA and intermediate CA)

2

I’m not able to replicate your experience. Are you sure the leaf certificate is descendant from the configured intermediate and root?

Moreover, this log message is only possible if you use the trusted_leaf_cert or trusted_leaf_cert_file sub-sub-directives of the tls directive. Those directives aren’t in your config. Are you sure the config you’re running is the same one you think is running?

1 Like
3

Thanks for the reply Mohammed. I simplified my config a little to the minimal example that fails (I also removed the acme configuration line for let’s encrypt to not share credentials). But I did take the logging later from my normal configuration which has several trusted_leaf_cert_file which do work.

Most of the TLS logging I saw with debug were regarding picking the server certificate to return to the client. But for the validation of client, I couldn’t find much.

Is there a way to get additional logging on the certificate validation process? Maybe using an xcaddy build?

4

You say you’re experiencing this via a browser. Browsers are troublesome because they have all kinds of cache. If you can make it fail with curl, that’d be great. The command would be something as follows:

curl --cert <leaf-cert-file> --key <cert-key-file> https://<hostname>

Most browsers nowadays allow you to copy a request line of the Network tab as curl command. While trying curl, check Caddy logs on the other end and share those logs without redactions (except for obvious credentials).

2 Likes
5

So… This is when computers drive me crazy. I figured out the problem. The curl command really came in handy.

The issue was: if there are any trusted_leaf_cert_file in the client_auth directive, it will ignore the trust_pool directive and only validate against the leaf_cert.

So when my config was:

client_auth {
        mode require_and_verify
        trusted_leaf_cert_file /etc/caddy/cert_that_is_not_being_tested.crt
        # trusted_leaf_cert_file /etc/caddy/cert_that_IS_being_tested.crt
        trust_pool file {
                pem_file /etc/caddy/root-ca.crt
                pem_file /etc/caddy/intermediate-ca.crt
        }
}

curl would fail when run like:

$ curl -I --cert cert_being_tested.crt --key cert_being_tested.key https://mydomain.com
curl: (56) OpenSSL SSL_read: OpenSSL/3.2.1: error:0A000412:SSL routines::ssl/tls alert bad certificate, errno 0

When I removed ALL of the trusted_leaf_cert_file directives, everything worked correctly.

client_auth {
        mode require_and_verify
        # trusted_leaf_cert_file /etc/caddy/cert_that_is_not_being_tested.crt
        # trusted_leaf_cert_file /etc/caddy/cert_that_IS_being_tested.crt
        trust_pool file {
                pem_file /etc/caddy/root-ca.crt
                pem_file /etc/caddy/intermediate-ca.crt
        }
}

Curl response

$ curl -I --cert cert_being_tested.crt --key cert_being_tested.key https://mydomain.com
HTTP/2 401 
alt-svc: h3=":443"; ma=2592000
date: Thu, 19 Sep 2024 17:38:04 GMT
server: Caddy
server: Kestrel

I don’t know if this is intended behavior or a bug. The documentation for tls at tls (Caddyfile directive) — Caddy Documentation doesn’t mention that these two directives are mutually exclusive though.

Thanks @Mohammed90 for your help in figuring this out

1 Like
6

They’re not mutually exclusive. They work in layers. The (now deprecated) trusted_leaf_cert_file directive translates to verifier module, which checks the provided certificate in aspects not related to its chain (e.g. revocation, exact match, etc.). The trust_pool is only for certificate chain checks to ensure the submitted cert is anchored to a specific known roots.

We should expand the docs around that to clarify how they’re different.

1 Like
7

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.