Client cert auth using trust_pool

Help
1

1. The problem I’m having:

I’m trialling caddy as a reverse proxy and have attempted to set up client cert auth as per the docs using the trust_pool file directive. However, I seem to get some strange results that I’m struggling to diagnose. The Home Assistant app which has the client cert works fine. Chrome does not; it throws an error stating ERR_BAD_SSL_CLIENT_AUTH_CERT and never asks me to present a client cert. I’m new to caddy so it’s probably a misunderstanding on my part but I’ve been through the docs and can’t find what I’m doing wrong.

2. Error messages and/or full log output:

Chrome errors:

This site can't provide a secure connection
<FQDN> didn't accept your login certificate, or one may not have been provided
Try contacting the system admin
ERR_BAD_SSL_CLIENT_AUTH_CERT

Caddy access logs look fine, I can’t see anything bouncing there and the caddyfile obviously loads without complaint.

Worryingly, because I don’t understand this issue, the Home Assistant app seems to work fine both across ipv4 and ipv6. Chrome exhibits the same behaviour across both stacks.

3. Caddy version:

v2.8.4 from apt

4. How I installed and ran Caddy:

Installed from apt as per the docs: Install — Caddy Documentation

a. System environment:

AWS Lightsail instance running Ubuntu 22.04 exposing both ipv4 and ipv6 endpoints

b. Command:

$ caddy start

$ caddy reload --config /etc/caddy/Caddyfile
2024/07/21 20:25:39.921	INFO	using config from file	{"file": "/etc/caddy/Caddyfile"}
2024/07/21 20:25:39.922	INFO	adapted config to JSON	{"adapter": "caddyfile"}

c. Service/unit/compose file:

n/a

d. My complete Caddy config:

(mutual_tls) {
        tls {
                protocols tls1.2 tls1.2
                client_auth {
                        mode require_and_verify
                        #trusted_ca_cert_file /etc/caddy/ca.crt
                        #trusted_leaf_cert_file /etc/caddy/hass.crt
                        trust_pool file {
                                pem_file /etc/caddy/ca.crt
                        }
                }
        }
}

# HomeAssistant
<fqdn of caddy> {
        log {
                output file /var/log/caddy/access.log {
                        roll_size 10mb
                        roll_keep 5
                        roll_keep_for 168h
                }
        }
        import mutual_tls
        reverse_proxy https://192.168.100.1:443 {
                transport http {
                        tls_server_name <fqdn of wireguard remote peer which port forwards on to the nginx origin server>
                }
        }

My ca.crt file is a PEM encoded openssl generated CA with no intermediate and is valid.

5. Links to relevant resources:

n/a

2

I’m not able to reproduce it. You’re not seeing anything in the access logs because the connection was rejected before it reaches the HTTP layer. Debug logs may have more info, though I doubt it’ll say much. Anyways, enable debug logging and share them. You can enable debug logging by adding debug in the global options section (very top part of your Caddyfile) as such:

{
    debug
}

If you’ve used the same FQDN without mTLS before, then you’ll have to clear the browser’s cache behavior. Also, double check the personal certificate that’s added to the browser store.

1 Like
3

I’m shocked you can’t reproduce based on my caddyfile. I’ve reset all devices and now none are asking for client certs so I’m totally locked out, my only option being to disable client cert auth on the server. Surely my caddyfile has some errors?!

4

Latest log:

{"level":"error","ts":1721679426.3073099,"logger":"http.log.acc
ess.log0","msg":"handled request","request":{"remote_ip":"ipv4addr","remote_port":"44157","client_ip":"ipv4addr","pr
oto":"HTTP/1.1","method":"GET","host":"fqd ","uri":
"/api/websocket","headers":{"Upgrade":["websocket"],"Accept-Enc
oding":["gzip"],"Cookie":["REDACTED"],"User-Agent":["Home Assis
tant/2024.6.1-12936 (Android 14; Pixel 6)"],"Connection":["Upgr
ade"],"Sec-Websocket-Key":["stuff"],"Sec-Web
socket-Version":["13"],"Sec-Websocket-Extensions":["permessage-
deflate"]},"tls":{"resumed":false,"version":772,"cipher_suite":
4865,"proto":"http/1.1","server_name":fqdn","clien
t_common_name":"fqdn","client_serial":"1"}},"bytes_
read":0,"user_id":"","duration":0.000511441,"size":0,"status":5
02,"resp_headers":{"Server":["Caddy"],"Alt-Svc":["h3=\":443\";
ma=2592000"]}}
5

But that still shows connections from the app which are fine, it doesn’t show failed requests from Chrome which are completely borked.

6

Browsers have a strong cache and retention of prior configuration. For instance, for me to test it, I had to use the Firefox Incognito mode. The browser doesn’t give me the certificate prompt when reusing an existing domain name which the browser has connected to before without a certificate. This is not something Caddy can handle. The browser settings need to be amended. I’m not sure if clearing site settings will work but try that.

1 Like
7

I tend to test using Chrome and Firefox incognito sessions to avoid any of those issues. Another weird thing is that enabling debug logging doesn’t seem to change the verbosity of logs written into my access.log file; either the debug logs are written somewhere else that I’ve not found or there is some underlying issue with how I’ve deployed and run caddy. grepping the logs for debug returns nothing, the only level written seems to be info. Could all of these issues point to something I’ve done wrong with the deployment of caddy?

8

Another interesting thing I’ve just found is that if I uncomment the trusted_leaf_cert_file directive and comment out the trust_pool directive then the browser will prompt for a client cert. It fails client cert auth so I can’t connect but it’s the only way I can get it to prompt for a cert. So my caddyfile looks like:

{
	debug
}

(mutual_tls) {
	tls {
		protocols tls1.2 tls1.3
		client_auth {
			mode require_and_verify
			#trusted_ca_cert_file /etc/caddy/ca.crt
			trusted_leaf_cert_file /etc/caddy/hass.crt
			#trust_pool file {
			#	pem_file /etc/caddy/ca.crt
			#}
		}
	}
}

# HomeAssistant
<fqdn> {
	log {
		output file /var/log/caddy/access.log {
			roll_size 10mb
			roll_keep 5
			roll_keep_for 168h
		}
	}
	import mutual_tls
	reverse_proxy https://192.168.100.1:443 {
		transport http {
			tls_server_name <fqdn>
		}
	}

Is that behaviour ringing any bells with you?

9

We only write INFO and ERROR level access logs, based on the status code of the response.

Your runtime logs are printed to stderr by default, and can be configured via the log global option (not the log directive which appears inside site blocks, which only configures access logs).

You couldn’t use caddy start in production. It doesn’t start back up if your reboot the machine, and your logs will get printed out to the void if you detach your terminal. It’s only meant for quick-and-dirty dev usecases.

Run Caddy as a systemd service. Then your logs will be found in the journal. See Keep Caddy Running — Caddy Documentation Make sure to stop any Caddy instance you may have started with caddy start with caddy stop, then only use systemd commands from here-on.

2 Likes
10

Moved to systemd, good shout.

Some more debugging:

  1. Using trust_pool file doesn’t prompt for a client cert in the browser and gives the error:
Jul 23 09:51:02 ip-172-26-4-233 caddy[2221]: {"level":"debug","ts":1721728262.6301224,"logger":"http.stdlib","msg":"http: TLS handshake error from IPv4:8703: EOF"}
Jul 23 09:51:02 ip-172-26-4-233 caddy[2221]: {"level":"debug","ts":1721728262.792029,"logger":"http.stdlib","msg":"http: TLS handshake error from IPv4:17544: tls: client didn't provide a certificate"}
  1. Using trusted_leaf_cert_file does prompt for a client cert in the browser but gives the following error:
Jul 23 09:52:13 ip-172-26-4-233 caddy[2221]: {"level":"debug","ts":1721728333.158188,"logger":"http.stdlib","msg":"http: TLS handshake error from IPv4:19531: EOF"}
Jul 23 09:52:14 ip-172-26-4-233 caddy[2221]: {"level":"debug","ts":1721728334.5041795,"logger":"http.stdlib","msg":"http: TLS handshake error from IPv4:50360: tls: failed to verify certificate: x509: certificate signed by unknown authority"}

My certs have always worked previously in nginx and I can verify using openssl:

openssl verify -CAfile ca.crt hass.crt
hass.crt: OK

I’m not clear on which directive I should be using but trusted_leaf_cert_file gets me closest to the behaviour that I expect (but obviously fails on the cert chain validation for some reason). Switching back to trust_pool file at least allows the Home Assistant Android app to connect and throws out the following debug log:

Jul 23 09:57:38 ip-172-26-4-233 caddy[2221]: {"level":"debug","ts":1721728658.9430428,"logger":"http.stdlib","msg":"http: TLS handshake error from IPv4:59550: EOF"}
:["*/*"],"X-Requested-With":["io.homeassistant.companion.android"],"Sec-Fetch-Dest":["serviceworker"],"Referer":["https://<fqdn>/service_worker.js"],"Priority":["u=4, i"],"Cache-Control":["max-age=0"],"Sec-Fetch-Mode":["same-origin"],"User-Agent":["Mozilla/5.0 (Linux; Android 14; Pixel 6 Build/AP2A.240605.024; wv) AppleWebKit/537.36 (KHTML, like Gecko) Version/4.0 Chrome/126.0.6478.134 Mobile Safari/537.36"],"X-Forwarded-For":["IPv4"]},"tls":{"resumed":false,"version":772,"cipher_suite":4865,"proto":"h2","server_name":"<fqdn>","client_common_name":"fqdn>","client_serial":"1"}},"headers":{"X-Frame-Options":["SAMEORIGIN"],"Etag":["\"17e3ae3a10de4600-b4e1\""],"Last-Modified":["Fri, 19 Jul 2024 17:44:07 GMT"],"Strict-Transport-Security":["max-age=31536000; includeSubDomains"],"Server":["nginx"],"Date":["Tue, 23 Jul 2024 09:58:08 GMT"],"Referrer-Policy":["no-referrer"],"X-Content-Type-Options":["nosniff"]},"status":304}
Jul 23 09:58:42 ip-172-26-4-233 caddy[2221]: {"level":"debug","ts":1721728722.312133,"logger":"http.handlers.reverse_proxy","msg":"selected upstream","dial":"192.168.100.1:443","total_upstreams":1}
Jul 23 09:58:42 ip-172-26-4-233 caddy[2221]: {"level":"debug","ts":1721728722.3369036,"logger":"http.handlers.reverse_proxy","msg":"upstream roundtrip","upstream":"192.168.100.1:443","duration":0.024676159,"request":{"remote_ip":"IPv4","remote_port":"18259","client_ip":"IPv4","proto":"HTTP/2.0","method":"GET","host":"<fqdn>","uri":"/lovelace-main?external_auth=1","headers":{"X-Forwarded-Proto":["https"],"Sec-Fetch-Mode":["navigate"],"Upgrade-Insecure-Requests":["1"],"Cookie":["REDACTED"],"X-Forwarded-Host":["<fqdn>"],"Sec-Ch-Ua":["\"Not/A)Brand\";v=\"8\", \"Chromium\";v=\"126\", \"Android WebView\";v=\"126\""],"Priority":["u=0, i"],"Accept-Language":["en-GB,en-US;q=0.9,en;q=0.8"],"X-Requested-With":["io.homeassistant.companion.android"],"Referer":["https://<fqdn>/lovelace-main?external_auth=1"],"Accept-Encoding":["gzip, deflate, br, zstd"],"Sec-Ch-Ua-Mobile":["?1"],"User-Agent":["Mozilla/5.0 (Linux; Android 14; Pixel 6 Build/AP2A.240605.024; wv) AppleWebKit/537.36 (KHTML, like Gecko) Version/4.0 Chrome/126.0.6478.134 Mobile Safari/537.36"],"Cache-Control":["max-age=0"],"Sec-Fetch-Dest":["empty"],"Accept":["text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.7"],"Sec-Ch-Ua-Platform":["\"Android\""],"X-Forwarded-For":["IPv4"],"Sec-Fetch-Site":["same-origin"]},"tls":{"resumed":false,"version":772,"cipher_suite":4865,"proto":"h2","server_name":"<fqdn>","client_common_name":"<fqdn>","client_serial":"1"}},"headers":{"Date":["Tue, 23 Jul 2024 09:58:42 GMT"],"Content-Length":["1830"],"X-Content-Type-Options":["nosniff"],"X-Frame-Options":["SAMEORIGIN"],"Strict-Transport-Security":["max-age=31536000; includeSubDomains"],"Server":["nginx"],"Referrer-Policy":["no-referrer"],"Content-Encoding":["deflate"],"Content-Type":["text/html; charset=utf-8"]},"status":200}
11

Fixed, the problem was user error! The openssl verify in the previous comment was done on my local machine where CA exists. I scp’d the file across to my VPS the other day but somehow must have flipped the filenames so my caddyfile was effectively loading the leaf file into the trust_pool and not the root file. :woman_facepalming: Caddy didn’t pick up on the fact it wasn’t a root file though - but I guess it was a valid PEM so caddy just ran with it. Thanks for all your tips and pointers, much appreciated.

1 Like
12

Glad it’s resolved now! I was planning on asking for the text output of openssl to validate the certificate chain, but I guess it was going to be useful depending on how you’ve mixed them up :grin:

1 Like
13

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.