Tls_client_auth is not adding certificates

1. Caddy version:

v2.6.2 h1:wKoFIxpmOJLGl3QXoo6PNbYvGW4xLEgo32GPBEjWL8o=

2. How I installed, and run Caddy:

a. System environment:

Docker in WSL
I created a custom certificate for client authentication in /shared

b. Command:

docker-compose up

c. Service/unit/compose file:

docker-compose.yml

version: "3.9"

services:

  server:
    image: caddy
    volumes:
      - '$PWD/caddyfile_server:/etc/caddy/Caddyfile:ro'
      - '$PWD/imports:/etc/caddy/imports:ro'
      - '$PWD/data_server:/data:rw'
      - '$PWD/data_client:/data_client:rw'
      - '$PWD/shared:/shared:rw'
    restart: "no"

  client:
    image: caddy
    volumes:
      - '$PWD/caddyfile_client:/etc/caddy/Caddyfile:ro'
      - '$PWD/imports:/etc/caddy/imports:ro'
      - '$PWD/data_client:/data:rw'
      - '$PWD/data_server:/data_server:rw'
      - '$PWD/shared:/shared:rw'
    restart: "no"

  user:
    image: ellerbrock/alpine-bash-curl-ssl
    volumes:
      - '$PWD/shared:/shared:rw'
      - '$PWD/data_server:/data_server:rw'
    command:
      - /bin/sh
      - -c
      - |
        echo "sleep for 5 sec"
        sleep 5
        curl http://client/ -v -k -L

imports/logger

log {
  output stdout
  format console
  level  debug
}

d. My complete Caddy config:

caddyfile_client

{
  acme_ca "https://server/acme/root_ca/directory"
  import imports/logger
}

client {
  import imports/logger

  tls {
    issuer acme {
      dir "http://localhost:8080/acme/root_ca/directory"
      trusted_roots "/data_server/caddy/pki/authorities/root_ca/root.crt"
    }

    key_type "rsa2048"
    on_demand
  }
}

http://localhost:8080 {
  import imports/logger

  handle /acme/* {
    reverse_proxy {
        transport http {
          tls_client_auth "/shared/client.crt" "/shared/client.key"
          tls_trusted_ca_certs "/data_server/caddy/pki/authorities/root_ca/root.crt"
        }
        header_up Host "server"
        to server:443
    }
  }
}

caddyfile_server

{
  import imports/logger
  pki {
	ca root_ca {
	  name "Caddy Root CA"
	}
  }
}

server {
  import imports/logger

  tls {
    issuer internal {
	  ca root_ca
	}
    client_auth {
		mode require_and_verify
		trusted_ca_cert_file "/shared/client.crt"
		# trusted_leaf_cert_file "/shared/client.crt"
	}
    key_type "rsa2048"
    on_demand
  }

  acme_server {
    ca root_ca
  }
}

3. The problem I’m having:

I was trying to create an certificate for client using ACME. This is tested by user by sending a request to client. To achieve mTLS between server and client, I tried adding client authentication using the custom leaf certificate in /shared. This is realized on client by sending the ACME requests to a proxy, which adds the client authentication and sends the request to server.

  1. Problem: adding just trusted_leaf_cert_file to the client_auth module on server does not result in trusting the specified leaf certificate. This would require also adding the CA of the certificate in trusted_ca_cert_file. I would expect the fields to be ORed (as described in tls (Caddyfile directive) — Caddy Documentation (caddyserver.com)), instead they are ANDed.
    → related issue: trusted_leaf_cert_file does not work as documented · Issue #4518 · caddyserver/caddy · GitHub
    → solution: specify the leaf certificate in trusted_ca_cert_file

  2. Problem: client does not add the specified certificates to the TLS handshake. This is probably a problem with the reverse_proxy/transport/tls_client_auth field.

4. Error messages and/or full log output:

These are the resulted logs of the configuration I used:

-> % docker-compose up
[+] Running 3/0
 ⠿ Container acme-test-client-1  Created                                                                                                                                                                                                0.0s
 ⠿ Container acme-test-user-1    Created                                                                                                                                                                                                0.0s
 ⠿ Container acme-test-server-1  Created                                                                                                                                                                                                0.0s
Attaching to acme-test-client-1, acme-test-server-1, acme-test-user-1
acme-test-user-1    | sleep for 5 sec
acme-test-server-1  | {"level":"info","ts":1677251502.3590624,"msg":"using provided configuration","config_file":"/etc/caddy/Caddyfile","config_adapter":"caddyfile"}
acme-test-server-1  | {"level":"warn","ts":1677251502.3685856,"msg":"Caddyfile input is not formatted; run the 'caddy fmt' command to fix inconsistencies","adapter":"caddyfile","file":"/etc/caddy/Caddyfile","line":2}
acme-test-server-1  | {"level":"info","ts":1677251502.3688877,"msg":"redirected default logger","from":"stderr","to":"stdout"}
acme-test-server-1  | 2023/02/24 15:11:42.369   INFO    admin   admin endpoint started  {"address": "localhost:2019", "enforce_origin": false, "origins": ["//localhost:2019", "//[::1]:2019", "//127.0.0.1:2019"]}
acme-test-server-1  | 2023/02/24 15:11:42.369   INFO    tls.cache.maintenance   started background certificate maintenance                                                                                                                  {"cache": "0xc0001e0c40"}
acme-test-server-1  | 2023/02/24 15:11:42.389   INFO    http    enabling automatic HTTP->HTTPS redirects        {"server_name": "srv0"}
acme-test-server-1  | 2023/02/24 15:11:42.389   WARN    http    enabling strict SNI-Host enforcement because TLS client auth is configured                                                                                                  {"server_id": "srv0"}
acme-test-client-1  | {"level":"info","ts":1677251502.4116614,"msg":"using provided configuration","config_file":"/etc/caddy/Caddyfile","config_adapter":"caddyfile"}
acme-test-client-1  | {"level":"warn","ts":1677251502.4230883,"msg":"Caddyfile input is not formatted; run the 'caddy fmt' command to fix inconsistencies","adapter":"caddyfile","file":"/etc/caddy/Caddyfile","line":2}
acme-test-client-1  | {"level":"info","ts":1677251502.4233344,"msg":"redirected default logger","from":"stderr","to":"stdout"}
acme-test-client-1  | 2023/02/24 15:11:42.423   INFO    admin   admin endpoint started  {"address": "localhost:2019", "enforce_origin": false, "origins": ["//localhost:2019", "//[::1]:2019", "//127.0.0.1:2019"]}
acme-test-client-1  | 2023/02/24 15:11:42.423   INFO    tls.cache.maintenance   started background certificate maintenance                                                                                                                  {"cache": "0xc0002129a0"}
acme-test-client-1  | 2023/02/24 15:11:42.428   INFO    http    server is listening only on the HTTPS port but has no TLS connection policies; adding one to enable TLS                                                                     {"server_name": "srv0", "https_port": 443}
acme-test-client-1  | 2023/02/24 15:11:42.428   INFO    http    enabling automatic HTTP->HTTPS redirects        {"server_name": "srv0"}
acme-test-server-1  | 2023/02/24 15:11:42.436   INFO    pki.ca.root_ca  root certificate is already trusted by system  {"path": "storage:pki/authorities/root_ca/root.crt"}
acme-test-server-1  | 2023/02/24 15:11:42.436   WARN    tls     YOUR SERVER MAY BE VULNERABLE TO ABUSE: on-demand TLS is enabled, but no protections are in place                                                                           {"docs": "https://caddyserver.com/docs/automatic-https#on-demand-tls"}
acme-test-server-1  | 2023/02/24 15:11:42.436   INFO    http    enabling HTTP/3 listener        {"addr": ":443"}
acme-test-server-1  | 2023/02/24 15:11:42.437   INFO    tls     cleaning storage unit   {"description": "FileStorage:/data/caddy"}
acme-test-server-1  | {"level":"info","ts":1677251502.4370415,"msg":"failed to sufficiently increase receive buffer size (was: 208 kiB, wanted: 2048 kiB, got: 416 kiB). See https://github.com/lucas-clemente/quic-go/wiki/UDP-Receive-Buffer-Size for details."}
acme-test-server-1  | 2023/02/24 15:11:42.437   DEBUG   http    starting server loop    {"address": "[::]:443", "tls": true, "http3": true}
acme-test-server-1  | 2023/02/24 15:11:42.437   INFO    http.log        server running  {"name": "srv0", "protocols": ["h1", "h2", "h3"]}
acme-test-server-1  | 2023/02/24 15:11:42.437   DEBUG   http    starting server loop    {"address": "[::]:80", "tls": false, "http3": false}
acme-test-server-1  | 2023/02/24 15:11:42.437   INFO    http.log        server running  {"name": "remaining_auto_https_redirects", "protocols": ["h1", "h2", "h3"]}
acme-test-server-1  | 2023/02/24 15:11:42.437   INFO    http    enabling automatic TLS certificate management   {"domains": ["server"]}
acme-test-server-1  | 2023/02/24 15:11:42.438   INFO    autosaved config (load with --resume flag)      {"file": "/config/caddy/autosave.json"}
acme-test-server-1  | 2023/02/24 15:11:42.438   INFO    serving initial configuration
acme-test-client-1  | 2023/02/24 15:11:42.446   WARN    tls     YOUR SERVER MAY BE VULNERABLE TO ABUSE: on-demand TLS is enabled, but no protections are in place                                                                           {"docs": "https://caddyserver.com/docs/automatic-https#on-demand-tls"}
acme-test-client-1  | 2023/02/24 15:11:42.446   DEBUG   http    starting server loop    {"address": "[::]:80", "tls": false, "http3": false}
acme-test-client-1  | 2023/02/24 15:11:42.446   INFO    http.log        server running  {"name": "remaining_auto_https_redirects", "protocols": ["h1", "h2", "h3"]}
acme-test-client-1  | 2023/02/24 15:11:42.446   INFO    http    enabling HTTP/3 listener        {"addr": ":443"}
acme-test-client-1  | 2023/02/24 15:11:42.446   INFO    tls     cleaning storage unit   {"description": "FileStorage:/data/caddy"}
acme-test-client-1  | {"level":"info","ts":1677251502.446371,"msg":"failed to sufficiently increase receive buffer size (was: 208 kiB, wanted: 2048 kiB, got: 416 kiB). See https://github.com/lucas-clemente/quic-go/wiki/UDP-Receive-Buffer-Size for details."}
acme-test-client-1  | 2023/02/24 15:11:42.446   DEBUG   http    starting server loop    {"address": "[::]:443", "tls": true, "http3": true}
acme-test-client-1  | 2023/02/24 15:11:42.446   INFO    http.log        server running  {"name": "srv0", "protocols": ["h1", "h2", "h3"]}
acme-test-client-1  | 2023/02/24 15:11:42.446   DEBUG   http    starting server loop    {"address": "[::]:8080", "tls": false, "http3": false}
acme-test-client-1  | 2023/02/24 15:11:42.446   INFO    http.log        server running  {"name": "srv1", "protocols": ["h1", "h2", "h3"]}
acme-test-client-1  | 2023/02/24 15:11:42.446   INFO    http    enabling automatic TLS certificate management   {"domains": ["client"]}
acme-test-client-1  | 2023/02/24 15:11:42.446   INFO    autosaved config (load with --resume flag)      {"file": "/config/caddy/autosave.json"}
acme-test-client-1  | 2023/02/24 15:11:42.446   INFO    serving initial configuration
acme-test-client-1  | 2023/02/24 15:11:42.518   INFO    tls     finished cleaning storage units
acme-test-server-1  | 2023/02/24 15:11:42.547   INFO    tls     finished cleaning storage units
acme-test-user-1    |   % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
acme-test-user-1    |                                  Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0*   Trying 172.18.0.4...
acme-test-user-1    | * TCP_NODELAY set
acme-test-user-1    | * Connected to client (172.18.0.4) port 80 (#0)
acme-test-user-1    | > GET / HTTP/1.1
acme-test-user-1    | > Host: client
acme-test-user-1    | > User-Agent: curl/7.61.1
acme-test-user-1    | > Accept: */*
acme-test-user-1    | >
acme-test-user-1    | < HTTP/1.1 308 Permanent Redirect
acme-test-client-1  | 2023/02/24 15:11:47.176   INFO    http.log.access.log0    handled request {"request": {"remote_ip": "172.18.0.3", "remote_port": "41288", "proto": "HTTP/1.1", "method": "GET", "host": "client", "uri": "/", "headers": {"User-Agent": ["curl/7.61.1"], "Accept": ["*/*"]}}, "user_id": "", "duration": 0.000057844, "size": 0, "status": 308, "resp_headers": {"Server": ["Caddy"], "Connection": ["close"], "Location": ["https://client/"], "Content-Type": []}}
acme-test-user-1    | < Connection: close
acme-test-user-1    | < Location: https://client/
acme-test-user-1    | < Server: Caddy
acme-test-user-1    | < Date: Fri, 24 Feb 2023 15:11:47 GMT
acme-test-user-1    | < Content-Length: 0
acme-test-user-1    | <
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
acme-test-user-1    | * Closing connection 0
acme-test-user-1    | * Issue another request to this URL: 'https://client/'
acme-test-user-1    | *   Trying 172.18.0.4...
acme-test-user-1    | * TCP_NODELAY set
acme-test-user-1    | * Connected to client (172.18.0.4) port 443 (#1)
acme-test-user-1    | * ALPN, offering h2
acme-test-user-1    | * ALPN, offering http/1.1
acme-test-user-1    | * successfully set certificate verify locations:
acme-test-user-1    | *   CAfile: /etc/ssl/certs/ca-certificates.crt
acme-test-user-1    |   CApath: none
acme-test-user-1    | * TLSv1.2 (OUT), TLS handshake, Client hello (1):
acme-test-user-1    | } [224 bytes data]
acme-test-client-1  | 2023/02/24 15:11:47.182   DEBUG   events  event   {"name": "tls_get_certificate", "id": "ccbab60e-8a8c-4978-ac91-df0d2e8d9692", "origin": "tls", "data": {"client_hello":{"CipherSuites":[49200,49196,49192,49188,49172,49162,159,107,57,52393,52392,52394,65413,196,136,129,157,61,53,192,132,49199,49195,49191,49187,49171,49161,158,103,51,190,69,156,60,47,186,65,49169,49159,5,4,49170,49160,22,10,21,9,255],"ServerName":"client","SupportedCurves":[29,23,24],"SupportedPoints":"AA==","SignatureSchemes":[1537,1539,61423,1281,1283,1025,1027,61166,60909,769,771,513,515],"SupportedProtos":["h2","http/1.1"],"SupportedVersions":[771,770,769],"Conn":{}}}}
acme-test-client-1  | 2023/02/24 15:11:47.183   DEBUG   tls.handshake   no matching certificates and no custom selection logic                                                                                                              {"identifier": "client"}
acme-test-client-1  | 2023/02/24 15:11:47.183   DEBUG   tls.handshake   no matching certificates and no custom selection logic                                                                                                              {"identifier": "*"}
acme-test-client-1  | 2023/02/24 15:11:47.183   DEBUG   tls.handshake   all external certificate managers yielded no certificates and no errors                                                                                             {"remote_ip": "172.18.0.3", "remote_port": "54450", "sni": "client"}
acme-test-client-1  | 2023/02/24 15:11:47.187   INFO    tls.on_demand   obtaining new certificate       {"remote_ip": "172.18.0.3", "remote_port": "54450", "server_name": "client"}
acme-test-client-1  | 2023/02/24 15:11:47.198   INFO    tls.obtain      acquiring lock  {"identifier": "client"}
acme-test-client-1  | 2023/02/24 15:11:47.218   INFO    tls.obtain      lock acquired   {"identifier": "client"}
acme-test-client-1  | 2023/02/24 15:11:47.221   INFO    tls.obtain      obtaining certificate   {"identifier": "client"}
acme-test-client-1  | 2023/02/24 15:11:47.221   DEBUG   events  event   {"name": "cert_obtaining", "id": "4b137bf8-7f88-4631-acbb-8cf42d0e8c8f", "origin": "tls", "data": {"identifier":"client"}}
acme-test-client-1  | 2023/02/24 15:11:47.422   DEBUG   tls.obtain      trying issuer 1/1       {"issuer": "localhost:8080-acme-root_ca-directory"}
acme-test-client-1  | 2023/02/24 15:11:47.457   INFO    http    waiting on internal rate limiter        {"identifiers": ["client"], "ca": "http://localhost:8080/acme/root_ca/directory", "account": ""}
acme-test-client-1  | 2023/02/24 15:11:47.457   INFO    http    done waiting on internal rate limiter   {"identifiers": ["client"], "ca": "http://localhost:8080/acme/root_ca/directory", "account": ""}
acme-test-client-1  | 2023/02/24 15:11:47.458   DEBUG   http.handlers.reverse_proxy     selected upstream       {"dial": "server:443", "total_upstreams": 1}
acme-test-server-1  | 2023/02/24 15:11:47.459   DEBUG   events  event   {"name": "tls_get_certificate", "id": "d339ef16-a47f-4562-9e8e-b7d5299d728a", "origin": "tls", "data": {"client_hello":{"CipherSuites":[49195,49199,49196,49200,52393,52392,49161,49171,49162,49172,156,157,47,53,49170,10,4865,4866,4867],"ServerName":"server","SupportedCurves":[29,23,24,25],"SupportedPoints":"AA==","SignatureSchemes":[2052,1027,2055,2053,2054,1025,1281,1537,1283,1539,513,515],"SupportedProtos":["h2","http/1.1"],"SupportedVersions":[772,771],"Conn":{}}}}
acme-test-server-1  | 2023/02/24 15:11:47.459   DEBUG   tls.handshake   no matching certificates and no custom selection logic                                                                                                              {"identifier": "server"}
acme-test-server-1  | 2023/02/24 15:11:47.459   DEBUG   tls.handshake   no matching certificates and no custom selection logic                                                                                                              {"identifier": "*"}
acme-test-server-1  | 2023/02/24 15:11:47.459   DEBUG   tls.handshake   all external certificate managers yielded no certificates and no errors                                                                                             {"remote_ip": "172.18.0.4", "remote_port": "39238", "sni": "server"}
acme-test-server-1  | 2023/02/24 15:11:47.475   WARN    tls     stapling OCSP   {"error": "no OCSP stapling for [server]: no OCSP server specified in certificate", "identifiers": ["server"]}
acme-test-server-1  | 2023/02/24 15:11:47.475   DEBUG   tls.cache       added certificate to cache      {"subjects": ["server"], "expiration": "2023/02/25 00:43:51.000", "managed": true, "issuer_key": "root_ca", "hash": "958839288a97246868bfffa8d589e7f0f28cc49acafdb850c287b5853585b764", "cache_size": 1, "cache_capacity": 10000}
acme-test-server-1  | 2023/02/24 15:11:47.475   DEBUG   events  event   {"name": "cached_managed_cert", "id": "1c720e86-5ebf-4ed8-883e-0affc2107291", "origin": "tls", "data": {"sans":["server"]}}
acme-test-server-1  | 2023/02/24 15:11:47.475   DEBUG   tls.handshake   loaded certificate from storage {"remote_ip": "172.18.0.4", "remote_port": "39238", "subjects": ["server"], "managed": true, "expiration": "2023/02/25 00:43:51.000", "hash": "958839288a97246868bfffa8d589e7f0f28cc49acafdb850c287b5853585b764"}
acme-test-server-1  | 2023/02/24 15:11:47.477   DEBUG   http.stdlib     http: TLS handshake error from 172.18.0.4:39238: tls: client didn't provide a certificate
acme-test-client-1  | 2023/02/24 15:11:47.477   DEBUG   http.handlers.reverse_proxy     upstream roundtrip      {"upstream": "server:443", "duration": 0.018400662, "request": {"remote_ip": "127.0.0.1", "remote_port": "33302", "proto": "HTTP/1.1", "method": "GET", "host": "server", "uri": "/acme/root_ca/directory", "headers": {"Accept-Encoding": ["gzip"], "X-Forwarded-For": ["127.0.0.1"], "X-Forwarded-Proto": ["http"], "X-Forwarded-Host": ["localhost:8080"], "User-Agent": ["Caddy/2.6.2 CertMagic acmez (linux; amd64)"]}}, "error": "remote error: tls: bad certificate"}
acme-test-client-1  | 2023/02/24 15:11:47.477   ERROR   http.log.error.log1     remote error: tls: bad certificate     {"request": {"remote_ip": "127.0.0.1", "remote_port": "33302", "proto": "HTTP/1.1", "method": "GET", "host": "localhost:8080", "uri": "/acme/root_ca/directory", "headers": {"User-Agent": ["Caddy/2.6.2 CertMagic acmez (linux; amd64)"], "Accept-Encoding": ["gzip"]}}, "duration": 0.019083937, "status": 502, "err_id": "3v5cm1frr", "err_trace": "reverseproxy.statusError (reverseproxy.go:1272)"}
acme-test-client-1  | 2023/02/24 15:11:47.477   ERROR   http.log.access.log1    handled request {"request": {"remote_ip": "127.0.0.1", "remote_port": "33302", "proto": "HTTP/1.1", "method": "GET", "host": "localhost:8080", "uri": "/acme/root_ca/directory", "headers": {"User-Agent": ["Caddy/2.6.2 CertMagic acmez (linux; amd64)"], "Accept-Encoding": ["gzip"]}}, "user_id": "", "duration": 0.019083937, "size": 0, "status": 502, "resp_headers": {"Server": ["Caddy"]}}
acme-test-client-1  | 2023/02/24 15:11:47.477   DEBUG   http.acme_client        http request    {"method": "GET", "url": "http://localhost:8080/acme/root_ca/directory", "headers": {"User-Agent":["Caddy/2.6.2 CertMagic acmez (linux; amd64)"]}, "response_headers": {"Content-Length":["0"],"Date":["Fri, 24 Feb 2023 15:11:47 GMT"],"Server":["Caddy"]}, "status_code": 502}
acme-test-client-1  | 2023/02/24 15:11:47.477   ERROR   tls.obtain      could not get certificate from issuer   {"identifier": "client", "issuer": "localhost:8080-acme-root_ca-directory", "error": "[client] creating new order: provisioning client: HTTP 502:  (ca=http://localhost:8080/acme/root_ca/directory)"}
acme-test-client-1  | 2023/02/24 15:11:47.477   DEBUG   events  event   {"name": "cert_failed", "id": "8d39a076-efb3-4fd9-a55f-0e38b071f432", "origin": "tls", "data": {"error":{},"identifier":"client","issuers":["localhost:8080-acme-root_ca-directory"],"renewal":false}}
acme-test-client-1  | 2023/02/24 15:11:47.477   ERROR   tls.obtain      will retry      {"error": "[client] Obtain: [client] creating new order: provisioning client: HTTP 502:  (ca=http://localhost:8080/acme/root_ca/directory)", "attempt": 1, "retrying_in": 60, "elapsed": 0.25941168, "max_duration": 2592000}

5. What I already tried:

For debugging purposes, I tried reaching the acme_server endpoint using the user container. The needed certificates were added in a curl command. This resulted in no errors, even though I used the same certificates. This is why suspect the client caddyfile to be incorrect. Am I not seeing something here? Or is there an issue with the reverse_proxy/transport/tls_client_auth field?

user logs

-> % docker exec -it acme-test-user-1 sh
~ $ curl https://server --cacert /data_server/caddy/pki/authorities/root_ca/root.crt --cert /shared/client.crt --key /sh
ared/client.key -v
* Rebuilt URL to: https://server/
*   Trying 172.18.0.4...
* TCP_NODELAY set
* Connected to server (172.18.0.4) port 443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* successfully set certificate verify locations:
*   CAfile: /data_server/caddy/pki/authorities/root_ca/root.crt
  CApath: none
* TLSv1.2 (OUT), TLS handshake, Client hello (1):
* TLSv1.2 (IN), TLS handshake, Server hello (2):
* TLSv1.2 (IN), TLS handshake, Certificate (11):
* TLSv1.2 (IN), TLS handshake, Server key exchange (12):
* TLSv1.2 (IN), TLS handshake, Request CERT (13):
* TLSv1.2 (IN), TLS handshake, Server finished (14):
* TLSv1.2 (OUT), TLS handshake, Certificate (11):
* TLSv1.2 (OUT), TLS handshake, Client key exchange (16):
* TLSv1.2 (OUT), TLS handshake, CERT verify (15):
* TLSv1.2 (OUT), TLS change cipher, Change cipher spec (1):
* TLSv1.2 (OUT), TLS handshake, Finished (20):
* TLSv1.2 (IN), TLS change cipher, Change cipher spec (1):
* TLSv1.2 (IN), TLS handshake, Finished (20):
* SSL connection using TLSv1.2 / ECDHE-RSA-AES128-GCM-SHA256
* ALPN, server accepted to use h2
* Server certificate:
*  subject: [NONE]
*  start date: Feb 24 12:43:50 2023 GMT
*  expire date: Feb 25 00:43:50 2023 GMT
*  subjectAltName: host "server" matched cert's "server"
*  issuer: CN=Caddy Root CA - ECC Intermediate
*  SSL certificate verify ok.
* Using HTTP2, server supports multi-use
* Connection state changed (HTTP/2 confirmed)
* Copying HTTP/2 data in stream buffer to connection buffer after upgrade: len=0
* Using Stream ID: 1 (easy handle 0x5577fe4427a0)
> GET / HTTP/2
> Host: server
> User-Agent: curl/7.61.1
> Accept: */*
>
* Connection state changed (MAX_CONCURRENT_STREAMS == 250)!
< HTTP/2 200
< alt-svc: h3=":443"; ma=2592000
< server: Caddy
< content-length: 0
< date: Fri, 24 Feb 2023 15:42:27 GMT
<
* Connection #0 to host server left intact

server logs

acme-test-server-1  | 2023/02/24 15:42:27.618   DEBUG   events  event   {"name": "tls_get_certificate", "id": "f2df696b-ffaf-4f69-88f3-7332c39f3216", "origin": "tls", "data": {"client_hello":{"CipherSuites":[49200,49196,49192,49188,49172,49162,159,107,57,52393,52392,52394,65413,196,136,129,157,61,53,192,132,49199,49195,49191,49187,49171,49161,158,103,51,190,69,156,60,47,186,65,49169,49159,5,4,49170,49160,22,10,21,9,255],"ServerName":"server","SupportedCurves":[29,23,24],"SupportedPoints":"AA==","SignatureSchemes":[1537,1539,61423,1281,1283,1025,1027,61166,60909,769,771,513,515],"SupportedProtos":["h2","http/1.1"],"SupportedVersions":[771,770,769],"Conn":{}}}}
acme-test-server-1  | 2023/02/24 15:42:27.618   DEBUG   tls.handshake   choosing certificate    {"identifier": "server", "num_choices": 1}
acme-test-server-1  | 2023/02/24 15:42:27.618   DEBUG   tls.handshake   default certificate selection results   {"identifier": "server", "subjects": ["server"], "managed": true, "issuer_key": "root_ca", "hash": "958839288a97246868bfffa8d589e7f0f28cc49acafdb850c287b5853585b764"}
acme-test-server-1  | 2023/02/24 15:42:27.618   DEBUG   tls.handshake   matched certificate in cache    {"remote_ip": "172.18.0.2", "remote_port": "36936", "subjects": ["server"], "managed": true, "expiration": "2023/02/25 00:43:51.000", "hash": "958839288a97246868bfffa8d589e7f0f28cc49acafdb850c287b5853585b764"}
acme-test-server-1  | 2023/02/24 15:42:27.629   INFO    http.log.access.log0    handled request {"request": {"remote_ip": "172.18.0.2", "remote_port": "36936", "proto": "HTTP/2.0", "method": "GET", "host": "server", "uri": "/", "headers": {"Accept": ["*/*"], "User-Agent": ["curl/7.61.1"]}, "tls": {"resumed": false, "version": 771, "cipher_suite": 49199, "proto": "h2", "server_name": "server", "client_common_name": "client1", "client_serial": "656240866094280870725999024418798869542958736418"}}, "user_id": "", "duration": 0.000005673, "size": 0, "status": 0, "resp_headers": {"Alt-Svc": ["h3=\":443\"; ma=2592000"], "Server": ["Caddy"]}}

6. Links to relevant resources:

This is a lot deeper than I have time to dig into tbh.

But I’ll point you to trusted_leaf_cert_file does not work as documented · Issue #4518 · caddyserver/caddy · GitHub which might clarify some things for you, maybe. I don’t really know if there’s a conclusion.

Ultimately, I personally have no need for client auth, so it’s not a priority for me to look into it. I’m maintaining Caddy as a volunteer, so I have to draw the line somewhere to keep my sanity sometimes. This is one of those cases for me, I think.

I don’t think client auth is a priority for Matt either, right now. But if you need this for specifics business reasons, then I’d say you should consider getting a support contract at Caddy for Business to fund the work on it.

1 Like

Thank you for still taking time to answer @francislavoie

The main concern from my point of view is this config snippet:

It’s basically straight from the documentation and nothing special. It should work, right? Or am I missing something here?

It’s probably not possible to use server certificates as client ones. But I’m not sure.

Honestly, I don’t know that I can answer this, because I don’t ever play with client certs, like I said.

We tested this configuration also with 2 local caddy instances with 3 different ports for client, client reverse proxy and server.
We noticed that the client reverse_proxy actually added the client_auth certificate to the first message to the server. The server then replied with a message containing the ACME endpoint URLs of the server. But because these URLs contained the port of the server, and not the port of the reverse proxy, all further messages from the client were sent straight to the server, passing by the reverse proxy. This explains why, starting from the second message from the client, no client_auth certificate was added to the communication.

For example, if localhost:4444 is the acme server and localhost :5555 is the reverse proxy, sending a request to http://localhost:5555/acme/root_ca/directory results in following response:

{"newNonce":"https://localhost:4444/acme/root_ca/new-nonce","newAccount":"https://localhost:4444/acme/root_ca/new-account","newOrder":"https://localhost:4444/acme/root_ca/new-order","revokeCert":"https://localhost:4444/acme/root_ca/revoke-cert","keyChange":"https://localhost:4444/acme/root_ca/key-change"}

In this case the client getting this response would automatically send the next message to https://localhost:4444/acme/root_ca/new-nonce.

Unfortunately, finding out every point in the communication where these URLs are used and replacing them is not practicable for us right now.

Ah, interesting. So basically acme_server probably needs a “base hostname” config or something is what you’re suggesting?

@marcell Aha, thanks for investigating!

That sounds very related to this issue:

That issue is about changing the ACME server address, but proxying it is probably a similar idea.

I’ll see if there’s anything that can be done about it.

We figured out two ways to fix this issue:

  1. Set the host field on the acme_server to the hostname of the client reverse proxy:
    This way, the server populated the acme endpoints using the right hostname.
    Downside: this field is marked with a compatibility warning.

  2. Add a reverse proxy on the acme server side which sets the “host” header to the hostname of the client reverse proxy
    This produces the same result, because if the host field on the acme server is not set, the Host header of the request will be used.

We decided to go with the second solution, as we don’t need to use a feature marked with a warning and we could use the server reverse proxy to handle the client_auth certificate (mTLS).

1 Like

This is the complete setup to get certificates over ACME using mTLS:

docker-compose.yml

version: "3.9"

services:

  server:
    image: caddy
    volumes:
      - '$PWD/caddyfile_server:/etc/caddy/Caddyfile:ro'
      - '$PWD/imports:/etc/caddy/imports:ro'
      - '$PWD/data_server:/data:rw'
      - '$PWD/data_client:/data_client:rw'
      - '$PWD/shared:/shared:rw'
    restart: "no"

  client:
    image: caddy
    volumes:
      - '$PWD/caddyfile_client:/etc/caddy/Caddyfile:ro'
      - '$PWD/imports:/etc/caddy/imports:ro'
      - '$PWD/data_client:/data:rw'
      - '$PWD/data_server:/data_server:rw'
      - '$PWD/shared:/shared:rw'
    restart: "no"

  user:
    image: ellerbrock/alpine-bash-curl-ssl
    volumes:
      - '$PWD/shared:/shared:rw'
      - '$PWD/data_server:/data_server:rw'
    command:
      - /bin/sh
      - -c
      - |
        echo "sleep for 5 sec"
        sleep 5
        curl https://client/ -v -k -L

caddyfile_client

{
  import imports/logger
}

https://localhost:5555 {
  import imports/logger

  handle /acme/* {
    reverse_proxy {
      to https://server
      transport http {
        tls_client_auth "/shared/client_auth_cert/client.cert.pem" "/shared/client_auth_cert/client.key.pem"
        tls_trusted_ca_certs "/data_server/caddy/pki/authorities/root_ca/root.crt"
      }
      header_up Host server
    }
  }

  tls "/shared/client_auth_cert/client.cert.pem" "/shared/client_auth_cert/client.key.pem"
}

https://client {
  import imports/logger

  tls {
    issuer acme {
      dir "https://localhost:5555/acme/root_ca/directory"
      trusted_roots "/shared/client_auth_cert/client.cert.pem" "/data_server/caddy/pki/authorities/root_ca/root.crt"
    }

    key_type "rsa2048"
    on_demand
  }
}

caddyfile_server

{
  import imports/logger
  
  pki {
	  ca root_ca {
	    name "Caddy Root CA"
	  }
  }
}

https://localhost:4480 {
  import imports/logger

  tls {
    issuer internal {
	    ca root_ca
	  }
    key_type "rsa2048"
  }

  acme_server {
    ca root_ca
  }
}

https://server {
  import imports/logger

  tls {
    issuer internal {
	    ca root_ca
	  }
    client_auth {
		  mode verify_if_given
		  trusted_ca_cert_file "/shared/client_auth_cert/client.cert.pem"
	  }
    key_type "rsa2048"
  }

  @acmemtls {
    vars_regexp {http.request.tls.client.subject} (.*CN=ACMEClient.*)
  }
  handle /acme/* {
    reverse_proxy @acmemtls {
      to https://localhost:4480
      header_up Host localhost:5555
    }
  }
}

The client side consists of the acme_client and a reverse proxy. The reverse proxy adds a custom client_auth certificate to the messages directed at the acme_server. The trust between acme_client and client reverse proxy is also based on the client_auth certificate. On top of that, both trust the root_ca certificate (issued by the acme_server).

The server side consists of the acme_server and another reverse proxy. Both get their SLL certificates internally. The reverse proxy first checks if the client_auth certificate is valid (if it is given). Then all requests to “/acme/*”, that contain a client_auth certificate, are redirected to the acme_server. This way it is ensured that only requests to the acme endpoint require a client_auth certificate.

1 Like