Tls_client_auth is not adding certificates

marcsal · February 24, 2023, 3:48pm

1. Caddy version:

v2.6.2 h1:wKoFIxpmOJLGl3QXoo6PNbYvGW4xLEgo32GPBEjWL8o=

2. How I installed, and run Caddy:

a. System environment:

Docker in WSL
I created a custom certificate for client authentication in /shared

b. Command:

docker-compose up

c. Service/unit/compose file:

docker-compose.yml

version: "3.9"

services:

  server:
    image: caddy
    volumes:
      - '$PWD/caddyfile_server:/etc/caddy/Caddyfile:ro'
      - '$PWD/imports:/etc/caddy/imports:ro'
      - '$PWD/data_server:/data:rw'
      - '$PWD/data_client:/data_client:rw'
      - '$PWD/shared:/shared:rw'
    restart: "no"

  client:
    image: caddy
    volumes:
      - '$PWD/caddyfile_client:/etc/caddy/Caddyfile:ro'
      - '$PWD/imports:/etc/caddy/imports:ro'
      - '$PWD/data_client:/data:rw'
      - '$PWD/data_server:/data_server:rw'
      - '$PWD/shared:/shared:rw'
    restart: "no"

  user:
    image: ellerbrock/alpine-bash-curl-ssl
    volumes:
      - '$PWD/shared:/shared:rw'
      - '$PWD/data_server:/data_server:rw'
    command:
      - /bin/sh
      - -c
      - |
        echo "sleep for 5 sec"
        sleep 5
        curl http://client/ -v -k -L

imports/logger

log {
  output stdout
  format console
  level  debug
}

d. My complete Caddy config:

caddyfile_client

{
  acme_ca "https://server/acme/root_ca/directory"
  import imports/logger
}

client {
  import imports/logger

  tls {
    issuer acme {
      dir "http://localhost:8080/acme/root_ca/directory"
      trusted_roots "/data_server/caddy/pki/authorities/root_ca/root.crt"
    }

    key_type "rsa2048"
    on_demand
  }
}

http://localhost:8080 {
  import imports/logger

  handle /acme/* {
    reverse_proxy {
        transport http {
          tls_client_auth "/shared/client.crt" "/shared/client.key"
          tls_trusted_ca_certs "/data_server/caddy/pki/authorities/root_ca/root.crt"
        }
        header_up Host "server"
        to server:443
    }
  }
}

caddyfile_server

{
  import imports/logger
  pki {
	ca root_ca {
	  name "Caddy Root CA"
	}
  }
}

server {
  import imports/logger

  tls {
    issuer internal {
	  ca root_ca
	}
    client_auth {
		mode require_and_verify
		trusted_ca_cert_file "/shared/client.crt"
		# trusted_leaf_cert_file "/shared/client.crt"
	}
    key_type "rsa2048"
    on_demand
  }

  acme_server {
    ca root_ca
  }
}

3. The problem I’m having:

I was trying to create an certificate for client using ACME. This is tested by user by sending a request to client. To achieve mTLS between server and client, I tried adding client authentication using the custom leaf certificate in /shared. This is realized on client by sending the ACME requests to a proxy, which adds the client authentication and sends the request to server.

Problem: adding just trusted_leaf_cert_file to the client_auth module on server does not result in trusting the specified leaf certificate. This would require also adding the CA of the certificate in trusted_ca_cert_file. I would expect the fields to be ORed (as described in tls (Caddyfile directive) — Caddy Documentation (caddyserver.com)), instead they are ANDed.
→ related issue: trusted_leaf_cert_file does not work as documented · Issue #4518 · caddyserver/caddy · GitHub
→ solution: specify the leaf certificate in trusted_ca_cert_file
Problem: client does not add the specified certificates to the TLS handshake. This is probably a problem with the reverse_proxy/transport/tls_client_auth field.

4. Error messages and/or full log output:

These are the resulted logs of the configuration I used:

-> % docker-compose up
[+] Running 3/0
 ⠿ Container acme-test-client-1  Created                                                                                                                                                                                                0.0s
 ⠿ Container acme-test-user-1    Created                                                                                                                                                                                                0.0s
 ⠿ Container acme-test-server-1  Created                                                                                                                                                                                                0.0s
Attaching to acme-test-client-1, acme-test-server-1, acme-test-user-1
acme-test-user-1    | sleep for 5 sec
acme-test-server-1  | {"level":"info","ts":1677251502.3590624,"msg":"using provided configuration","config_file":"/etc/caddy/Caddyfile","config_adapter":"caddyfile"}
acme-test-server-1  | {"level":"warn","ts":1677251502.3685856,"msg":"Caddyfile input is not formatted; run the 'caddy fmt' command to fix inconsistencies","adapter":"caddyfile","file":"/etc/caddy/Caddyfile","line":2}
acme-test-server-1  | {"level":"info","ts":1677251502.3688877,"msg":"redirected default logger","from":"stderr","to":"stdout"}
acme-test-server-1  | 2023/02/24 15:11:42.369   INFO    admin   admin endpoint started  {"address": "localhost:2019", "enforce_origin": false, "origins": ["//localhost:2019", "//[::1]:2019", "//127.0.0.1:2019"]}
acme-test-server-1  | 2023/02/24 15:11:42.369   INFO    tls.cache.maintenance   started background certificate maintenance                                                                                                                  {"cache": "0xc0001e0c40"}
acme-test-server-1  | 2023/02/24 15:11:42.389   INFO    http    enabling automatic HTTP->HTTPS redirects        {"server_name": "srv0"}
acme-test-server-1  | 2023/02/24 15:11:42.389   WARN    http    enabling strict SNI-Host enforcement because TLS client auth is configured                                                                                                  {"server_id": "srv0"}
acme-test-client-1  | {"level":"info","ts":1677251502.4116614,"msg":"using provided configuration","config_file":"/etc/caddy/Caddyfile","config_adapter":"caddyfile"}
acme-test-client-1  | {"level":"warn","ts":1677251502.4230883,"msg":"Caddyfile input is not formatted; run the 'caddy fmt' command to fix inconsistencies","adapter":"caddyfile","file":"/etc/caddy/Caddyfile","line":2}
acme-test-client-1  | {"level":"info","ts":1677251502.4233344,"msg":"redirected default logger","from":"stderr","to":"stdout"}
acme-test-client-1  | 2023/02/24 15:11:42.423   INFO    admin   admin endpoint started  {"address": "localhost:2019", "enforce_origin": false, "origins": ["//localhost:2019", "//[::1]:2019", "//127.0.0.1:2019"]}
acme-test-client-1  | 2023/02/24 15:11:42.423   INFO    tls.cache.maintenance   started background certificate maintenance                                                                                                                  {"cache": "0xc0002129a0"}
acme-test-client-1  | 2023/02/24 15:11:42.428   INFO    http    server is listening only on the HTTPS port but has no TLS connection policies; adding one to enable TLS                                                                     {"server_name": "srv0", "https_port": 443}
acme-test-client-1  | 2023/02/24 15:11:42.428   INFO    http    enabling automatic HTTP->HTTPS redirects        {"server_name": "srv0"}
acme-test-server-1  | 2023/02/24 15:11:42.436   INFO    pki.ca.root_ca  root certificate is already trusted by system  {"path": "storage:pki/authorities/root_ca/root.crt"}
acme-test-server-1  | 2023/02/24 15:11:42.436   WARN    tls     YOUR SERVER MAY BE VULNERABLE TO ABUSE: on-demand TLS is enabled, but no protections are in place                                                                           {"docs": "https://caddyserver.com/docs/automatic-https#on-demand-tls"}
acme-test-server-1  | 2023/02/24 15:11:42.436   INFO    http    enabling HTTP/3 listener        {"addr": ":443"}
acme-test-server-1  | 2023/02/24 15:11:42.437   INFO    tls     cleaning storage unit   {"description": "FileStorage:/data/caddy"}
acme-test-server-1  | {"level":"info","ts":1677251502.4370415,"msg":"failed to sufficiently increase receive buffer size (was: 208 kiB, wanted: 2048 kiB, got: 416 kiB). See https://github.com/lucas-clemente/quic-go/wiki/UDP-Receive-Buffer-Size for details."}
acme-test-server-1  | 2023/02/24 15:11:42.437   DEBUG   http    starting server loop    {"address": "[::]:443", "tls": true, "http3": true}
acme-test-server-1  | 2023/02/24 15:11:42.437   INFO    http.log        server running  {"name": "srv0", "protocols": ["h1", "h2", "h3"]}
acme-test-server-1  | 2023/02/24 15:11:42.437   DEBUG   http    starting server loop    {"address": "[::]:80", "tls": false, "http3": false}
acme-test-server-1  | 2023/02/24 15:11:42.437   INFO    http.log        server running  {"name": "remaining_auto_https_redirects", "protocols": ["h1", "h2", "h3"]}
acme-test-server-1  | 2023/02/24 15:11:42.437   INFO    http    enabling automatic TLS certificate management   {"domains": ["server"]}
acme-test-server-1  | 2023/02/24 15:11:42.438   INFO    autosaved config (load with --resume flag)      {"file": "/config/caddy/autosave.json"}
acme-test-server-1  | 2023/02/24 15:11:42.438   INFO    serving initial configuration
acme-test-client-1  | 2023/02/24 15:11:42.446   WARN    tls     YOUR SERVER MAY BE VULNERABLE TO ABUSE: on-demand TLS is enabled, but no protections are in place                                                                           {"docs": "https://caddyserver.com/docs/automatic-https#on-demand-tls"}
acme-test-client-1  | 2023/02/24 15:11:42.446   DEBUG   http    starting server loop    {"address": "[::]:80", "tls": false, "http3": false}
acme-test-client-1  | 2023/02/24 15:11:42.446   INFO    http.log        server running  {"name": "remaining_auto_https_redirects", "protocols": ["h1", "h2", "h3"]}
acme-test-client-1  | 2023/02/24 15:11:42.446   INFO    http    enabling HTTP/3 listener        {"addr": ":443"}
acme-test-client-1  | 2023/02/24 15:11:42.446   INFO    tls     cleaning storage unit   {"description": "FileStorage:/data/caddy"}
acme-test-client-1  | {"level":"info","ts":1677251502.446371,"msg":"failed to sufficiently increase receive buffer size (was: 208 kiB, wanted: 2048 kiB, got: 416 kiB). See https://github.com/lucas-clemente/quic-go/wiki/UDP-Receive-Buffer-Size for details."}
acme-test-client-1  | 2023/02/24 15:11:42.446   DEBUG   http    starting server loop    {"address": "[::]:443", "tls": true, "http3": true}
acme-test-client-1  | 2023/02/24 15:11:42.446   INFO    http.log        server running  {"name": "srv0", "protocols": ["h1", "h2", "h3"]}
acme-test-client-1  | 2023/02/24 15:11:42.446   DEBUG   http    starting server loop    {"address": "[::]:8080", "tls": false, "http3": false}
acme-test-client-1  | 2023/02/24 15:11:42.446   INFO    http.log        server running  {"name": "srv1", "protocols": ["h1", "h2", "h3"]}
acme-test-client-1  | 2023/02/24 15:11:42.446   INFO    http    enabling automatic TLS certificate management   {"domains": ["client"]}
acme-test-client-1  | 2023/02/24 15:11:42.446   INFO    autosaved config (load with --resume flag)      {"file": "/config/caddy/autosave.json"}
acme-test-client-1  | 2023/02/24 15:11:42.446   INFO    serving initial configuration
acme-test-client-1  | 2023/02/24 15:11:42.518   INFO    tls     finished cleaning storage units
acme-test-server-1  | 2023/02/24 15:11:42.547   INFO    tls     finished cleaning storage units
acme-test-user-1    |   % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
acme-test-user-1    |                                  Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0*   Trying 172.18.0.4...
acme-test-user-1    | * TCP_NODELAY set
acme-test-user-1    | * Connected to client (172.18.0.4) port 80 (#0)
acme-test-user-1    | > GET / HTTP/1.1
acme-test-user-1    | > Host: client
acme-test-user-1    | > User-Agent: curl/7.61.1
acme-test-user-1    | > Accept: */*
acme-test-user-1    | >
acme-test-user-1    | < HTTP/1.1 308 Permanent Redirect
acme-test-client-1  | 2023/02/24 15:11:47.176   INFO    http.log.access.log0    handled request {"request": {"remote_ip": "172.18.0.3", "remote_port": "41288", "proto": "HTTP/1.1", "method": "GET", "host": "client", "uri": "/", "headers": {"User-Agent": ["curl/7.61.1"], "Accept": ["*/*"]}}, "user_id": "", "duration": 0.000057844, "size": 0, "status": 308, "resp_headers": {"Server": ["Caddy"], "Connection": ["close"], "Location": ["https://client/"], "Content-Type": []}}
acme-test-user-1    | < Connection: close
acme-test-user-1    | < Location: https://client/
acme-test-user-1    | < Server: Caddy
acme-test-user-1    | < Date: Fri, 24 Feb 2023 15:11:47 GMT
acme-test-user-1    | < Content-Length: 0
acme-test-user-1    | <
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
acme-test-user-1    | * Closing connection 0
acme-test-user-1    | * Issue another request to this URL: 'https://client/'
acme-test-user-1    | *   Trying 172.18.0.4...
acme-test-user-1    | * TCP_NODELAY set
acme-test-user-1    | * Connected to client (172.18.0.4) port 443 (#1)
acme-test-user-1    | * ALPN, offering h2
acme-test-user-1    | * ALPN, offering http/1.1
acme-test-user-1    | * successfully set certificate verify locations:
acme-test-user-1    | *   CAfile: /etc/ssl/certs/ca-certificates.crt
acme-test-user-1    |   CApath: none
acme-test-user-1    | * TLSv1.2 (OUT), TLS handshake, Client hello (1):
acme-test-user-1    | } [224 bytes data]
acme-test-client-1  | 2023/02/24 15:11:47.182   DEBUG   events  event   {"name": "tls_get_certificate", "id": "ccbab60e-8a8c-4978-ac91-df0d2e8d9692", "origin": "tls", "data": {"client_hello":{"CipherSuites":[49200,49196,49192,49188,49172,49162,159,107,57,52393,52392,52394,65413,196,136,129,157,61,53,192,132,49199,49195,49191,49187,49171,49161,158,103,51,190,69,156,60,47,186,65,49169,49159,5,4,49170,49160,22,10,21,9,255],"ServerName":"client","SupportedCurves":[29,23,24],"SupportedPoints":"AA==","SignatureSchemes":[1537,1539,61423,1281,1283,1025,1027,61166,60909,769,771,513,515],"SupportedProtos":["h2","http/1.1"],"SupportedVersions":[771,770,769],"Conn":{}}}}
acme-test-client-1  | 2023/02/24 15:11:47.183   DEBUG   tls.handshake   no matching certificates and no custom selection logic                                                                                                              {"identifier": "client"}
acme-test-client-1  | 2023/02/24 15:11:47.183   DEBUG   tls.handshake   no matching certificates and no custom selection logic                                                                                                              {"identifier": "*"}
acme-test-client-1  | 2023/02/24 15:11:47.183   DEBUG   tls.handshake   all external certificate managers yielded no certificates and no errors                                                                                             {"remote_ip": "172.18.0.3", "remote_port": "54450", "sni": "client"}
acme-test-client-1  | 2023/02/24 15:11:47.187   INFO    tls.on_demand   obtaining new certificate       {"remote_ip": "172.18.0.3", "remote_port": "54450", "server_name": "client"}
acme-test-client-1  | 2023/02/24 15:11:47.198   INFO    tls.obtain      acquiring lock  {"identifier": "client"}
acme-test-client-1  | 2023/02/24 15:11:47.218   INFO    tls.obtain      lock acquired   {"identifier": "client"}
acme-test-client-1  | 2023/02/24 15:11:47.221   INFO    tls.obtain      obtaining certificate   {"identifier": "client"}
acme-test-client-1  | 2023/02/24 15:11:47.221   DEBUG   events  event   {"name": "cert_obtaining", "id": "4b137bf8-7f88-4631-acbb-8cf42d0e8c8f", "origin": "tls", "data": {"identifier":"client"}}
acme-test-client-1  | 2023/02/24 15:11:47.422   DEBUG   tls.obtain      trying issuer 1/1       {"issuer": "localhost:8080-acme-root_ca-directory"}
acme-test-client-1  | 2023/02/24 15:11:47.457   INFO    http    waiting on internal rate limiter        {"identifiers": ["client"], "ca": "http://localhost:8080/acme/root_ca/directory", "account": ""}
acme-test-client-1  | 2023/02/24 15:11:47.457   INFO    http    done waiting on internal rate limiter   {"identifiers": ["client"], "ca": "http://localhost:8080/acme/root_ca/directory", "account": ""}
acme-test-client-1  | 2023/02/24 15:11:47.458   DEBUG   http.handlers.reverse_proxy     selected upstream       {"dial": "server:443", "total_upstreams": 1}
acme-test-server-1  | 2023/02/24 15:11:47.459   DEBUG   events  event   {"name": "tls_get_certificate", "id": "d339ef16-a47f-4562-9e8e-b7d5299d728a", "origin": "tls", "data": {"client_hello":{"CipherSuites":[49195,49199,49196,49200,52393,52392,49161,49171,49162,49172,156,157,47,53,49170,10,4865,4866,4867],"ServerName":"server","SupportedCurves":[29,23,24,25],"SupportedPoints":"AA==","SignatureSchemes":[2052,1027,2055,2053,2054,1025,1281,1537,1283,1539,513,515],"SupportedProtos":["h2","http/1.1"],"SupportedVersions":[772,771],"Conn":{}}}}
acme-test-server-1  | 2023/02/24 15:11:47.459   DEBUG   tls.handshake   no matching certificates and no custom selection logic                                                                                                              {"identifier": "server"}
acme-test-server-1  | 2023/02/24 15:11:47.459   DEBUG   tls.handshake   no matching certificates and no custom selection logic                                                                                                              {"identifier": "*"}
acme-test-server-1  | 2023/02/24 15:11:47.459   DEBUG   tls.handshake   all external certificate managers yielded no certificates and no errors                                                                                             {"remote_ip": "172.18.0.4", "remote_port": "39238", "sni": "server"}
acme-test-server-1  | 2023/02/24 15:11:47.475   WARN    tls     stapling OCSP   {"error": "no OCSP stapling for [server]: no OCSP server specified in certificate", "identifiers": ["server"]}
acme-test-server-1  | 2023/02/24 15:11:47.475   DEBUG   tls.cache       added certificate to cache      {"subjects": ["server"], "expiration": "2023/02/25 00:43:51.000", "managed": true, "issuer_key": "root_ca", "hash": "958839288a97246868bfffa8d589e7f0f28cc49acafdb850c287b5853585b764", "cache_size": 1, "cache_capacity": 10000}
acme-test-server-1  | 2023/02/24 15:11:47.475   DEBUG   events  event   {"name": "cached_managed_cert", "id": "1c720e86-5ebf-4ed8-883e-0affc2107291", "origin": "tls", "data": {"sans":["server"]}}
acme-test-server-1  | 2023/02/24 15:11:47.475   DEBUG   tls.handshake   loaded certificate from storage {"remote_ip": "172.18.0.4", "remote_port": "39238", "subjects": ["server"], "managed": true, "expiration": "2023/02/25 00:43:51.000", "hash": "958839288a97246868bfffa8d589e7f0f28cc49acafdb850c287b5853585b764"}
acme-test-server-1  | 2023/02/24 15:11:47.477   DEBUG   http.stdlib     http: TLS handshake error from 172.18.0.4:39238: tls: client didn't provide a certificate
acme-test-client-1  | 2023/02/24 15:11:47.477   DEBUG   http.handlers.reverse_proxy     upstream roundtrip      {"upstream": "server:443", "duration": 0.018400662, "request": {"remote_ip": "127.0.0.1", "remote_port": "33302", "proto": "HTTP/1.1", "method": "GET", "host": "server", "uri": "/acme/root_ca/directory", "headers": {"Accept-Encoding": ["gzip"], "X-Forwarded-For": ["127.0.0.1"], "X-Forwarded-Proto": ["http"], "X-Forwarded-Host": ["localhost:8080"], "User-Agent": ["Caddy/2.6.2 CertMagic acmez (linux; amd64)"]}}, "error": "remote error: tls: bad certificate"}
acme-test-client-1  | 2023/02/24 15:11:47.477   ERROR   http.log.error.log1     remote error: tls: bad certificate     {"request": {"remote_ip": "127.0.0.1", "remote_port": "33302", "proto": "HTTP/1.1", "method": "GET", "host": "localhost:8080", "uri": "/acme/root_ca/directory", "headers": {"User-Agent": ["Caddy/2.6.2 CertMagic acmez (linux; amd64)"], "Accept-Encoding": ["gzip"]}}, "duration": 0.019083937, "status": 502, "err_id": "3v5cm1frr", "err_trace": "reverseproxy.statusError (reverseproxy.go:1272)"}
acme-test-client-1  | 2023/02/24 15:11:47.477   ERROR   http.log.access.log1    handled request {"request": {"remote_ip": "127.0.0.1", "remote_port": "33302", "proto": "HTTP/1.1", "method": "GET", "host": "localhost:8080", "uri": "/acme/root_ca/directory", "headers": {"User-Agent": ["Caddy/2.6.2 CertMagic acmez (linux; amd64)"], "Accept-Encoding": ["gzip"]}}, "user_id": "", "duration": 0.019083937, "size": 0, "status": 502, "resp_headers": {"Server": ["Caddy"]}}
acme-test-client-1  | 2023/02/24 15:11:47.477   DEBUG   http.acme_client        http request    {"method": "GET", "url": "http://localhost:8080/acme/root_ca/directory", "headers": {"User-Agent":["Caddy/2.6.2 CertMagic acmez (linux; amd64)"]}, "response_headers": {"Content-Length":["0"],"Date":["Fri, 24 Feb 2023 15:11:47 GMT"],"Server":["Caddy"]}, "status_code": 502}
acme-test-client-1  | 2023/02/24 15:11:47.477   ERROR   tls.obtain      could not get certificate from issuer   {"identifier": "client", "issuer": "localhost:8080-acme-root_ca-directory", "error": "[client] creating new order: provisioning client: HTTP 502:  (ca=http://localhost:8080/acme/root_ca/directory)"}
acme-test-client-1  | 2023/02/24 15:11:47.477   DEBUG   events  event   {"name": "cert_failed", "id": "8d39a076-efb3-4fd9-a55f-0e38b071f432", "origin": "tls", "data": {"error":{},"identifier":"client","issuers":["localhost:8080-acme-root_ca-directory"],"renewal":false}}
acme-test-client-1  | 2023/02/24 15:11:47.477   ERROR   tls.obtain      will retry      {"error": "[client] Obtain: [client] creating new order: provisioning client: HTTP 502:  (ca=http://localhost:8080/acme/root_ca/directory)", "attempt": 1, "retrying_in": 60, "elapsed": 0.25941168, "max_duration": 2592000}

5. What I already tried:

For debugging purposes, I tried reaching the acme_server endpoint using the user container. The needed certificates were added in a curl command. This resulted in no errors, even though I used the same certificates. This is why suspect the client caddyfile to be incorrect. Am I not seeing something here? Or is there an issue with the reverse_proxy/transport/tls_client_auth field?

user logs

-> % docker exec -it acme-test-user-1 sh
~ $ curl https://server --cacert /data_server/caddy/pki/authorities/root_ca/root.crt --cert /shared/client.crt --key /sh
ared/client.key -v
* Rebuilt URL to: https://server/
*   Trying 172.18.0.4...
* TCP_NODELAY set
* Connected to server (172.18.0.4) port 443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* successfully set certificate verify locations:
*   CAfile: /data_server/caddy/pki/authorities/root_ca/root.crt
  CApath: none
* TLSv1.2 (OUT), TLS handshake, Client hello (1):
* TLSv1.2 (IN), TLS handshake, Server hello (2):
* TLSv1.2 (IN), TLS handshake, Certificate (11):
* TLSv1.2 (IN), TLS handshake, Server key exchange (12):
* TLSv1.2 (IN), TLS handshake, Request CERT (13):
* TLSv1.2 (IN), TLS handshake, Server finished (14):
* TLSv1.2 (OUT), TLS handshake, Certificate (11):
* TLSv1.2 (OUT), TLS handshake, Client key exchange (16):
* TLSv1.2 (OUT), TLS handshake, CERT verify (15):
* TLSv1.2 (OUT), TLS change cipher, Change cipher spec (1):
* TLSv1.2 (OUT), TLS handshake, Finished (20):
* TLSv1.2 (IN), TLS change cipher, Change cipher spec (1):
* TLSv1.2 (IN), TLS handshake, Finished (20):
* SSL connection using TLSv1.2 / ECDHE-RSA-AES128-GCM-SHA256
* ALPN, server accepted to use h2
* Server certificate:
*  subject: [NONE]
*  start date: Feb 24 12:43:50 2023 GMT
*  expire date: Feb 25 00:43:50 2023 GMT
*  subjectAltName: host "server" matched cert's "server"
*  issuer: CN=Caddy Root CA - ECC Intermediate
*  SSL certificate verify ok.
* Using HTTP2, server supports multi-use
* Connection state changed (HTTP/2 confirmed)
* Copying HTTP/2 data in stream buffer to connection buffer after upgrade: len=0
* Using Stream ID: 1 (easy handle 0x5577fe4427a0)
> GET / HTTP/2
> Host: server
> User-Agent: curl/7.61.1
> Accept: */*
>
* Connection state changed (MAX_CONCURRENT_STREAMS == 250)!
< HTTP/2 200
< alt-svc: h3=":443"; ma=2592000
< server: Caddy
< content-length: 0
< date: Fri, 24 Feb 2023 15:42:27 GMT
<
* Connection #0 to host server left intact

server logs

acme-test-server-1  | 2023/02/24 15:42:27.618   DEBUG   events  event   {"name": "tls_get_certificate", "id": "f2df696b-ffaf-4f69-88f3-7332c39f3216", "origin": "tls", "data": {"client_hello":{"CipherSuites":[49200,49196,49192,49188,49172,49162,159,107,57,52393,52392,52394,65413,196,136,129,157,61,53,192,132,49199,49195,49191,49187,49171,49161,158,103,51,190,69,156,60,47,186,65,49169,49159,5,4,49170,49160,22,10,21,9,255],"ServerName":"server","SupportedCurves":[29,23,24],"SupportedPoints":"AA==","SignatureSchemes":[1537,1539,61423,1281,1283,1025,1027,61166,60909,769,771,513,515],"SupportedProtos":["h2","http/1.1"],"SupportedVersions":[771,770,769],"Conn":{}}}}
acme-test-server-1  | 2023/02/24 15:42:27.618   DEBUG   tls.handshake   choosing certificate    {"identifier": "server", "num_choices": 1}
acme-test-server-1  | 2023/02/24 15:42:27.618   DEBUG   tls.handshake   default certificate selection results   {"identifier": "server", "subjects": ["server"], "managed": true, "issuer_key": "root_ca", "hash": "958839288a97246868bfffa8d589e7f0f28cc49acafdb850c287b5853585b764"}
acme-test-server-1  | 2023/02/24 15:42:27.618   DEBUG   tls.handshake   matched certificate in cache    {"remote_ip": "172.18.0.2", "remote_port": "36936", "subjects": ["server"], "managed": true, "expiration": "2023/02/25 00:43:51.000", "hash": "958839288a97246868bfffa8d589e7f0f28cc49acafdb850c287b5853585b764"}
acme-test-server-1  | 2023/02/24 15:42:27.629   INFO    http.log.access.log0    handled request {"request": {"remote_ip": "172.18.0.2", "remote_port": "36936", "proto": "HTTP/2.0", "method": "GET", "host": "server", "uri": "/", "headers": {"Accept": ["*/*"], "User-Agent": ["curl/7.61.1"]}, "tls": {"resumed": false, "version": 771, "cipher_suite": 49199, "proto": "h2", "server_name": "server", "client_common_name": "client1", "client_serial": "656240866094280870725999024418798869542958736418"}}, "user_id": "", "duration": 0.000005673, "size": 0, "status": 0, "resp_headers": {"Alt-Svc": ["h3=\":443\"; ma=2592000"], "Server": ["Caddy"]}}

6. Links to relevant resources:

francislavoie · February 26, 2023, 3:23am

This is a lot deeper than I have time to dig into tbh.

But I’ll point you to trusted_leaf_cert_file does not work as documented · Issue #4518 · caddyserver/caddy · GitHub which might clarify some things for you, maybe. I don’t really know if there’s a conclusion.

Ultimately, I personally have no need for client auth, so it’s not a priority for me to look into it. I’m maintaining Caddy as a volunteer, so I have to draw the line somewhere to keep my sanity sometimes. This is one of those cases for me, I think.

I don’t think client auth is a priority for Matt either, right now. But if you need this for specifics business reasons, then I’d say you should consider getting a support contract at Caddy for Business to fund the work on it.

ypnos · February 26, 2023, 9:30pm

Thank you for still taking time to answer @francislavoie

The main concern from my point of view is this config snippet:

It’s basically straight from the documentation and nothing special. It should work, right? Or am I missing something here?

francislavoie · February 26, 2023, 9:35pm

It’s probably not possible to use server certificates as client ones. But I’m not sure.

Honestly, I don’t know that I can answer this, because I don’t ever play with client certs, like I said.

marcsal · March 7, 2023, 9:39am

We tested this configuration also with 2 local caddy instances with 3 different ports for client, client reverse proxy and server.
We noticed that the client reverse_proxy actually added the client_auth certificate to the first message to the server. The server then replied with a message containing the ACME endpoint URLs of the server. But because these URLs contained the port of the server, and not the port of the reverse proxy, all further messages from the client were sent straight to the server, passing by the reverse proxy. This explains why, starting from the second message from the client, no client_auth certificate was added to the communication.

For example, if localhost:4444 is the acme server and localhost :5555 is the reverse proxy, sending a request to http://localhost:5555/acme/root_ca/directory results in following response:

{"newNonce":"https://localhost:4444/acme/root_ca/new-nonce","newAccount":"https://localhost:4444/acme/root_ca/new-account","newOrder":"https://localhost:4444/acme/root_ca/new-order","revokeCert":"https://localhost:4444/acme/root_ca/revoke-cert","keyChange":"https://localhost:4444/acme/root_ca/key-change"}

In this case the client getting this response would automatically send the next message to https://localhost:4444/acme/root_ca/new-nonce.

Unfortunately, finding out every point in the communication where these URLs are used and replacing them is not practicable for us right now.

francislavoie · March 7, 2023, 9:57am

Ah, interesting. So basically acme_server probably needs a “base hostname” config or something is what you’re suggesting?

matt · March 7, 2023, 10:29pm

@marcell Aha, thanks for investigating!

That sounds very related to this issue:

github.com/smallstep/certificates

Changing ACME server host or port causes existing accounts to craft malformed requests

opened 07:59PM - 01 Jul 21 UTC

mholt

bug area/acme needs triage

### Subject of the issue **Please note, I am not sure if this is a bug.** It …is just unexpected behavior I found after wasting a little of Max Furman's time (sorry about that) troubleshooting #193. Basically, the `kid` used with JWS contain the scheme, host, port, and path (i.e. `https://localhost:9000/acme/myca/account/...`) from the ACME server's configuration at the time. However, if the ACME server changes any part of the address, then the kid prefix check fails here: https://github.com/smallstep/certificates/blob/6476eb45a7e97ac43a37da7ab4893555625de249/acme/api/middleware.go#L324-L331 ### Steps to reproduce 1. Create an ACME account with the server at one URL 2. Move the ACME server to a new URL (any of scheme, host, port, or path). 3. Try another ACME transaction: `urn:ietf:params:acme:error:malformed - The request message was malformed` ### Expected behaviour I am not sure. Maybe that error is expected. ### Actual behaviour `urn:ietf:params:acme:error:malformed - The request message was malformed` ### Additional context I do think this is unexpected, especially with internal/development ACME servers, since in my case I originally had the account created at `localhost:9000` then set up a reverse proxy for the ACME server at `localhost:9001` and started pointing the client to use that one, but reuse the same account. Since the client stores the account `kid` from when it created the account, it retained the `localhost:9000` value. Essentially, this means the ACME server can never change its address after it is being used, unless new accounts are created _or_ clients change their stored account locations to reflect the new server address. ~~Maybe that is the best way to handle it.~~ Edit: RFC 8555 suggests otherwise, see discussion below. I am not familar enough with JWK/JWS to know the value of the prefix check, but I assume it wasn't invented for nothing. So this might very well be on the client to fix. (Incidentally, setting STEPDEBUG=1 was not enough to emit detailed error messages. I had to add my own debug logging to find this.) Again, I am not sure if this is a server bug. Wanted to open a discussion anyway.

That issue is about changing the ACME server address, but proxying it is probably a similar idea.

I’ll see if there’s anything that can be done about it.

marcsal · March 15, 2023, 9:58am

We figured out two ways to fix this issue:

Set the host field on the acme_server to the hostname of the client reverse proxy:
This way, the server populated the acme endpoints using the right hostname.
Downside: this field is marked with a compatibility warning.
Add a reverse proxy on the acme server side which sets the “host” header to the hostname of the client reverse proxy
This produces the same result, because if the host field on the acme server is not set, the Host header of the request will be used.

We decided to go with the second solution, as we don’t need to use a feature marked with a warning and we could use the server reverse proxy to handle the client_auth certificate (mTLS).

marcsal · March 15, 2023, 10:29am

This is the complete setup to get certificates over ACME using mTLS:

docker-compose.yml

version: "3.9"

services:

  server:
    image: caddy
    volumes:
      - '$PWD/caddyfile_server:/etc/caddy/Caddyfile:ro'
      - '$PWD/imports:/etc/caddy/imports:ro'
      - '$PWD/data_server:/data:rw'
      - '$PWD/data_client:/data_client:rw'
      - '$PWD/shared:/shared:rw'
    restart: "no"

  client:
    image: caddy
    volumes:
      - '$PWD/caddyfile_client:/etc/caddy/Caddyfile:ro'
      - '$PWD/imports:/etc/caddy/imports:ro'
      - '$PWD/data_client:/data:rw'
      - '$PWD/data_server:/data_server:rw'
      - '$PWD/shared:/shared:rw'
    restart: "no"

  user:
    image: ellerbrock/alpine-bash-curl-ssl
    volumes:
      - '$PWD/shared:/shared:rw'
      - '$PWD/data_server:/data_server:rw'
    command:
      - /bin/sh
      - -c
      - |
        echo "sleep for 5 sec"
        sleep 5
        curl https://client/ -v -k -L

caddyfile_client

{
  import imports/logger
}

https://localhost:5555 {
  import imports/logger

  handle /acme/* {
    reverse_proxy {
      to https://server
      transport http {
        tls_client_auth "/shared/client_auth_cert/client.cert.pem" "/shared/client_auth_cert/client.key.pem"
        tls_trusted_ca_certs "/data_server/caddy/pki/authorities/root_ca/root.crt"
      }
      header_up Host server
    }
  }

  tls "/shared/client_auth_cert/client.cert.pem" "/shared/client_auth_cert/client.key.pem"
}

https://client {
  import imports/logger

  tls {
    issuer acme {
      dir "https://localhost:5555/acme/root_ca/directory"
      trusted_roots "/shared/client_auth_cert/client.cert.pem" "/data_server/caddy/pki/authorities/root_ca/root.crt"
    }

    key_type "rsa2048"
    on_demand
  }
}

caddyfile_server

{
  import imports/logger
  
  pki {
	  ca root_ca {
	    name "Caddy Root CA"
	  }
  }
}

https://localhost:4480 {
  import imports/logger

  tls {
    issuer internal {
	    ca root_ca
	  }
    key_type "rsa2048"
  }

  acme_server {
    ca root_ca
  }
}

https://server {
  import imports/logger

  tls {
    issuer internal {
	    ca root_ca
	  }
    client_auth {
		  mode verify_if_given
		  trusted_ca_cert_file "/shared/client_auth_cert/client.cert.pem"
	  }
    key_type "rsa2048"
  }

  @acmemtls {
    vars_regexp {http.request.tls.client.subject} (.*CN=ACMEClient.*)
  }
  handle /acme/* {
    reverse_proxy @acmemtls {
      to https://localhost:4480
      header_up Host localhost:5555
    }
  }
}

The client side consists of the acme_client and a reverse proxy. The reverse proxy adds a custom client_auth certificate to the messages directed at the acme_server. The trust between acme_client and client reverse proxy is also based on the client_auth certificate. On top of that, both trust the root_ca certificate (issued by the acme_server).

The server side consists of the acme_server and another reverse proxy. Both get their SLL certificates internally. The reverse proxy first checks if the client_auth certificate is valid (if it is given). Then all requests to “/acme/*”, that contain a client_auth certificate, are redirected to the acme_server. This way it is ensured that only requests to the acme endpoint require a client_auth certificate.

system · April 14, 2023, 10:30am

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.