Proxy/loadbalance kubernetes api

[jakoberpf]

1. Caddy version (caddy version):

v2.4.6

2. How I run Caddy:

On a OCI Instance with Ubuntu 20.04.

a. System environment:

Package install with cloudflare-dns-plugin

b. Command:

kubectl get nodes ${domain}

c. Service/unit/compose file:

d. My complete Caddyfile or JSON config:

api.name.kube.example.com {
    reverse_proxy 123.456.789:6443 {
        transport http {
            tls
            tls_insecure_skip_verify
        }
        header_up X-Forwarded-Proto https
    }
    log {
        output file ...
    }
}

3. The problem I’m having:

I want to proxy (and then also loadbalance) to a kubernetes api. But i keep getting the error Error while proxying request: x509: certificate signed by unknown authority because the certificate of the api/cluster is not forwarded correctly.

I found this post which accieved what i want to do, but am unsure what to do exactly.

• build caddy with caddy-l4 plugin
• what does my l4 config look like then?
• will the caddyfile config still work as normal?

4. Error messages and/or full log output:

Error while proxying request: x509: certificate signed by unknown authority

[francislavoie]

I’m not sure I understand.

Please provide full logs, not just the error message. The context matters. Make a request with curl -v and show what happens.

@jakoberpf

reverse_proxy ${host}:${port}

The Caddyfile doesn’t support variables like ${host}. Are you using another tool to make those replacements?

[jakoberpf]

Ah very sorry, that is my template how i generate my caddyfiles. Updated it to represent an example.

The output of a curl -v https://api.name.kube.example.com look like…

Trying 130.61.92.35:443...
* TCP_NODELAY set
* Connected to api.name.kube.example.com (130.61.92.35) port 443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* successfully set certificate verify locations:
*   CAfile: /etc/ssl/certs/ca-certificates.crt
  CApath: /etc/ssl/certs
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* TLSv1.3 (IN), TLS handshake, Server hello (2):
* TLSv1.3 (IN), TLS handshake, Encrypted Extensions (8):
* TLSv1.3 (IN), TLS handshake, Certificate (11):
* TLSv1.3 (IN), TLS handshake, CERT verify (15):
* TLSv1.3 (IN), TLS handshake, Finished (20):
* TLSv1.3 (OUT), TLS change cipher, Change cipher spec (1):
* TLSv1.3 (OUT), TLS handshake, Finished (20):
* SSL connection using TLSv1.3 / TLS_AES_128_GCM_SHA256
* ALPN, server accepted to use h2
* Server certificate:
*  subject: CN=api.athena.k8s.erpf.de
*  start date: May  2 17:08:47 2022 GMT
*  expire date: Jul 31 17:08:46 2022 GMT
*  subjectAltName: host "api.name.kube.example.com" matched cert's "api.name.kube.example.com"
*  issuer: C=US; O=Let's Encrypt; CN=R3
*  SSL certificate verify ok.
* Using HTTP2, server supports multi-use
* Connection state changed (HTTP/2 confirmed)
* Copying HTTP/2 data in stream buffer to connection buffer after upgrade: len=0
* Using Stream ID: 1 (easy handle 0x5631ec4aa800)
> GET / HTTP/2
> Host: api.name.kube.example.com
> user-agent: curl/7.68.0
> accept: */*
> 
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
* Connection state changed (MAX_CONCURRENT_STREAMS == 250)!
< HTTP/2 403 
< audit-id: 5a49ee54-c7a4-436a-9d95-797e2e3fa049
< cache-control: no-cache, private
< content-type: application/json
< date: Wed, 04 May 2022 18:47:40 GMT
< server: Caddy
< x-content-type-options: nosniff
< x-kubernetes-pf-flowschema-uid: 2c8cb18e-abdb-4777-9796-ce893501080d
< x-kubernetes-pf-prioritylevel-uid: 4614b8ed-a4b7-47f0-9e57-950bc7b07054
< content-length: 217
< 
{
  "kind": "Status",
  "apiVersion": "v1",
  "metadata": {},
  "status": "Failure",
  "message": "forbidden: User \"system:anonymous\" cannot get path \"/\"",
  "reason": "Forbidden",
  "details": {},
  "code": 403
* Connection #0 to host api.name.kube.example.com left intact

and the appropiate log output is…

{"level":"error","ts":1651690060.5785034,"logger":"http.log.access.log0","msg":"handled request","request":{"remote_addr":"130.61.92.35:58974","proto":"HTTP/2.0","method":"GET","host":"api.name.kube.example.com","uri":"/","headers":{"User-Agent":["curl/7.68.0"],"Accept":["*/*"]},"tls":{"resumed":false,"version":772,"cipher_suite":4865,"proto":"h2","proto_mutual":true,"server_name":"api.name.kube.example.com"}},"common_log":"130.61.92.35 - - [04/May/2022:18:47:40 +0000] \"GET / HTTP/2.0\" 403 217","user_id":"","duration":3.142099836,"size":217,"status":403,"resp_headers":{"Server":["Caddy"],"X-Content-Type-Options":["nosniff"],"X-Kubernetes-Pf-Prioritylevel-Uid":["4614b8ed-a4b7-47f0-9e57-950bc7b07054"],"Content-Length":["217"],"Cache-Control":["no-cache, private"],"X-Kubernetes-Pf-Flowschema-Uid":["2c8cb18e-abdb-4777-9796-ce893501080d"],"Date":["Wed, 04 May 2022 18:47:40 GMT"],"Audit-Id":["5a49ee54-c7a4-436a-9d95-797e2e3fa049"],"Content-Type":["application/json"]}}

The json I get in return actually looks like, the https requests reaches the kubernetes api.

[francislavoie]

I’m not seeing an issue with your Caddy setup here, and clearly Caddy was able to connect to your upstream. The error on that request is coming from your upstream, so you’ll need to dig into that.

[jakoberpf]

I do. I think because caddy does not forward the client-side certificates, the user its unable to authenticate to the cluster.

Is there a flag or config in caddy to allow client side passthrough?

[francislavoie]

There’s no way to make Caddy connect using the client cert, because Caddy doesn’t have the private key associated with the client cert.

What you can do though is pass through the client cert via an HTTP header, something like this:

header_up Client-Cert {tls_client_certificate_der_base64}

Then your backend app can take that and validate it as normal.