Protecting Prometheus endpoint (OAuth2 and basic auth)?

[Easing4868 (Jonas)]

Hi,

I have a self-hosted Prometheus (prometheus.example.org) that I wish to protect with Caddy. Caddy is already acting as a reverse proxy for Prometheus.

This is what I have in mind:

(A) Grafana should be able to query prometheus.example.org by forwarding my Grafana user’s OAuth2 access token on every request. Caddy should examine the token and accept/reject the request.
(B) Ansible should be able to access prometheus.example.org (in particular the /-/reload endpoint) using basic auth. (I use Ansible for deploying Prometheus).
(C) I should be able to access prometheus.example.org using my browser and be prompted for login in Keycloak (the same OIDC provider that Grafana is using).

I am unsure of how to achieve it. Pointers in the right direction are deeply appreciated.

Here are my incomplete thoughts:

Part (A): Use the JWT auth provider module (http.authentication.providers.jwt)

Part (B): Something like this:

{
  acme_ca https://vault.example.org/v1/pki_int/acme/directory
  email foo@example.org
}
prometheus.example.org {
  @promprotect {
    path /-/*
    not path /-/ready
  }
  basicauth @promprotect {
    ansible MY_BCRYPTED_HASH
  }
  log {
    output stdout
  }
  reverse_proxy prometheus:9090
}

Part (C): Somehow use Caddy Security to integrate with Keycloak (the same IdP that Grafana uses) without clashing with (A)?

[francislavoie]

I think you’d have trouble with basicauth. It’s pretty incompatible with a lot of apps that want to have their own auth (often API tokens etc) using the Authorization header.

You could probably use forward_auth (Caddyfile directive) — Caddy Documentation though which allows using apps like Authelia to do the auth, and it uses cookies for subsequent requests.