Grafana behind Caddy2

Help
1

1. The problem I’m having:

Grafana seems to have introduced a CSRF security measure that does not work with my current reverse proxy setup with Caddy2. Their docs only provide examples for nginx and friends.

While Caddy is working fine, it seems Grafana (now) wants a difference Host header.

Origin Not Allowed

403 on POST or PUT

For comparison, the nginx config looks like this:

server {
  listen 80;
  root /usr/share/nginx/www;
  index index.html index.htm;

  location /grafana/ {
    rewrite  ^/grafana/(.*)  /$1 break;
    proxy_set_header Host $http_host;
    proxy_pass http://grafana;
  }

  # Proxy Grafana Live WebSocket connections.
  location /grafana/api/live/ {
    rewrite  ^/grafana/(.*)  /$1 break;
    proxy_http_version 1.1;
    proxy_set_header Upgrade $http_upgrade;
    proxy_set_header Connection $connection_upgrade;
    proxy_set_header Host $http_host;
    proxy_pass http://grafana;
  }
}

From what I understood the proxy_set_header Host $http_host; is what is required.

I am not quite sure what could be wrong with the current
header_up Host {http.reverse_proxy.upstream.hostport}
setup. I thought that would be the equivalent.

2. Error messages and/or full log output:

The error is actually in the Grafana UI.
The linked resources provide the details.

3. Caddy version:

caddy:2.6-alpine
v2.6.4 h1:2hwYqiRwk1tf3VruhMpLcYTg+11fCdr8S3jhNAdnPy8=

4. How I installed and ran Caddy:

docker-compose up

a. System environment:

docker on x86

b. Command:

NA

c. Service/unit/compose file:

NA

d. My complete Caddy config:

{
	servers {
		metrics
	}
	log {
		format json
		level ERROR
	}
}

(proxy) {
	tls /servers/{args.0}.home/server.crt /servers/{args.0}.home/server.key {
	}

	# header / {
	# 	X-Content-Type-Options nosniff
	# 	X-XSS-Protection "1; mode=block"
	# 	Strict-Transport-Security "max-age=31536000;"
	# }

	reverse_proxy {
		to {args.1}
		header_up Host {http.reverse_proxy.upstream.hostport}
		header_down -server
	}
}

grafana.home {
	import proxy grafana http://127.0.0.1:3000
}

5. Links to relevant resources:

2

Some progress.

When I set the Host header explicitly to grafana.home it works.
Do I have some wrong expectations about the upstream variables?

http.reverse_proxy.upstream.hostport # grafana.home:443
http.reverse_proxy.upstream.host # grafana.home

I couldn’t find more details in the docs.

3

You should only need to change the Host if your upstream address is HTTPS and the upstream expects the Host to match TLS-SNI.

Caddy sets the Host header automatically to the original incoming host (so in your case, grafana.home) so it shouldn’t be necessary to change.

Enable the debug global option and see the proxy logs, you’ll see the Host header’s value.

Are you sure your Grafana instance is configured to expect grafana.home or is it expecting something else?

4

I am confused. Because the upstream is http://grafana:3000.

And in fact when I remove the header_up Grafana works as expected. :man_shrugging:

But I don’t understand why there is a difference. At least when I just use host instead of hostport

header_up Host {http.reverse_proxy.upstream.host} # grafana.home ?
header_up Host {http.reverse_proxy.upstream.hostport} # grafana.home:443 ?

But isn’t the default basically the same as this:

header_up Host {http.reverse_proxy.upstream.host}

I didn’t find it in the docs on http.reverse_proxy.upstream.*.

This will come in handy. Thanks!

According to the docs I looked at, it does not not need to know where it is hosted.
Which makes this all so very confusing.

Since removing the header_up Host made it work I am OK.
But I would still love to understand the why.

Thanks again for the help!

5

Right, but your site address is grafana.home.

Caddy passes through the original Host header by default, which is whatever domain the client requested and whatever matches the site address (which is a host matcher).

Those placeholders use the proxy upstream address, i.e. grafana:3000, not the original Host. Overriding the header doesn’t make sense in your situation. Like I said, that placeholder is typically only used when proxying over HTTPS.

6

The domain is grafana.home but grafana is the host address (in docker)
My point was - it’s http instead of https.

But I think now I get it. It probably is like this:

header_up Host {http.reverse_proxy.upstream.host} # grafana
header_up Host {http.reverse_proxy.upstream.hostport} # grafana:30000

and that suddenly make sense.
Thanks!

1 Like