tcurdt
(Torsten Curdt)
March 12, 2023, 4:21pm
1
1. The problem I’m having:
Grafana seems to have introduced a CSRF security measure that does not work with my current reverse proxy setup with Caddy2. Their docs only provide examples for nginx and friends.
While Caddy is working fine, it seems Grafana (now) wants a difference Host
header.
Origin Not Allowed
403 on POST or PUT
For comparison, the nginx config looks like this:
server {
listen 80;
root /usr/share/nginx/www;
index index.html index.htm;
location /grafana/ {
rewrite ^/grafana/(.*) /$1 break;
proxy_set_header Host $http_host;
proxy_pass http://grafana;
}
# Proxy Grafana Live WebSocket connections.
location /grafana/api/live/ {
rewrite ^/grafana/(.*) /$1 break;
proxy_http_version 1.1;
proxy_set_header Upgrade $http_upgrade;
proxy_set_header Connection $connection_upgrade;
proxy_set_header Host $http_host;
proxy_pass http://grafana;
}
}
From what I understood the proxy_set_header Host $http_host;
is what is required.
I am not quite sure what could be wrong with the current
header_up Host {http.reverse_proxy.upstream.hostport}
setup. I thought that would be the equivalent.
2. Error messages and/or full log output:
The error is actually in the Grafana UI.
The linked resources provide the details.
3. Caddy version:
caddy:2.6-alpine
v2.6.4 h1:2hwYqiRwk1tf3VruhMpLcYTg+11fCdr8S3jhNAdnPy8=
4. How I installed and ran Caddy:
docker-compose up
a. System environment:
docker on x86
b. Command:
NA
c. Service/unit/compose file:
NA
d. My complete Caddy config:
{
servers {
metrics
}
log {
format json
level ERROR
}
}
(proxy) {
tls /servers/{args.0}.home/server.crt /servers/{args.0}.home/server.key {
}
# header / {
# X-Content-Type-Options nosniff
# X-XSS-Protection "1; mode=block"
# Strict-Transport-Security "max-age=31536000;"
# }
reverse_proxy {
to {args.1}
header_up Host {http.reverse_proxy.upstream.hostport}
header_down -server
}
}
grafana.home {
import proxy grafana http://127.0.0.1:3000
}
5. Links to relevant resources:
opened 01:01PM - 02 Sep 22 UTC
closed 12:28PM - 05 Sep 22 UTC
area/backend
<!--
Please use this template to create your bug report. By providing as much i… nfo as possible you help us understand the issue, reproduce it and resolve it for you quicker. Therefore take a couple of extra minutes to make sure you have provided all info needed.
PROTIP: record your screen and attach it as a gif to showcase the issue.
- Questions should be posted to: https://community.grafana.com
- Use query inspector to troubleshoot issues: https://bit.ly/2XNF6YS
- How to record and attach gif: https://bit.ly/2Mi8T6K
-->
**What happened**: Receiving an "Origin Not Allowed" 403 error when POST or PUT requests are made from the browser after upgrading from 7.5.7
**What you expected to happen**: Requests (such as query to CloudWatch datasource) to go through. Calls are working when made to the Grafana API manually from command line and do not contain the "Origin" header
**How to reproduce it (as minimally and precisely as possible)**:
**Anything else we need to know?**: Greafana is hosted in a docker container on ECS. Traffic to it passes through an ALB and Zuul Gateway. ALB integration support appears to require Grafana to seek another means to the CSRF, most likely via x-forwarded-for header.
**Environment**:
- Grafana version: 9.1.1
- Data source type & version: Cloud Watch 5.0.0 and InfluxDB
- OS Grafana is installed on: Ubuntu 20.0.4
- User OS & Browser: Windows Chrome/Firefox
- Grafana plugins:
- Others:
opened 04:36AM - 09 Feb 22 UTC
closed 06:25PM - 10 Feb 22 UTC
type/docs
**What happened**:
Unable to create a new dashboard or save changes to existing… dashboards following update to Grafana v8.3.5
**What you expected to happen**:
Ability to create/modify dashboards in Grafana
**How to reproduce it (as minimally and precisely as possible)**:
- Log into Grafana instance and click on "+" button and create new dashboard
- Go to save the dashboard into a folder
- Attempt to save fails, "Origin Not Allowed" error shows up in upper right corner
**Anything else we need to know?**:
Attached is what I'm seeing in the console upon trying to save; looks like it is getting a 403 Forbidden for FQDN/api/dashboards/db and this is preventing dashboard from saving. Clicking the "Save" button prompts one of these errors in the console. I'm also seeing a 403 Forbidden on FQDN/api/frontend-metrics and a screenshot for that is attached too. I'm not seeing anything of particular use in the /var/log/grafana/grafana.log at INFO level, let me know if I should try debug level.
<img width="1676" alt="Screen Shot 2022-02-08 at 10 26 05 PM" src="https://user-images.githubusercontent.com/16138910/153122711-779a4b11-48f3-4071-a5da-b7dd218859de.png">
<img width="1675" alt="Screen Shot 2022-02-08 at 10 31 46 PM" src="https://user-images.githubusercontent.com/16138910/153122716-9a93814d-2712-43ee-83cf-6883646cfb9c.png">
**Environment**:
- Grafana version: 8.3.5
- Data source type & version: InfluxDB
- OS Grafana is installed on: RHEL 8
- User OS & Browser: Safari, Chrome
- Grafana plugins: None
- Others: None
tcurdt
(Torsten Curdt)
March 12, 2023, 11:19pm
2
Some progress.
When I set the Host
header explicitly to grafana.home
it works.
Do I have some wrong expectations about the upstream variables?
http.reverse_proxy.upstream.hostport # grafana.home:443
http.reverse_proxy.upstream.host # grafana.home
I couldn’t find more details in the docs.
tcurdt:
header_up Host
You should only need to change the Host
if your upstream address is HTTPS and the upstream expects the Host to match TLS-SNI.
Caddy sets the Host
header automatically to the original incoming host (so in your case, grafana.home
) so it shouldn’t be necessary to change.
Enable the debug
global option and see the proxy logs, you’ll see the Host
header’s value.
Are you sure your Grafana instance is configured to expect grafana.home
or is it expecting something else?
tcurdt
(Torsten Curdt)
March 13, 2023, 2:55pm
4
I am confused. Because the upstream is http://grafana:3000
.
And in fact when I remove the header_up
Grafana works as expected.
But I don’t understand why there is a difference. At least when I just use host
instead of hostport
header_up Host {http.reverse_proxy.upstream.host} # grafana.home ?
header_up Host {http.reverse_proxy.upstream.hostport} # grafana.home:443 ?
But isn’t the default basically the same as this:
header_up Host {http.reverse_proxy.upstream.host}
I didn’t find it in the docs on http.reverse_proxy.upstream.*
.
This will come in handy. Thanks!
According to the docs I looked at, it does not not need to know where it is hosted.
Which makes this all so very confusing.
Since removing the header_up Host
made it work I am OK.
But I would still love to understand the why.
Thanks again for the help!
Right, but your site address is grafana.home
.
Caddy passes through the original Host header by default, which is whatever domain the client requested and whatever matches the site address (which is a host matcher).
tcurdt:
And in fact when I remove the header_up
Grafana works as expected.
But I don’t understand why there is a difference. At least when I just use host
instead of hostport
header_up Host {http.reverse_proxy.upstream.host} # grafana.home ?
header_up Host {http.reverse_proxy.upstream.hostport} # grafana.home:443 ?
Those placeholders use the proxy upstream address, i.e. grafana:3000
, not the original Host. Overriding the header doesn’t make sense in your situation. Like I said, that placeholder is typically only used when proxying over HTTPS.
tcurdt
(Torsten Curdt)
March 14, 2023, 12:31pm
6
francislavoie:
Right, but your site address is grafana.home
.
Caddy passes through the original Host header by default, which is whatever domain the client requested and whatever matches the site address (which is a host matcher).
The domain is grafana.home
but grafana
is the host address (in docker)
My point was - it’s http instead of https.
But I think now I get it. It probably is like this:
header_up Host {http.reverse_proxy.upstream.host} # grafana
header_up Host {http.reverse_proxy.upstream.hostport} # grafana:30000
and that suddenly make sense.
Thanks!
1 Like