Caddy and Grafana sub directory

Help
1

So I am running into an issue with grafana and caddy with grafana as a sub-directory. I saw this post from awhile ago and I tried what as listed as a working config here but it is not working.

I can see the http GET request in my grafana container logs but caddy is saying there is no such site on port 80.

Caddy log:

2019/02/01 01:44:49 [INFO] xx.xx.xx.xx - No such site at :80 (Remote: xx.xx.xx.xx, Referer: )

Grafana log:

t=2019-02-01T01:53:14+0000 lvl=info msg=“HTTP Server Listen” logger=http.server address=0.0.0.0:3000 protocol=http subUrl=/grafana socket=

t=2019-02-01T01:55:44+0000 lvl=info msg=“Request Completed” logger=context userId=0 orgId=0 uname= method=GET path=/grafana status=404 remote_addr=xx.xx.xx.xx time_ms=7 size=26532 referer=

t=2019-02-01T01:55:45+0000 lvl=info msg=“Request Completed” logger=context userId=0 orgId=0 uname= method=GET path=/grafana/public/build/vendor.469095018b321ef1da7c.js status=404 remote_addr=xx.xx.xx.xx time_ms=5 size=26532 referer=https://domain.net/grafana

t=2019-02-01T01:55:45+0000 lvl=info msg=“Request Completed” logger=context userId=0 orgId=0 uname= method=GET path=/grafana/public/build/grafana.dark.469095018b321ef1da7c.css status=404 remote_addr=xx.xx.xx.xx time_ms=3 size=26532 referer=https://domain.net/grafana

t=2019-02-01T01:55:45+0000 lvl=info msg=“Request Completed” logger=context userId=0 orgId=0 uname= method=GET path=/grafana/public/build/app.469095018b321ef1da7c.js status=404 remote_addr=xx.xx.xx.xx time_ms=3 size=26532 referer=https://domain.net/grafana

t=2019-02-01T02:02:52+0000 lvl=info msg=“Request Completed” logger=context userId=0 orgId=0 uname= method=GET path=/grafana status=404 remote_addr=xx.xx.xx.xx time_ms=3 size=26532 referer=

t=2019-02-01T02:02:52+0000 lvl=info msg=“Request Completed” logger=context userId=0 orgId=0 uname= method=GET path=/grafana/public/build/grafana.dark.469095018b321ef1da7c.css status=404 remote_addr=xx.xx.xx.xx time_ms=4 size=26532 referer=https://domain.net/grafana

Caddyfile (note: I also tried without the http:// and got the same result):

www.domain.net, domain.net {
        gzip
        header / {
            Content-Security-Policy
            default-src "*"
            X-Frame-Options "DENY"
            X-Content-Type-Options "nosniff"
            X-XSS-Protection "1; mode=block"
            Strict-Transport-Security "max-age=31536000;"
            Referrer-Policy "same-origin"
            Feature-Policy "self"
        }
        tls certs@domain.net
        proxy /grafana http://10.9.9.132:3000 {
                transparent
                websocket
        }

and my grafana container docker run command:

docker run -d --user 1000 --volume "/home/alexander/grafana:/var/lib/grafana" -p 3000:3000 -e "GF_SERVER_ROOT_URL=%(protocol)s://domain.net/grafana" -e "GF_SERVER_DOMAIN=domain.net" -e "GF_SECURITY_ADMIN_PASSWORD=admin" -e "GF_INSTALL_PLUGINS=natel-influx-admin-panel,ryantxu-annolist-panel,jdbranham-diagram-panel,grafana-worldmap-panel,grafana-piechart-panel" grafana/grafana

GF_SERVER_ROOT_URL and GF_SERVER_DOMAIN are defined here as required env variables when setting up grafana as a sub directory. However when I navigate to the directory I get:

If you’re seeing this Grafana has failed to load its application files

  1. This could be caused by your reverse proxy settings.
  2. If you host grafana under subpath make sure your grafana.ini root_path setting includes subpath
  3. If you have a local dev build make sure you build frontend using: npm run dev, npm run watch, or npm run build
  4. Sometimes restarting grafana-server can help
2

This seems very likely to be a Grafana configuration issue; possibly the environmental variables aren’t being read as expected.

There is unfortunately no simple way to get Caddy to deal with an unruly upstream that doesn’t take to a base path very well. If you can’t configure Grafana properly, your last resort is http.filter to rewrite the resource locations in the HTML/JS/CSS on the fly. Can’t promise it’ll work very well, especially if there’s websockets involved, etc.

https://caddyserver.com/docs/http.filter

I don’t believe it’s possible for Caddy to both respond to client with “No such site” and proxy the request upstream for the same request. Either Caddy finds a matching configuration (which will include the proxy configuration), or it finds no match for the requested site and issues the “No such site” 404.

3

I wonder if I have to build Grafana with those variables instead of passing them at runtime.

4

I don’t know if they’d be compiled in.

You might consider also copying out the ini files, editing them, and mounting them back in when you run the container.

5

I may try mounting the config files (like caddy) first to see if its possible to just edit them that way.

1 Like
6

Okay that “worked” though its still not working. Its odd because others have gotten it to work behind nginx and apache but for the life of me I can’t get it to work behind caddy.

In the dev console on chrome I can see its trying to load resources at the correct path but they just won’t load.

7

Why not? What response are these requests getting back?

8

404s though the resources are in the right directory and the config ini is setup so that resources are loaded from it.

9

Can you initiate a request directly for one of these resources, but swap in the upstream server directly, e.g.:

curl -I http://10.9.9.132:3000/grafana/path/to/resource.file

Just to check whether it’s Caddy getting in the way or the Grafana still isn’t happy with its configuration?

10

Okay I don’t know what it was but I wiped out the container and re-did the mounts for grafana.ini and edited/saved the file before I re-ran the docker run command to spin up a new container. Now its working…

lol.

so grafana.ini looks like:

#################################### Server ####################################
[server]
# Protocol (http, https, socket)
;protocol = http

# The ip address to bind to, empty will bind to all interfaces
;http_addr =

# The http port  to use
;http_port = 3000

# The public facing domain name used to access grafana from a browser
domain = domain.net

# Redirect to correct domain if host header does not match domain
# Prevents DNS rebinding attacks
;enforce_domain = false

# The full public facing url you use in browser, used for redirects and emails
# If you use reverse proxy and sub path specify full url (with sub path)
root_url = http://domain.net/grafana

# Log web requests
;router_logging = false

# the path relative working path
;static_root_path = public

# enable gzip
;enable_gzip = false

# https certs & key file
;cert_file =
;cert_key =

# Unix socket path
;socket =

Caddy config is still same as original op.

1 Like
11

Okay so its actually not working correctly. I can only get it to work with without /grafana/. I dropped a post in the grafana forums to see if this is a grafana config issue as doing the curl to the resource returns a 404…

1 Like
12

This topic was automatically closed 90 days after the last reply. New replies are no longer allowed.