Grafana behind Caddy

Help
1

1. The problem I’m having:

I’ve been trying to use Caddy to serve Grafana over HTTPs, however having issues with it when using HTTPS specifically. I’ve tried all the examples I can find on the forums without luck.

It works as expected directly to Grafana via HTTP. If I turn off TLS in Caddy, serve it as plain HTTP only, it also works perfectly.

With HTTPS however, graphs load sporadically, and most of the time a couple of the graphs just sit there with loading indicator forever. The more I refresh, the more it seems to get worse and less graphs will load. If I restart caddy, it is sluggish and takes 10+ seconds to stop.
e.g.

image
image2548×1296 180 KB

2. Error messages and/or full log output:

Oct 07 12:17:16 caddy caddy[29173]: {"level":"debug","ts":1728265636.3363388,"logger":"http.handlers.reverse_proxy","msg":"upstream roundtrip","upstream":"192.168.10.7:3000","duration":0.051233035,"request":{"remote_ip":"192.168.10.100","remote_port":"49634","client_ip":"192.168.10.100","proto":"HTTP/1.1","method":"GET","host":"grafana.pi.localdomain","uri":"/grafana/public/build/4998.ed066aa322d971e471f0.js","headers":{"Pragma":["no-cache"],"X-Forwarded-Host":["grafana.pi.localdomain"],"Sec-Fetch-Dest":["script"],"Cookie":["REDACTED"],"Accept":["*/*"],"Sec-Fetch-Mode":["no-cors"],"Cache-Control":["no-cache"],"X-Forwarded-Proto":["https"],"Accept-Encoding":["gzip, deflate, br, zstd"],"Referer":["https://grafana.pi.localdomain/grafana/d/gwEYOHWRk/graphs?orgId=1"],"X-Forwarded-For":["192.168.10.100"],"Dnt":["1"],"User-Agent":["Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:130.0) Gecko/20100101 Firefox/130.0"],"Accept-Language":["en-US,en;q=0.5"],"Sec-Fetch-Site":["same-origin"]},"tls":{"resumed":false,"version":772,"cipher_suite":4865,"proto":"http/1.1","server_name":"grafana.pi.localdomain"}},"headers":{"Date":["Mon, 07 Oct 2024 01:47:16 GMT"],"Accept-Ranges":["bytes"],"Cache-Control":["public, max-age=31536000"],"Content-Encoding":["gzip"],"Content-Type":["text/javascript; charset=utf-8"],"Last-Modified":["Tue, 30 Jul 2024 15:22:55 GMT"],"Vary":["Accept-Encoding"]},"status":200}
Oct 07 12:17:16 caddy caddy[29173]: {"level":"debug","ts":1728265636.3499272,"logger":"http.handlers.reverse_proxy","msg":"upstream roundtrip","upstream":"192.168.10.7:3000","duration":0.094200848,"request":{"remote_ip":"192.168.10.100","remote_port":"49631","client_ip":"192.168.10.100","proto":"HTTP/1.1","method":"GET","host":"grafana.pi.localdomain","uri":"/grafana/public/build/1247.34022e0fa8ad25da7b32.js","headers":{"Accept":["*/*"],"Sec-Fetch-Site":["same-origin"],"Accept-Encoding":["gzip, deflate, br, zstd"],"X-Forwarded-For":["192.168.10.100"],"Cookie":["REDACTED"],"Sec-Fetch-Mode":["no-cors"],"Dnt":["1"],"Referer":["https://grafana.pi.localdomain/grafana/d/gwEYOHWRk/graphs?orgId=1"],"Accept-Language":["en-US,en;q=0.5"],"Pragma":["no-cache"],"Cache-Control":["no-cache"],"Sec-Fetch-Dest":["script"],"User-Agent":["Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:130.0) Gecko/20100101 Firefox/130.0"],"X-Forwarded-Proto":["https"],"X-Forwarded-Host":["grafana.pi.localdomain"]},"tls":{"resumed":true,"version":772,"cipher_suite":4865,"proto":"http/1.1","server_name":"grafana.pi.localdomain"}},"headers":{"Date":["Mon, 07 Oct 2024 01:47:16 GMT"],"Accept-Ranges":["bytes"],"Cache-Control":["public, max-age=31536000"],"Content-Encoding":["gzip"],"Content-Type":["text/javascript; charset=utf-8"],"Last-Modified":["Tue, 30 Jul 2024 15:22:55 GMT"],"Vary":["Accept-Encoding"]},"status":200}
Oct 07 12:17:16 caddy caddy[29173]: {"level":"debug","ts":1728265636.3555205,"logger":"http.handlers.reverse_proxy","msg":"upstream roundtrip","upstream":"192.168.10.7:3000","duration":0.04526787,"request":{"remote_ip":"192.168.10.100","remote_port":"49630","client_ip":"192.168.10.100","proto":"HTTP/1.1","method":"GET","host":"grafana.pi.localdomain","uri":"/grafana/public/fonts/inter/Inter-Regular.woff2","headers":{"X-Forwarded-Host":["grafana.pi.localdomain"],"Sec-Fetch-Mode":["cors"],"Sec-Fetch-Site":["same-origin"],"User-Agent":["Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:130.0) Gecko/20100101 Firefox/130.0"],"X-Forwarded-Proto":["https"],"Pragma":["no-cache"],"Accept-Encoding":["identity"],"Accept":["application/font-woff2;q=1.0,application/font-woff;q=0.9,*/*;q=0.8"],"Sec-Fetch-Dest":["font"],"Dnt":["1"],"Cookie":["REDACTED"],"Referer":["https://grafana.pi.localdomain/grafana/public/build/grafana.dark.ac9d572e376a8330bd1a.css"],"Accept-Language":["en-US,en;q=0.5"],"X-Forwarded-For":["192.168.10.100"],"Cache-Control":["no-cache"]},"tls":{"resumed":true,"version":772,"cipher_suite":4865,"proto":"http/1.1","server_name":"grafana.pi.localdomain"}},"headers":{"Content-Length":["108488"],"Content-Type":["font/woff2"],"Last-Modified":["Tue, 30 Jul 2024 15:22:55 GMT"],"Date":["Mon, 07 Oct 2024 01:47:16 GMT"],"Accept-Ranges":["bytes"],"Cache-Control":["public, max-age=3600"]},"status":200}
Oct 07 12:17:16 caddy caddy[29173]: {"level":"debug","ts":1728265636.3971038,"logger":"http.handlers.reverse_proxy","msg":"upstream roundtrip","upstream":"192.168.10.7:3000","duration":0.124023535,"request":{"remote_ip":"192.168.10.100","remote_port":"49629","client_ip":"192.168.10.100","proto":"HTTP/1.1","method":"GET","host":"grafana.pi.localdomain","uri":"/grafana/public/build/app.8222b35d5a14e3d0ea28.js","headers":{"Referer":["https://grafana.pi.localdomain/grafana/d/gwEYOHWRk/graphs?orgId=1"],"Sec-Fetch-Site":["same-origin"],"User-Agent":["Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:130.0) Gecko/20100101 Firefox/130.0"],"Cookie":["REDACTED"],"Accept-Encoding":["gzip, deflate, br, zstd"],"Pragma":["no-cache"],"Sec-Fetch-Dest":["script"],"X-Forwarded-For":["192.168.10.100"],"Sec-Fetch-Mode":["no-cors"],"Accept-Language":["en-US,en;q=0.5"],"Dnt":["1"],"X-Forwarded-Proto":["https"],"Accept":["*/*"],"Cache-Control":["no-cache"],"X-Forwarded-Host":["grafana.pi.localdomain"]},"tls":{"resumed":true,"version":772,"cipher_suite":4865,"proto":"http/1.1","server_name":"grafana.pi.localdomain"}},"headers":{"Last-Modified":["Tue, 30 Jul 2024 15:22:55 GMT"],"Vary":["Accept-Encoding"],"Date":["Mon, 07 Oct 2024 01:47:16 GMT"],"Accept-Ranges":["bytes"],"Cache-Control":["public, max-age=31536000"],"Content-Encoding":["gzip"],"Content-Type":["text/javascript; charset=utf-8"]},"status":200}
Oct 07 12:17:17 caddy caddy[29173]: {"level":"debug","ts":1728265637.7894757,"logger":"http.handlers.reverse_proxy","msg":"selected upstream","dial":"192.168.10.7:3000","total_upstreams":1}
Oct 07 12:17:17 caddy caddy[29173]: {"level":"debug","ts":1728265637.803527,"logger":"http.handlers.reverse_proxy","msg":"selected upstream","dial":"192.168.10.7:3000","total_upstreams":1}
Oct 07 12:17:17 caddy caddy[29173]: {"level":"debug","ts":1728265637.8041866,"logger":"http.handlers.reverse_proxy","msg":"upstream roundtrip","upstream":"192.168.10.7:3000","duration":0.014029112,"request":{"remote_ip":"192.168.10.100","remote_port":"49631","client_ip":"192.168.10.100","proto":"HTTP/1.1","method":"GET","host":"grafana.pi.localdomain","uri":"/grafana/public/build/9521.9e42436d765de344d7fc.js","headers":{"Cache-Control":["no-cache"],"Referer":["https://grafana.pi.localdomain/grafana/d/gwEYOHWRk/graphs?orgId=1"],"Sec-Fetch-Site":["same-origin"],"Sec-Fetch-Mode":["no-cors"],"X-Forwarded-Host":["grafana.pi.localdomain"],"User-Agent":["Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:130.0) Gecko/20100101 Firefox/130.0"],"Accept-Encoding":["gzip, deflate, br, zstd"],"X-Forwarded-Proto":["https"],"Accept-Language":["en-US,en;q=0.5"],"Cookie":["REDACTED"],"Accept":["*/*"],"X-Forwarded-For":["192.168.10.100"],"Sec-Fetch-Dest":["script"],"Dnt":["1"],"Pragma":["no-cache"]},"tls":{"resumed":true,"version":772,"cipher_suite":4865,"proto":"http/1.1","server_name":"grafana.pi.localdomain"}},"headers":{"Date":["Mon, 07 Oct 2024 01:47:17 GMT"],"Accept-Ranges":["bytes"],"Cache-Control":["public, max-age=31536000"],"Content-Encoding":["gzip"],"Content-Type":["text/javascript; charset=utf-8"],"Last-Modified":["Tue, 30 Jul 2024 15:22:55 GMT"],"Vary":["Accept-Encoding"]},"status":200}
Oct 07 12:17:17 caddy caddy[29173]: {"level":"debug","ts":1728265637.8154032,"logger":"http.handlers.reverse_proxy","msg":"upstream roundtrip","upstream":"192.168.10.7:3000","duration":0.011086454,"request":{"remote_ip":"192.168.10.100","remote_port":"49635","client_ip":"192.168.10.100","proto":"HTTP/1.1","method":"GET","host":"grafana.pi.localdomain","uri":"/grafana/api/live/ws","headers":{"Sec-Websocket-Extensions":["permessage-deflate"],"Connection":["Upgrade"],"Sec-Websocket-Key":["9FZ5KslLVFx8NawrUlwDng=="],"Cookie":["REDACTED"],"X-Forwarded-Proto":["https"],"Origin":["https://grafana.pi.localdomain"],"Sec-Websocket-Version":["13"],"Dnt":["1"],"Pragma":["no-cache"],"Accept-Encoding":["gzip, deflate, br, zstd"],"Sec-Fetch-Mode":["websocket"],"Cache-Control":["no-cache"],"Sec-Fetch-Site":["same-origin"],"Accept-Language":["en-US,en;q=0.5"],"X-Forwarded-For":["192.168.10.100"],"X-Forwarded-Host":["grafana.pi.localdomain"],"User-Agent":["Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:130.0) Gecko/20100101 Firefox/130.0"],"Upgrade":["websocket"],"Accept":["*/*"],"Sec-Fetch-Dest":["empty"]},"tls":{"resumed":true,"version":772,"cipher_suite":4865,"proto":"http/1.1","server_name":"grafana.pi.localdomain"}},"headers":{"Upgrade":["websocket"],"Connection":["Upgrade"],"Sec-Websocket-Accept":["w1CncahWEa7UTxQnmAevvuEuqDg="]},"status":101}
Oct 07 12:17:17 caddy caddy[29173]: {"level":"debug","ts":1728265637.816011,"logger":"http.handlers.reverse_proxy","msg":"upgrading connection","upstream":"192.168.10.7:3000","duration":0.011086454,"request":{"remote_ip":"192.168.10.100","remote_port":"49635","client_ip":"192.168.10.100","proto":"HTTP/1.1","method":"GET","host":"grafana.pi.localdomain","uri":"/grafana/api/live/ws","headers":{"Sec-Websocket-Extensions":["permessage-deflate"],"Connection":["Upgrade"],"Sec-Websocket-Key":["9FZ5KslLVFx8NawrUlwDng=="],"Cookie":["REDACTED"],"X-Forwarded-Proto":["https"],"Origin":["https://grafana.pi.localdomain"],"Sec-Websocket-Version":["13"],"Dnt":["1"],"Pragma":["no-cache"],"Accept-Encoding":["gzip, deflate, br, zstd"],"Sec-Fetch-Mode":["websocket"],"Cache-Control":["no-cache"],"Sec-Fetch-Site":["same-origin"],"Accept-Language":["en-US,en;q=0.5"],"X-Forwarded-For":["192.168.10.100"],"X-Forwarded-Host":["grafana.pi.localdomain"],"User-Agent":["Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:130.0) Gecko/20100101 Firefox/130.0"],"Upgrade":["websocket"],"Accept":["*/*"],"Sec-Fetch-Dest":["empty"]},"tls":{"resumed":true,"version":772,"cipher_suite":4865,"proto":"http/1.1","server_name":"grafana.pi.localdomain"}}}
Oct 07 12:17:17 caddy caddy[29173]: {"level":"debug","ts":1728265637.8223455,"logger":"http.handlers.reverse_proxy","msg":"selected upstream","dial":"192.168.10.7:3000","total_upstreams":1}
Oct 07 12:17:17 caddy caddy[29173]: {"level":"debug","ts":1728265637.823707,"logger":"http.handlers.reverse_proxy","msg":"selected upstream","dial":"192.168.10.7:3000","total_upstreams":1}
Oct 07 12:17:17 caddy caddy[29173]: {"level":"debug","ts":1728265637.8223455,"logger":"http.handlers.reverse_proxy","msg":"selected upstream","dial":"192.168.10.7:3000","total_upstreams":1}
Oct 07 12:17:17 caddy caddy[29173]: {"level":"debug","ts":1728265637.823707,"logger":"http.handlers.reverse_proxy","msg":"selected upstream","dial":"192.168.10.7:3000","total_upstreams":1}
Oct 07 12:17:17 caddy caddy[29173]: {"level":"debug","ts":1728265637.8295712,"logger":"http.handlers.reverse_proxy","msg":"upstream roundtrip","upstream":"192.168.10.7:3000","duration":0.006622291,"request":{"remote_ip":"192.168.10.100","remote_port":"49633","client_ip":"192.168.10.100","proto":"HTTP/1.1","method":"GET","host":"grafana.pi.localdomain","uri":"/grafana/public/img/fav32.png","headers":{"Accept-Encoding":["gzip, deflate, br, zstd"],"Pragma":["no-cache"],"X-Forwarded-For":["192.168.10.100"],"Sec-Fetch-Mode":["no-cors"],"Sec-Fetch-Site":["same-origin"],"Cookie":["REDACTED"],"Sec-Fetch-Dest":["image"],"Priority":["u=6"],"Referer":["https://grafana.pi.localdomain/grafana/d/gwEYOHWRk/graphs?orgId=1"],"X-Forwarded-Host":["grafana.pi.localdomain"],"Accept":["image/avif,image/webp,image/png,image/svg+xml,image/*;q=0.8,*/*;q=0.5"],"X-Forwarded-Proto":["https"],"User-Agent":["Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:130.0) Gecko/20100101 Firefox/130.0"],"Dnt":["1"],"Accept-Language":["en-US,en;q=0.5"],"Cache-Control":["no-cache"]},"tls":{"resumed":false,"version":772,"cipher_suite":4865,"proto":"http/1.1","server_name":"grafana.pi.localdomain"}},"headers":{"Cache-Control":["public, max-age=3600"],"Content-Encoding":["gzip"],"Content-Type":["image/png"],"Last-Modified":["Tue, 30 Jul 2024 15:22:55 GMT"],"Vary":["Accept-Encoding"],"Date":["Mon, 07 Oct 2024 01:47:17 GMT"],"Content-Length":["1146"],"Accept-Ranges":["bytes"]},"status":200}
Oct 07 12:17:17 caddy caddy[29173]: {"level":"debug","ts":1728265637.8302116,"logger":"http.handlers.reverse_proxy","msg":"upstream roundtrip","upstream":"192.168.10.7:3000","duration":0.006196729,"request":{"remote_ip":"192.168.10.100","remote_port":"49631","client_ip":"192.168.10.100","proto":"HTTP/1.1","method":"GET","host":"grafana.pi.localdomain","uri":"/grafana/public/img/apple-touch-icon.png","headers":{"Sec-Fetch-Mode":["no-cors"],"Sec-Fetch-Site":["same-origin"],"X-Forwarded-Host":["grafana.pi.localdomain"],"Accept-Language":["en-US,en;q=0.5"],"Accept":["image/avif,image/webp,image/png,image/svg+xml,image/*;q=0.8,*/*;q=0.5"],"X-Forwarded-Proto":["https"],"Sec-Fetch-Dest":["image"],"Accept-Encoding":["gzip, deflate, br, zstd"],"Dnt":["1"],"Pragma":["no-cache"],"Cookie":["REDACTED"],"User-Agent":["Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:130.0) Gecko/20100101 Firefox/130.0"],"Referer":["https://grafana.pi.localdomain/grafana/d/gwEYOHWRk/graphs?orgId=1"],"Priority":["u=6"],"Cache-Control":["no-cache"],"X-Forwarded-For":["192.168.10.100"]},"tls":{"resumed":true,"version":772,"cipher_suite":4865,"proto":"http/1.1","server_name":"grafana.pi.localdomain"}},"headers":{"Date":["Mon, 07 Oct 2024 01:47:17 GMT"],"Accept-Ranges":["bytes"],"Cache-Control":["public, max-age=3600"],"Content-Encoding":["gzip"],"Content-Type":["image/png"],"Last-Modified":["Tue, 30 Jul 2024 15:22:55 GMT"],"Vary":["Accept-Encoding"]},"status":200}
Oct 07 12:17:17 caddy caddy[29173]: {"level":"debug","ts":1728265637.8980308,"logger":"http.handlers.reverse_proxy","msg":"selected upstream","dial":"192.168.10.7:3000","total_upstreams":1}
Oct 07 12:17:17 caddy caddy[29173]: {"level":"debug","ts":1728265637.899331,"logger":"http.handlers.reverse_proxy","msg":"selected upstream","dial":"192.168.10.7:3000","total_upstreams":1}
Oct 07 12:17:17 caddy caddy[29173]: {"level":"debug","ts":1728265637.9000115,"logger":"http.handlers.reverse_proxy","msg":"selected upstream","dial":"192.168.10.7:3000","total_upstreams":1}
Oct 07 12:17:17 caddy caddy[29173]: {"level":"debug","ts":1728265637.900645,"logger":"http.handlers.reverse_proxy","msg":"selected upstream","dial":"192.168.10.7:3000","total_upstreams":1}
Oct 07 12:17:17 caddy caddy[29173]: {"level":"debug","ts":1728265637.90919,"logger":"http.handlers.reverse_proxy","msg":"upstream roundtrip","upstream":"192.168.10.7:3000","duration":0.008381121,"request":{"remote_ip":"192.168.10.100","remote_port":"49633","client_ip":"192.168.10.100","proto":"HTTP/1.1","method":"GET","host":"grafana.pi.localdomain","uri":"/grafana/public/build/660.d976140acbafb98e22ef.js","headers":{"Dnt":["1"],"X-Forwarded-Host":["grafana.pi.localdomain"],"User-Agent":["Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:130.0) Gecko/20100101 Firefox/130.0"],"Accept":["*/*"],"Cookie":["REDACTED"],"Sec-Fetch-Dest":["script"],"Accept-Encoding":["gzip, deflate, br, zstd"],"Cache-Control":["no-cache"],"Sec-Fetch-Mode":["no-cors"],"Sec-Fetch-Site":["same-origin"],"Pragma":["no-cache"],"X-Forwarded-For":["192.168.10.100"],"Referer":["https://grafana.pi.localdomain/grafana/d/gwEYOHWRk/graphs?orgId=1"],"Accept-Language":["en-US,en;q=0.5"],"X-Forwarded-Proto":["https"]},"tls":{"resumed":false,"version":772,"cipher_suite":4865,"proto":"http/1.1","server_name":"grafana.pi.localdomain"}},"headers":{"Content-Encoding":["gzip"],"Content-Type":["text/javascript; charset=utf-8"],"Last-Modified":["Tue, 30 Jul 2024 15:22:55 GMT"],"Vary":["Accept-Encoding"],"Date":["Mon, 07 Oct 2024 01:47:17 GMT"],"Accept-Ranges":["bytes"],"Cache-Control":["public, max-age=31536000"]},"status":200}
Oct 07 12:17:17 caddy caddy[29173]: {"level":"debug","ts":1728265637.9125152,"logger":"http.handlers.reverse_proxy","msg":"upstream roundtrip","upstream":"192.168.10.7:3000","duration":0.010670932,"request":{"remote_ip":"192.168.10.100","remote_port":"49630","client_ip":"192.168.10.100","proto":"HTTP/1.1","method":"GET","host":"grafana.pi.localdomain","uri":"/grafana/public/build/6263.8e49ed37c2dc63f6b065.js","headers":{"Accept-Language":["en-US,en;q=0.5"],"Sec-Fetch-Site":["same-origin"],"User-Agent":["Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:130.0) Gecko/20100101 Firefox/130.0"],"Cookie":["REDACTED"],"Referer":["https://grafana.pi.localdomain/grafana/d/gwEYOHWRk/graphs?orgId=1"],"Sec-Fetch-Mode":["no-cors"],"X-Forwarded-Host":["grafana.pi.localdomain"],"Pragma":["no-cache"],"Sec-Fetch-Dest":["script"],"Cache-Control":["no-cache"],"X-Forwarded-Proto":["https"],"Dnt":["1"],"Accept-Encoding":["gzip, deflate, br, zstd"],"Accept":["*/*"],"X-Forwarded-For":["192.168.10.100"]},"tls":{"resumed":true,"version":772,"cipher_suite":4865,"proto":"http/1.1","server_name":"grafana.pi.localdomain"}},"headers":{"Content-Encoding":["gzip"],"Content-Type":["text/javascript; charset=utf-8"],"Last-Modified":["Tue, 30 Jul 2024 15:22:55 GMT"],"Vary":["Accept-Encoding"],"Date":["Mon, 07 Oct 2024 01:47:17 GMT"],"Accept-Ranges":["bytes"],"Cache-Control":["public, max-age=31536000"]},"status":200}
Oct 07 12:17:17 caddy caddy[29173]: {"level":"debug","ts":1728265637.913106,"logger":"http.handlers.reverse_proxy","msg":"upstream roundtrip","upstream":"192.168.10.7:3000","duration":0.011858181,"request":{"remote_ip":"192.168.10.100","remote_port":"49631","client_ip":"192.168.10.100","proto":"HTTP/1.1","method":"GET","host":"grafana.pi.localdomain","uri":"/grafana/public/build/9569.81cb8bd2d9ed9b9b3389.js","headers":{"Dnt":["1"],"Pragma":["no-cache"],"X-Forwarded-For":["192.168.10.100"],"Sec-Fetch-Mode":["no-cors"],"Cache-Control":["no-cache"],"X-Forwarded-Host":["grafana.pi.localdomain"],"Cookie":["REDACTED"],"Sec-Fetch-Site":["same-origin"],"Accept":["*/*"],"Sec-Fetch-Dest":["script"],"Accept-Encoding":["gzip, deflate, br, zstd"],"Referer":["https://grafana.pi.localdomain/grafana/d/gwEYOHWRk/graphs?orgId=1"],"X-Forwarded-Proto":["https"],"User-Agent":["Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:130.0) Gecko/20100101 Firefox/130.0"],"Accept-Language":["en-US,en;q=0.5"]},"tls":{"resumed":true,"version":772,"cipher_suite":4865,"proto":"http/1.1","server_name":"grafana.pi.localdomain"}},"headers":{"Content-Type":["text/javascript; charset=utf-8"],"Last-Modified":["Tue, 30 Jul 2024 15:22:55 GMT"],"Vary":["Accept-Encoding"],"Date":["Mon, 07 Oct 2024 01:47:17 GMT"],"Accept-Ranges":["bytes"],"Cache-Control":["public, max-age=31536000"],"Content-Encoding":["gzip"]},"status":200}
Oct 07 12:17:17 caddy caddy[29173]: {"level":"debug","ts":1728265637.9164693,"logger":"http.handlers.reverse_proxy","msg":"upstream roundtrip","upstream":"192.168.10.7:3000","duration":0.014883177,"request":{"remote_ip":"192.168.10.100","remote_port":"49629","client_ip":"192.168.10.100","proto":"HTTP/1.1","method":"GET","host":"grafana.pi.localdomain","uri":"/grafana/public/build/DashboardPageProxy.53725fe76b78fada005e.js","headers":{"Pragma":["no-cache"],"Cache-Control":["no-cache"],"User-Agent":["Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:130.0) Gecko/20100101 Firefox/130.0"],"Sec-Fetch-Mode":["no-cors"],"Referer":["https://grafana.pi.localdomain/grafana/d/gwEYOHWRk/graphs?orgId=1"],"Sec-Fetch-Site":["same-origin"],"Accept-Encoding":["gzip, deflate, br, zstd"],"Accept-Language":["en-US,en;q=0.5"],"Dnt":["1"],"X-Forwarded-For":["192.168.10.100"],"Accept":["*/*"],"Sec-Fetch-Dest":["script"],"X-Forwarded-Proto":["https"],"X-Forwarded-Host":["grafana.pi.localdomain"],"Cookie":["REDACTED"]},"tls":{"resumed":true,"version":772,"cipher_suite":4865,"proto":"http/1.1","server_name":"grafana.pi.localdomain"}},"headers":{"Accept-Ranges":["bytes"],"Cache-Control":["public, max-age=31536000"],"Content-Encoding":["gzip"],"Content-Type":["text/javascript; charset=utf-8"],"Last-Modified":["Tue, 30 Jul 2024 15:22:55 GMT"],"Vary":["Accept-Encoding"],"Date":["Mon, 07 Oct 2024 01:47:17 GMT"]},"status":200}
Oct 07 12:17:17 caddy caddy[29173]: {"level":"debug","ts":1728265637.9507685,"logger":"http.handlers.reverse_proxy","msg":"selected upstream","dial":"192.168.10.7:3000","total_upstreams":1}
Oct 07 12:17:17 caddy caddy[29173]: {"level":"debug","ts":1728265637.9517632,"logger":"http.handlers.reverse_proxy","msg":"selected upstream","dial":"192.168.10.7:3000","total_upstreams":1}
Oct 07 12:17:17 caddy caddy[29173]: {"level":"debug","ts":1728265637.952416,"logger":"http.handlers.reverse_proxy","msg":"selected upstream","dial":"192.168.10.7:3000","total_upstreams":1}
Oct 07 12:17:17 caddy caddy[29173]: {"level":"debug","ts":1728265637.953049,"logger":"http.handlers.reverse_proxy","msg":"selected upstream","dial":"192.168.10.7:3000","total_upstreams":1}
Oct 07 12:17:17 caddy caddy[29173]: {"level":"debug","ts":1728265637.952416,"logger":"http.handlers.reverse_proxy","msg":"selected upstream","dial":"192.168.10.7:3000","total_upstreams":1}
Oct 07 12:17:17 caddy caddy[29173]: {"level":"debug","ts":1728265637.953049,"logger":"http.handlers.reverse_proxy","msg":"selected upstream","dial":"192.168.10.7:3000","total_upstreams":1}
Oct 07 12:17:17 caddy caddy[29173]: {"level":"debug","ts":1728265637.95734,"logger":"http.handlers.reverse_proxy","msg":"upstream roundtrip","upstream":"192.168.10.7:3000","duration":0.002865353,"request":{"remote_ip":"192.168.10.100","remote_port":"49629","client_ip":"192.168.10.100","proto":"HTTP/1.1","method":"GET","host":"grafana.pi.localdomain","uri":"/grafana/public/fonts/inter/Inter-Medium.woff2","headers":{"Sec-Fetch-Site":["same-origin"],"Accept-Language":["en-US,en;q=0.5"],"Referer":["https://grafana.pi.localdomain/grafana/public/build/grafana.dark.ac9d572e376a8330bd1a.css"],"Dnt":["1"],"Sec-Fetch-Dest":["font"],"X-Forwarded-Proto":["https"],"User-Agent":["Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:130.0) Gecko/20100101 Firefox/130.0"],"Cookie":["REDACTED"],"Accept":["application/font-woff2;q=1.0,application/font-woff;q=0.9,*/*;q=0.8"],"Accept-Encoding":["identity"],"X-Forwarded-Host":["grafana.pi.localdomain"],"Cache-Control":["no-cache"],"Pragma":["no-cache"],"X-Forwarded-For":["192.168.10.100"],"Sec-Fetch-Mode":["cors"]},"tls":{"resumed":true,"version":772,"cipher_suite":4865,"proto":"http/1.1","server_name":"grafana.pi.localdomain"}},"headers":{"Cache-Control":["public, max-age=3600"],"Content-Length":["111380"],"Content-Type":["font/woff2"],"Last-Modified":["Tue, 30 Jul 2024 15:22:55 GMT"],"Date":["Mon, 07 Oct 2024 01:47:17 GMT"],"Accept-Ranges":["bytes"]},"status":200}
Oct 07 12:17:17 caddy caddy[29173]: {"level":"debug","ts":1728265637.961906,"logger":"http.handlers.reverse_proxy","msg":"upstream roundtrip","upstream":"192.168.10.7:3000","duration":0.008335953,"request":{"remote_ip":"192.168.10.100","remote_port":"49630","client_ip":"192.168.10.100","proto":"HTTP/1.1","method":"GET","host":"grafana.pi.localdomain","uri":"/grafana/public/img/icons/unicons/spinner.svg","headers":{"X-Forwarded-Host":["grafana.pi.localdomain"],"Accept-Encoding":["gzip, deflate, br, zstd"],"Cache-Control":["no-cache"],"Sec-Fetch-Mode":["cors"],"X-Forwarded-Proto":["https"],"Sec-Fetch-Site":["same-origin"],"User-Agent":["Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:130.0) Gecko/20100101 Firefox/130.0"],"Dnt":["1"],"Sec-Fetch-Dest":["empty"],"Cookie":["REDACTED"],"Referer":["https://grafana.pi.localdomain/grafana/d/gwEYOHWRk/graphs?orgId=1"],"X-Forwarded-For":["192.168.10.100"],"Accept":["*/*"],"Pragma":["no-cache"],"Priority":["u=4"],"Accept-Language":["en-US,en;q=0.5"]},"tls":{"resumed":true,"version":772,"cipher_suite":4865,"proto":"http/1.1","server_name":"grafana.pi.localdomain"}},"headers":{"Content-Length":["413"],"Accept-Ranges":["bytes"],"Cache-Control":["public, max-age=3600"],"Content-Encoding":["gzip"],"Content-Type":["image/svg+xml"],"Last-Modified":["Tue, 30 Jul 2024 15:22:55 GMT"],"Vary":["Accept-Encoding"],"Date":["Mon, 07 Oct 2024 01:47:17 GMT"]},"status":200}
Oct 07 12:17:17 caddy caddy[29173]: {"level":"debug","ts":1728265637.9807627,"logger":"http.handlers.reverse_proxy","msg":"upstream roundtrip","upstream":"192.168.10.7:3000","duration":0.026485579,"request":{"remote_ip":"192.168.10.100","remote_port":"49631","client_ip":"192.168.10.100","proto":"HTTP/1.1","method":"GET","host":"grafana.pi.localdomain","uri":"/grafana/api/dashboards/uid/gwEYOHWRk","headers":{"Dnt":["1"],"X-Forwarded-Proto":["https"],"Sec-Fetch-Dest":["empty"],"Accept":["application/json, text/plain, */*"],"X-Grafana-Org-Id":["1"],"Pragma":["no-cache"],"Sec-Fetch-Site":["same-origin"],"User-Agent":["Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:130.0) Gecko/20100101 Firefox/130.0"],"X-Forwarded-For":["192.168.10.100"],"Priority":["u=4"],"Cache-Control":["no-cache"],"X-Grafana-Device-Id":["4e5bf113b66df136308e462ae011c4a9"],"Sec-Fetch-Mode":["cors"],"Accept-Encoding":["gzip, deflate, br, zstd"],"Cookie":["REDACTED"],"Accept-Language":["en-US,en;q=0.5"],"X-Forwarded-Host":["grafana.pi.localdomain"],"Referer":["https://grafana.pi.localdomain/grafana/d/gwEYOHWRk/graphs?orgId=1"]},"tls":{"resumed":true,"version":772,"cipher_suite":4865,"proto":"http/1.1","server_name":"grafana.pi.localdomain"}},"headers":{"Content-Type":["application/json"],"X-Xss-Protection":["1; mode=block"],"Date":["Mon, 07 Oct 2024 01:47:18 GMT"],"X-Frame-Options":["deny"],"Cache-Control":["no-store"],"Content-Encoding":["gzip"],"Vary":["Accept-Encoding"],"X-Content-Type-Options":["nosniff"]},"status":200}
Oct 07 12:17:17 caddy caddy[29173]: {"level":"debug","ts":1728265637.9834895,"logger":"http.handlers.reverse_proxy","msg":"upstream roundtrip","upstream":"192.168.10.7:3000","duration":0.029412781,"request":{"remote_ip":"192.168.10.100","remote_port":"49633","client_ip":"192.168.10.100","proto":"HTTP/1.1","method":"GET","host":"grafana.pi.localdomain","uri":"/grafana/api/dashboards/uid/gwEYOHWRk","headers":{"Referer":["https://grafana.pi.localdomain/grafana/d/gwEYOHWRk/graphs?orgId=1"],"Priority":["u=4"],"Cache-Control":["no-cache"],"Accept-Language":["en-US,en;q=0.5"],"Accept-Encoding":["gzip, deflate, br, zstd"],"Cookie":["REDACTED"],"Sec-Fetch-Dest":["empty"],"Accept":["application/json, text/plain, */*"],"X-Grafana-Org-Id":["1"],"Pragma":["no-cache"],"X-Forwarded-For":["192.168.10.100"],"X-Forwarded-Proto":["https"],"X-Grafana-Device-Id":["4e5bf113b66df136308e462ae011c4a9"],"Sec-Fetch-Site":["same-origin"],"User-Agent":["Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:130.0) Gecko/20100101 Firefox/130.0"],"Dnt":["1"],"Sec-Fetch-Mode":["cors"],"X-Forwarded-Host":["grafana.pi.localdomain"]},"tls":{"resumed":false,"version":772,"cipher_suite":4865,"proto":"http/1.1","server_name":"grafana.pi.localdomain"}},"headers":{"Cache-Control":["no-store"],"Content-Type":["application/json"],"X-Content-Type-Options":["nosniff"],"Content-Encoding":["gzip"],"Vary":["Accept-Encoding"],"X-Frame-Options":["deny"],"X-Xss-Protection":["1; mode=block"],"Date":["Mon, 07 Oct 2024 01:47:18 GMT"]},"status":200}
Oct 07 12:17:18 caddy caddy[29173]: {"level":"debug","ts":1728265638.017644,"logger":"http.handlers.reverse_proxy","msg":"upstream roundtrip","upstream":"192.168.10.7:3000","duration":0.010750059,"request":{"remote_ip":"192.168.10.100","remote_port":"49631","client_ip":"192.168.10.100","proto":"HTTP/1.1","method":"GET","host":"grafana.pi.localdomain","uri":"/grafana/api/prometheus/grafana/api/v1/rules?dashboard_uid=gwEYOHWRk","headers":{"Accept-Language":["en-US,en;q=0.5"],"X-Forwarded-Proto":["https"],"User-Agent":["Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:130.0) Gecko/20100101 Firefox/130.0"],"X-Grafana-Device-Id":["4e5bf113b66df136308e462ae011c4a9"],"Accept":["application/json, text/plain, */*"],"Referer":["https://grafana.pi.localdomain/grafana/d/gwEYOHWRk/graphs?orgId=1"],"Sec-Fetch-Dest":["empty"],"X-Forwarded-Host":["grafana.pi.localdomain"],"Accept-Encoding":["gzip, deflate, br, zstd"],"Priority":["u=4"],"X-Grafana-Org-Id":["1"],"Sec-Fetch-Mode":["cors"],"Sec-Fetch-Site":["same-origin"],"Dnt":["1"],"Cookie":["REDACTED"],"X-Forwarded-For":["192.168.10.100"]},"tls":{"resumed":true,"version":772,"cipher_suite":4865,"proto":"http/1.1","server_name":"grafana.pi.localdomain"}},"headers":{"Vary":["Accept-Encoding"],"X-Frame-Options":["deny"],"X-Xss-Protection":["1; mode=block"],"Content-Length":["65"],"Content-Type":["application/json"],"Content-Encoding":["gzip"],"X-Content-Type-Options":["nosniff"],"Date":["Mon, 07 Oct 2024 01:47:18 GMT"],"Cache-Control":["no-store"]},"status":200}
Oct 07 12:17:18 caddy caddy[29173]: {"level":"debug","ts":1728265638.0270858,"logger":"http.handlers.reverse_proxy","msg":"selected upstream","dial":"192.168.10.7:3000","total_upstreams":1}
Oct 07 12:17:18 caddy caddy[29173]: {"level":"debug","ts":1728265638.044102,"logger":"http.handlers.reverse_proxy","msg":"upstream roundtrip","upstream":"192.168.10.7:3000","duration":0.010082829,"request":{"remote_ip":"192.168.10.100","remote_port":"49631","client_ip":"192.168.10.100","proto":"HTTP/1.1","method":"GET","host":"grafana.pi.localdomain","uri":"/grafana/avatar/46d229b033af06a191ff2267bca9ae56","headers":{"User-Agent":["Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:130.0) Gecko/20100101 Firefox/130.0"],"Priority":["u=5, i"],"Pragma":["no-cache"],"X-Forwarded-Proto":["https"],"X-Forwarded-Host":["grafana.pi.localdomain"],"Cookie":["REDACTED"],"Referer":["https://grafana.pi.localdomain/grafana/d/gwEYOHWRk/graphs?orgId=1"],"Accept":["image/avif,image/webp,image/png,image/svg+xml,image/*;q=0.8,*/*;q=0.5"],"X-Forwarded-For":["192.168.10.100"],"Cache-Control":["no-cache"],"Dnt":["1"],"Sec-Fetch-Mode":["no-cors"],"Sec-Fetch-Dest":["image"],"Sec-Fetch-Site":["same-origin"],"Accept-Language":["en-US,en;q=0.5"],"Accept-Encoding":["gzip, deflate, br, zstd"]},"tls":{"resumed":true,"version":772,"cipher_suite":4865,"proto":"http/1.1","server_name":"grafana.pi.localdomain"}},"error":"context canceled"}
Oct 07 12:17:18 caddy caddy[29173]: {"level":"debug","ts":1728265638.049703,"logger":"http.handlers.reverse_proxy","msg":"selected upstream","dial":"192.168.10.7:3000","total_upstreams":1}
Oct 07 12:17:18 caddy caddy[29173]: {"level":"debug","ts":1728265638.0505311,"logger":"http.handlers.reverse_proxy","msg":"selected upstream","dial":"192.168.10.7:3000","total_upstreams":1}
Oct 07 12:17:18 caddy caddy[29173]: {"level":"debug","ts":1728265638.0505311,"logger":"http.handlers.reverse_proxy","msg":"selected upstream","dial":"192.168.10.7:3000","total_upstreams":1}
Oct 07 12:17:18 caddy caddy[29173]: {"level":"debug","ts":1728265638.0622342,"logger":"http.handlers.reverse_proxy","msg":"upstream roundtrip","upstream":"192.168.10.7:3000","duration":0.011582518,"request":{"remote_ip":"192.168.10.100","remote_port":"49629","client_ip":"192.168.10.100","proto":"HTTP/1.1","method":"GET","host":"grafana.pi.localdomain","uri":"/grafana/api/user/orgs","headers":{"Sec-Fetch-Site":["same-origin"],"X-Grafana-Org-Id":["1"],"Priority":["u=4"],"Accept-Encoding":["gzip, deflate, br, zstd"],"Sec-Fetch-Dest":["empty"],"User-Agent":["Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:130.0) Gecko/20100101 Firefox/130.0"],"X-Forwarded-Proto":["https"],"X-Forwarded-For":["192.168.10.100"],"X-Forwarded-Host":["grafana.pi.localdomain"],"Accept":["application/json, text/plain, */*"],"Accept-Language":["en-US,en;q=0.5"],"Cookie":["REDACTED"],"X-Grafana-Device-Id":["4e5bf113b66df136308e462ae011c4a9"],"Dnt":["1"],"Referer":["https://grafana.pi.localdomain/grafana/d/gwEYOHWRk/graphs?orgId=1"],"Sec-Fetch-Mode":["cors"]},"tls":{"resumed":true,"version":772,"cipher_suite":4865,"proto":"http/1.1","server_name":"grafana.pi.localdomain"}},"headers":{"X-Content-Type-Options":["nosniff"],"Date":["Mon, 07 Oct 2024 01:47:18 GMT"],"Cache-Control":["no-store"],"Content-Encoding":["gzip"],"Content-Type":["application/json"],"Vary":["Accept-Encoding"],"X-Frame-Options":["deny"],"X-Xss-Protection":["1; mode=block"],"Content-Length":["69"]},"status":200}
Oct 07 12:17:18 caddy caddy[29173]: {"level":"debug","ts":1728265638.0663276,"logger":"http.handlers.reverse_proxy","msg":"selected upstream","dial":"192.168.10.7:3000","total_upstreams":1}
Oct 07 12:17:18 caddy caddy[29173]: {"level":"debug","ts":1728265638.0686882,"logger":"http.handlers.reverse_proxy","msg":"upstream roundtrip","upstream":"192.168.10.7:3000","duration":0.016700772,"request":{"remote_ip":"192.168.10.100","remote_port":"49633","client_ip":"192.168.10.100","proto":"HTTP/1.1","method":"GET","host":"grafana.pi.localdomain","uri":"/grafana/avatar/46d229b033af06a191ff2267bca9ae56","headers":{"Dnt":["1"],"Priority":["u=4, i"],"Sec-Fetch-Mode":["no-cors"],"Cache-Control":["no-cache"],"X-Forwarded-Host":["grafana.pi.localdomain"],"Sec-Fetch-Site":["same-origin"],"Pragma":["no-cache"],"User-Agent":["Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:130.0) Gecko/20100101 Firefox/130.0"],"Accept-Language":["en-US,en;q=0.5"],"Cookie":["REDACTED"],"Sec-Fetch-Dest":["image"],"Accept":["image/avif,image/webp,image/png,image/svg+xml,image/*;q=0.8,*/*;q=0.5"],"Accept-Encoding":["gzip, deflate, br, zstd"],"X-Forwarded-For":["192.168.10.100"],"X-Forwarded-Proto":["https"],"Referer":["https://grafana.pi.localdomain/grafana/d/gwEYOHWRk/graphs?orgId=1"]},"tls":{"resumed":false,"version":772,"cipher_suite":4865,"proto":"http/1.1","server_name":"grafana.pi.localdomain"}},"headers":{"Content-Encoding":["gzip"],"Content-Type":["image/jpeg"],"X-Content-Type-Options":["nosniff"],"Date":["Mon, 07 Oct 2024 01:47:18 GMT"],"Content-Length":["327"],"Cache-Control":["private, max-age=3600"],"X-Frame-Options":["deny"],"X-Xss-Protection":["1; mode=block"],"Vary":["Accept-Encoding"]},"status":200}
Oct 07 12:17:18 caddy caddy[29173]: {"level":"debug","ts":1728265638.076054,"logger":"http.handlers.reverse_proxy","msg":"upstream roundtrip","upstream":"192.168.10.7:3000","duration":0.009599621,"request":{"remote_ip":"192.168.10.100","remote_port":"49630","client_ip":"192.168.10.100","proto":"HTTP/1.1","method":"GET","host":"grafana.pi.localdomain","uri":"/grafana/api/search?limit=5&type=dash-db&dashboardUID=gwEYOHWRk","headers":{"Sec-Fetch-Dest":["empty"],"X-Grafana-Org-Id":["1"],"Accept":["application/json, text/plain, */*"],"User-Agent":["Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:130.0) Gecko/20100101 Firefox/130.0"],"Accept-Encoding":["gzip, deflate, br, zstd"],"X-Forwarded-Host":["grafana.pi.localdomain"],"X-Forwarded-Proto":["https"],"Cookie":["REDACTED"],"Accept-Language":["en-US,en;q=0.5"],"X-Grafana-Device-Id":["4e5bf113b66df136308e462ae011c4a9"],"Sec-Fetch-Site":["same-origin"],"Referer":["https://grafana.pi.localdomain/grafana/d/gwEYOHWRk/graphs?orgId=1"],"Priority":["u=4"],"Sec-Fetch-Mode":["cors"],"X-Forwarded-For":["192.168.10.100"],"Dnt":["1"]},"tls":{"resumed":true,"version":772,"cipher_suite":4865,"proto":"http/1.1","server_name":"grafana.pi.localdomain"}},"headers":{"Content-Encoding":["gzip"],"Vary":["Accept-Encoding"],"X-Content-Type-Options":["nosniff"],"X-Xss-Protection":["1; mode=block"],"Content-Length":["145"],"Cache-Control":["no-store"],"X-Frame-Options":["deny"],"Date":["Mon, 07 Oct 2024 01:47:18 GMT"],"Content-Type":["application/json"]},"status":200}
Oct 07 12:17:18 caddy caddy[29173]: {"level":"debug","ts":1728265638.1049685,"logger":"http.handlers.reverse_proxy","msg":"selected upstream","dial":"192.168.10.7:3000","total_upstreams":1}
Oct 07 12:17:18 caddy caddy[29173]: {"level":"debug","ts":1728265638.1203332,"logger":"http.handlers.reverse_proxy","msg":"upstream roundtrip","upstream":"192.168.10.7:3000","duration":0.014568442,"request":{"remote_ip":"192.168.10.100","remote_port":"49633","client_ip":"192.168.10.100","proto":"HTTP/1.1","method":"GET","host":"grafana.pi.localdomain","uri":"/grafana/api/annotations?from=1720493239570&to=1728265639570&limit=100&matchAny=false&dashboardUID=gwEYOHWRk","headers":{"Sec-Fetch-Mode":["cors"],"X-Forwarded-Proto":["https"],"X-Grafana-Device-Id":["4e5bf113b66df136308e462ae011c4a9"],"X-Grafana-Org-Id":["1"],"Dnt":["1"],"Accept-Language":["en-US,en;q=0.5"],"User-Agent":["Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:130.0) Gecko/20100101 Firefox/130.0"],"Cookie":["REDACTED"],"Sec-Fetch-Dest":["empty"],"Sec-Fetch-Site":["same-origin"],"Accept":["application/json, text/plain, */*"],"Accept-Encoding":["gzip, deflate, br, zstd"],"X-Forwarded-For":["192.168.10.100"],"Referer":["https://grafana.pi.localdomain/grafana/d/gwEYOHWRk/graphs?orgId=1"],"Priority":["u=4"],"X-Forwarded-Host":["grafana.pi.localdomain"]},"tls":{"resumed":false,"version":772,"cipher_suite":4865,"proto":"http/1.1","server_name":"grafana.pi.localdomain"}},"headers":{"X-Frame-Options":["deny"],"Content-Type":["application/json"],"Vary":["Accept-Encoding"],"X-Content-Type-Options":["nosniff"],"Date":["Mon, 07 Oct 2024 01:47:18 GMT"],"Content-Length":["26"],"Cache-Control":["no-store"],"Content-Encoding":["gzip"],"X-Xss-Protection":["1; mode=block"]},"status":200}
Oct 07 12:17:18 caddy caddy[29173]: {"level":"debug","ts":1728265638.1337922,"logger":"http.handlers.reverse_proxy","msg":"selected upstream","dial":"192.168.10.7:3000","total_upstreams":1}
Oct 07 12:17:18 caddy caddy[29173]: {"level":"debug","ts":1728265638.1354368,"logger":"http.handlers.reverse_proxy","msg":"selected upstream","dial":"192.168.10.7:3000","total_upstreams":1}
Oct 07 12:17:18 caddy caddy[29173]: {"level":"debug","ts":1728265638.1366198,"logger":"http.handlers.reverse_proxy","msg":"selected upstream","dial":"192.168.10.7:3000","total_upstreams":1}
Oct 07 12:17:18 caddy caddy[29173]: {"level":"debug","ts":1728265638.191289,"logger":"http.handlers.reverse_proxy","msg":"upstream roundtrip","upstream":"192.168.10.7:3000","duration":0.053866316,"request":{"remote_ip":"192.168.10.100","remote_port":"49629","client_ip":"192.168.10.100","proto":"HTTP/1.1","method":"POST","host":"grafana.pi.localdomain","uri":"/grafana/api/ds/query?ds_type=mysql&requestId=Q101","headers":{"X-Forwarded-Host":["grafana.pi.localdomain"],"Content-Type":["application/json"],"Accept-Encoding":["gzip, deflate, br, zstd"],"X-Grafana-Device-Id":["4e5bf113b66df136308e462ae011c4a9"],"Dnt":["1"],"Sec-Fetch-Site":["same-origin"],"X-Forwarded-Proto":["https"],"X-Panel-Plugin-Id":["timeseries"],"Origin":["https://grafana.pi.localdomain"],"Accept-Language":["en-US,en;q=0.5"],"Sec-Fetch-Mode":["cors"],"X-Forwarded-For":["192.168.10.100"],"Cookie":["REDACTED"],"Referer":["https://grafana.pi.localdomain/grafana/d/gwEYOHWRk/graphs?orgId=1"],"X-Grafana-Org-Id":["1"],"Content-Length":["455"],"X-Panel-Id":["19"],"Sec-Fetch-Dest":["empty"],"User-Agent":["Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:130.0) Gecko/20100101 Firefox/130.0"],"X-Datasource-Uid":["CkrpG-f4k"],"X-Plugin-Id":["mysql"],"Accept":["application/json, text/plain, */*"],"X-Dashboard-Uid":["gwEYOHWRk"],"Priority":["u=4"]},"tls":{"resumed":true,"version":772,"cipher_suite":4865,"proto":"http/1.1","server_name":"grafana.pi.localdomain"}},"headers":{"Content-Type":["application/json"],"Vary":["Accept-Encoding"],"X-Content-Type-Options":["nosniff"],"Cache-Control":["no-store"],"X-Frame-Options":["deny"],"X-Xss-Protection":["1; mode=block"],"Date":["Mon, 07 Oct 2024 01:47:18 GMT"],"Content-Encoding":["gzip"]},"status":200}

3. Caddy version:

v2.8.4 h1:q3pe0wpBj1OcHFZ3n/1nl4V4bxBrYoSoab7rL9BMYNk=

4. How I installed and ran Caddy:

Install as per Debian Stable release method. Updated through APT.

a. System environment:

Running inside Debian 12.7 VM, x86_64, no docker

b. Command:

Systemd

[Unit]
Description=Caddy
Documentation=https://caddyserver.com/docs/
After=network.target network-online.target
Requires=network-online.target

[Service]
Type=notify
User=caddy
Group=caddy
ExecStart=/usr/bin/caddy run --environ --config /etc/caddy/Caddyfile
ExecReload=/usr/bin/caddy reload --config /etc/caddy/Caddyfile --force
TimeoutStopSec=5s
LimitNOFILE=1048576
PrivateTmp=true
ProtectSystem=full
AmbientCapabilities=CAP_NET_ADMIN CAP_NET_BIND_SERVICE

[Install]
WantedBy=multi-user.target

c. Service/unit/compose file:

PASTE OVER THIS, BETWEEN THE ``` LINES.
Please use the preview pane to ensure it looks nice.

d. My complete Caddy config:

I’ve tried various combinations of turning off HTTPS header protections, HTTP versions and encodings etc, so there are some commented out options. They did not improve anything.

# The Caddyfile is an easy way to configure your Caddy web server.
#
# Unless the file starts with a global options block, the first
# uncommented line is always the address of your site.
#
# To use your own domain name (with automatic HTTPS), first make
# sure your domain's A/AAAA DNS records are properly pointed to
# this machine's public IP, then replace ":80" below with your
# domain name.
{
        auto_https off
        admin off
}

(local_only) {
        @local_subnets {
                not remote_ip 192.168.101.0/24 192.168.10.0/23 192.168.1.0/24 192.168.100.7/32
        }
        respond @local_subnets 403
}

(iot_only) {
        @iot_subnets {
                not remote_ip 192.168.100.0/24
        }
        respond @iot_subnets 403
}

(logs) {
        log {
                output file /var/log/caddy/caddy.log {
                        roll_size 5MiB
                        roll_keep 5
                        roll_keep_for 168h
                }
                format console
        }
}

(http_header) {
        header {
                X-XSS-Protection "1; mode=block"
                X-Content-Type-Options "nosniff"
                Referrer-Policy "same-origin"
                -Server
                Content-Security-Policy "frame-ancestors pi.localdomain *.pi.localdomain"
                Permissions-Policy "geolocation=(self) , microphone=()"
        }
}

(http_header_generic) {
        header {
                X-XSS-Protection "1; mode=block"
                X-Content-Type-Options "nosniff"
                Referrer-Policy "same-origin"
        }
}

(https_header) {
        header {
                Strict-Transport-Security "max-age=31536000; includeSubdomains; preload"
                X-XSS-Protection "1; mode=block"
                X-Content-Type-Options "nosniff"
                Referrer-Policy "same-origin"
                -Server
                Content-Security-Policy "frame-ancestors pi.localdomain *.pi.localdomain"
                Permissions-Policy "geolocation=(self) , microphone=()"
        }
}

(https_header_generic) {
        header {
                Strict-Transport-Security "max-age=31536000; includeSubdomains; preload"
                X-XSS-Protection "1; mode=block"
                X-Content-Type-Options "nosniff"
                Referrer-Policy "same-origin"
        }
}

(exploits) {
        @exploits {
                path */actuator/*
                path */env
        }
        respond @exploits 403
}

## Internal Only Names ##
https://grafana.pi.localdomain {
        #import https_header
        tls /etc/caddy/self_signed/serverchaincert.pem /etc/caddy/self_signed/serverkey.pem {
                alpn http/1.1
        }
        reverse_proxy 192.168.10.7:3000 {
                transport http {
                        versions 1.1
                }
        }
        import logs
        #encode zstd gzip
        #import local_only
}

## Default block pages ##
http://:80 {
        import http_header_generic
        respond /* 403
        import logs
}

https://*:443 {
        import https_header_generic
        tls internal
        respond /* 403
        import logs
}

5. Links to relevant resources:

I’ve noticed in a browser, the main culprits seem to be the query requests, they seem to never complete.

image
image2556×422 50.2 KB

If I F5 to refresh the page, I get a bunch of NS_BINDING_ABORTED. Seems like the request is hanging?

image
image2556×421 65.3 KB

2

This doesn’t make sense: the * in a host matcher only matches a single label (parts separated by dots), so it would only match like localhost but never anything like example.com. Also, tls internal can’t do anything with this because it’s not an actual site address it can use to issue a cert for. You can just remove that block altogether, it doesn’t do anything useful for you.

NS_BINDING_ABORTED is just Firefox saying “I gave up and cancelled the request”. The reason for that

This logs seems to suggest it worked fine?

Maybe you have some browser extension blocking things (overzealous adblocker?) You can try in incognito mode without the extensions to see if it helps, or try in Chrome or w/e.

1 Like
3

Its for security; its a catch all for when someone hit’s caddy without knowing my DNS address (e.g. by IP only). Bots/someone probes my open ports, they only get a self-signed cert and caddy sends them back a 403. The 403’s are then dropped on the way out by my IDS; results in them getting a time-out rather than making a successful connection.

Yes, but the query will stay stuck for minutes, its not until after you refresh they get cancelled. The graph(s) as shown in the picture just spins saying its loading for as long as you let it. 80% of the time, every time you refresh a different graph breaks.

As best as I can tell, yes the caddy logs look OK, but the page is not. The firefox debugger, shows certain queries are never responded to, as shown in the second pick. The connection stays open, but no response is received.

The odd thing is, if I go into caddyfile and comment out the TLS part, and serve the page purely as http. It works great.

Just tested, chrome and edge, same problem. Both incognito, and neither have any addons installed.

4

99.9999% of bots fail the TLS handshake anyway because they don’t use a known domain in SNI. That site block does not help at all with that.

Hmm :thinking: I’m reaching out to someone who might be able to help, but it sounds like a problem with Grafana, I don’t see much evidence as-is of a problem with Caddy.

1 Like
5

@Callifo what does your Grafana config look like? Specifically, did you update the server.root_url setting?

It’s odd that only /grafana/api/ds/query is hanging…

6

I initially had it as stock no changes, but I did try changing it to this as well, neither improved the situation.

# The public facing domain name used to access grafana from a browser
domain = pi.localhost

# Redirect to correct domain if host header does not match domain
# Prevents DNS rebinding attacks
enforce_domain = false

# The full public facing url you use in browser, used for redirects and emails
# If you use reverse proxy and sub path specify full url (with sub path)
root_url = https://grafana.pi.localdomain/grafana

# Serve Grafana from subpath specified in `root_url` setting. By default it is set to `false` for compatibility reasons.
serve_from_sub_path = true

Also tried it with, and without, the /grafana/ sub-path.

7

@Callifo This may be a Grafana issue rather than a Caddy issue - I’d recommend posting to the Grafana community forums: https://community.grafana.com/

8

Ok will try there.

You don’t think the TLS proxing in caddy has anything to do with it, given serving Grafana via caddy as HTTP works?

9

Found it, was indeed not caddy, nor was it Grafana.

Suricata had a SQL injection signature that was dropping some of Grafana queries.

WEB_SERVER Possible SQL Injection SELECT CAST in HTTP Request Body rule. Disabled and working again now.

Thanks for the help

3 Likes
10

Good catch! Weird that it applied to HTTPS and not HTTP, but I’m glad you found the issue!

11

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.