Referer Header missing using Caddy with Sentry

nalo · February 21, 2024, 10:24am

1. The problem I’m having:

I recently installed Sentry Self-Hosted on my server. The installation script create every Docker container at once, ready to be used.
In order to access it, the documentation indicates to install a reverse proxy in front of it.

Using the recommanded configuration for NGINX, and a Certbot configuration, it worked. I was able to access the website through https//sentry․example․com, and the workers installed using python module Sentry-SDK on my websites were able to communicate with Sentry.

I then discovered the power of Caddy, simplifying a lot my configurations to handle certificates and routes automatically.

I created the docker compose file that will power Caddy, and configured the Caddyfile as the Sentry documentation indicates (see 4.d).

When launched up, I was still able to access the website, but the workers were not able to communicate with Sentry anymore.

2. Error messages and/or full log output:

Caddy logs (debug mode), from docker compose -f caddy.yaml up (attached):

caddy  | {"level":"debug","ts":1708445472.6647162,"logger":"http.handlers.reverse_proxy","msg":"upstream roundtrip","upstream":"sentry-self-hosted-web-1:9000","duration":0.095230196,"request":{"remote_ip":"185.31.40.179","remote_port":"43910","client_ip":"185.31.40.179","proto":"HTTP/1.1","method":"POST","host":"sentry-self-hosted-web-1:9000","uri":"/api/6/envelope/","headers":{"User-Agent":["sentry.python/1.40.4"],"X-Forwarded-Host":["sentry․example․com"],"Content-Type":["application/x-sentry-envelope"],"X-Sentry-Auth":["Sentry sentry_key=REDACTED, sentry_version=7, sentry_client=sentry.python/1.40.4"],"X-Forwarded-Proto":["https"],"Content-Length":["4465"],"Accept-Encoding":["identity"],"Content-Encoding":["gzip"],"X-Forwarded-For":["185.31.40.179"]},"tls":{"resumed":false,"version":772,"cipher_suite":4865,"proto":"http/1.1","server_name":"sentry․example․com"}},"headers":{"Content-Language":["en"],"Content-Length":["8519"],"X-Frame-Options":["deny"],"X-Content-Type-Options":["nosniff"],"X-Xss-Protection":["1; mode=block"],"Content-Security-Policy-Report-Only":["default-src 'none'; font-src 'self' data:; media-src *; script-src 'self' 'unsafe-inline' 'report-sample' 'nonce-BDKIaMz6vUYphEuFiWO7dg=='; frame-ancestors 'none'; object-src 'none'; img-src blob: data: *; base-uri 'none'; connect-src 'self' *․algolia․net *․algolianet․com *․algolia․io; style-src 'unsafe-inline' *"],"Content-Type":["text/html"],"Vary":["Accept-Language, Cookie"]},"status":403}

Django’s Sentry logs, from docker compose logs web -f :

web-1  | 16:10:50 [WARNING] django.security.csrf: Forbidden (Referer checking failed - no Referer.): /api/6/envelope/ (status_code=403 request=<WSGIRequest: POST '/api/6/envelope/'>)

3. Caddy version:

From docker compose -f caddy.yaml exec caddy caddy version:

v2.7.6 h1:w0NymbG2m9PcvKWsrXO6EEkY9Ru4FJK8uQbYcev1p3A=

4. How I installed and ran Caddy:

a. System environment:

OS: Debian GNU/Linux 12 (bookworm)
CPU: Intel Xeon CPU E5-1620 v2 @ 3.70GHz x86_64
Docker: 25.0.3, build 4debf41
Docker compose: version v2.24.5

b. Command:

Running Sentry :

cd sentry-self-host
docker compose up -d

Running Caddy (after Sentry, so that the network exists):

docker compose -f caddy.yaml up -d

c. Service/unit/compose file:

caddy.yaml

version: "3.8"

services:
  caddy:
    container_name: caddy
    image: caddy:alpine
    restart: unless-stopped
    volumes:
      - ./Caddyfile:/etc/caddy/Caddyfile
      - caddy_data:/data
      - caddy_config:/config
    ports:
      - 80:80
      - 443:443
    networks:
      - caddy
      - sentry-self-hosted_default

volumes:
  caddy_data:
  caddy_config:

networks:
  caddy:
    name: caddy
  sentry-self-hosted_default:
    name: sentry-self-hosted_default
    driver: bridge
    external: true

You can see here that I put Caddy in the same network that Sentry container created, in order for Caddy to be able to communicate.
I couldn’t get Sentry to use Caddy’s network (it broke Sentry), so I was forced to configure Caddy to use Sentry’s network.

Sentry configuration

sentry/config.yml:

# ...
system.url-prefix: 'https://sentry․example․com
system.internal-url-prefix: 'http://web:9000'
# ...

sentry/sentry.conf.py:

# ...

###########
# SSL/TLS #
###########

SECURE_PROXY_SSL_HEADER = ('HTTP_X_FORWARDED_PROTO', 'https')
USE_X_FORWARDED_HOST = True
SESSION_COOKIE_SECURE = True
CSRF_COOKIE_SECURE = True
SOCIAL_AUTH_REDIRECT_IS_HTTPS = True

# ...

d. My complete Caddy config:

As from the Sentry documentation:

sentry.example.com {
  reverse_proxy sentry-self-hosted-web-1:9000 {
    health_uri /_health/
    health_status 2xx
    header_up Host {upstream_hostport}
  }
 
  tls contact@example.com
 
  header {
    # Delete "Server" header
    -Server
  }
}

5. What I tried:

a. Referer header configuration

The error clearly indicates no Referer. I then tried to add the following instruction in the reverse_proxy section of my Caddyfile:

header_up Referer {header.Referer}

After that, the Referer Header was set up, but to an empty string, as if the header was not present before reaching Caddy, giving the following Django error: Referer is malformed

web-1  | 16:12:02 [WARNING] django.security.csrf: Forbidden (Referer checking failed - Referer is malformed.): /api/6/envelope/ (status_code=403 request=<WSGIRequest: POST '/api/6/envelope/'>)

b. No Sentry SSL configuration

I went up to my sentry/sentry.conf.py file to remove every SSL/TLS configuration, but nothing changed, it was as if nothing was edited (of course I restarted Sentry containers in order to apply the changes)

c. Django reqanalysis

Something that caught my eyes was that requests at route /api/<id>/envelope where failing, but the ones at /api/0/relays where working.
Looking at the Django code, it appears that the Referer header is checked only if the request is considered as secured (HTTPS).
And in fact, every request I make to Sentry has its request.is_secure() to False, except the ones coming from the Sentry workers.

Maybe that’s why it works using NGINX, but not with caddy: NGINX passes every request as HTTP (not secure). But Caddy documentation says the same:

transport defines how to communicate with the backend. Default is http.

6. Links to relevant resources:

(remove space after https://)
https:// caddy.community/t/django-csrf-issues-with-reverse-proxy/16957

Note:

I know I shouldn’t redact the domains from the logs and configuration, but as this page will be indexed by search engines and is publicly accessible, I don’t want my domain to be find here. That’s why I replaced it with sentry․example․com everywhere, so that the logic is still there.

Also, I had to remove a lot of links that could have help you, but I’m limited to 4 as a new user…

Thanks in advance for any help you can bring me!

francislavoie · February 21, 2024, 11:34am

This is a CORS error, it means that whatever client is connecting to your server isn’t sending the Referer header. It doesn’t mean that Caddy isn’t proxying it.

Caddy passes through all headers transparently by default, so it’s wrong to add a header_up line, since the original request didn’t have it in the first place.

Remove this, you only want this if you’re proxying to an HTTPS upstream, otherwise you’re overriding the Host with sentry-self-hosted-web-1 which isn’t right.

There’s no reason to do this, it doesn’t have any benefit. It doesn’t hide anything sensitive.

From this config, it looks like it proxies API requests to the relay instead of web.

		location ~ ^/api/[1-9]\d*/ {
			proxy_pass http://relay;
		}

You should probably try and do the same with Caddy. You can add this to your config:

@relay `path_regexp('^/api/[1-9]\d*') || path('/api/store/*')`
reverse_proxy @relay sentry-self-hosted-relay-1:3000

I’m guessing the relay doesn’t check for CORS so this might solve it.

If not, then you’ll need to get help from the Sentry community. I don’t know the recommended setup.

nalo · February 21, 2024, 2:02pm

Thanks a lot!

The fix was actually to implement the relay routes.
The fact is that I tried doing it before, and it wasn’t working. But know that I see your solution, it’s because I wasn’t doing regex right.

And in fact, your solution is giving me a lot of issues at Caddy startup:

caddy  | Error: loading initial config: loading new config: loading http app module: provision http: server srv0: setting up route handlers: route 0: loading handler modules: position 0: loading module 'subroute': provision http.handlers.subroute: setting up subroutes: route 1: loading matcher modules: module name 'expression': provision http.matchers.expression: compiling CEL program: ERROR: <input>:1:13: Syntax error: token recognition error at: ''^/api/[1-9]\d'
caddy  |  | path_regexp('^/api/[1-9]\d*/*') || path_regexp('/api/store/*')
caddy  |  | ............^
caddy  | ERROR: <input>:1:27: Syntax error: mismatched input '*' expecting {'[', '{', '(', ')', '.', '-', '!', 'true', 'false', 'null', NUM_FLOAT, NUM_INT, NUM_UINT, STRING, BYTES, IDENTIFIER}
caddy  |  | path_regexp('^/api/[1-9]\d*/*') || path_regexp('/api/store/*')
caddy  |  | ..........................^
caddy  | ERROR: <input>:1:28: Syntax error: mismatched input '/' expecting {'[', '{', '(', '.', '-', '!', 'true', 'false', 'null', NUM_FLOAT, NUM_INT, NUM_UINT, STRING, BYTES, IDENTIFIER}
caddy  |  | path_regexp('^/api/[1-9]\d*/*') || path_regexp('/api/store/*')
caddy  |  | ...........................^
caddy  | ERROR: <input>:1:29: Syntax error: extraneous input '*' expecting {'[', '{', '(', '.', '-', '!', 'true', 'false', 'null', NUM_FLOAT, NUM_INT, NUM_UINT, STRING, BYTES, IDENTIFIER}
caddy  |  | path_regexp('^/api/[1-9]\d*/*') || path_regexp('/api/store/*')
caddy  |  | ............................^
caddy  | ERROR: <input>:1:60: Syntax error: mismatched input '*' expecting {'[', '{', '(', '.', '-', '!', 'true', 'false', 'null', NUM_FLOAT, NUM_INT, NUM_UINT, STRING, BYTES, IDENTIFIER}
caddy  |  | path_regexp('^/api/[1-9]\d*/*') || path_regexp('/api/store/*')
caddy  |  | ...........................................................^
caddy  | ERROR: <input>:1:61: Syntax error: token recognition error at: '')'
caddy  |  | path_regexp('^/api/[1-9]\d*/*') || path_regexp('/api/store/*')
caddy  |  | ............................................................^
caddy  | ERROR: <input>:1:63: Syntax error: mismatched input '<EOF>' expecting {'[', '{', '(', '.', '-', '!', 'true', 'false', 'null', NUM_FLOAT, NUM_INT, NUM_UINT, STRING, BYTES, IDENTIFIER}
caddy  |  | path_regexp('^/api/[1-9]\d*/*') || path_regexp('/api/store/*')
caddy  |  | ..............................................................^
caddy exited with code 0

I don’t get why, because the synthax is good according to the doc about it.

I found a workaround by doing the following:

@api path_regexp ^/api/[1-9]\d*/*
@store path_regexp ^/api/store/*  
reverse_proxy @api sentry-self-hosted-relay-1:3000
reverse_proxy @store srentry-self-hosted-relay-1:3000

Thanks a lot for the help! It is now perfectly working

francislavoie · February 21, 2024, 7:01pm

Ah oops, the problem is CEL needs us to escape the \ so that it can be passed through correctly to the matcher. So this works (\d → \\d):

@relay `path_regexp('^/api/[1-9]\\d*') || path('/api/store/*')`
reverse_proxy @relay sentry-self-hosted-relay-1:3000

This is wrong, actually. Your matcher is matching multiple / at the end of the string, not any character. You need to do .* to match any character, not just *

system · March 22, 2024, 7:01pm

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.