Headers are not sent for graphql websocket connections

1. Caddy version (v2):

2. How I run Caddy:

docker-compose start caddy

a. System environment:

Ubuntu 20.04

b. Command:

docker-compose restart caddy

c. Service/unit/compose file:

version: '3.6'
services:
  postgres:
    image: postgres:12
    restart: always
    volumes:
    - db_data:/var/lib/postgresql/data
    environment:
      POSTGRES_PASSWORD: *****
  graphql-engine:
    image: hasura/graphql-engine:v2.9.0-beta.1
    depends_on:
    - "postgres"
    restart: always
    environment:
      ## postgres database to store Hasura metadata
      HASURA_GRAPHQL_METADATA_DATABASE_URL: postgres://postgres:*****@postgres:5432/postgres
      ## enable the console served by server
      HASURA_GRAPHQL_ENABLE_CONSOLE: "true" # set "false" to disable console
      ## enable debugging mode. It is recommended to disable this in production
      HASURA_GRAPHQL_DEV_MODE: "true"
      ## uncomment next line to set an admin secret
      HASURA_GRAPHQL_ADMIN_SECRET: ********************
      HASURA_GRAPHQL_LIVE_QUERIES_MULTIPLEXED_REFETCH_INTERVAL: 50
      HASURA_GRAPHQL_STREAMING_QUERIES_MULTIPLEXED_REFETCH_INTERVAL: 50
      HASURA_GRAPHQL_UNAUTHORIZED_ROLE: public
      HASURA_GRAPHQL_EXPERIMENTAL_FEATURES: streaming_subscriptions
    command:
    - graphql-engine
    - serve
  caddy:
    image: caddy/caddy
    depends_on:
    - "graphql-engine"
    restart: always
    ports:
    - "80:80"
    - "443:443"
    volumes:
    - ./Caddyfile:/etc/caddy/Caddyfile
    - caddy_certs:/root/.caddy
volumes:
  db_data:
  caddy_certs:

d. My complete Caddyfile or JSON config:

(common) {
    header {
        -Server
    }
}
https://mywebsite.com {
    reverse_proxy * graphql-engine:8080 {
        header_up X-Hasura-User-Id {http.request.header.Access-Token}
    }
    import common
}

3. The problem I’m having:

Hello everyone, I have a GraphQL engine setup using Hasura docker image and it includes the caddy configuration. I manipulated the reverse proxy header so I can pass the Access-Token to the X-Hasura-User-Id.
Caddyfile works well and sending the headers with GraphQL query (get request through HTTPS), however, the {http.request.header.*} including Access-Token are not being passed when using graphQL subscription (websocket WSS connection).
WSS works perfectly though but headers are not being sent.

4. Error messages and/or full log output:

5. What I already tried:

  • @websockets matchers
  • request_header before reverse_proxy

6. Links to relevant resources:

What’s your evidence that it isn’t being sent? What’s in your logs? Turn on the debug global option to see the proxy debug logs, which will show you the headers being sent upstream.

{http.request.header.Access-Token} is returning blank when using GQL subscriptions (wss), and is returning the Access-Token when using query (https).

Unfortunately, for some reason, the debug is not working for me because I couldn’t access the docker container.

> docker exec -it 08d37b10f6d2 bash
OCI runtime exec failed: exec failed: container_linux.go:346: starting container process caused "exec: \"bash\": executable file not found in $PATH": unknown

Basically the reverse_proxy works on both connections https and wss. The only issue is that the headers are accessible through {header.*} for https but not wss

header_up X-Hasura-User-Id "Some hardcoded token" :green_circle: works with both https and wss
header_up X-Hasura-User-Id {http.request.header.Access-Token} :red_circle: works with https but not wss

No, you add this at the top of your Caddyfile:

{
	debug
}

That’s the debug global option.

Then, check your container’s logs and see what it emitted.

No errors on wss subscription request but the output as follows with empty "X-Hasura-User-Id":[""] header value

{"level":"debug","ts":1657579042.3522673,"logger":"http.handlers.reverse_proxy","msg":"upstream roundtrip","upstream":"graphql-engine:8080","request":{"method":"GET","uri":"/v1/graphql","proto":"HTTP/1.1","remote_addr":"*******:51840","host":"*******.com","headers":{"Origin":["https://*******.com"],"Accept-Language":["en-US,en;q=0.9,ar;q=0.8,de;q=0.7"],"X-Hasura-User-Id":[""],"Cache-Control":["no-cache"],"Cookie":["_ga=*******; _gid=*******"],"Sec-Websocket-Key":["*******"],"Sec-Websocket-Extensions":["permessage-deflate; client_max_window_bits"],"Upgrade":["websocket"],"User-Agent":["Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/102.0.5005.61 Safari/537.36"],"Connection":["Upgrade"],"Accept-Encoding":["gzip, deflate, br"],"Pragma":["no-cache"],"Sec-Websocket-Version":["13"],"Sec-Websocket-Protocol":["graphql-ws"],"X-Forwarded-For":["*******"]},"tls":{"resumed":true,"version":772,"ciphersuite":4865,"proto":"http/1.1","proto_mutual":true,"server_name":"*******.com"}},"headers":{"Sec-Websocket-Protocol":["graphql-ws"],"Upgrade":["websocket"],"Connection":["Upgrade"],"Sec-Websocket-Accept":["rrO/c7LUqOuoJbhx3QsrbglnmE4="]},"duration":0.001453859,"status":101}

Here is the debug output with GQL query https request

{"level":"debug","ts":1657579313.7087102,"logger":"http.handlers.reverse_proxy","msg":"upstream roundtrip","upstream":"graphql-engine:8080","request":{"method":"POST","uri":"/v1/graphql","proto":"HTTP/2.0","remote_addr":"******:51719","host":"******.com","headers":{"Sec-Fetch-Dest":["empty"],"Accept-Encoding":["gzip, deflate, br"],"Sec-Fetch-Site":["same-origin"],"Sec-Fetch-Mode":["cors"],"Accept-Language":["en-US,en;q=0.9,ar;q=0.8,de;q=0.7"],"Origin":["https://******.com"],"User-Agent":["Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/102.0.5005.61 Safari/537.36"],"Referer":["https://******.com/console"],"X-Hasura-User-Id":["w6GWo******E9sozq******pXamD******Mnb2I"],"Sec-Ch-Ua":["\" Not A;Brand\";v=\"99\", \"Chromium\";v=\"102\", \"Google Chrome\";v=\"102\""],"Accept":["*/*"],"Cookie":["_ga=******; _gid=******"],"X-Forwarded-For":["*****"],"Sec-Ch-Ua-Mobile":["?0"],"Sec-Ch-Ua-Platform":["\"macOS\""],"Content-Type":["text/plain;charset=UTF-8"],"Content-Length":["179"],"Access-Token":["w6GWo******E9sozq******pXamD******Mnb2I"]},"tls":{"resumed":true,"version":772,"ciphersuite":4865,"proto":"h2","proto_mutual":true,"server_name":"******.com"}},"headers":{"Content-Type":["application/json; charset=utf-8"],"Access-Control-Allow-Origin":["https://******.com"],"Access-Control-Allow-Credentials":["true"],"Access-Control-Allow-Methods":["GET,POST,PUT,PATCH,DELETE,OPTIONS"],"X-Request-Id":["3fc00c3f-7aee-4bf9-a720-e608924a2ead"],"Content-Encoding":["gzip"],"Date":["Mon, 11 Jul 2022 22:41:53 GMT"],"Server":["Warp/3.3.19"]},"duration":0.00419763,"status":200}

It looks like the websocket request just doesn’t have the Access-Token header in it.

That’s not a Caddy problem, you need to figure out why your client isn’t passing along that header.

The GraphQL request looks exactly the same for both query and subscription (you just swipe the type). No specific client and that can be produced by any graphql explorer or simple curl/console.

I can’t remember my nginx setting where I had the exact setup and it was working, but not with Caddy.

Well, clearly the request doesn’t have Access-Token in it.

Look at the log. If Caddy doesn’t see Access-Token, it can’t copy it to another header with header_up.

I don’t know what else to tell you. I don’t know how your client is making the requests, but Caddy seems to working exactly as intended.

Makes sense. I will investigate further. Thanks for the prompt response!

The reason I suspect caddy is because the header manipulation header_down or header (-Server) is not taking effect on the wss request. Server: Caddy always exist.

Also, please do not use the caddy/caddy Docker Hub image but instead caddy.

The caddy/caddy is the CI target and specifically the :latest tag is very much outdated:

❯ docker run -it --rm caddy/caddy caddy version
v2.0.0-rc.3 h1:z2H/QnaRscip6aZJxwTbghu3zhC88Vo8l/K57WUce4Q=

# the proper official image on the other hand
❯ docker run -it --rm caddy caddy version
v2.5.1 h1:bAWwslD1jNeCzDa+jDCNwb8M3UJ2tPa8UZFFzPVmGKs=
3 Likes

This topic was automatically closed after 30 days. New replies are no longer allowed.