"malformed HTTP response" when proxying WebSockets to Heroku

Help
1

1. The problem I’m having:

I am trying to proxy our site through Caddy to Heroku. Long-term we want to migrate to other provider.

While HTTP(S) proxying works fine, I have issues with WSS/WebSockets.

When we are running

wscat -c wss://our.staging.host/cable

It hangs for slightly under 6 minutes and in Caddy logs I can see logs pasted in next section.

2. Error messages and/or full log output:

{
    "duration": 352.062551103,
    "err_id": "x7zksa8bt",
    "err_trace": "reverseproxy.statusError (reverseproxy.go:1299)",
    "level": "error",
    "logger": "http.log.error",
    "msg": "net/http: HTTP/1.x transport connection broken: malformed HTTP response \"\\x15\\x00\\x00\\x00\\x02\\x01\\x00\"",
    "request":
    {
        "headers":
        {
            ...
            "Connection":
            [
                "Upgrade"
            ],
            ...
            "Upgrade":
            [
                "websocket"
            ],
            ...
        },
        "host": "our.staging.host",
        "method": "GET",
        "proto": "HTTP/1.1",
        "remote_ip": "10.31.12.1",
        "remote_port": "58020",
        "tls":
        {
            "cipher_suite": 4865,
            "proto": "",
            "resumed": false,
            "server_name": "our.staging.host",
            "version": 772
        },
        "uri": "/cable"
    },
    "status": 502,
    "ts": 1686306304.828851
}

and

{
    "duration": 352.062551103,
    "level": "error",
    "logger": "http.log.access",
    "msg": "handled request",
    "request":
    {
        "headers":
        {
          ...
        },
        "host": "our.staging.host",
        "method": "GET",
        "proto": "HTTP/1.1",
        "remote_ip": "10.31.12.1",
        "remote_port": "58020",
        "tls":
        {
            "cipher_suite": 4865,
            "proto": "",
            "resumed": false,
            "server_name": "our.staging.host",
            "version": 772
        },
        "uri": "/cable"
    },
    "resp_headers":
    {
        "Server":
        [
            "Caddy"
        ]
    },
    "size": 0,
    "status": 502,
    "ts": 1686306304.82889,
    "user_id": ""
}

3. Caddy version:

2.6.4

4. How I installed and ran Caddy:

a. System environment:

EKS 1.26, ARM, custom Helm chart

b. Command:

/usr/bin/caddy run --config /etc/caddy/config/content --adapter caddyfile

c. Service/unit/compose file:

not relevant I guess(?)

d. My complete Caddy config:

(server_defaults) {
  metrics

  protocols h1 h2
}

{
  acme_ca https://acme.zerossl.com/v2/DV90
  acme_eab {
    key_id  {$EAB_KEY_ID}
    mac_key {$EAB_HMAC_KEY}
  }

  admin 0.0.0.0:2019

  grace_period 10s

  log {
    output stdout
    format filter {
      wrap json
    }
  }

  on_demand_tls {
    ask http://caddy-ask-api.default/internal/account_domain_validation/{$API_TOKEN}
    burst    10
    interval 1s
  }

  servers {
    import server_defaults
  }

  servers :80 {
    name http

    import server_defaults
  }

  servers :443 {
    name https

    import server_defaults
  }

  shutdown_delay 10s

  storage dynamodb caddy-staging-table
}

# We use it only for probes, we do not log anything here
internal:80 {
  @goingDown vars {http.shutting_down} true
  respond @goingDown "Bye-bye in {http.time_until_shutdown}" 503
  respond "OK!"
}

:80, :443 {
  log

  @websockets {
    header Connection *Upgrade*
    header Upgrade    websocket
  }

  reverse_proxy @websockets wss://our-staging.herokuapp.com:443 {
    header_up Host {upstream_hostport}
  }

  reverse_proxy https://our-staging.herokuapp.com {
    header_up Host {upstream_hostport}
  }

  tls "caddy@our.domain.io" {
    on_demand
  }
}

5. Links to relevant resources:

It behaves the same if I drop wss:// from the reverse proxy. When I switch to https://, I am getting 404 from Heroku.

2

wss:// is not a valid scheme for Caddy’s proxy. See the docs which explain: reverse_proxy (Caddyfile directive) — Caddy Documentation. Because it’s not valid, it’s the same as the default which is HTTP.

In fact, you don’t need that reverse_proxy at all, it’s a duplicate of your other one with https://. You don’t need the websockets matcher in this case because you’re proxying both websocket and HTTP traffic to the same place. A matcher is only needed if you need to split the traffic (i.e. websockets proxied to one place, HTTP to another).

I can’t answer why the 404 happens, that’s a problem with your upstream that you need to figure out.

3

Thanks for clarification! I thought it is an upstream issue, but wasn’t sure. Heroku support is sometimes hard to talk to.

4

If someone comes here with similar issue. In our case, it was due to the Rails’ request_forgery_protection implementation. It only checks Host header, where it should consider X-Forwarded-* when available. At least other web servers/frameworks do it.

2 Likes
5

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.