How to make h2c to h2c proxy work

[k_o (Karsten)]

1. The problem I’m having:

I need to setup a h2c to h2c proxy with the goal to manipulate the requests and responses in a later step. No configuration I found works. What is the correct way to fulfill the first step of h2c ↔ h2c

2. Error messages and/or full log output:

127.0.0.200:7778

log {
    level debug
}

reverse_proxy {
    to h2c://127.0.0.201:7777
    transport http {
        versions h2c 2
    }
}

Here I get no errors and not any output, the proxy seems to do nothing and the app request targeting the proxies are failing.

sudo caddy reverse-proxy -v --access-log --from h2c://127.0.0.200:7778 --to h2c://127.0.0.201:7777

But I get:

Caddy proxying h2c://127.0.0.200 → 127.0.0.201:7777
2023/12/30 20:39:11.455 DEBUG http.stdlib http: TLS handshake error from 127.0.0.1:36540: tls: first record does not look like a TLS handshake

Somehow caddy tries to use https although this is not the case.

3. Caddy version:

v2.7.6 h1:w0NymbG2m9PcvKWsrXO6EEkY9Ru4FJK8uQbYcev1p3A=

4. How I installed and ran Caddy:

sudo apt-get install caddy

a. System environment:

Ubuntu 22.04

b. Command:

sudo caddy reverse-proxy -v --access-log --from h2c://127.0.0.200:7778 --to h2c://127.0.0.201:7777

or with the Daddyfile:

caddy run

d. My complete Caddy config:

127.0.0.200:7778

log {
    level debug
}

reverse_proxy {
    to h2c://127.0.0.201:7777
    transport http {
        versions h2c 2
    }
}

[Mohammed90 (Mohammed Al Sahaf)]

To enable h2c as the reverse_proxy transport, all you need is for the upstream address to have h2c://. You don’t need to manually configure the transport segment. More on this here:

reverse_proxy (Caddyfile directive) - Caddy Documentation

To enable h2c on the server listener side, you need customize the enabled protocols list in the global options to have h2c. Caddy defaults to h1 h2 h3 because h2c is not encrypted so it’s not part of the default setup. See here:

Global options (Caddyfile) - Caddy Documentation

[k_o (Karsten)]

Thanks for the quick reply. I’m now using this and get a different error:

{
    debug
    admin   off
    log default {
        level debug
        output stdout
        format console
    }
    servers :7778 {
        protocols h2c
    }
}

:7778 {
    bind 127.0.0.200
    reverse_proxy * h2c://127.0.0.201:7777 {
    }
    log default {
        level debug
        output stdout
        format console
    }
}

I get in caddy:

2023/12/31 01:05:25.180 DEBUG http.handlers.reverse_proxy selected upstream {“dial”: “127.0.0.201:7777”, “total_upstreams”: 1}
2023/12/31 01:05:25.180 DEBUG http.handlers.reverse_proxy upstream roundtrip {“upstream”: “127.0.0.201:7777”, “duration”: 0.000479471, “request”: {“remote_ip”: “127.0.0.1”, “remote_port”: “36684”, “client_ip”: “127.0.0.1”, “proto”: “HTTP/2.0”, “method”: “PRI”, “host”: “”, “uri”: ““, “headers”: {“X-Forwarded-Proto”: [“http”], “X-Forwarded-Host”: [”“], “User-Agent”: [”"], “X-Forwarded-For”: [“127.0.0.1”]}}, “error”: “http2: Transport: cannot retry err [stream error: stream ID 3; PROTOCOL_ERROR; received from peer] after Request.Body was written; define Request.GetBody to avoid this error”}
2023/12/31 01:05:25.181 ERROR http.log.error.default http2: Transport: cannot retry err [stream error: stream ID 3; PROTOCOL_ERROR; received from peer] after Request.Body was written; define Request.GetBody to avoid this error {“request”: {“remote_ip”: “127.0.0.1”, “remote_port”: “36684”, “client_ip”: “127.0.0.1”, “proto”: “HTTP/2.0”, “method”: “PRI”, “host”: “”, “uri”: "”, “headers”: {}}, “duration”: 0.000594763, “status”: 502, “err_id”: “u7td0w0b4”, “err_trace”: “reverseproxy.statusError (reverseproxy.go:1267)”}
2023/12/31 01:05:25.181 ERROR http.log.access.default handled request {“request”: {“remote_ip”: “127.0.0.1”, “remote_port”: “36684”, “client_ip”: “127.0.0.1”, “proto”: “HTTP/2.0”, “method”: “PRI”, “host”: “”, “uri”: “*”, “headers”: {}}, “bytes_read”: 0, “user_id”: “”, “duration”: 0.000594763, “size”: 0, “status”: 502, “resp_headers”: {“Server”: [“Caddy”]}}

I get in my app:

ERROR: [127.0.0.1]:55972 invalid frame (-532:Violation in HTTP messaging rule)

[francislavoie]

@k_o

    servers :7778 {
        protocols h2c
    }
}

:7778 {
    bind 127.0.0.200

Since you’re using bind, you need to do servers 127.0.0.200:7778 to match the listener address.

Alternatively you can omit the listener address from the global option which will apply it to all servers (which is just the one in your case).

Also, if you set protocols to h2c, this also disables h1, h2 and h3 I think, which may be undesirable.

@k_o

    log default {
        level debug
        output stdout
        format console
    }

You can simplify this to just log (no options) which has the same effect. There’s no debug level access logs.

@k_o

    reverse_proxy * h2c://127.0.0.201:7777 {
    }

You can remove the { } braces here, they don’t do anything for you.

[k_o (Karsten)]

Awesome! This works. Fastest community on this planet. Thanks Mohammed and Francis.

Here my working config as reference. I had to add also the h1 protocol

{
    debug
    admin   off
    servers 127.0.0.200:7778 {
        protocols h1 h2c
    }
}

:7778 {
    bind 127.0.0.200
    reverse_proxy * h2c://127.0.0.201:7777
    log default
}