Proxying streaming GRPC with Caddy 2

Help
1

1. Caddy version (caddy version):

v2.3.0 h1:fnrqJLa3G5vfxcxmOH/+kJOcunPLhSBnjgIvjXV/QTA=

2. How I run Caddy:

macOS launchctl:

<!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN"
	"http://www.apple.com/DTDs/PropertyList-1.0.dtd" >
<plist version='1.0'>
  <dict>
    <key>Label</key>
    <string>com.caddyserver.caddy</string>
    <key>ProgramArguments</key>
    <array>
      <string>/usr/bin/env</string>
      <string>XDG_CONFIG_HOME=/opt/local/etc/caddy</string>
      <string>XDG_DATA_HOME=/opt/local/etc/caddy</string>
      <string>/opt/local/sbin/caddy</string>
      <string>run</string>
      <string>-config</string>
      <string>/opt/local/etc/caddy/caddy.conf</string>
      <string>-adapter</string>
      <string>caddyfile</string>
    </array>
    <key>KeepAlive</key>
    <true/>
  </dict>
</plist>

a. System environment:

macOS 10.14.6 (18G7016)

b. Command:

N/A

c. Service/unit/compose file:

N/A

d. My complete Caddyfile or JSON config:

{
    servers {
	protocol {
	    experimental_http3
	}
    }
}

www.timeheart.net, timeheart.net, ipv6.timeheart.net {
    root * /Library/WebServer/Documents
    file_server browse

    reverse_proxy {
	to grpc://localhost:50051
	transport http {
	    versions h2c
	}
    }

    header / Strict-Transport-Security "max-age=31536000;"

    log {
	output file /opt/local/var/log/caddy/main.log
    }
}

3. The problem I’m having:

I’m experimenting with using Caddy as a reverse proxy to accept secure GRPC requests and forward them in the clear to a GRPC server on a different port on localhost. I have this working just fine for simple GRPC requests. However, I noticed something that I wanted to report.

It appears that Caddy can’t properly handle streaming requests when the client tries to read any of the response before finishing sending the request (which is common when using streaming RPC). Related to this, it appears it doesn’t handle a client trying to read the server metadata (which is communicated in h2 response headers) before sending its request. I believe this latter issue could apply to both streaming and unary requests.

This is not unexpected, as web servers typically forward the entirety of the client request to the server before beginning to forward any of the server’s response back to the client. However, for GRPC specifically, that’s not good enough to handle all cases.

Since Caddy seems to be explicitly supporting the “grpc://” scheme in its reverse_proxy, are there any plans to support this kind of interleaved streaming request/response or early request for metadata?

4. Error messages and/or full log output:

When trying to read response data before sending the complete request, the grpc client simply hangs waiting for response data from CaddyServer which never arrives.

5. What I already tried:

I confirmed that it works perfectly when I send the entire request before reading any response data. However, if I try to read either the server metadata or server response data prior to marking my request complete, it hangs.

6. Links to relevant resources:

2

Where are you seeing support for this? I don’t think reverse_proxy directly supports grpc://. I don’t see anything in the codebase for this. The protocol matcher does support grpc, but that’s totally separate from reverse_proxy.

You probably want h2c://, which is supported.

3

I got that from Caddy 2 with GRPC (specifically tweeniev2’s response), and as I mentioned it works fine when I send the complete request in the client before trying to access anything in the response.

Changing to “h2c://” results in identical behavior.

4

Here are some clients and a server I built to illustrate the problem, using the Python asyncio grpc interface:

echo.proto:

syntax = "proto3";

message EchoMessage {
  string message = 1;
}

service Echo {
  rpc Echo(stream EchoMessage) returns (stream EchoMessage);
}

Non-secure server:

import asyncio
import logging

import grpc
import grpc.aio

from echo_pb2_grpc import EchoServicer, add_EchoServicer_to_server

logging.basicConfig(level='DEBUG')

class EchoService(EchoServicer):
    async def Echo(self, request_iter, context):
        requests = []

        print('Client metadata:')
        for key, value in context.invocation_metadata():
            print(f'  {key}: {value}')

        await context.send_initial_metadata(context.invocation_metadata())

        async for request in request_iter:
            print(f'Echoing "{request.message}"')
            await context.write(request)

        context.set_trailing_metadata((('foo', 'yyy'), ('bar', 'zzz')))

        print()

async def main():
    server = grpc.aio.server()

    add_EchoServicer_to_server(EchoService(), server)

    server.add_insecure_port('localhost:50051')
    await server.start()
    await server.wait_for_termination()

if __name__ == '__main__':
    asyncio.run(main())

Client going directly to server with interleaving of request and response (works fine):

import asyncio

import grpc
import grpc.aio
import logging

from echo_pb2 import EchoMessage
from echo_pb2_grpc import EchoStub

logging.basicConfig(level='DEBUG')

async def main():
    async with grpc.aio.insecure_channel('localhost:50051') as channel:
        stub = EchoStub(channel)

        stream = stub.Echo(metadata=(('foo', 'abc'), ('bar', 'def')))

        print('Initial metadata:')
        for key, value in await stream.initial_metadata():
            print(f'  {key}: {value}')

        for data in ('aaa', 'bbb', 'ccc'):
            await stream.write(EchoMessage(message=data))
            response = await stream.read()
            print(f'Received "{response.message}"')

        await stream.done_writing()

        print('Trailing metadata:')
        for key, value in await stream.trailing_metadata():
            print(f'  {key}: {value}')

if __name__ == '__main__':
    asyncio.run(main())

Same client changed to open secure channel to Caddy as a reverse proxy (doesn’t work):

import asyncio

import grpc
import grpc.aio
import logging

from echo_pb2 import EchoMessage
from echo_pb2_grpc import EchoStub

logging.basicConfig(level='DEBUG')

async def main():
    with open('/opt/local/share/curl/curl-ca-bundle.crt', 'rb') as f:
        trusted_roots = f.read()

    credentials = grpc.ssl_channel_credentials(root_certificates=trusted_roots)
    async with grpc.aio.secure_channel('www.timeheart.net:443',
                                       credentials) as channel:
        stub = EchoStub(channel)

        stream = stub.Echo(metadata=(('foo', 'abc'), ('bar', 'def')))

        print('Initial metadata:')
        for key, value in await stream.initial_metadata():
            print(f'  {key}: {value}')

        for data in ('aaa', 'bbb', 'ccc'):
            await stream.write(EchoMessage(message=data))
            response = await stream.read()
            print(f'Received "{response.message}"')

        await stream.done_writing()

        print('Trailing metadata:')
        for key, value in await stream.trailing_metadata():
            print(f'  {key}: {value}')

if __name__ == '__main__':
    asyncio.run(main())

Client modified to not interleave request & response going through Caddy (works fine):

import asyncio

import grpc
import grpc.aio
import logging

from echo_pb2 import EchoMessage
from echo_pb2_grpc import EchoStub

logging.basicConfig(level='DEBUG')

async def main():
    with open('/opt/local/share/curl/curl-ca-bundle.crt', 'rb') as f:
        trusted_roots = f.read()

    credentials = grpc.ssl_channel_credentials(root_certificates=trusted_roots)
    async with grpc.aio.secure_channel('www.timeheart.net:443',
                                       credentials) as channel:
        stub = EchoStub(channel)

        stream = stub.Echo(metadata=(('foo', 'abc'), ('bar', 'def')))

        for data in ('aaa', 'bbb', 'ccc'):
            await stream.write(EchoMessage(message=data))

        await stream.done_writing()

        print('Initial metadata:')
        for key, value in await stream.initial_metadata():
            print(f'  {key}: {value}')

        async for response in stream:
            print(f'Received "{response.message}"')

        print('Trailing metadata:')
        for key, value in await stream.trailing_metadata():
            print(f'  {key}: {value}')

if __name__ == '__main__':
    asyncio.run(main())

I imagine the non-interleave case might stop working if too much data was written, since the server would probably block at some point with the client not reading any of the response data. However, for this simple example, it does work, suggesting that the issue may be with Caddy not supporting bidirectional I/O when it proxies this traffic.

In the case where it doesn’t work, the server still sees the client’s invocation metadata and the first message to echo, but the client blocks trying to display the server’s initial metadata before it sends any data to to be echoed. If the metadata code is commented out, it blocks after sending the first message to echo, waiting to see the response, even though the server side shows that it got the request and responded.

This interleaved version works just fine when going direct to the server (bypassing Caddy).

5

What happens if you set flush_interval -1 explicitly?

6

Thanks for the suggestion. No change, unfortunately.

7

Caddy GRPC H2C Passthrough
Had a similar issue, adding allow_h2c in the Caddyfile’s server section and using h2c:// in the proxy destination fixed the issue for me and streaming gRPC works fine now.

2 Likes
8

Yes - I’ve tried using h2c:// as well, and it behaves the same as using grpc://. Streaming RPC works, but only if the client writes all of the messages in the request stream before attempting to read any of the messages in the response stream (or other information like server metadata). Have you tried a streaming RPC where you alternate between reading & writing messages within a single streaming RPC call?

9

This topic was automatically closed after 30 days. New replies are no longer allowed.