How to reverse_proxy a gRPC service?

1. Caddy version:

$ caddy version
v2.6.2 h1:wKoFIxpmOJLGl3QXoo6PNbYvGW4xLEgo32GPBEjWL8o=

2. How I installed, and run Caddy:

Installed via apt install caddy following the Caddy docs for Debian, Ubuntu, Raspbian.

a. System environment:

$ cat /etc/os-release
PRETTY_NAME="Ubuntu 22.04.1 LTS"
NAME="Ubuntu"
VERSION_ID="22.04"
VERSION="22.04.1 LTS (Jammy Jellyfish)"
VERSION_CODENAME=jammy
ID=ubuntu
ID_LIKE=debian
HOME_URL="https://www.ubuntu.com/"
SUPPORT_URL="https://help.ubuntu.com/"
BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/"
PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy"
UBUNTU_CODENAME=jammy

b. Command:

# systemctl start caddy

c. Service/unit/compose file:

$ systemctl cat caddy.service
# /lib/systemd/system/caddy.service
# caddy.service
#
# For using Caddy with a config file.
#
# Make sure the ExecStart and ExecReload commands are correct
# for your installation.
#
# See https://caddyserver.com/docs/install for instructions.
#
# WARNING: This service does not use the --resume flag, so if you
# use the API to make changes, they will be overwritten by the
# Caddyfile next time the service is restarted. If you intend to
# use Caddy's API to configure it, add the --resume flag to the
# `caddy run` command or use the caddy-api.service file instead.

[Unit]
Description=Caddy
Documentation=https://caddyserver.com/docs/
After=network.target network-online.target
Requires=network-online.target

[Service]
Type=notify
User=caddy
Group=caddy
ExecStart=/usr/bin/caddy run --environ --config /etc/caddy/Caddyfile
ExecReload=/usr/bin/caddy reload --config /etc/caddy/Caddyfile --force
TimeoutStopSec=5s
LimitNOFILE=1048576
LimitNPROC=512
PrivateTmp=true
ProtectSystem=full
AmbientCapabilities=CAP_NET_BIND_SERVICE

[Install]
WantedBy=multi-user.target

d. My complete Caddy config:

$ cat /etc/caddy/Caddyfile
foo.example.com {
        encode zstd gzip

        log {
                output stdout
                format json
                level DEBUG
        }

        handle /grpc* {
                #rewrite * {path}
                uri strip_prefix /grpc
                reverse_proxy 127.0.0.1:9090 {
                        transport http {
                                versions h2c 2
                        }
                }
        }
        #reverse_proxy /grpc/* grpc://127.0.0.1:9090

        handle_path /rest/* {
                rewrite * {path}
                reverse_proxy /* 127.0.0.1:1317
        }

        handle_path /rpc/* {
                rewrite * {path}
                reverse_proxy /* 127.0.0.1:26657
        }
        reverse_proxy /* 127.0.0.1:26657
}

3. The problem I’m having:

I’m trying to create a /grpc/ path that is reverse_proxied to a service running on port 9090 on the machine. This service is working and I can check via $ grpc-client-cli -V discover foo.example.com:9090. But when I try to connect via gRPC to the endpoint I get and error:

$ grpc-client-cli -V discover prod-sei-node-1.lavapro.xyz/grpc/

Method:
Status: 14 Unavailable

Request Headers:

Request duration: 1.566µs
Request size: 0 bytes
Response size: 0 bytes

rpc error: code = Unavailable desc = last resolver error: produced zero addresses

The other endpoints (/rest/, /rpc/, and /) are working fine.

4. Error messages and/or full log output:

When I use the gRPC CLI client to interact with the /grpc/ endpoint, I see nothing logged in the Caddy logs. If I curl any one of the other endpoints I can see the request in the logs.

5. What I already tried:

I tried multiple variations of reverse_proxy, handle, handle_path and had no success to get the gRPC endpoint routed correctly. Some of my attempts are commented out in the Caddyfile above.

6. Links to relevant resources:

Is this trying to connect over HTTP, or HTTPS? Try specifying https://. I don’t know how the gRPC client works.

Turn on the debug global option, you should see more details about the proxy traffic.

Lots of little things can be simplified here. There’s a bunch of redundant/useless bits.

foo.example.com {
	encode zstd gzip

	log {
		format json
	}

	handle_path /grpc* {
		reverse_proxy h2c://127.0.0.1:9090
	}

	handle_path /rest/* {
		reverse_proxy 127.0.0.1:1317
	}

	handle_path /rpc/* {
		reverse_proxy 127.0.0.1:26657
	}

	handle {
		reverse_proxy 127.0.0.1:26657
	}
}

I suspected that were many things to simplify, thanks for answering many side questions I had :slight_smile:

Well, the funny thing is: i’m not sure:

$ grpcurl -vv -plaintext $URL:443/grpc list
Failed to dial target host "foo.example.com:443/grpc": dial tcp: lookup tcp/443/grpc: Servname not supported for ai_socktype

$ grpcurl -vv -plaintext $URL:443/grpc/ list
Failed to dial target host "foo.example.com:443/grpc/": dial tcp: lookup tcp/443/grpc/: Servname not supported for ai_socktype

$ grpcurl -vv -plaintext $URL:80/grpc list
Failed to dial target host "foo.example.com:80/grpc": dial tcp: lookup tcp/80/grpc: Servname not supported for ai_socktype

$ grpcurl -vv -plaintext $URL:80/grpc/ list
Failed to dial target host "foo.example.com:80/grpc/": dial tcp: lookup tcp/80/grpc/: Servname not supported for ai_socktype

$ grpcurl -vv -plaintext $URL:9090/ list
Failed to dial target host "foo.example.com:9090/": dial tcp: lookup tcp/9090/: Servname not supported for ai_socktype

$ grpcurl -vv -plaintext $URL:9090 list
cosmos.auth.v1beta1.Query
cosmos.bank.v1beta1.Query
cosmos.base.reflection.v1beta1.ReflectionService
cosmos.base.reflection.v2alpha1.ReflectionService
cosmos.base.tendermint.v1beta1.Service
cosmos.distribution.v1beta1.Query
cosmos.evidence.v1beta1.Query
cosmos.feegrant.v1beta1.Query
cosmos.gov.v1beta1.Query
cosmos.params.v1beta1.Query
cosmos.slashing.v1beta1.Query
cosmos.staking.v1beta1.Query
cosmos.tx.v1beta1.Service
cosmos.upgrade.v1beta1.Query
cosmwasm.wasm.v1.Query
grpc.reflection.v1alpha.ServerReflection
ibc.applications.transfer.v1.Query
ibc.core.channel.v1.Query
ibc.core.client.v1.Query
ibc.core.connection.v1.Query
seiprotocol.seichain.dex.Query
seiprotocol.seichain.epoch.Query
seiprotocol.seichain.mint.Query
seiprotocol.seichain.nitro.Query
seiprotocol.seichain.oracle.Query
seiprotocol.seichain.tokenfactory.Query

This gRPC service is very sensitive to trailing slashes.

Even with DEBUG in the global settings, I see nothing in the logs about these requests. Any ideas on how to debug this further?

Well… as far as I can tell, gRPC is very opinionated about how URLs should be constructed.

Another idea, instead of trying to use a subpath, you could match on the Content-Type which should be application/grpc for gRPC.

@grpc header Content-Type application/grpc
handle @grpc {
	reverse_proxy h2c://127.0.0.1:9090
}

If that doesn’t work either, I guess the best option is to use a different domain or subdomain for gRPC.

1 Like

Thanks @francislavoie! You pointed me to the directions I needed!

I got the gRPC reverse proxied to where I want now :slight_smile:

@grpc protocol grpc
handle @grpc {
	reverse_proxy h2c://127.0.0.1:9090
}

Have a nice day!

1 Like