gRPC proxying in plaintext

Help
1

1. Caddy version (caddy version):

v2.3.0 h1:fnrqJLa3G5vfxcxmOH/+kJOcunPLhSBnjgIvjXV/QTA=

2. How I run Caddy:

a. System environment:

Mac

b. Command:

caddy run --config CONFIG

c. Service/unit/compose file:

N/A

d. My complete Caddyfile or JSON config:

I tried both

localhost {
   reverse_proxy 192.168.64.36:31245
}

and

localhost {
    reverse_proxy {
        to h2c://192.168.64.36:31245
        transport http {
            versions h2c 2
        }
    }
}

3. The problem I’m having:

I have a service running with plaintext gRPC at 192.168.64.36:31245:

grpc-health-probe -addr=192.168.64.36:31245
status: SERVING

I’m trying to get Caddy to proxy to it. I tried both configurations above. Neither of them work (testing via GitHub - grpc-ecosystem/grpc-health-probe: A command-line tool to perform health-checks for gRPC applications in Kubernetes etc.):

$ grpc-health-probe -addr=localhost:80
timeout: failed to connect service "localhost:80" within 1s
$  grpc-health-probe -addr=localhost:443
timeout: failed to connect service "localhost:443" within 1s
$ grpc-health-probe -tls -tls-no-verify -addr=localhost:443
error: health rpc failed: rpc error: code = Unavailable desc = Bad Gateway: HTTP status code 502; transport: missing content-type field

$ grpc-health-probe -h
Usage:
  -addr string
        (required) tcp host:port to connect
  -connect-timeout duration
        timeout for establishing connection (default 1s)
  -rpc-timeout duration
        timeout for health check rpc (default 1s)
  -service string
        service name to check (default: "")
  -tls
        use TLS (default: false, INSECURE plaintext transport)
  -tls-ca-cert string
        (with -tls, optional) file containing trusted certificates for verifying server
  -tls-client-cert string
        (with -tls, optional) client certificate for authenticating to the server (requires -tls-client-key)
  -tls-client-key string
        (with -tls) client private key for authenticating to the server (requires -tls-client-cert)
  -tls-no-verify
        (with -tls) don't verify the certificate (INSECURE) presented by the server (default: false)
  -tls-server-name string
        (with -tls) override the hostname used to verify the server certificate
  -user-agent string
        user-agent header value of health check requests (default "grpc_health_probe")
  -v    verbose logs

I’m wondering what I miss. Thanks!

4. Error messages and/or full log output:

2021/01/30 01:34:08.758 INFO    using provided configuration    {"config_file": "Caddyfile", "config_adapter": ""}
2021/01/30 01:34:08.759 INFO    admin   admin endpoint started  {"address": "tcp/localhost:2019", "enforce_origin": false, "origins": ["localhost:2019", "[::1]:2019", "127.0.0.1:2019"]}
2021/01/30 01:34:08.759 INFO    tls.cache.maintenance   started background certificate maintenance      {"cache": "0xc0003a9ab0"}
2021/01/30 01:34:08.759 INFO    http    server is listening only on the HTTPS port but has no TLS connection policies; adding one to enable TLS {"server_name": "srv0", "https_port": 443}
2021/01/30 01:34:08.760 INFO    http    enabling automatic HTTP->HTTPS redirects        {"server_name": "srv0"}
2021/01/30 01:34:08.775 INFO    tls     cleaned up storage units
2021/01/30 01:34:08.885 INFO    pki.ca.local    root certificate is already trusted by system   {"path": "storage:pki/authorities/local/root.crt"}
2021/01/30 01:34:08.886 INFO    http    enabling automatic TLS certificate management   {"domains": ["localhost"]}
2021/01/30 01:34:08.886 WARN    tls     stapling OCSP   {"error": "no OCSP stapling for [localhost]: no OCSP server specified in certificate"}
2021/01/30 01:34:08.886 INFO    autosaved config        {"file": "/Users/owen/Library/Application Support/Caddy/autosave.json"}
2021/01/30 01:34:08.886 INFO    serving initial configuration
2021/01/30 01:35:39.425 ERROR   http.log.error  net/http: HTTP/1.x transport connection broken: malformed HTTP response "\x00\x00\x06\x04\x00\x00\x00\x00\x00\x00\x05\x00\x00@\x00"     {"request": {"remote_addr": "[::1]:49680", "proto": "HTTP/2.0", "method": "POST", "host": "localhost:443", "uri": "/grpc.health.v1.Health/Check", "headers": {"Grpc-Timeout": ["999971u"], "Content-Type": ["application/grpc"], "User-Agent": ["grpc_health_probe grpc-go/1.35.0-dev"], "Te": ["trailers"]}, "tls": {"resumed": false, "version": 772, "cipher_suite": 4865, "proto": "h2", "proto_mutual": true, "server_name": "localhost"}}, "duration": 0.001378605, "status": 502, "err_id": "pdfm91ysu", "err_trace": "reverseproxy.statusError (reverseproxy.go:783)"}
2021/01/30 01:37:22.692 ERROR   http.log.error  net/http: HTTP/1.x transport connection broken: malformed HTTP response "\x00\x00\x06\x04\x00\x00\x00\x00\x00\x00\x05\x00\x00@\x00"     {"request": {"remote_addr": "[::1]:49696", "proto": "HTTP/2.0", "method": "POST", "host": "localhost:443", "uri": "/grpc.health.v1.Health/Check", "headers": {"Content-Type": ["application/grpc"], "User-Agent": ["grpc_health_probe grpc-go/1.35.0-dev"], "Te": ["trailers"], "Grpc-Timeout": ["999968u"]}, "tls": {"resumed": false, "version": 772, "cipher_suite": 4865, "proto": "h2", "proto_mutual": true, "server_name": "localhost"}}, "duration": 0.001004614, "status": 502, "err_id": "20k02j536", "err_trace": "reverseproxy.statusError (reverseproxy.go:783)"}

5. What I already tried:

N/A

6. Links to relevant resources:

2

Hmm…Do I need GitHub - mholt/caddy-l4: Layer 4 (TCP/UDP) app for Caddy to proxy the gRPC traffic to upstream? It doesn’t seem Caddy has good gRPC proxying support out of the box.

3

If you specify h2c://, you don’t need the transport config, so you can reduce this to a single line.

But I’m not sure why you’re getting that error. @matt may have a clue.

4

I’m not really familar with gRPC. It uses HTTP/2 frames, yes? Many other users have reported success proxying gRPC using HTTP/2 (including h2c) with Caddy just fine, so you’d probably have to ask someone who knows more about gRPC than myself (I don’t use it).

5

This topic was automatically closed after 30 days. New replies are no longer allowed.