Proxy grpc with caddy (ultimate goal: cisco telemetry -> caddy proxy -> telegraf -> disk store metrics)

Hi there,

I have a grpc client/server hello world toy example running fine in my local machine:

Server:

➜ make server
go run greeter_server/main.go
2023/03/16 09:45:50 server listening at [::]:50051
2023/03/16 09:46:02 Received: Drio

Client:

➜ make client
go run greeter_client/main.go --name=Drio
2023/03/16 09:46:34 Greeting: Hello Drio

Now I want to proxy the grpc server requests with Caddy (v2.6.4). For that I use the following Caddyfile:

localhost {
  handle /ping {
    respond "pong"
  }  
  reverse_proxy  grpc://localhost:50051 
}

On the grpc client code, I just change the addr from: localhost:50051 to localhost:443. Then I start caddy and the grpc service/server. Then I run the client again but I get the following error:

➜ make client
go run greeter_client/main.go --name=Drio
2023/03/16 09:50:26 could not greet: rpc error: code = Unavailable desc = connection error: desc = "error reading server preface: EOF"
exit status 1
make: *** [client] Error 1

Does anyone know what may be happening?

I must admit I have zero expertise in grpc. I am just trying to stream cisco telemetry to a telegraf instance to store the data but I wanted to move in small steps.

I see there is another grpc mode (streaming) which requires a different caddy configuration (h2c). What is exactly the difference between traditional grpc vs grpc streaming?

Thank you,
-drd

grpc:// is not a valid upstream scheme. See the docs, you probably want h2c:// instead.

Thank you, francis.

With:

localhost {
  reverse_proxy h2c://localhost:50051
}

I still get the same error when proxying via Caddy:

➜ go run greeter_client/main.go --addr localhost:443;
2023/03/16 12:45:48 could not greet: rpc error: code = Unavailable desc = connection error: desc = "error reading server preface: EOF"
exit status 1

But the client works correctly if I connect directly to the server.

➜ go run greeter_client/main.go --addr localhost:50051;
2023/03/16 12:45:42 Greeting: Hello world

The client/server code is this one.

Turn off the debug global option in Caddy. What do you see in Caddy’s logs?

2023/03/16 17:44:39.665 DEBUG   http.stdlib     http: TLS handshake error from [::1]:53779: tls: first record does not look like a TLS handshake

You probably need to use https://localhost instead then, to instruct your grpc client to use TLS.

With:

{
  debug off
}

https://localhost {
  reverse_proxy h2c://localhost:50051
}

I get this on the client side:

➜ go run greeter_client/main.go --addr localhost:443;
2023/03/16 14:17:23 could not greet: rpc error: code = Unavailable desc = connection error: desc = "error reading server preface: EOF"
exit status 1

And the Caddy log gives the same error:

023/03/16 18:19:59.866 DEBUG   http.stdlib     http: TLS handshake error from [::1]:54512: tls: first record does not look like a TLS handshake

More error details in the client side (enable debug in client library):

➜ GRPC_TRACE=all
GRPC_VERBOSITY=DEBUG
GRPC_GO_LOG_VERBOSITY_LEVEL=2
GRPC_GO_LOG_SEVERITY_LEVEL=info go run greeter_client/main.go --addr localhost:443;
2023/03/16 14:23:15 INFO: [core] [Channel #1] Channel created
2023/03/16 14:23:15 INFO: [core] [Channel #1] original dial target is: "localhost:443"
2023/03/16 14:23:15 INFO: [core] [Channel #1] parsed dial target is: {Scheme:localhost Authority: URL:{Scheme:localhost Opaque:443 User: Host: Path: RawPath: OmitHost:false ForceQuery:false RawQuery: Fragment: RawFragment:}}
2023/03/16 14:23:15 INFO: [core] [Channel #1] fallback to scheme "passthrough"
2023/03/16 14:23:15 INFO: [core] [Channel #1] parsed dial target is: {Scheme:passthrough Authority: URL:{Scheme:passthrough Opaque: User: Host: Path:/localhost:443 RawPath: OmitHost:false ForceQuery:false RawQuery: Fragment: RawFragment:}}
2023/03/16 14:23:15 INFO: [core] [Channel #1] Channel authority set to "localhost:443"
2023/03/16 14:23:15 INFO: [core] [Channel #1] Resolver state updated: {
  "Addresses": [
    {
      "Addr": "localhost:443",
      "ServerName": "",
      "Attributes": null,
      "BalancerAttributes": null,
      "Type": 0,
      "Metadata": null
    }
  ],
  "ServiceConfig": null,
  "Attributes": null
} (resolver returned new addresses)
2023/03/16 14:23:15 INFO: [core] [Channel #1] Channel switches to new LB policy "pick_first"
2023/03/16 14:23:15 INFO: [core] [Channel #1 SubChannel #2] Subchannel created
2023/03/16 14:23:15 INFO: [core] [Channel #1] Channel Connectivity change to CONNECTING
2023/03/16 14:23:15 INFO: [core] [Channel #1 SubChannel #2] Subchannel Connectivity change to CONNECTING
2023/03/16 14:23:15 INFO: [core] [Channel #1 SubChannel #2] Subchannel picks a new address "localhost:443" to connect
2023/03/16 14:23:15 WARNING: [core] [Channel #1 SubChannel #2] grpc: addrConn.createTransport failed to connect to {
  "Addr": "localhost:443",
  "ServerName": "localhost:443",
  "Attributes": null,
  "BalancerAttributes": null,
  "Type": 0,
  "Metadata": null
}. Err: connection error: desc = "error reading server preface: EOF"
2023/03/16 14:23:15 INFO: [core] [Channel #1 SubChannel #2] Subchannel Connectivity change to TRANSIENT_FAILURE
2023/03/16 14:23:15 INFO: [core] [Channel #1] Channel Connectivity change to TRANSIENT_FAILURE
2023/03/16 14:23:15 could not greet: rpc error: code = Unavailable desc = connection error: desc = "error reading server preface: EOF"
exit status 1

What I’m saying is that your client is connecting with HTTP, not HTTPS. That’s what the log message in Caddy is saying.

I think I know what is going on.
The server code looks like this:

...
func main() {
	flag.Parse()
	lis, err := net.Listen("tcp", fmt.Sprintf(":%d", *port))
	if err != nil {
		log.Fatalf("failed to listen: %v", err)
	}
	s := grpc.NewServer()
	pb.RegisterGreeterServer(s, &server{})
	log.Printf("server listening at %v", lis.Addr())
	if err := s.Serve(lis); err != nil {
		log.Fatalf("failed to serve: %v", err)
	}
}

So, the server is talking plain http2.

If I connect to port 80 (served by caddy), the client gives me this error:

➜ go run greeter_client/main.go --addr localhost:80;
2023/03/16 14:34:16 could not greet: rpc error: code = Unavailable desc = connection error: desc = "error reading server preface: http2: frame too large"
exit status 1

With this, Caddy will redirect HTTP requests to HTTPS.

Did you actually try with https://localhost instead of localhost:443?

My caddy file:

{
  debug off
}

https://localhost {
  reverse_proxy h2c://localhost:50051
}

➜ go run greeter_client/main.go --addr localhost:443

Is the only one that actually connects to Caddy (but Caddy does not proxy the request).
And in the caddy logs I see:

http.stdlib     http: TLS handshake error from [::1]:51567: tls: first record does not look like a TLS handshake

With https://localhost I get:

➜ go run greeter_client/main.go --addr https://localhost
2023/03/17 06:55:27 could not greet: rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing: dial tcp: lookup tcp///localhost: nodename nor servname provided, or not known"
exit status 1

And if I connect directly to the server/service:

➜ go run greeter_client/main.go --addr localhost:50051
2023/03/17 06:57:53 Greeting: Hello world

Like I said, this is because your client is connecting using HTTP, and not HTTPS.

You need to make sure your client uses HTTPS.

Okay, your client program doesn’t understand schemes. That’s strange.

Ultimately this problem is with your client program, not with Caddy. Your Caddy config is correct. Your program is only able to connect using HTTP and not HTTPS.

Thank you, Francis.

I modified the client to talk TLS:

	// BEFORE
	//conn, err := grpc.Dial(*addr, grpc.WithTransportCredentials(insecure.NewCredentials()))
        // AFTER
	certFile := "/Users/drio/Library/Application Support/Caddy/pki/authorities/local/root.crt"
	creds, _ := credentials.NewClientTLSFromFile(certFile, "")
	conn, err := grpc.Dial(*addr, grpc.WithTransportCredentials(creds))

And now :fire::

➜ go run greeter_client/main.go --addr localhost:443
2023/03/17 09:21:37 Greeting: Hello world

1 Like

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.