Caddy plaintext grpc reverse proxy without TLS certificate

Help
1

1. The problem I’m having:

I have a working caddy reverse proxy to my insecure grpc service written in golang
it’s working with SSL termination on caddy side
eg:

grpcurl -cacert certificate.pem 127.0.0.1:8443 list

but does not work when I use plaintext

grpcurl -plaintext 127.0.0.1:8443 list
Failed to dial target host "127.0.0.1:8443": context deadline exceeded

Caddy log reports about TLS handshake error even I’m not using any TLS certificate.
I tried auto_https disable_redirects and disable_certs , but it does not seem to make any difference.

I saw multiple examples on that forum when people used exactly the same config and plaintext worked. So it’s not clear why it happens. It’s much more convenient to test things without TLS using plaintext

2. Error messages and/or full log output:

2023/11/18 10:27:50.173 INFO    using provided configuration    {"config_file": "/home/sd/Build/ytdlp-ssh/cmd/caddy/Caddyfile", "config_adapter": ""}
2023/11/18 10:27:50.173 WARN    Caddyfile input is not formatted; run 'caddy fmt --overwrite' to fix inconsistencies    {"adapter": "caddyfile", "file": "/home/sd/Build/ytdlp-ssh/cmd/caddy/Caddyfile", "line": 2}
2023/11/18 10:27:50.174 INFO    admin   admin endpoint started  {"address": "localhost:2019", "enforce_origin": false, "origins": ["//[::1]:2019", "//127.0.0.1:2019", "//localhost:2019"]}
2023/11/18 10:27:50.174 INFO    http.auto_https server is listening only on the HTTPS port but has no TLS connection policies; adding one to enable TLS {"server_name": "srv0", "https_port": 8443}
2023/11/18 10:27:50.174 INFO    http.auto_https enabling automatic HTTP->HTTPS redirects        {"server_name": "srv0"}
2023/11/18 10:27:50.174 DEBUG   http.auto_https adjusted config {"tls": {"automation":{"policies":[{}]}}, "http": {"http_port":8000,"https_port":8443,"servers":{"remaining_auto_https_redirects":{"listen":[":8000"],"routes":[{},{}]},"srv0":{"listen":[":8443"],"routes":[{"handle":[{"handler":"reverse_proxy","transport":{"protocol":"http","versions":["h2c","2"]},"upstreams":[{"dial":"127.0.0.1:5051"}]}]}],"tls_connection_policies":[{}],"automatic_https":{}}}}}
2023/11/18 10:27:50.174 INFO    tls.cache.maintenance   started background certificate maintenance      {"cache": "0xc000546200"}
2023/11/18 10:27:50.174 INFO    tls     cleaning storage unit   {"description": "FileStorage:/home/sd/.local/share/caddy"}
2023/11/18 10:27:50.174 INFO    http    enabling HTTP/3 listener        {"addr": ":8443"}
2023/11/18 10:27:50.174 INFO    tls     finished cleaning storage units
2023/11/18 10:27:50.174 DEBUG   http    starting server loop    {"address": "[::]:8443", "tls": true, "http3": true}
2023/11/18 10:27:50.174 INFO    http.log        server running  {"name": "srv0", "protocols": ["h1", "h2", "h3"]}
2023/11/18 10:27:50.174 DEBUG   http    starting server loop    {"address": "[::]:8000", "tls": false, "http3": false}
2023/11/18 10:27:50.174 INFO    http.log        server running  {"name": "remaining_auto_https_redirects", "protocols": ["h1", "h2", "h3"]}
2023/11/18 10:27:50.174 INFO    autosaved config (load with --resume flag)      {"file": "/home/sd/.config/caddy/autosave.json"}
2023/11/18 10:27:50.174 INFO    serving initial configuration
2023/11/18 10:27:55.529 DEBUG   http.stdlib     http: TLS handshake error from 127.0.0.1:42772: tls: first record does not look like a TLS handshake
2023/11/18 10:27:56.530 DEBUG   http.stdlib     http: TLS handshake error from 127.0.0.1:42788: tls: first record does not look like a TLS handshake
2023/11/18 10:27:58.066 DEBUG   http.stdlib     http: TLS handshake error from 127.0.0.1:42804: tls: first record does not look like a TLS handshake
2023/11/18 10:28:00.178 DEBUG   http.stdlib     http: TLS handshake error from 127.0.0.1:42818: tls: first record does not look like a TLS handshake

3. Caddy version:

2.7.5/devel version

4. How I installed and ran Caddy:

Linux binary / Run from sources

a. System environment:

Debian 12 amd64
Caddy run directly on the host

b. Command:

./caddy_linux_amd64 run --config Caddyfile

c. Service/unit/compose file:

None

d. My complete Caddy config:

{
debug
https_port 8443
http_port 8000
}
:8443 {
   reverse_proxy h2c://127.0.0.1:5051  
}

5. Links to relevant resources:

None

2

You told Caddy that the HTTPS port is 8443 so Caddy is expecting TLS connections on that port. But yes you didn’t tell Caddy how to get a certificate, so connections will be impossible.

You’ll need to use a different port for plaintext, you can’t multiplex HTTP and HTTPS on the same port.

You can enable h2c by configuring the global options (see Global options (Caddyfile) — Caddy Documentation):

{
	servers {
		protocols h1 h2 h2c h3
	}
}
3

Thank you Francis!
it’s very descriptive response.
Including protocols h1 h2 h2c did the trick.

Here is working config:

{
debug
https_port 8443
http_port 8000

servers {
		protocols h1 h2 h2c
	}
}

:8443 {
       tls certificate.pem private_key.pem
       reverse_proxy h2c://127.0.0.1:5051  
}

:8000 {
       reverse_proxy h2c://127.0.0.1:5051  
}

So both ssl and plaintext work on different ports.

grpcurl -cacert certificate.pem 127.0.0.1:8443 list
grpcurl -plaintext -vv 127.0.0.1:8000 list

Do I understand correctly that h2c is not enabled by default and h2 used instead when I try :8000 http port?
Also how caddy understands to pick h2c instead of h2 when I specify protocols h1 h2 h2c ?

:8000 {
       reverse_proxy h2c://127.0.0.1:5051  
}

Without specifying protocols h1 h2 h2c it gives me

   http.handlers.reverse_proxy     upstream roundtrip      {"upstream": "127.0.0.1:5051", "duration": 0.000333368, "request": {"remote_ip": "127.0.0.1", "remote_port": "37942", "client_ip": "127.0.0.1", "proto": "HTTP/2.0", "method": "PRI", "host": "", "uri": "*", "headers": {"User-Agent": [""], "X-Forwarded-For": ["127.0.0.1"], "X-Forwarded-Proto": ["http"], "X-Forwarded-Host": [""]}}, "headers": {"Content-Type": ["application/grpc"], "Grpc-Status": ["3"], "Grpc-Message": ["invalid gRPC request content-type \"\""]}, "status": 415}
4

Correct that h2c is not enabled by default, but it’s because HTTP/2 requires TLS. H2C is basically “breaking” the protocol by ignoring the TLS requirement.

Enabling h2c allows HTTP/2 to be used plaintext (i.e. HTTP, not HTTPS).

h2c is HTTP, and h2 is HTTPS.

Yeah, gRPC is picky and requires HTTP/2 at the protocol level to work correctly. I find that honestly pretty ridiculous, but anyway :man_shrugging:

Keep in mind that you disabled h3 by doing this (because the default is h1 h2 h3) just in case you eventually want HTTP/3 support (i.e. HTTPS over UDP, instead of TCP).

5

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.