1. The problem I’m having:
I am trying to reverse proxy darkgrpc.dd84ai.com to my running GRPC server written in Go from a quickstart tutorial.
- Caddy is running as Caddy Docker Proxy
As far as i copy caddy config from running caddy container it is currently looking like this
darkgrpc.dd84ai.com {
reverse_proxy h2c://10.0.3.20:50051
}
Full caddy configuration infra/tf/modules/caddy/main.tf at master · darklab8/infra · GitHub
i am running project darkstat
That has basically helloworld of grpc here fl-darkstat/darkgrpc at master · darklab8/fl-darkstat · GitHub
its docker configuration with caddy docker proxy labels are here fl-darkstat/tf/modules/darkstat/main.tf at 09aeec13b42763a07fab42ea4910f5088b2701b7 · darklab8/fl-darkstat · GitHub
Solution works with 50051 port if i bypass caddy, this command prints all the
naa@naa-MS-7C89:~/repos/pet_projects/fl-darkstat$ grpcurl -plaintext darkgrpc.dd84ai.com:50051 darkgrpc.DarkGRpc/GetBases
{
// different json stuff
"NicknameHash": "2262346050",
"System": "Cologne",
"SystemNickname": "rh07",
"SystemNicknameHash": "2281143426",
"Region": "ALSACE PASSAGE",
"File": "universe/systems/playerbase/bases/rh_proxy_base.ini",
"BGCSBaseRunBy": "W02bF35",
"Pos": {
"Y": -100000
},
"SectorCoord": "E-5",
"IsTransportUnreachable": true
}
]
}
and etc stuff
and it does not work if i use 80/443 ports as i wish. grpcurl shows
naa@naa-MS-7C89:~/repos/pet_projects/fl-darkstat/darkgrpc/client_test$ grpcurl -plaintext darkgrpc.dd84ai.com:443 darkgrpc.DarkGRpc/GetBases
Failed to dial target host "darkgrpc.dd84ai.com:443": context deadline exceeded
naa@naa-MS-7C89:~/repos/pet_projects/fl-darkstat/darkgrpc/client_test$ grpcurl -plaintext darkgrpc.dd84ai.com:80 darkgrpc.DarkGRpc/GetBases
Failed to dial target host "darkgrpc.dd84ai.com:80": context deadline exceeded
and golang client code from darklab8/fl-darkstat/blob/master/darkgrpc/client_test/client.go shows
naa@naa-MS-7C89:~/repos/pet_projects/fl-darkstat/darkgrpc/client_test$ go run .
<nil>
err= rpc error: code = Unavailable desc = connection error: desc = "error reading server preface: EOF"
2025/02/04 14:29:10 could not greet: rpc error: code = Unavailable desc = connection error: desc = "error reading server preface: EOF"
exit status 1
2. Error messages and/or full log output:
naa@naa-MS-7C89:~/repos/pet_projects/fl-darkstat/darkgrpc/client_test$ grpcurl -plaintext darkgrpc.dd84ai.com:443 darkgrpc.DarkGRpc/GetBases
Failed to dial target host "darkgrpc.dd84ai.com:443": context deadline exceeded
naa@naa-MS-7C89:~/repos/pet_projects/fl-darkstat/darkgrpc/client_test$ grpcurl -plaintext darkgrpc.dd84ai.com:80 darkgrpc.DarkGRpc/GetBases
Failed to dial target host "darkgrpc.dd84ai.com:80": context deadline exceeded
naa@naa-MS-7C89:~/repos/pet_projects/fl-darkstat/darkgrpc/client_test$ go run .
<nil>
err= rpc error: code = Unavailable desc = connection error: desc = "error reading server preface: EOF"
2025/02/04 14:29:10 could not greet: rpc error: code = Unavailable desc = connection error: desc = "error reading server preface: EOF"
exit status 1
3. Caddy version:
docker exec -it caddy caddy version
v2.8.4 h1:q3pe0wpBj1OcHFZ3n/1nl4V4bxBrYoSoab7rL9BMYNk=
4. How I installed and ran Caddy:
caddy docker proxyvia terraform docker darklab8/infra/blob/master/tf/modules/caddy/main.tf
connected to labels of app darklab8/fl-darkstat/blob/09aeec13b42763a07fab42ea4910f5088b2701b7/tf/modules/darkstat/main.tf#L48
a. System environment:
Hetzner server
root@node-darklab:/var/lib/caddy/caddy# cat /etc/os-release
PRETTY_NAME="Ubuntu 24.04.1 LTS"
NAME="Ubuntu"
VERSION_ID="24.04"
VERSION="24.04.1 LTS (Noble Numbat)"
root@node-darklab:/var/lib/caddy/caddy# docker version
Client: Docker Engine - Community
Version: 27.3.1
b. Command:
as far as docker inspect caddy shows
"Env": [
"PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin",
"XDG_CONFIG_HOME=/config",
"XDG_DATA_HOME=/data"
],
"Cmd": [
"docker-proxy"
],
"Entrypoint": [
"/bin/caddy"
],
Probably more useful to see infra code for terraform docker container from infra repo
c. Service/unit/compose file:
Caddy service:
resource "docker_network" "network" {
name = "caddy"
attachable = true
driver = "overlay"
}
resource "docker_image" "caddy" {
name = "lucaslorentz/caddy-docker-proxy:2.9.1"
}
resource "docker_container" "caddy" {
name = "caddy"
image = docker_image.caddy.image_id
restart = "always"
networks_advanced {
name = docker_network.network.id
}
ports {
internal = "80"
external = "80"
}
ports {
internal = "443"
external = "443"
}
volumes {
host_path = "/var/run/docker.sock"
container_path = "/var/run/docker.sock"
read_only = true
}
volumes {
volume_name = "caddy_data"
container_path = "/data"
}
lifecycle {
ignore_changes = [
memory_swap,
network_mode,
]
}
}
My service
resource "docker_network" "network" {
name = "darkstat-${var.environment}"
attachable = true
driver = "overlay"
}
locals {
tag = var.tag != null ? var.tag : var.environment
}
resource "docker_image" "darkstat" {
name = "darkwind8/darkstat:${local.tag}"
keep_locally = true
}
data "docker_network" "caddy" {
name = "caddy"
}
resource "docker_service" "darkstat" {
name = "${var.environment}-darkstat-app"
task_spec {
networks_advanced {
name = docker_network.network.id
}
networks_advanced {
name = data.docker_network.caddy.id
}
container_spec {
image = docker_image.darkstat.name
env = local.envs
# args = ["sleep", "infinity"]
labels {
label = "caddy_0"
value = "${var.stat_prefix}.${var.zone}"
}
labels {
label = "caddy_0.reverse_proxy"
value = "{{upstreams 8000}}"
}
labels {
label = "caddy_1"
value = "${var.relay_prefix}.${var.zone}"
}
labels {
label = "caddy_1.reverse_proxy"
value = "{{upstreams 8080}}"
}
labels {
label = "caddy_2"
value = "${var.rpc_prefix}.${var.zone}"
}
labels {
label = "caddy_2.reverse_proxy"
value = "{{upstreams h2c 50051}}"
}
mounts {
target = "/data"
source = var.discovery_path
type = "bind"
read_only = false
bind_options {
propagation = "rprivate"
}
}
mounts { // darkstat socks
target = "/tmp/darkstat"
source = "/tmp/darkstat-${var.environment}"
type = "bind"
read_only = false
bind_options {
propagation = "rprivate"
}
}
}
restart_policy {
condition = "any"
delay = "20s"
}
resources {
limits {
memory_bytes = 1000 * 1000 * 3000 # 1 gb
}
}
}
lifecycle {
ignore_changes = [
task_spec[0].restart_policy[0].window,
task_spec[0].container_spec[0].env,
]
}
# with usage of docker networking, this is no longer necessary
endpoint_spec {
mode = "vip"
ports {
target_port = "50051"
published_port = tostring(var.rpc_port)
}
}
}
d. My complete Caddy config:
that was generated by caddy docker proxy from my labels
root@node-darklab:/var/lib/caddy/caddy# cat Caddyfile.autosave
darkgrpc-dev.dd84ai.com {
reverse_proxy h2c://10.0.3.21:50051
}
darkgrpc.dd84ai.com {
reverse_proxy h2c://10.0.3.20:50051
}
darkrelay-dev.dd84ai.com {
reverse_proxy 10.0.3.21:8080
}
darkrelay.dd84ai.com {
reverse_proxy 10.0.3.20:8080
}
darkstat-dev.dd84ai.com {
reverse_proxy 10.0.3.21:8000
}
darkstat.dd84ai.com {
reverse_proxy 10.0.3.20:8000
}
5. Links to relevant resources:
Site has limit 4 links per post. i removed https github com part from some links going over the limit
So summarizing all the stuff about.
grpcurl -plaintext darkgrpc.dd84ai.com:50051 darkgrpc.DarkGRpc/GetBases
// works without caddy. just using docker swarm opened port directly
grpcurl -plaintext darkgrpc.dd84ai.com:443 darkgrpc.DarkGRpc/GetBase
// does not work if trying to use caddy to access it
Failed to dial target host "darkgrpc.dd84ai.com:443": context deadline exceeded
With Caddyfile
darkgrpc-dev.dd84ai.com {
reverse_proxy h2c://10.0.3.21:50051
}
And a bit difficult to write caddy docker proxy settings. I tried outdated some issues written in this forum, but servers { protocols { allow_h2c }}}
did not work with error
{"level":"info","ts":1738674206.124626,"logger":"docker-proxy","msg":"Process Caddyfile","logs":"[ERROR] Removing invalid block: parsing caddyfile tokens for 'servers': unrecognized servers option 'protocol', at Caddyfile:3\n{\n\tservers {\n\t\tprotocol {\n\t\t\tallow_h2c\n\t\t}\n\t}\n}\n\n"}
What could be an up to date option to resolve it?
P.S. another project already works from same Caddy container at same server, the one with prefix darkstat to the dmain. regular https golang web app. All is accessable okay.
Golang GRPC app was written according to helloworld taken from grpc quickstart for golang
at github path, it is accessable at github com / grpc/grpc-go/tree/master/examples/helloworld
UPD:
Caught some caddy logs
{"level":"info","ts":1738682279.9113512,"logger":"http.log.access","msg":"handled request","request":{"remote_ip":"52.47.113.81","remote_port":"48244","client_ip":"52.47.113.81","proto":"HTTP/2.0","method":"PRI","host":"","uri":"*","headers":{}},"bytes_read":0,"user_id":"","duration":0.00007736,"size":0,"status":308,"resp_headers":{"Connection":["close"],"Location":["https://*"],"Content-Type":[],"Server":["Caddy"]}}
{"level":"info","ts":1738682281.069223,"logger":"http.log.access","msg":"handled request","request":{"remote_ip":"52.47.113.81","remote_port":"48256","client_ip":"52.47.113.81","proto":"HTTP/2.0","method":"PRI","host":"","uri":"*","headers":{}},"bytes_read":0,"user_id":"","duration":0.00004484,"size":0,"status":308,"resp_headers":{"Connection":["close"],"Location":["https://*"],"Content-Type":[],"Server":["Caddy"]}}
{"level":"info","ts":1738682282.712738,"logger":"http.log.access","msg":"handled request","request":{"remote_ip":"52.47.113.81","remote_port":"48270","client_ip":"52.47.113.81","proto":"HTTP/2.0","method":"PRI","host":"","uri":"*","headers":{}},"bytes_read":0,"user_id":"","duration":0.00003696,"size":0,"status":308,"resp_headers":{"Server":["Caddy"],"Connection":["close"],"Location":["https://*"],"Content-Type":[]}}
{"level":"info","ts":1738682285.262502,"logger":"http.log.access","msg":"handled request","request":{"remote_ip":"52.47.113.81","remote_port":"48282","client_ip":"52.47.113.81","proto":"HTTP/2.0","method":"PRI","host":"","uri":"*","headers":{}},"bytes_read":0,"user_id":"","duration":0.000074041,"size":0,"status":308,"resp_headers":{"Server":["Caddy"],"Connection":["close"],"Location":["https://*"],"Content-Type":[]}}