Caddy Grpc configuration for hello world Golang app

1. The problem I’m having:

I am trying to reverse proxy darkgrpc.dd84ai.com to my running GRPC server written in Go from a quickstart tutorial.

  • Caddy is running as Caddy Docker Proxy
    As far as i copy caddy config from running caddy container it is currently looking like this
darkgrpc.dd84ai.com {
        reverse_proxy h2c://10.0.3.20:50051
}

Full caddy configuration infra/tf/modules/caddy/main.tf at master · darklab8/infra · GitHub

i am running project darkstat
That has basically helloworld of grpc here fl-darkstat/darkgrpc at master · darklab8/fl-darkstat · GitHub
its docker configuration with caddy docker proxy labels are here fl-darkstat/tf/modules/darkstat/main.tf at 09aeec13b42763a07fab42ea4910f5088b2701b7 · darklab8/fl-darkstat · GitHub

Solution works with 50051 port if i bypass caddy, this command prints all the

naa@naa-MS-7C89:~/repos/pet_projects/fl-darkstat$ grpcurl -plaintext darkgrpc.dd84ai.com:50051 darkgrpc.DarkGRpc/GetBases

{
    // different json stuff
    "NicknameHash": "2262346050",
      "System": "Cologne",
      "SystemNickname": "rh07",
      "SystemNicknameHash": "2281143426",
      "Region": "ALSACE PASSAGE",
      "File": "universe/systems/playerbase/bases/rh_proxy_base.ini",
      "BGCSBaseRunBy": "W02bF35",
      "Pos": {
        "Y": -100000
      },
      "SectorCoord": "E-5",
      "IsTransportUnreachable": true
    }
  ]
}

and etc stuff

and it does not work if i use 80/443 ports as i wish. grpcurl shows

naa@naa-MS-7C89:~/repos/pet_projects/fl-darkstat/darkgrpc/client_test$ grpcurl -plaintext darkgrpc.dd84ai.com:443 darkgrpc.DarkGRpc/GetBases
Failed to dial target host "darkgrpc.dd84ai.com:443": context deadline exceeded
naa@naa-MS-7C89:~/repos/pet_projects/fl-darkstat/darkgrpc/client_test$ grpcurl -plaintext darkgrpc.dd84ai.com:80 darkgrpc.DarkGRpc/GetBases
Failed to dial target host "darkgrpc.dd84ai.com:80": context deadline exceeded

and golang client code from darklab8/fl-darkstat/blob/master/darkgrpc/client_test/client.go shows

naa@naa-MS-7C89:~/repos/pet_projects/fl-darkstat/darkgrpc/client_test$ go run .
<nil>
err= rpc error: code = Unavailable desc = connection error: desc = "error reading server preface: EOF"
2025/02/04 14:29:10 could not greet: rpc error: code = Unavailable desc = connection error: desc = "error reading server preface: EOF"
exit status 1

2. Error messages and/or full log output:

naa@naa-MS-7C89:~/repos/pet_projects/fl-darkstat/darkgrpc/client_test$ grpcurl -plaintext darkgrpc.dd84ai.com:443 darkgrpc.DarkGRpc/GetBases
Failed to dial target host "darkgrpc.dd84ai.com:443": context deadline exceeded
naa@naa-MS-7C89:~/repos/pet_projects/fl-darkstat/darkgrpc/client_test$ grpcurl -plaintext darkgrpc.dd84ai.com:80 darkgrpc.DarkGRpc/GetBases
Failed to dial target host "darkgrpc.dd84ai.com:80": context deadline exceeded
naa@naa-MS-7C89:~/repos/pet_projects/fl-darkstat/darkgrpc/client_test$ go run .
<nil>
err= rpc error: code = Unavailable desc = connection error: desc = "error reading server preface: EOF"
2025/02/04 14:29:10 could not greet: rpc error: code = Unavailable desc = connection error: desc = "error reading server preface: EOF"
exit status 1

3. Caddy version:

docker exec -it caddy caddy version

v2.8.4 h1:q3pe0wpBj1OcHFZ3n/1nl4V4bxBrYoSoab7rL9BMYNk=

4. How I installed and ran Caddy:

caddy docker proxyvia terraform docker darklab8/infra/blob/master/tf/modules/caddy/main.tf
connected to labels of app darklab8/fl-darkstat/blob/09aeec13b42763a07fab42ea4910f5088b2701b7/tf/modules/darkstat/main.tf#L48

a. System environment:

Hetzner server

root@node-darklab:/var/lib/caddy/caddy# cat /etc/os-release 
PRETTY_NAME="Ubuntu 24.04.1 LTS"
NAME="Ubuntu"
VERSION_ID="24.04"
VERSION="24.04.1 LTS (Noble Numbat)"
root@node-darklab:/var/lib/caddy/caddy# docker version
Client: Docker Engine - Community
 Version:           27.3.1

b. Command:

as far as docker inspect caddy shows

            "Env": [
                "PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin",
                "XDG_CONFIG_HOME=/config",
                "XDG_DATA_HOME=/data"
            ],
            "Cmd": [
                "docker-proxy"
            ],
            "Entrypoint": [
                "/bin/caddy"
            ],

Probably more useful to see infra code for terraform docker container from infra repo :slight_smile:

c. Service/unit/compose file:

Caddy service:

resource "docker_network" "network" {
  name       = "caddy"
  attachable = true
  driver     = "overlay"
}

resource "docker_image" "caddy" {
  name = "lucaslorentz/caddy-docker-proxy:2.9.1"
}

resource "docker_container" "caddy" {
  name    = "caddy"
  image   = docker_image.caddy.image_id
  restart = "always"

  networks_advanced {
    name = docker_network.network.id
  }

  ports {
    internal = "80"
    external = "80"
  }
  ports {
    internal = "443"
    external = "443"
  }

  volumes {
    host_path      = "/var/run/docker.sock"
    container_path = "/var/run/docker.sock"
    read_only      = true
  }

  volumes {
    volume_name    = "caddy_data"
    container_path = "/data"
  }


  lifecycle {
    ignore_changes = [
      memory_swap,
      network_mode,
    ]
  }
}

My service

resource "docker_network" "network" {
  name       = "darkstat-${var.environment}"
  attachable = true
  driver     = "overlay"
}

locals {
  tag = var.tag != null ? var.tag : var.environment
}

resource "docker_image" "darkstat" {
  name         = "darkwind8/darkstat:${local.tag}"
  keep_locally = true
}

data "docker_network" "caddy" {
  name = "caddy"
}

resource "docker_service" "darkstat" {
  name = "${var.environment}-darkstat-app"

  task_spec {
    networks_advanced {
      name = docker_network.network.id
    }
    networks_advanced {
      name = data.docker_network.caddy.id
    }

    container_spec {
      image = docker_image.darkstat.name
      env   = local.envs
      #   args = ["sleep", "infinity"]

      labels {
        label = "caddy_0"
        value = "${var.stat_prefix}.${var.zone}"
      }
      labels {
        label = "caddy_0.reverse_proxy"
        value = "{{upstreams 8000}}"
      }
      labels {
        label = "caddy_1"
        value = "${var.relay_prefix}.${var.zone}"
      }
      labels {
        label = "caddy_1.reverse_proxy"
        value = "{{upstreams 8080}}"
      }
      labels {
        label = "caddy_2"
        value = "${var.rpc_prefix}.${var.zone}"
      }
      labels {
        label = "caddy_2.reverse_proxy"
        value = "{{upstreams h2c 50051}}"
      }


      mounts {
        target    = "/data"
        source    = var.discovery_path
        type      = "bind"
        read_only = false

        bind_options {
          propagation = "rprivate"
        }
      }
      mounts { // darkstat socks
        target    = "/tmp/darkstat"
        source    = "/tmp/darkstat-${var.environment}"
        type      = "bind"
        read_only = false
        bind_options {
          propagation = "rprivate"
        }
      }
    }
    restart_policy {
      condition = "any"
      delay     = "20s"
    }
    resources {
      limits {
        memory_bytes = 1000 * 1000 * 3000 # 1 gb
      }
    }
  }
  lifecycle {
    ignore_changes = [
      task_spec[0].restart_policy[0].window,
      task_spec[0].container_spec[0].env,
    ]
  }
  # with usage of docker networking, this is no longer necessary
  endpoint_spec {
    mode = "vip"

    ports {
      target_port    = "50051"
      published_port = tostring(var.rpc_port)
    }
  }
}

d. My complete Caddy config:

that was generated by caddy docker proxy from my labels

root@node-darklab:/var/lib/caddy/caddy# cat Caddyfile.autosave 
darkgrpc-dev.dd84ai.com {
        reverse_proxy h2c://10.0.3.21:50051
}
darkgrpc.dd84ai.com {
        reverse_proxy h2c://10.0.3.20:50051
}
darkrelay-dev.dd84ai.com {
        reverse_proxy 10.0.3.21:8080
}
darkrelay.dd84ai.com {
        reverse_proxy 10.0.3.20:8080
}
darkstat-dev.dd84ai.com {
        reverse_proxy 10.0.3.21:8000
}
darkstat.dd84ai.com {
        reverse_proxy 10.0.3.20:8000
}

5. Links to relevant resources:

Site has limit 4 links per post. i removed https github com part from some links going over the limit

So summarizing all the stuff about.
grpcurl -plaintext darkgrpc.dd84ai.com:50051 darkgrpc.DarkGRpc/GetBases // works without caddy. just using docker swarm opened port directly
grpcurl -plaintext darkgrpc.dd84ai.com:443 darkgrpc.DarkGRpc/GetBase // does not work if trying to use caddy to access it

Failed to dial target host "darkgrpc.dd84ai.com:443": context deadline exceeded

With Caddyfile

darkgrpc-dev.dd84ai.com {
        reverse_proxy h2c://10.0.3.21:50051
}

And a bit difficult to write caddy docker proxy settings. I tried outdated some issues written in this forum, but servers { protocols { allow_h2c }}} did not work with error

{"level":"info","ts":1738674206.124626,"logger":"docker-proxy","msg":"Process Caddyfile","logs":"[ERROR]  Removing invalid block: parsing caddyfile tokens for 'servers': unrecognized servers option 'protocol', at Caddyfile:3\n{\n\tservers {\n\t\tprotocol {\n\t\t\tallow_h2c\n\t\t}\n\t}\n}\n\n"}

What could be an up to date option to resolve it? :yum:

P.S. another project already works from same Caddy container at same server, the one with prefix darkstat to the dmain. regular https golang web app. All is accessable okay.

Golang GRPC app was written according to helloworld taken from grpc quickstart for golang
at github path, it is accessable at github com / grpc/grpc-go/tree/master/examples/helloworld

UPD:
Caught some caddy logs

{"level":"info","ts":1738682279.9113512,"logger":"http.log.access","msg":"handled request","request":{"remote_ip":"52.47.113.81","remote_port":"48244","client_ip":"52.47.113.81","proto":"HTTP/2.0","method":"PRI","host":"","uri":"*","headers":{}},"bytes_read":0,"user_id":"","duration":0.00007736,"size":0,"status":308,"resp_headers":{"Connection":["close"],"Location":["https://*"],"Content-Type":[],"Server":["Caddy"]}}
{"level":"info","ts":1738682281.069223,"logger":"http.log.access","msg":"handled request","request":{"remote_ip":"52.47.113.81","remote_port":"48256","client_ip":"52.47.113.81","proto":"HTTP/2.0","method":"PRI","host":"","uri":"*","headers":{}},"bytes_read":0,"user_id":"","duration":0.00004484,"size":0,"status":308,"resp_headers":{"Connection":["close"],"Location":["https://*"],"Content-Type":[],"Server":["Caddy"]}}
{"level":"info","ts":1738682282.712738,"logger":"http.log.access","msg":"handled request","request":{"remote_ip":"52.47.113.81","remote_port":"48270","client_ip":"52.47.113.81","proto":"HTTP/2.0","method":"PRI","host":"","uri":"*","headers":{}},"bytes_read":0,"user_id":"","duration":0.00003696,"size":0,"status":308,"resp_headers":{"Server":["Caddy"],"Connection":["close"],"Location":["https://*"],"Content-Type":[]}}
{"level":"info","ts":1738682285.262502,"logger":"http.log.access","msg":"handled request","request":{"remote_ip":"52.47.113.81","remote_port":"48282","client_ip":"52.47.113.81","proto":"HTTP/2.0","method":"PRI","host":"","uri":"*","headers":{}},"bytes_read":0,"user_id":"","duration":0.000074041,"size":0,"status":308,"resp_headers":{"Server":["Caddy"],"Connection":["close"],"Location":["https://*"],"Content-Type":[]}}
{"level":"info","ts":1738682279.9113512,"logger":"http.log.access","msg":"handled request","request":{"remote_ip":"52.47.113.81","remote_port":"48244","client_ip":"52.47.113.81","proto":"HTTP/2.0","method":"PRI","host":"","uri":"*","headers":{}},"bytes_read":0,"user_id":"","duration":0.00007736,"size":0,"status":308,"resp_headers":{"Connection":["close"],"Location":["https://*"],"Content-Type":[],"Server":["Caddy"]}}
{"level":"info","ts":1738682281.069223,"logger":"http.log.access","msg":"handled request","request":{"remote_ip":"52.47.113.81","remote_port":"48256","client_ip":"52.47.113.81","proto":"HTTP/2.0","method":"PRI","host":"","uri":"*","headers":{}},"bytes_read":0,"user_id":"","duration":0.00004484,"size":0,"status":308,"resp_headers":{"Connection":["close"],"Location":["https://*"],"Content-Type":[],"Server":["Caddy"]}}
{"level":"info","ts":1738682282.712738,"logger":"http.log.access","msg":"handled request","request":{"remote_ip":"52.47.113.81","remote_port":"48270","client_ip":"52.47.113.81","proto":"HTTP/2.0","method":"PRI","host":"","uri":"*","headers":{}},"bytes_read":0,"user_id":"","duration":0.00003696,"size":0,"status":308,"resp_headers":{"Server":["Caddy"],"Connection":["close"],"Location":["https://*"],"Content-Type":[]}}
{"level":"info","ts":1738682285.262502,"logger":"http.log.access","msg":"handled request","request":{"remote_ip":"52.47.113.81","remote_port":"48282","client_ip":"52.47.113.81","proto":"HTTP/2.0","method":"PRI","host":"","uri":"*","headers":{}},"bytes_read":0,"user_id":"","duration":0.000074041,"size":0,"status":308,"resp_headers":{"Server":["Caddy"],"Connection":["close"],"Location":["https://*"],"Content-Type":[]}}

the found logs with 308 status code is some kind of a clue

Found related potentially issue but it is not having a solution

according to this issue by default GRPC is not resolving any redirects

It may be possible with custom Serverinterceptor… but there are no code examples how

You shouldn’t use -plaintext with HTTPS/443 because HTTPS is not plaintext.

Fascinating :smile:
grpcurl darkgrpc.dd84ai.com:443 darkgrpc.DarkGRpc/GetBases

Just in such simple step away from happiness :slight_smile: thanks.

My main issues then apperently at client golang code then to perform https i guess connection auth correctly. Could be good not having automated https may be, so i could connect without https/tls stuff

A big vague how to turn it off for h2c to test the solution

Anyway… that is already way easier resolvable, now that i know that my issues only in TLS Auth authorizations for gRPC

t$ grpcurl -plaintext darkgrpc.dd84ai.com:80 darkgrpc.DarkGRpc/GetBases
Failed to dial target host "darkgrpc.dd84ai.com:80": context deadline exceeded
http://darkgrpc.dd84ai.com, https://darkgrpc.dd84ai.com {
        encode zstd gzip
        log {
                output stdout
                format json
                level DEBUG
        }
        reverse_proxy {
                to h2c://10.0.3.50:50051
                transport http {
                        versions h2c 2
                }
        }
}

this one did not help

http://darkgrpc.dd84ai.com {
   encode zstd gzip
   log {
      output stdout
      format json
      level DEBUG
    }

    reverse_proxy {
      to h2c://10.0.3.50:50051
      transport http {
          versions h2c 2
      }
    }
}

This one did not work too

 grpcurl -plaintext darkgrpc.dd84ai.com:80 darkgrpc.DarkGRpc/GetBases
Failed to dial target host "darkgrpc.dd84ai.com:80": context deadline exceeded

not passing checks

{
    auto_https disable_redirects
}

darkgrpc.dd84ai.com:80, darkgrpc.dd84ai.com:888 {
   log {
      output stdout
      format json
      level DEBUG
    }

    reverse_proxy h2c://production-darkstat-app:50051
}

That one too.
Works with
grpcurl darkgrpc.dd84ai.com:888 darkgrpc.DarkGRpc/GetBases
does not work with grpcurl -plaintext darkgrpc.dd84ai.com:888 darkgrpc.DarkGRpc/GetBases

:see_no_evil: i tried disabling TLS in all the ways. i can’t turn it off for h2c.
This is why default golang client i think not able to connect. That is proved by
grpcurl darkgrpc.dd84ai.com:888 darkgrpc.DarkGRpc/GetBases not able to work with -plaintext argument, but it works with it.

I FOUND THE SOLUTION!!!

  client := credentials.NewTLS(&tls.Config{InsecureSkipVerify: false}) // to use any TLS, caddy one too.
    // creds := insecure.NewCredentials() // for darkstat.dd84ai.com:50051, only without tls!
  conn, err := grpc.NewClient(*addr, grpc.WithTransportCredentials(creds))

in this way we can turn off Golang client default TLS auth, yet having it working with Caddy not disablable TLS for h2c reverseproxy!

Also for a future, more logging on golang side can be turn on with env vars like

                "GRPC_GO_LOG_SEVERITY_LEVEL": "info",
                "GRPC_GO_LOG_VERBOSITY_LEVEL": "6",
                "GRPC_TRACE": "all",
                "GRPC_VERBOSITY": "DEBUG",

Final caddyfile is used just

darkgrpc.dd84ai.com {
    reverse_proxy h2c://production-darkstat-app:50051
}

left without tls port at 50051 opened for the app through Docker means directly to by pass caddy stuff, for other people debugging sanity :slight_smile:

1 Like

Caddy doesn’t enable h2c for external clients by default. It can be enabled, but I’m not sure I should share it to not encourage its use :wink: Forcing HTTPS forced you to learn how to amend your app to connect properly, right :slight_smile:

@Mohammed90 Better to share :stuck_out_tongue: Infra devs like a bit of debugging sanity. gRPC is very small in amount of examples represented for caddy in general. More examples would help more using gRPC with caddy.

Also it would be nice if was possible having an option of h2c working without tls at 80th port, and only at 443 port with tls.
For the purpose of making grpc API friendly for everyone trying to use it.

The api in question just dumps all game data and has no data sensitivity. I would prefer there was an option of its usage without tls just for the sake of… Any level of dev able to use it. The community in question is very low level in skills, unit testing is already concept beyond their grasp. Anything that makes things more friendly for them is worthy to add
I would wish encourage them usage of gRPC and make it as most friendly as it is maximum possible. :innocent:
I am forced to leave open TLS free port by docker directly for those goals of user friendliness. it would be nice to avoid needing host direct port binding and routing everything through caddy

Very questionable if it is possible at this point though. Apperently Caddy pretends to dial TCL if using plaintext h2c? eh. Smth weird is happening

Holy Jesus. I made it

{
  servers {
    protocols h1 h2 h2c
  }
}

darkgrpc.dd84ai.com:80, darkgrpc.dd84ai.com:888 {
    reverse_proxy {
      to production-darkstat-app:50051
      transport http {
          versions h1 h2c
      }
    }
}

darkgrpc.dd84ai.com:443 {
    reverse_proxy {
      to h2c://production-darkstat-app:50051
      transport http {
          versions h2c 2
      }
    }
}

For some reason In this way config writing, plaintext gcrp works
grpcurl -plaintext darkgrpc.dd84ai.com:80 darkgrpc.DarkGRpc/GetBases
And it does not work even if u just remove servers section

Final version:

{
  servers {
    protocols h1 h2 h2c
  }
}

darkgrpc.dd84ai.com:80 {
    reverse_proxy {
      to production-darkstat-app:50051
      transport http {
          versions h1 h2c
      }
    }
}

darkgrpc.dd84ai.com:443 {
    reverse_proxy h2c://production-darkstat-app:50051
}

This one works for Caddy to make gRCP working for both without TCL and with TCL
both for grpcurl and for golang client :smirk:
Any deviation and it will not work already highly likely

  • server protocols h1 h2 h2c must be enabled
  • 80 and 443 instructions must be separately
  • at 80 port h1/h2c instructions need to be specified for Golang client to work with it (the one from quickstart helloworld for grpc
	// creds := credentials.NewTLS(&tls.Config{InsecureSkipVerify: false}) // for darkstat.dd84ai.com:443
	creds := insecure.NewCredentials() // for darkstat.dd84ai.com:80
	conn, err := grpc.NewClient(*addr, grpc.WithTransportCredentials(creds))

The final version is able to use those both methods of golang client auth.
And grpcurl works both in -plaintext and without it

1 Like