Reverse Proxy H2C Stoped to Function

[mbn18 (Michael Ben-Nes)]

1. Caddy version (caddy version):

Docker caddy:2.2.1-alpine

Also tested with v2.4.5

2. How I run Caddy:

Using docker

a. System environment:

Docker system on Ubuntu

b. Command:

docker-compose up

c. compose file:

docker-compose file:

version: "3.2"
services:
  api:
    container_name: mapper-api
    image: docker-repo-/mapper-api:v1.2.3
    volumes:
      - /tmp:/host
    ports:
      - 8008:8008
    restart: on-failure
    command: serve -dgAddr 172.31.0.2:9080
    networks:
      - api_net
  gw:
    container_name: caddy
    image: caddy:2.2.1-alpine
    volumes:
      - /tmp:/host
      - ${PWD}/Caddyfile:/etc/caddy/Caddyfile:ro
      - ${PWD}/caddy_data:/data
      - ${PWD}/caddy_config:/config
    ports:
      - 443:443
    restart: on-failure
    networks:
      - api_net
networks:
  api_net:
    driver: bridge

d. My complete Caddyfile or JSON config:

# The Caddyfile is an easy way to configure your Caddy web server.
#
# Unless the file starts with a global options block, the first
# uncommented line is always the address of your site.
#
# To use your own domain name (with automatic HTTPS), first make
# sure your domain's A/AAAA DNS records are properly pointed to
# this machine's public IP, then replace the line below with your
# domain name.
grpc.mapper.com

log {
    level INFO
}

reverse_proxy {
    to h2c://api:8008
    transport http {
        versions h2c 2
    }
}

# Refer to the Caddy docs for more information:
# https://caddyserver.com/docs/caddyfile

3. The problem I’m having:

This configuration worked since Jan 2021.
It listen to port 443 then using h2c it forward the request to the grpc server in clear-text.

Last day (Sep 30) around 5pm GMT grpc cli started to return error:

{
  "error": "14 UNAVAILABLE: failed to connect to all addresses"
}

• I validated the cert and that the port is open (using firefox), and it is open and valid till end of Dec.
• I opened the port to the insecure endpoint (skipping caddy) and the grpc server behaved correctly.
• The dockers was not restarted or changed in the last month
• I wiped the caddy data dir and restarted to get a new cert/reset/whatever. Didn’t helped.
• Restarted other dockers / upgrade caddy to latest. None helped.

4. Error messages and/or full log output:

Reverse proxy to GRPC didn’t yield any logs whatsoever and that’s since I started to use Caddy.
Did tried to change Caddy log level ro DEBUG. nothing changed.
Maybe the log directive in my Caddy is incorrect???

6. Links to relevant resources:

My initial discuss on how to implament h2c:

Reverse proxy to H2C GRPC

1. Caddy version (caddy version): Tried both 2.2.1 & 2.3 using the caddy alpine docker 2. How I run Caddy: start docker: docker run --rm -it -h node2 --name node2 --network my_net -p 443:443 -v $PWD/Caddyfile:/etc/caddy/Caddyfile -v caddy_data:/data --link node1 caddy:2.3.0-alpine a. System environment: docker on fedora linux: Client: Docker Engine - Community Version: 19.03.8 API version: 1.40 Go version: go1.12.17 Git commit: afacb8b Built: …

Another discussion about h2c:

Caddy GRPC H2C Passthrough

1. Caddy version (caddy version): caddy-2.3.0-alpine 2. How I run Caddy: In a docker container, on Docker Desktop for Windows using WSL2. Proxying to a GRPC service running on server:5300, defined in the docker-compose via which caddy is run. (Caddy also proxies the HTTP API on server:8000 which works great) a. System environment: Docker Desktop For Windows using WSL2. b. Command: c. Service/unit/compose file: d. My complete Caddyfile or JSON config: :80 { route /host.Service/* { …

[francislavoie]

@mbn18

reverse_proxy {
    to h2c://api:8008
    transport http {
        versions h2c 2
    }
}

FYI you can simplify this to just this config:

reverse_proxy h2c://api:8008

The h2c:// scheme is a shortcut to setting versions.

@mbn18

Last day (Sep 30) around 5pm GMT grpc cli started to return error:

{
  "error": "14 UNAVAILABLE: failed to connect to all addresses"
}

Doesn’t sound like an issue with Caddy, then. If you didn’t change anything in Caddy, it can’t be a problem with Caddy.

@mbn18

Did tried to change Caddy log level ro DEBUG. nothing changed.

You need to set the debug global option, not change the log directive’s level – the log directive only configures access logs. Additional debug logs are not emitted via access logging.

Add this at the top of your Caddyfile (before the site address):

{
	debug
}

Global options (Caddyfile) - Caddy Documentation

[mbn18 (Michael Ben-Nes)]

Thanks for the tips.

Turned out the the problem was with an outdated root certificate files that come as part of the grpc stack and is used by most grpc clients/servers implementation ( ref ).

[francislavoie]

Ah, that makes sense. Thanks for following up!