Proxying Argo CD grpc no reply

Help
1

1. The problem I’m having:

I am setting up Argo CD via Caddy. Argo CD is deployed in K8s and when port forwarding directly to the Nginx Ingress all protocols are working. Problem here is Argo’s mixing of protocols that I can’t seem to figure out.

If I access Argo CD via the Caddy proxy, all http requests are working but gRPC requests never get a response, and usually they are done and closed within tenths of a second.

2. Error messages and/or full log output:

{"level":"error","ts":1747322643.351713,"logger":"http.handlers.reverse_proxy","msg":"aborting with incomplete response","upstream":"10.1.3.232:443","duration":4.362483883,"request":{"remote_ip":"178.174.139.42","remote_port":"50780","client_ip":"178.174.139.42","proto":"HTTP/2.0","method":"GET","host":"dev.corporate.com","uri":"/test/argo-cd/api/v1/stream/applications/pdf-creator-service/resource-tree?appNamespace=","headers":{"Accept":["text/event-stream"],"Cookie":[],"Priority":["u=1, i"],"Cache-Control":["no-cache"],"Accept-Language":["en-US,en;q=0.9,sv;q=0.8,pt;q=0.7"],"X-Forwarded-For":["178.174.139.42"],"User-Agent":["Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/136.0.0.0 Safari/537.36"],"Referer":["dev.corporate.com/test/argo-cd/applications/pdf-creator-service?resource="],"X-Forwarded-Proto":["https"],"X-Forwarded-Host":["dev.corporate.com"],"Sec-Ch-Ua":["\"Chromium\";v=\"136\", \"Google Chrome\";v=\"136\", \"Not.A/Brand\";v=\"99\""],"Sec-Ch-Ua-Mobile":["?0"],"Sec-Fetch-Mode":["cors"],"Sec-Fetch-Dest":["empty"],"Accept-Encoding":["gzip, deflate, br, zstd"],"Sec-Ch-Ua-Platform":["\"Linux\""],"Sec-Fetch-Site":["same-origin"]},"tls":{"resumed":true,"version":772,"cipher_suite":4865,"proto":"h2","server_name":"dev.corporate.com","client_common_name":"dev.corporate.com","client_serial":"5330010024230036372"}},"error":"reading: context canceled"}

3. Caddy version:

2.7.8 Docker build

4. How I installed and ran Caddy:

Docker containers in Azure.

d. My complete Caddy config:

We have quite a volume of endpoints… so, no. :slight_smile:

First I tried this setup:

handle /test/argo-cd/* {
  handle /api/v1/stream/* {
          reverse_proxy h2c://10.1.3.232 {
            transport http {
              versions h2c 2  # With and without this transport block
            }
          }
  }

  handle {
          reverse_proxy https://10.1.3.232 {
                  transport http {
                          tls_insecure_skip_verify
                  }
          }
  }
}

Then I tried:

        @grpctestargocd {
                protocol grpc
                path /test/argo-cd/*
        }
        handle @grpctestargocd {
                reverse_proxy h2://10.1.3.232  # also tried h2c:// with no success
        }
        handle /test/argo-cd/* {
                handle {
                        reverse_proxy https://10.1.3.232 {
                                transport http {
                                        tls_insecure_skip_verify
                                }
                        }
                }
        }

But unfortunately I can’t get the gRPC calls to run successfully via Caddy.

Requests like:

https://dev.corporate.com/test/argo-cd/api/v1/stream/applications/customer-creator-service/resource-tree?appNamespace=
-----
:method GET
:path /test/argo-cd/api/v1/stream/applications/pdf-creator-service/resource-tree?appNamespace=
:scheme https
accept text/event-stream
accept-encoding gzip, deflate, br, zstd
accept-language en-US,en;q=0.9,sv;q=0.8,pt;q=0.7
cache-control no-cache

is unfortunately no bueno.
What should I do here?

5. Links to relevant resources:

2

Please upgrade to latest

Please enable debug logs and share more than this line. It doesn’t seem to be a gRPC request.

Does Argo know it’s served on a sub-path?

What do you see in Argo logs?

3

Yes, Argo CD works fine on the root path when I port-forward directly to the cluster ingress.

That log line is actually what Caddy logs when the request is being proxied. Unfortunately I can’t play around too much with this environment as it is running production load. Gotta see if I can setup a test environment.

4

Change this to handle_path

5

That will not do the trick. The backend server is serving on the same path and http requests are coming through. It is the gRPC-calls that fails, and they are on the same root path.

6

Well, we need more log lines around the issue, and to know what’s ArgoCD seeing through its logs.