Http2: Transport: cannot retry err [http2: Transport received Server's graceful shutdown GOAWAY] after Request.Body was written; define Request.GetBody to avoid this error

1. The problem I’m having:

Hey Everyone! I’m trying to configure Caddy to be a load balancer for multiple solana nodes I’m using (if you know what that is). Configuration seems valid and it works 95% of the time, but sometimes the Caddy fails. I’m not sure what the reason for that is, I’m assuming that Batch Reqests (around 200-300 simultanious req) that I sometimes send have something to do with it. I’d be really glad if somebody can help me out, I don’t want to go back to Nginx :wink:

2. Error messages and/or full log output:

Feb 11 10:45:31 hetzner-cloud-test caddy[905]: {"level":"error","ts":1739270731.0617454,"logger":"http.log.error.log0","msg":"http2: Transport: cannot retry err [http2: Transport received Server's graceful shutdown GOAWAY] after Request.Body was written; define Request.GetBody to avoid this error","request":{"remote_ip":"<private>","remote_port":"54450","client_ip":"<private>","proto":"HTTP/1.1","method":"POST","host":"example.com","uri":"/","headers":{"Content-Type":["application/json"],"Solana-Client":["js/1.0.0-maintenance"],"Connection":["keep-alive"],"User-Agent":["Bun/1.2.2"],"Accept":["*/*"],"Accept-Encoding":["gzip, deflate, br"],"Content-Length":["168"]},"tls":{"resumed":false,"version":772,"cipher_suite":4865,"proto":"http/1.1","server_name":"example.com"}},"duration":0.004944987,"status":502,"err_id":"fp93xhb76","err_trace":"reverseproxy.statusError (reverseproxy.go:1373)"}

3. Caddy version:

v2.9.1

4. How I installed and ran Caddy:

sudo apt install -y debian-keyring debian-archive-keyring apt-transport-https curl
curl -1sLf 'https://dl.cloudsmith.io/public/caddy/stable/gpg.key' | sudo gpg --dearmor -o /usr/share/keyrings/caddy-stable-archive-keyring.gpg
curl -1sLf 'https://dl.cloudsmith.io/public/caddy/stable/debian.deb.txt' | sudo tee /etc/apt/sources.list.d/caddy-stable.list
sudo apt update
sudo apt install caddy

a. System environment:

Distributor ID: Ubuntu
Description: Ubuntu 24.04.1 LTS
Release: 24.04
Codename: noble
systemd 255 (255.4-1ubuntu8.5)

b. Command:

I’m not sure what do I need to state here. I run caddy.service unit file via systemctl

d. My complete Caddy config:

example.com {
        log {
                output stdout
                format console {
                        time_format common_log
                        time_local
                }
                level WARN
        }

        # First enforce IP whitelist for ALL requests
        @allowed_ips remote_ip 127.0.0.1 # REPLACE WITH YOUR IPS
        handle @allowed_ips {
                # Protected reverse proxy configuration
                rewrite ?api-key=<private>
                reverse_proxy https://mainnet.helius-rpc.com https://rpc.shyft.to {
                        header_up Host {upstream_hostport}
                        header_down X-Backend-Host {upstream_hostport}

                        lb_policy least_conn

                        transport http {
                                tls
                        }
                }
        }

        # Block all non-whitelisted IPs
        handle {
                respond "Access denied" 403
        }
}

I’m not entirely sure why it looks like the request gets dropped or doesn’t finish. I wonder if one is unavailable and so it drops.

Try adding health check directives to your Caddyfile. I’d also add load balancing if you can.

Hi!
No, all nodes are healthy, they are really reliable.
Caddy health checks are not flexible so I can’t configure them for JSON RPC APIs.
I’m already using load balancing, it’s in the config

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.