Cloudflare randomly receiving 502

1. The problem I’m having:

Cloudflare is randomly receiving 502 from my webserver.

2. Error messages and/or full log output:

{
    "level": "error",
    "ts": 1715226311.48635,
    "logger": "http.log.error",
    "msg": "EOF",
    "request": {
        "remote_ip": "172.71.242.74",
        "remote_port": "62390",
        "client_ip": "",
        "proto": "HTTP/2.0",
        "method": "POST",
        "host": "example.domain.com",
        "uri": "/api/example/provider",
        "headers": {
            "Accept-Encoding": [
                "gzip, br"
            ],
            "X-Forwarded-For": [
                "156.208.176.145"
            ],
            "Content-Length": [
                "2769"
            ],
            "X-Forwarded-Proto": [
                "https"
            ],
            "Cf-Visitor": [
                "{\"scheme\":\"https\"}"
            ],
            "Sec-Ch-Ua-Mobile": [
                "?0"
            ],
            "Accept": [
                "*/*"
            ],
            "Origin": [
                "https://example.domain.com"
            ],
            "Referer": [
                "https://example.domain.com/663917b80afa0ff0351b5384"
            ],
            "Accept-Language": [
                "en-US,en;q=0.9"
            ],
            "Priority": [
                "u=1, i"
            ],
            "Cf-Ipcountry": [
                "EG"
            ],
            "Sec-Ch-Ua": [
                "\"Not-A.Brand\";v=\"99\", \"Chromium\";v=\"124\""
            ],
            "Dnt": [
                "1"
            ],
            "User-Agent": [
                "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/124.0.0.0 Safari/537.36"
            ],
            "Sec-Fetch-Mode": [
                "cors"
            ],
            "Sec-Fetch-Dest": [
                "empty"
            ],
            "Cdn-Loop": [
                "cloudflare"
            ],
            "Cf-Ray": [
                "880eb1fc186123b3-LHR"
            ],
            "Sec-Fetch-Site": [
                "same-origin"
            ],
            "Cookie": [],
            "Sec-Ch-Ua-Platform": [
                "\"macOS\""
            ],
            "Content-Type": [
                "text/plain;charset=UTF-8"
            ],
            "Cf-Connecting-Ip": [
                "156.208.176.145"
            ]
        },
        "tls": {
            "resumed": false,
            "version": 772,
            "cipher_suite": 4865,
            "proto": "h2",
            "server_name": "example.domain.com"
        }
    },
    "duration": 0.345228739,
    "status": 502,
    "err_id": "66u574fpk",
    "err_trace": "reverseproxy.statusError (reverseproxy.go:1267)"
}

3. Caddy version:

v2.7.6

4. How I installed and ran Caddy:

apt install -y debian-keyring debian-archive-keyring apt-transport-https
curl -1sLf 'https://dl.cloudsmith.io/public/caddy/stable/gpg.key' | gpg --dearmor -o /usr/share/keyrings/caddy-stable-archive-keyring.gpg
curl -1sLf 'https://dl.cloudsmith.io/public/caddy/stable/debian.deb.txt' | tee /etc/apt/sources.list.d/caddy-stable.list
apt update
apt install caddy
# Download and install go
snap install go --channel=1.22/stable --classic
go version
# Install xcaddy
curl -1sLf 'https://dl.cloudsmith.io/public/caddy/xcaddy/gpg.key' | sudo gpg --dearmor -o /usr/share/keyrings/caddy-xcaddy-archive-keyring.gpg
curl -1sLf 'https://dl.cloudsmith.io/public/caddy/xcaddy/debian.deb.txt' | sudo tee /etc/apt/sources.list.d/caddy-xcaddy.list
apt update
apt install xcaddy
sudo xcaddy build --output /usr/bin/caddy.custom --with github.com/WeidiDeng/caddy-cloudflare-ip 
# Prioritize custom caddy build over default
sudo dpkg-divert --divert /usr/bin/caddy.default --rename /usr/bin/caddy
sudo update-alternatives --install /usr/bin/caddy caddy /usr/bin/caddy.default 10
sudo update-alternatives --install /usr/bin/caddy caddy /usr/bin/caddy.custom 50
sudo systemctl restart caddy

a. System environment:

Ubuntu 22.04, systemd

b. Command:

PASTE OVER THIS, BETWEEN THE ``` LINES.
Please use the preview pane to ensure it looks nice.

c. Service/unit/compose file:

d. My complete Caddy config:

{
    servers {
        metrics
        trusted_proxies cloudflare {
                interval 12h
                timeout 15s
    }
    }
}


example.domain.com {

    # Define a matcher for restricted paths
    @restricted path /api/internal/*
    handle @restricted {
            abort
    }
    reverse_proxy :5001
}

5. Links to relevant resources:

Howdy @yehiaabdelm,

Your setup looks pretty reasonable.

I’d consider adding the debug global option to your Caddyfile in order to get reverse proxy request roundtrips to your upstream logged, which will help determine the exact nature of the problem.

This essentially means your upstream stopped writing data before Caddy expected it to.

The debug logs might show more info.

Thank you. I’ve added the debug statement. I haven’t gotten the 502 again, so I’ll report back when it happens again.