Uploading large files from Caddy to Nginx results in 499 - 502

[aiozufh (Caddy'Fellow)]

1. The problem I’m having:

I’m using Caddy as proxy, while Nginx + PHP-FPM is the receiver side. While uploading large files, Caddy reports 502 and Nginx 499.

What i don’t understand here is why Caddy is reporting an upstream fault (502) and nginx a client, so Caddy, fault, with 499.

2. Error messages and/or full log output:

Caddy and Nginx log line, respectively:

{"level":"error","ts":1755818208.0664213,"logger":"http.log.error.log1","msg":"readfrom tcp 100.74.154.104:47696->100.82.141.56:443: write tcp 100.74.154.104:47696->100.82.141.56:443: write: connection reset by peer","request":{"remote_ip":"212.114.19.17","remote_port":"54291","client_ip":"212.114.19.17","proto":"HTTP/1.1","method":"PUT","host":"my.server.com","uri":"/remote.php/dav/files/admin/record%20-%2017%20juillet%202023%20-%2001-52-24%20.mp4","headers":{"Authorization":["REDACTED"],"Accept":["*/*"],"Content-Length":["13324642653"],"Expect":["100-continue"]},"tls":{"resumed":false,"version":772,"cipher_suite":4865,"proto":"http/1.1","server_name":"my.server.com"}},"duration":57.773655214,"status":502,"err_id":"vdbien4zv","err_trace":"reverseproxy.statusError (reverseproxy.go:1390)"}

PUT /remote.php/dav/files/admin/record%20-%2017%20juillet%202023%20-%2001-52-24%20.mp4 499

3. Caddy version:

caddy --version

v2.10.0 h1:fonubSaQKF1YANl8TXqGcn4IbIRUDdfAkpcsfI/vX5U=

4. How I installed and ran Caddy:

Trough APT repositories https://dl.cloudsmith.io/public/caddy/stable/deb/debian

a. System environment:

Ubuntu 22.04 LTS

b. Command:

Unrelated

c. Service/unit/compose file:

PASTE OVER THIS, BETWEEN THE ``` LINES.
Please use the preview pane to ensure it looks nice.

d. My complete Caddy config:

# GLOBAL options
{
        #debug
        email server@email.com

        shutdown_delay 30s

        cert_issuer acme
        renew_interval 30m
        ocsp_stapling off

        storage file_system {
                root /mnt/caddyvol/caddy
        }

        servers :80 {
                strict_sni_host on
                protocols h1
                trusted_proxies static 10.0.0.0/16
                trusted_proxies_strict

                timeouts {
                        read_body 5s
                        read_header 5s
                        write 5s
                        idle 10s
                }

                listener_wrappers {
                        proxy_protocol {
                                allow 10.0.0.0/16
                                fallback_policy reject
                        }
                }
        }

        servers :443 {
                strict_sni_host on
                protocols h1
                trusted_proxies static 10.0.0.0/16
                trusted_proxies_strict
                enable_full_duplex

                listener_wrappers {
                        proxy_protocol {
                                allow 10.0.0.0/16
                                fallback_policy require
                        }
                        http_redirect
                        tls
                }
        }

        on_demand_tls {
                ask http://ask.localhost/check
        }
}

# auto_https "ASK" Server
import AllowedTLSDomains

# Common server parameters
(server_options) {
        header -Server
        header -Via
        tls {
                on_demand
        }

        log {
                output file /var/log/caddy/error.log
                level ERROR
        }
}

# Common options applied to Nextcloud servers
(nextcloud_config) {
        redir /.well-known/carddav /remote.php/dav 301
        redir /.well-known/caldav /remote.php/dav 301
        redir /.well-known/webfinger /index.php/.well-known/webfinger 301
        redir /.well-known/nodeinfo /index.php/.well-known/nodeinfo 301
}

(default_headers) {
        header * {
                Content-Security-Policy "default-src 'self'; style-src 'self' 'unsafe-inline';"
                Permissions-Policy "accelerometer=(self), ambient-light-sensor=(self), autoplay=(self), battery=(self), camera=(self), cross-origin-isolated=(self), display-capture=(self), document-domain=(self), encrypted-media=(self), execution-while-not-rendered=(self), execution-while-out-of-viewport=(self), fullscreen=(self), geolocation=(self), gyroscope=(self), keyboard-map=(self), magnetometer=(self), microphone=(self), midi=(self), navigation-override=(self), payment=(self), picture-in-picture=(self), publickey-credentials-get=(self), screen-wake-lock=(self), sync-xhr=(self), usb=(self), web-share=(self), xr-spatial-tracking=(self)"
                Strict-Transport-Security "max-age=31536000; includeSubDomains; preload;"
                Referrer-Policy "no-referrer"
                X-Content-Type-Options "nosniff"
                X-Frame-Options "SAMEORIGIN"
                X-Download-Options "noopen"
                X-Permitted-Cross-Domain-Policies "none"
                X-XSS-Protection "1; mode=block"
        }
}

# Proxy Options
(proxy_options) {
        reverse_proxy {
                to {args[0]}

                stream_close_delay 8h
                stream_timeout 24h
                flush_interval -1

                transport http {
                        versions 1.1
                        keepalive off
                        proxy_protocol v2
                        write_buffer 128k
                        dial_timeout 10s
                }
        }
}

# Common ports (no response)
:80 {
        abort
}

:443 {
        abort
}

# server sub-domains
*.server.com,
server.com {
        import server_options

        @rootredirect host www.server.com
        handle @rootredirect {
                redir https://server.com{uri} 308
        }

        @my host my.server.com
        handle @my {
                import nextcloud_config
                import proxy_options 100.65.100.74:443
        }
}

5. Links to relevant resources:

[aiozufh (Caddy'Fellow)]

Fixed upstream, there is a LBaaS in front of Caddy.