Caddy not reverse proxying large requests to Fast API server

1. The problem I’m having:

I have a caddyFile running to do https reverse proxy to a fast api (python) server, everything works fine most of the time, however my problem is sometimes when the payload is large (about 500KB - 1MB of json strings), my gunicorn server never picks up the request from caddy, i don’t see logs on the gunicorn logfile (for these large requests, i do see logs for succesfull small ones) and this is the last log from caddy:

DEBUG   http.handlers.reverse_proxy     selected upstream       {"dial": "localhost:8000", "total_upstreams": 1}
DEBUG   http.handlers.reverse_proxy     upstream roundtrip      {"upstream": "localhost:8000", "duration": 1.232420955, "request": {"remote_ip": "", "remote_port": "59966", "proto": "HTTP/3.0", "method": "POST", "host": "api.chatshape.com", "uri": "/embed/", "headers": {"X-Forwarded-For": ["104.173.33.232"], "X-Forwarded-Host": ["api.chatshape.com"], "User-Agent": ["Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/111.0.0.0 Safari/537.36"], "Sec-Fetch-Dest": ["empty"], "Content-Type": ["application/json"], "Accept-Encoding": ["gzip, deflate, br"], "Accept": ["application/json"], "X-Forwarded-Proto": ["https"], "Origin": ["chrome-extension://mmlalgciccemmiggjhgkhmklpmpficen"], "Sec-Fetch-Site": ["none"], "Sec-Fetch-Mode": ["cors"], "Accept-Language": ["en-US,en;q=0.9"]}, "tls": {"resumed": true, "version": 772, "cipher_suite": 4865, "proto": "h3", "server_name": "api.chatshape.com"}}, "headers": {"Access-Control-Allow-Credentials": ["true"], "Date": ["Fri, 24 Mar 2023 05:30:24 GMT"], "Server": ["uvicorn"], "Content-Length": ["77"], "Content-Type": ["application/json"], "Access-Control-Allow-Origin": ["*"]}, "status": 307}

Im trying to figure out if this is a caddy problem or my python gunicorn server problem, theres no specific logs in the python one so I’m assuming its the caddy layer that is the issue.

heres a successfull series of caddy logs in comparison (For smaller requests that succeed):

2023/03/24 05:47:06.731 DEBUG   http.handlers.reverse_proxy     upstream roundtrip      {"upstream": "localhost:8000", "duration": 0.001345782, "request": {"remote_ip": "", "remote_port": "59966", "proto": "HTTP/3.0", "method": "POST", "host": "api.chatshape.com", "uri": "/embed", "headers": {"Accept-Language": ["en-US,en;q=0.9"], "Content-Type": ["application/json"], "Origin": ["chrome-extension://mmlalgciccemmiggjhgkhmklpmpficen"], "X-Forwarded-For": ["104.173.33.232"], "X-Forwarded-Proto": ["https"], "Accept": ["application/json"], "Sec-Fetch-Dest": ["empty"], "Sec-Fetch-Mode": ["cors"], "Accept-Encoding": ["gzip, deflate, br"], "X-Forwarded-Host": ["api.chatshape.com"], "User-Agent": ["Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/111.0.0.0 Safari/537.36"], "Sec-Fetch-Site": ["none"]}, "tls": {"resumed": true, "version": 772, "cipher_suite": 4865, "proto": "h3", "server_name": "api.chatshape.com"}}, "headers": {"Date": ["Fri, 24 Mar 2023 05:47:06 GMT"], "Server": ["uvicorn"], "Content-Length": ["0"], "Location": ["https://api.chatshape.com/embed/"], "Access-Control-Allow-Origin": ["*"], "Access-Control-Allow-Credentials": ["true"]}, "status": 307}
2023/03/24 05:47:06.816 DEBUG   http.handlers.reverse_proxy     selected upstream       {"dial": "localhost:8000", "total_upstreams": 1}
2023/03/24 05:47:08.114 DEBUG   http.handlers.reverse_proxy     upstream roundtrip      {"upstream": "localhost:8000", "duration": 1.298061824, "request": {"remote_ip": "", "remote_port": "59966", "proto": "HTTP/3.0", "method": "POST", "host": "api.chatshape.com", "uri": "/embed/", "headers": {"Sec-Fetch-Dest": ["empty"], "Accept-Encoding": ["gzip, deflate, br"], "Accept-Language": ["en-US,en;q=0.9"], "User-Agent": ["Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/111.0.0.0 Safari/537.36"], "Content-Type": ["application/json"], "X-Forwarded-For": ["104.173.33.232"], "Sec-Fetch-Site": ["none"], "Origin": ["chrome-extension://mmlalgciccemmiggjhgkhmklpmpficen"], "Sec-Fetch-Mode": ["cors"], "X-Forwarded-Proto": ["https"], "X-Forwarded-Host": ["api.chatshape.com"], "Accept": ["application/json"]}, "tls": {"resumed": true, "version": 772, "cipher_suite": 4865, "proto": "h3", "server_name": "api.chatshape.com"}}, "headers": {"Access-Control-Allow-Credentials": ["true"], "Date": ["Fri, 24 Mar 2023 05:47:06 GMT"], "Server": ["uvicorn"], "Content-Length": ["77"], "Content-Type": ["application/json"], "Access-Control-Allow-Origin": ["*"]}, "status": 200}

3. Caddy version:

v2.6.4 h1:2hwYqiRwk1tf3VruhMpLcYTg+11fCdr8S3jhNAdnPy8=

4. How I installed and ran Caddy:

used Install — Caddy Documentation installed for linux thru command line, i run caddy with the below caddy file

b. Command:

caddy start

d. My complete Caddy config:

{
        debug
}

api.chatshape.com {
        reverse_proxy localhost:8000

        log {
                output file /var/log/caddy/access.log {
                        roll_size 100MiB
                        roll_keep 10
                        roll_keep_for 720h
                }
        }
}

:8080 {
        respond "Hello, world!"
}

As far as I know, gunicorn has some weird characteristics that make it that request/response buffering in Caddy is sometimes necessary for it to behave properly. Try enabling request buffering:

Just found the issue, it was a fastapi issue where i added my route as “/embed/” instead of it should be “/embed” , not caddys fault

One trailing slash cost me days LOL

1 Like

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.