1. The problem I’m having:
I have a caddyFile running to do https reverse proxy to a fast api (python) server, everything works fine most of the time, however my problem is sometimes when the payload is large (about 500KB - 1MB of json strings), my gunicorn server never picks up the request from caddy, i don’t see logs on the gunicorn logfile (for these large requests, i do see logs for succesfull small ones) and this is the last log from caddy:
DEBUG http.handlers.reverse_proxy selected upstream {"dial": "localhost:8000", "total_upstreams": 1}
DEBUG http.handlers.reverse_proxy upstream roundtrip {"upstream": "localhost:8000", "duration": 1.232420955, "request": {"remote_ip": "", "remote_port": "59966", "proto": "HTTP/3.0", "method": "POST", "host": "api.chatshape.com", "uri": "/embed/", "headers": {"X-Forwarded-For": ["104.173.33.232"], "X-Forwarded-Host": ["api.chatshape.com"], "User-Agent": ["Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/111.0.0.0 Safari/537.36"], "Sec-Fetch-Dest": ["empty"], "Content-Type": ["application/json"], "Accept-Encoding": ["gzip, deflate, br"], "Accept": ["application/json"], "X-Forwarded-Proto": ["https"], "Origin": ["chrome-extension://mmlalgciccemmiggjhgkhmklpmpficen"], "Sec-Fetch-Site": ["none"], "Sec-Fetch-Mode": ["cors"], "Accept-Language": ["en-US,en;q=0.9"]}, "tls": {"resumed": true, "version": 772, "cipher_suite": 4865, "proto": "h3", "server_name": "api.chatshape.com"}}, "headers": {"Access-Control-Allow-Credentials": ["true"], "Date": ["Fri, 24 Mar 2023 05:30:24 GMT"], "Server": ["uvicorn"], "Content-Length": ["77"], "Content-Type": ["application/json"], "Access-Control-Allow-Origin": ["*"]}, "status": 307}
Im trying to figure out if this is a caddy problem or my python gunicorn server problem, theres no specific logs in the python one so I’m assuming its the caddy layer that is the issue.
heres a successfull series of caddy logs in comparison (For smaller requests that succeed):
2023/03/24 05:47:06.731 DEBUG http.handlers.reverse_proxy upstream roundtrip {"upstream": "localhost:8000", "duration": 0.001345782, "request": {"remote_ip": "", "remote_port": "59966", "proto": "HTTP/3.0", "method": "POST", "host": "api.chatshape.com", "uri": "/embed", "headers": {"Accept-Language": ["en-US,en;q=0.9"], "Content-Type": ["application/json"], "Origin": ["chrome-extension://mmlalgciccemmiggjhgkhmklpmpficen"], "X-Forwarded-For": ["104.173.33.232"], "X-Forwarded-Proto": ["https"], "Accept": ["application/json"], "Sec-Fetch-Dest": ["empty"], "Sec-Fetch-Mode": ["cors"], "Accept-Encoding": ["gzip, deflate, br"], "X-Forwarded-Host": ["api.chatshape.com"], "User-Agent": ["Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/111.0.0.0 Safari/537.36"], "Sec-Fetch-Site": ["none"]}, "tls": {"resumed": true, "version": 772, "cipher_suite": 4865, "proto": "h3", "server_name": "api.chatshape.com"}}, "headers": {"Date": ["Fri, 24 Mar 2023 05:47:06 GMT"], "Server": ["uvicorn"], "Content-Length": ["0"], "Location": ["https://api.chatshape.com/embed/"], "Access-Control-Allow-Origin": ["*"], "Access-Control-Allow-Credentials": ["true"]}, "status": 307}
2023/03/24 05:47:06.816 DEBUG http.handlers.reverse_proxy selected upstream {"dial": "localhost:8000", "total_upstreams": 1}
2023/03/24 05:47:08.114 DEBUG http.handlers.reverse_proxy upstream roundtrip {"upstream": "localhost:8000", "duration": 1.298061824, "request": {"remote_ip": "", "remote_port": "59966", "proto": "HTTP/3.0", "method": "POST", "host": "api.chatshape.com", "uri": "/embed/", "headers": {"Sec-Fetch-Dest": ["empty"], "Accept-Encoding": ["gzip, deflate, br"], "Accept-Language": ["en-US,en;q=0.9"], "User-Agent": ["Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/111.0.0.0 Safari/537.36"], "Content-Type": ["application/json"], "X-Forwarded-For": ["104.173.33.232"], "Sec-Fetch-Site": ["none"], "Origin": ["chrome-extension://mmlalgciccemmiggjhgkhmklpmpficen"], "Sec-Fetch-Mode": ["cors"], "X-Forwarded-Proto": ["https"], "X-Forwarded-Host": ["api.chatshape.com"], "Accept": ["application/json"]}, "tls": {"resumed": true, "version": 772, "cipher_suite": 4865, "proto": "h3", "server_name": "api.chatshape.com"}}, "headers": {"Access-Control-Allow-Credentials": ["true"], "Date": ["Fri, 24 Mar 2023 05:47:06 GMT"], "Server": ["uvicorn"], "Content-Length": ["77"], "Content-Type": ["application/json"], "Access-Control-Allow-Origin": ["*"]}, "status": 200}
3. Caddy version:
v2.6.4 h1:2hwYqiRwk1tf3VruhMpLcYTg+11fCdr8S3jhNAdnPy8=
4. How I installed and ran Caddy:
used Install — Caddy Documentation installed for linux thru command line, i run caddy with the below caddy file
b. Command:
caddy start
d. My complete Caddy config:
{
debug
}
api.chatshape.com {
reverse_proxy localhost:8000
log {
output file /var/log/caddy/access.log {
roll_size 100MiB
roll_keep 10
roll_keep_for 720h
}
}
}
:8080 {
respond "Hello, world!"
}