1. The problem I’m having:
I’m experiencing errors when accessing a Minio server which is behind a caddy reverse_proxy.
It periodically returns writing: http2: stream closed (Gitlab accesses Minio 24/7) and if I try to manually execute some S3 operations using minio-client it retries a lot and reading: unexpected EOF appear on caddy’s log.
2. Error messages and/or full log output:
EOF error:
Apr 01 11:08:52 minio caddy[3177871]: {"level":"debug","ts":1743498532.3938632,"logger":"tls.handshake","msg":"matched certificate in cache","remote_ip":"10.200.1.5","remote_port":"59430","subjects":["minios3.example"],"managed":true,"expiration":1747537450,"hash":"b2c411f28ff8c308d54821c14b277d04b5b6d138aebb50055a4ea73005a95edd"}
Apr 01 11:08:52 minio caddy[3177871]: {"level":"debug","ts":1743498532.395753,"logger":"http.handlers.reverse_proxy","msg":"selected upstream","dial":"localhost:9000","total_upstreams":1}
Apr 01 11:08:52 minio caddy[3177871]: {"level":"debug","ts":1743498532.3996522,"logger":"events","msg":"event","name":"tls_get_certificate","id":"00ef4b76-43f7-4d23-87a8-67773c768089","origin":"tls","data":{"client_hello":{"CipherSuites":[4866,4867,4865,52393,52392,49195,49199,49196,49200,158,159,49187,49191,49161,49192,49171,49188,49162,49172,103,51,107,57,156,157,60,61,47,53],"ServerName":"minios3.example","SupportedCurves":[29,23,30,25,24,256,257,258,259,260],"SupportedPoints":"AAEC","SignatureSchemes":[1027,1283,1539,2055,2056,2074,2075,2076,2057,2058,2059,2052,2053,2054,1025,1281,1537,771,769,770,1026,1282,1538],"SupportedProtos":null,"SupportedVersions":[772,771],"RemoteAddr":{"IP":"10.200.1.5","Port":59442,"Zone":""},"LocalAddr":{"IP":"10.200.1.19","Port":443,"Zone":""}}}}
Apr 01 11:08:52 minio caddy[3177871]: {"level":"debug","ts":1743498532.3996654,"logger":"tls.handshake","msg":"choosing certificate","identifier":"minios3.example","num_choices":1}
Apr 01 11:08:52 minio caddy[3177871]: {"level":"debug","ts":1743498532.3996725,"logger":"tls.handshake","msg":"default certificate selection results","identifier":"minios3.example","subjects":["minios3.example"],"managed":true,"issuer_key":"acme-v02.api.letsencrypt.org-directory","hash":"b2c411f28ff8c308d54821c14b277d04b5b6d138aebb50055a4ea73005a95edd"}
Apr 01 11:08:52 minio caddy[3177871]: {"level":"debug","ts":1743498532.3996773,"logger":"tls.handshake","msg":"matched certificate in cache","remote_ip":"10.200.1.5","remote_port":"59442","subjects":["minios3.example"],"managed":true,"expiration":1747537450,"hash":"b2c411f28ff8c308d54821c14b277d04b5b6d138aebb50055a4ea73005a95edd"}
Apr 01 11:08:52 minio caddy[3177871]: {"level":"debug","ts":1743498532.4006639,"logger":"http.handlers.reverse_proxy","msg":"selected upstream","dial":"localhost:9000","total_upstreams":1}
Apr 01 11:08:52 minio caddy[3177871]: {"level":"error","ts":1743498532.408634,"logger":"http.handlers.reverse_proxy","msg":"reading from backend","error":"unexpected EOF"}
Apr 01 11:08:52 minio caddy[3177871]: {"level":"error","ts":1743498532.408692,"logger":"http.handlers.reverse_proxy","msg":"aborting with incomplete response","upstream":"localhost:9000","duration":0.001111054,"request":{"remote_ip":"10.200.1.19","remote_port":"45354","client_ip":"10.200.1.19","proto":"HTTP/1.1","method":"GET","host":"minios3.example","uri":"/gitlabdata-packages/7a/cc/7acc684a848a9b954959fdd22493f48cf44eed028275b6b9999c7cade8956fc7/packages/391/files/663/REDACTED","headers":{"Authorization":["REDACTED"],"X-Amz-Date":["20250401T090809Z"],"X-Forwarded-For":["{proxy_add_x_forwarded_for}"],"X-Forwarded-Host":["minios3.example"],"User-Agent":["MinIO (linux; amd64) minio-go/v7.0.88 mc/RELEASE.2025-03-12T17-29-24Z"],"X-Real-Ip":["{remote_addr}"],"Accept-Encoding":["identity"],"X-Forwarded-Proto":["https"],"Connection":[""],"X-Amz-Content-Sha256":["e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855"]},"tls":{"resumed":false,"version":772,"cipher_suite":4865,"proto":"","server_name":"minios3.example"}},"error":"reading: unexpected EOF"}
stream closed error:
Apr 01 11:24:06 minio caddy[3177871]: {"level":"debug","ts":1743499446.5888386,"logger":"http.handlers.reverse_proxy","msg":"selected upstream","dial":"localhost:9000","total_upstreams":1}
Apr 01 11:24:06 minio caddy[3177871]: {"level":"debug","ts":1743499446.596428,"logger":"events","msg":"event","name":"tls_get_certificate","id":"48699e5b-3d1f-439e-a4f5-0f4922eecd7b","origin":"tls","data":{"client_hello":{"CipherSuites":[49195,49199,49196,49200,52393,52392,49161,49171,49162,49172,4865,4866,4867],"ServerName":"minios3.example","SupportedCurves":[25497,29,23,24,25],"SupportedPoints":"AA==","SignatureSchemes":[2052,1027,2055,2053,2054,1025,1281,1537,1283,1539,513,515],"SupportedProtos":["h2","http/1.1"],"SupportedVersions":[772,771],"RemoteAddr":{"IP":"10.200.1.5","Port":39442,"Zone":""},"LocalAddr":{"IP":"10.200.1.19","Port":443,"Zone":""}}}}
Apr 01 11:24:06 minio caddy[3177871]: {"level":"debug","ts":1743499446.5964413,"logger":"tls.handshake","msg":"choosing certificate","identifier":"minios3.example","num_choices":1}
Apr 01 11:24:06 minio caddy[3177871]: {"level":"debug","ts":1743499446.5964494,"logger":"tls.handshake","msg":"default certificate selection results","identifier":"minios3.example","subjects":["minios3.example"],"managed":true,"issuer_key":"acme-v02.api.letsencrypt.org-directory","hash":"b2c411f28ff8c308d54821c14b277d04b5b6d138aebb50055a4ea73005a95edd"}
Apr 01 11:24:06 minio caddy[3177871]: {"level":"debug","ts":1743499446.596454,"logger":"tls.handshake","msg":"matched certificate in cache","remote_ip":"10.200.1.5","remote_port":"39442","subjects":["minios3.example"],"managed":true,"expiration":1747537450,"hash":"b2c411f28ff8c308d54821c14b277d04b5b6d138aebb50055a4ea73005a95edd"}
Apr 01 11:24:06 minio caddy[3177871]: {"level":"debug","ts":1743499446.5989547,"logger":"http.handlers.reverse_proxy","msg":"upstream roundtrip","upstream":"localhost:9000","duration":0.036320456,"request":{"remote_ip":"10.200.1.5","remote_port":"39400","client_ip":"10.200.1.5","proto":"HTTP/1.1","method":"HEAD","host":"minios3.example","uri":"/gitlabdata-artifacts/e6/29/e629fa6598d732768f7c726b4b621285f9c3b85303900aa912017db7617d8bdb/2023_05_22/414937/702851/job.log","headers":{"X-Forwarded-For":["{proxy_add_x_forwarded_for}"],"X-Forwarded-Proto":["https"],"Authorization":["REDACTED"],"Connection":[""],"User-Agent":["fog-core/2.1.0"],"X-Amz-Date":["20250401T092406Z"],"X-Forwarded-Host":["minios3.example"],"X-Amz-Content-Sha256":["e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855"],"Content-Length":["0"],"X-Real-Ip":["{remote_addr}"]},"tls":{"resumed":false,"version":772,"cipher_suite":4865,"proto":"","server_name":"minios3.example"}},"headers":{"Date":["Tue, 01 Apr 2025 09:24:06 GMT"],"Accept-Ranges":["bytes"],"Content-Length":["9665"],"Content-Type":["text/plain"],"X-Content-Type-Options":["nosniff"],"X-Xss-Protection":["1; mode=block"],"X-Amz-Expiration":["expiry-date=\"Wed, 13 May 2026 00:00:00 GMT\", rule-id=\"cp0fdahqehmcv8n1d330\""],"X-Ratelimit-Limit":["7876"],"Etag":["\"c76932c3ee964cc9ae545085bc966b76\""],"Last-Modified":["Sun, 12 May 2024 23:18:24 GMT"],"Server":["MinIO"],"Strict-Transport-Security":["max-age=31536000; includeSubDomains"],"Vary":["Origin","Accept-Encoding"],"X-Amz-Id-2":["dd9025bab4ad464b049177c95eb6ebf374d3b3fd1af9251148b658df7ac2e3e8"],"X-Amz-Request-Id":["183227825A7615FE"],"X-Ratelimit-Remaining":["7862"]},"status":200}
Apr 01 11:24:06 minio caddy[3177871]: {"level":"debug","ts":1743499446.6063988,"logger":"http.handlers.reverse_proxy","msg":"selected upstream","dial":"localhost:9000","total_upstreams":1}
Apr 01 11:24:06 minio caddy[3177871]: {"level":"debug","ts":1743499446.6106114,"logger":"http.handlers.reverse_proxy","msg":"upstream roundtrip","upstream":"localhost:9000","duration":0.004168468,"request":{"remote_ip":"10.200.1.5","remote_port":"39442","client_ip":"10.200.1.5","proto":"HTTP/2.0","method":"GET","host":"minios3.example","uri":"/gitlabdata-artifacts/e6/29/e629fa6598d732768f7c726b4b621285f9c3b85303900aa912017db7617d8bdb/%40final/7f/7b/4994fe97b7885636dbeff4a99bb49c1cf8bfaaeee9305a967b7e416dc0d5?X-Amz-Expires=15300&X-Amz-Date=20250401T092404Z&X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=2dA6TmvEbaff0TzBsdcg%2F20250401%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-SignedHeaders=host&X-Amz-Signature=b360dd634f2a6aa976686d633b5c78a8aae12436f1c1dcca779dfffb62f246c9","headers":{"X-Forwarded-Host":["minios3.example"],"User-Agent":["gitlab-workhorse (unknown)-(unknown)"],"X-Forwarded-For":["{proxy_add_x_forwarded_for}"],"X-Forwarded-Proto":["https"],"X-Real-Ip":["{remote_addr}"],"Connection":[""]},"tls":{"resumed":false,"version":772,"cipher_suite":4865,"proto":"h2","server_name":"minios3.example"}},"headers":{"Accept-Ranges":["bytes"],"Content-Length":["32303307"],"Etag":["\"cc91b333bf41182ae10153217e456632-7\""],"X-Amz-Id-2":["dd9025bab4ad464b049177c95eb6ebf374d3b3fd1af9251148b658df7ac2e3e8"],"Server":["MinIO"],"X-Xss-Protection":["1; mode=block"],"Date":["Tue, 01 Apr 2025 09:24:06 GMT"],"Content-Type":["binary/octet-stream"],"X-Amz-Request-Id":["183227825D157F0E"],"X-Content-Type-Options":["nosniff"],"X-Ratelimit-Limit":["7876"],"Last-Modified":["Tue, 01 Apr 2025 09:24:06 GMT"],"Strict-Transport-Security":["max-age=31536000; includeSubDomains"],"Vary":["Origin","Accept-Encoding"],"X-Ratelimit-Remaining":["7862"],"X-Amz-Expiration":["expiry-date=\"Fri, 02 Apr 2027 00:00:00 GMT\", rule-id=\"cp0fdahqehmcv8n1d330\""]},"status":200}
Apr 01 11:24:06 minio caddy[3177871]: {"level":"error","ts":1743499446.6110287,"logger":"http.handlers.reverse_proxy","msg":"aborting with incomplete response","upstream":"localhost:9000","duration":0.004168468,"request":{"remote_ip":"10.200.1.5","remote_port":"39442","client_ip":"10.200.1.5","proto":"HTTP/2.0","method":"GET","host":"minios3.example","uri":"/gitlabdata-artifacts/e6/29/e629fa6598d732768f7c726b4b621285f9c3b85303900aa912017db7617d8bdb/%40final/7f/7b/4994fe97b7885636dbeff4a99bb49c1cf8bfaaeee9305a967b7e416dc0d5?X-Amz-Expires=15300&X-Amz-Date=20250401T092404Z&X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=2dA6TmvEbaff0TzBsdcg%2F20250401%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-SignedHeaders=host&X-Amz-Signature=b360dd634f2a6aa976686d633b5c78a8aae12436f1c1dcca779dfffb62f246c9","headers":{"X-Forwarded-Host":["minios3.example"],"User-Agent":["gitlab-workhorse (unknown)-(unknown)"],"X-Forwarded-For":["{proxy_add_x_forwarded_for}"],"X-Forwarded-Proto":["https"],"X-Real-Ip":["{remote_addr}"],"Connection":[""]},"tls":{"resumed":false,"version":772,"cipher_suite":4865,"proto":"h2","server_name":"minios3.example"}},"error":"writing: http2: stream closed"}
3. Caddy version:
v2.8.4 h1:q3pe0wpBj1OcHFZ3n/1nl4V4bxBrYoSoab7rL9BMYNk=
(it is compiled with GitHub - caddy-dns/acmedns)
4. How I installed and ran Caddy:
Static binaries Install — Caddy Documentation
a. System environment:
Rocky Linux 8.9
b. Command:
/usr/local/bin/caddy run --environ --config /etc/caddy/Caddyfile
c. Service/unit/compose file:
[Unit]
Description=Caddy
Documentation=https://caddyserver.com/docs/
After=network.target network-online.target
Requires=network-online.target
[Service]
Type=notify
User=caddy
Group=caddy
ExecStart=/usr/local/bin/caddy run --environ --config /etc/caddy/Caddyfile
ExecReload=/usr/local/bin/caddy reload --config /etc/caddy/Caddyfile --force
TimeoutStopSec=5s
LimitNOFILE=1048576
PrivateTmp=true
ProtectSystem=strict
ReadWritePaths=/usr/local/bin /etc/caddy /var/lib/caddy
AmbientCapabilities=CAP_NET_ADMIN CAP_NET_BIND_SERVICE
# Above is upstream, below is option hardening
# Drop all capabilities minus these ones
CapabilityBoundingSet=CAP_NET_ADMIN CAP_NET_BIND_SERVICE
NoNewPrivileges=true
# Not sure if every caddy feature will work with this enabled
# RestrictAddressFamilies=AF_INET AF_INET6 AF_UNIX
RestrictSUIDSGID=true
RestrictNamespaces=true
RestrictRealtime=true
PrivateDevices=true
# If enabled caddy cannot bind to port 443 for some reason
# PrivateUsers=true
ProtectHome=true
ProtectKernelTunables=true
ProtectKernelModules=true
ProtectControlGroups=true
MemoryDenyWriteExecute=true
LockPersonality=true
RemoveIPC=true
SystemCallArchitectures=native
SystemCallFilter=@system-service
[Install]
WantedBy=multi-user.target
d. My complete Caddy config:
{
email REDACTED
debug
}
minios3.example {
tls {
resolvers 1.1.1.1
dns acmedns /etc/caddy/acme.json
}
reverse_proxy http://localhost:9000 {
header_up X-Real-IP {remote_addr}
header_up X-Forwarded-For {proxy_add_x_forwarded_for}
header_up Connection ""
flush_interval -1
}
request_body {
max_size 0
}
# header {
# Cache-Control "no-cache, no-store, must-revalidate"
# }
}
flush_internal -1 wasn’t originally set. I tried it to see if it fixed it, but it didn’t make any difference.
5. Links to relevant resources:
The reverse proxy settings are basically adapted from the nginx one here: Configure NGINX Proxy for MinIO Server — MinIO Object Storage for Linux