1. Caddy version (caddy version):
2.3.0
2. How I run Caddy:
a. System environment:
Running in Docker, Ubuntu Image
Caddy and Upstream services are running in Docker in a k8s cluster
b. Command:
Paste command here.
c. Service/unit/compose file:
Paste full file contents here.
Make sure backticks stay on their own lines,
and the post looks nice in the preview pane.
d. My complete Caddyfile or JSON config:
{
"admin": {
"disabled": false,
"listen": "0.0.0.0:2019"
},
"apps": {
"http": {
"servers": {
"srv0": {
"listen": [":443"],
"listener_wrappers": [{
"wrapper": "proxy_protocol",
"timeout": "5s"
},
{
"wrapper": "tls"
}
],
"routes": [
{
"match": [{
"host": ["api.domain"]
}],
"handle": [{
"handler": "subroute",
"routes": [{
"handle": [{
"handler": "headers",
"response": {
"set": {
"Server": ["OurServer"]
}
}
},
{
"handler": "custom_caddy_module"
},
{
"handler": "reverse_proxy",
"transport": {
"protocol": "http",
"tls": {
"client_certificate_file": "/etc/internal-certs/cert.pem",
"client_certificate_key_file": "/etc/internal-certs/key.pem",
"root_ca_pem_files": ["/etc/ca-certs/ca_crt.pem"]
}
},
"upstreams": [{
"dial": "api:443"
}]
}
]
}]
}],
"terminal": true
}
],
"tls_connection_policies": [{
"match": {
"sni": ["api.domain"]
},
"client_authentication": {
"mode": "verify_if_given",
"trusted_ca_certs_pem_files": ["/etc/certificates/ca.pem"]
}
},
{
"match": {
"sni": ["ui.domain"]
}
}
]
}
}
},
"tls": {
"certificates": {
"load_files": [{
"certificate": "/etc/certs/ui.pem",
"key": "/etc/certs/ui-key.pem",
"tags": ["ui"]
},
{
"certificate": "/etc/certs/api.pem",
"key": "/etc/certs/api-key.pem",
"tags": ["api"]
}
]
}
}
}
}
3. The problem I’m having:
We have a mTLS enabled server(upstream) and use Caddy as a reverse-proxy for it. We provide tls config in the caddy.json so that caddy can present certs while connecting with the upstream (api). This works fine in normal scenario/usage.
However, if we try running load of around 25 parallel users, we start seeing errors in Caddy. If we disable the mTLS and run the same load again, we do not see any errors.
4. Error messages and/or full log output:
Errors on Caddy:
ERROR http.log.error EOF {“request”: {“remote_addr”: “1x”, “proto”: “HTTP/1.1”, “method”: “POST”, “host”: “api.domain”, “uri”: “/api/…”, “headers”: {“User-Agent”: [“Apache-HttpClient/4.5.12 (Java/1.8.0_212)”], “Connection”: [“keep-alive”], “Content-Type”: [“application/json”], “Accept”: [“application/json”], “Authorization”: [“Bearer”], “Content-Length”: [“111”]}, “tls”: {“resumed”: false, “version”: 771, “cipher_suite”: 49200, “proto”: “”, “proto_mutual”: true, “server_name”: “api.domain”}}, “duration”: 0.017468929, “status”: 502, “err_id”: “dp6i1yxcc”, “err_trace”: “reverseproxy.statusError (reverseproxy.go:783)”}
Error log on the upstream server (Node JS server)
{
“name”: “api”,
“hostname”: “api-d9995-62g5r”,
“pid”: 1,
“level”: 50,
“type”: “ERROR”,
“error”: {
“message”: “request aborted”,
“code”: “ECONNABORTED”,
“expected”: 29,
“length”: 29,
“received”: 0,
“type”: “request.aborted”
},
“ip_address”: “1…”,
“request”: {
“method”: “PUT”,
“path”: “/api/…”,
“body”: {},
“query”: {},
“params”: {},
“headers”: {
“host”: “api.domain”,
“user-agent”: “Apache-HttpClient/4.5.12 (Java/1.8.0_212)”,
“content-length”: “29”,
“accept”: “application/json”,
“content-type”: “application/json”,
“x-forwarded-for”: “…”,
“x-forwarded-proto”: “https”,
“accept-encoding”: “gzip”
}
},
“msg”: “request aborted”,
}
5. What I already tried:
Tried disabling mTLS and running the same load, no errors observed.
Tried specifying max_conns_per_host to 15 to limit max connections, still same errors occur
6. Links to relevant resources:
Error Code on the node js upstream server body-parser/README.md at master · expressjs/body-parser · GitHub