502 under heavy load (benchmark test), with low load no problem

Hi, I am using caddy as a reverse proxy with SSL for my domains. The domains are served through nginx and php-fpm.

When I run benchmark from a second server like this:

wrk -t12 -c400 -d30s --latency PrivateBin
Running 30s test @ PrivateBin
12 threads and 400 connections
Thread Stats Avg Stdev Max +/- Stdev
Latency 112.84ms 7.95ms 919.42ms 98.74%
Req/Sec 291.15 40.09 333.00 83.36%
Latency Distribution
50% 111.88ms
75% 112.30ms
90% 114.99ms
99% 121.88ms
101458 requests in 30.09s, 21.20MB read
Socket errors: connect 0, read 0, write 0, timeout 1
Non-2xx or 3xx responses: 82890
Requests/sec: 3371.48
Transfer/sec: 721.37KB

Reloading a page in browser gives me 502 Bad Gateway and also the results from the benchmark show errors (Non-2xx or 3xx responses: 82890)

This is on a i3 with 8GB RAM running FreeBSD 11.0.

What information do you need to debug this some more?

Caddy 0.9.5
php-7.0.15
nginx-1.10.2

liguros.net *.liguros.net {
proxy / 10.0.0.8 {
transparent
}
tls {
max_certs 20
}
timeouts none
}

Thanks palica

Seems related to this: PHP-FPM via FastCGI throws 502 Bad Gateway after 10 Requests

And may be related to this: https://github.com/mholt/caddy/issues/1204

I would suggest a test:
Run benchmark on non-php page (an html page maybe?) and see if you get the 502 errors

You could also try to build Caddy with go 1.8, run the benchmark again and see if it corrects the error.

There seems to be an unconfirmed issue with the cgi middleware.

I tried it agains static html site and it does the same.

wrk -t12 -c400 -d30s --latency https://liguros.net
Running 30s test @ https://liguros.net
12 threads and 400 connections
Thread Stats Avg Stdev Max +/- Stdev
Latency 105.35ms 120.42ms 1.97s 96.38%
Req/Sec 258.41 173.38 650.00 53.01%
Latency Distribution
50% 80.56ms
75% 80.81ms
90% 113.33ms
99% 773.43ms
56902 requests in 30.10s, 55.16MB read
Socket errors: connect 1567, read 233, write 0, timeout 31
Non-2xx or 3xx responses: 51053
Requests/sec: 1890.46
Transfer/sec: 1.83MB

What are your OS for both servers (Caddy and the testing server)?

Did you change the descriptor limit on the server having Caddy to a limit above 1024?

I’ve had benchmark issues before if I didn’t up that to a higher number (maybe to something like 8192).

It’s related to the warning you first get when you start Caddy on Linux:

Warning: File descriptor limit 1024 is too low for production sites. At least 4096 is recommended. Set with “ulimit -n 4096”.

If it’s still at 1024, try putting it higher and run the benchmark again.

Both are FreeBSD 11.0

ulimit is higher than 1024
231903
and
57456

I have trouble building caddy with go-1.8 on freebsd

go get github.com/mholt/caddy/caddy
# github.com/lucas-clemente/quic-go/utils
go/src/github.com/lucas-clemente/quic-go/utils/atomic_bool.go:3: import /root/caddy-go/go/pkg/freebsd_amd64/sync/atomic.a: object is [freebsd amd64 go1.8rc3 X:framepointer] expected [freebsd amd64 go1.7.4 X:framepointer]
# github.com/mholt/caddy
go/src/github.com/mholt/caddy/assets.go:4: import /root/caddy-go/go/pkg/freebsd_amd64/os.a: object is [freebsd amd64 go1.8rc3 X:framepointer] expected [freebsd amd64 go1.7.4 X:framepointer]
# github.com/hashicorp/golang-lru
go/src/github.com/hashicorp/golang-lru/2q.go:4: import /root/caddy-go/go/pkg/freebsd_amd64/fmt.a: object is [freebsd amd64 go1.8rc3 X:framepointer] expected [freebsd amd64 go1.7.4 X:framepointer]
# golang.org/x/net/lex/httplex
go/src/golang.org/x/net/lex/httplex/httplex.go:13: import /root/caddy-go/go/pkg/freebsd_amd64/net.a: object is [freebsd amd64 go1.8rc3 X:framepointer] expected [freebsd amd64 go1.7.4 X:framepointer]
# github.com/russross/blackfriday
go/src/github.com/russross/blackfriday/block.go:17: import /root/caddy-go/go/pkg/freebsd_amd64/bytes.a: object is [freebsd amd64 go1.8rc3 X:framepointer] expected [freebsd amd64 go1.7.4 X:framepointer]
# gopkg.in/square/go-jose.v1
go/src/gopkg.in/square/go-jose.v1/asymmetric.go:20: import /root/caddy-go/go/pkg/freebsd_amd64/crypto.a: object is [freebsd amd64 go1.8rc3 X:framepointer] expected [freebsd amd64 go1.7.4 X:framepointer]
# github.com/naoina/toml
go/src/github.com/naoina/toml/decode.go:4: import /root/caddy-go/go/pkg/freebsd_amd64/fmt.a: object is [freebsd amd64 go1.8rc3 X:framepointer] expected [freebsd amd64 go1.7.4 X:framepointer]

can someone help me?

thanks

This topic was automatically closed 90 days after the last reply. New replies are no longer allowed.