Siege benchmarks - Nginx vs Caddy (identical systems)

Help
1

All benchmarks done on Upcloud 2 GB/ 2 core instance and a set of 60 urls. The same set of 60 html/php cached files/urls were used in the test.

First set on localhost, and second set on each other (using private IP).

FIRST SET OF RESULTS

siege -v -d1 -c250 -i -f siege.txt

Caddy

Transactions: 12145 hits
Availability: 99.44 %
Elapsed time: 35.07 secs
Data transferred: 635.66 MB
Response time: 0.69 secs
Transaction rate: 346.31 trans/sec
Throughput: 18.13 MB/sec
Concurrency: 239.72
Successful transactions: 11813
Failed transactions: 69
Longest transaction: 1.97
Shortest transaction: 0.00

Nginx

Transactions: 9909 hits
Availability: 99.77 %
Elapsed time: 34.35 secs
Data transferred: 455.19 MB
Response time: 0.84 secs
Transaction rate: 288.47 trans/sec
Throughput: 13.25 MB/sec
Concurrency: 242.80
Successful transactions: 9905
Failed transactions: 23
Longest transaction: 8.84
Shortest transaction: 0.00

SECOND SET

siege -v -d4 -c250 -i -f siege.txt

Caddy

Lifting the server siege…
Transactions: 68775 hits
Availability: 99.97 %
Elapsed time: 180.36 secs
Data transferred: 3558.97 MB
Response time: 0.58 secs
Transaction rate: 381.32 trans/sec
Throughput: 19.73 MB/sec
Concurrency: 221.55
Successful transactions: 66224
Failed transactions: 19
Longest transaction: 1.54
Shortest transaction: 0.00

Nginx
Lifting the server siege…
Transactions: 66938 hits
Availability: 99.98 %
Elapsed time: 180.54 secs
Data transferred: 3471.54 MB
Response time: 0.60 secs
Transaction rate: 370.77 trans/sec
Throughput: 19.23 MB/sec
Concurrency: 222.19
Successful transactions: 66892
Failed transactions: 14
Longest transaction: 5.48
Shortest transaction: 0.00

2 Likes
2

Hi @elos42,

I hope you don’t mind, I collated your results into a table for easier comparison.

siege -v -d1 -c250 -i -f siege.txt

Measurement Caddy result nginx result
Transactions 12145 hits 9909 hits
Availability 99.44 % 99.77 %
Elapsed time 35.07 secs 34.35 secs
Data transferred 635.66 MB 455.19 MB
Response time 0.69 secs 0.84 secs
Transaction rate 346.31 trans/sec 288.47 trans/sec
Throughput 18.13 MB/sec 13.25 MB/sec
Concurrency 239.72 242.80
Successful transactions 11813 9905
Failed transactions 69 23
Longest transaction 1.97 8.84
Shortest transaction 0.00 0.00

siege -v -d4 -c250 -i -f siege.txt

Measurement Caddy result nginx result
Transactions 68775 hits 66938 hits
Availability 99.97 % 99.98 %
Elapsed time 180.36 secs 180.54 secs
Data transferred 3558.97 MB 3471.54 MB
Response time 0.58 secs 0.60 secs
Transaction rate 381.32 trans/sec 370.77 trans/sec
Throughput 19.73 MB/sec 19.23 MB/sec
Concurrency 221.55 222.19
Successful transactions 66224 66892
Failed transactions 19 14
Longest transaction 1.54 5.48
Shortest transaction 0.00 0.00
5 Likes
3

Sure. Thanks.

I wonder why the difference increased so much when the interval was brought down to 1 second from 4 seconds, and why Nginx manages to show the same level of concurrency even as it was clearly left behind (in the 1-second test) in other parameters.

4

@elos42

  • Might also want to test HTTP/2 HTTPS performance and not just HTTP/1.1 HTTPS performance that most tools like apachebench, siege, wrk2 all test. AFAIK, only nghttp2’s h2load load tester supports testing HTTP/2 based HTTPS
  • For siege also pay attention to number of failed transactions.
  • Also might want to separate the test files to just static files vs dynamic php files to for clarity on the actual web server tested versus web server + php server.

my h2load benchmarks at https://community.centminmod.com/threads/caddy-http-2-server-benchmarks-part-2.12873/

h2load https://nghttp2.org/documentation/h2load-howto.html

h2load(1) is benchmarking tool for HTTP/2 and HTTP/1.1. If built with spdylay (Spdylay - SPDY C Library — Spdylay 1.4.0 documentation) library, it also supports SPDY protocol. It supports SSL/TLS and clear text for all supported protocols.

2 Likes
5

@eva2000 you seem to have done some comprehensive testing, but sorry for saying, the report is next to unreadable.
You may want to put it in a side-by-side table, or make some summary of al the tests. I dont think many people will want to sift through it all.

6

each post is a separate sub comparison test of sorts for most posts - the very last set of posts are all the previous combined so if you want to skip to the last heh

all tests weren’t all done in one sitting and evolved over days of testing so not really a single summary

7

These tests do not mean anything. Or everything. Depends on the machine to the machine. And the creator of the system. For example, I have a different system than anyone else. A special kernel mainline patched on Debian . Network Drivers patched. Nginx mainline last version from source with modules in my choice.
You can benchmark Nginx 1.13.7 vs. Nginx 1.13.7 and have completely different results on the same machine with different modules and kernel.
One needs to sit down and write the whole operating system in Go programming language. (Caddy OS :grin:) That would be a difference in speed.

1 Like