Seeking Performance Suggestions for Optimized Caddyfile

matt · September 5, 2022, 10:27pm

You might try setting flush_interval -1 but I have no idea if that’s going to help or hurt performance for your tests.

Btw while you’re testing, you might be interested in capturing profiles during some tests (but note that collecting a profile will slightly degrade performance). Use http://localhost:2019/debug/pprof to get some basic tools for inspecting heap and CPU/goroutines. Profiles can be captured to help assess what is impacting performance.

Please also make sure you’re using Caddy 2.6 beta 3 released just a few minutes ago: Release v2.6.0-beta.3 · caddyserver/caddy · GitHub

We’d really like that new version to undergo scrutiny. It doesn’t contain a lot of optimizations but it DOES contain significant optimizations in the php_fastcgi directive, if that happens to be relevant.

I’m not sure what to tell you about tuning your system though. That’s something I have no expertise in.

I’d also be interested in the test results with metrics disabled (or even commented from the source code entirely) – I know it has significant performance impact and there’s an issue to improve it:

github.com/caddyserver/caddy

Prometheus metrics are redundant and slow

opened 05:07AM - 18 Mar 22 UTC

renbou

bug help wanted discussion optimization

While digging into Caddy's source code I've noticed that every (!) route handler… is wrapped in a `metricsInstrumentedHandler` which updates Prometheus metrics during request execution. While it is a great feature and should definitely be enabled by default in Caddy, it currently uses up way to much CPU time and the metrics provided are quite redundant. Since Caddy tries out every route handler in order until it gets an answer to the request, metric instrumentation is called for each handler, even if it didn't actually partake (which is also quite difficult to define in this context) in resolving the request, so handler-specific metrics are constantly updated with unrelated data and as a result pretty much all of the handler-specific metrics are meaningless, making them only usable to track server-wide stats. As an example, here are the metrics for a simple Caddy server with 2 hosts, 1 of which only has a `reverse_proxy` handler, and the other has 2 `respond` handlers, 1 `reverse_proxy` handler and 1 `file_server` handler. The metrics were taken after running an http load-testing tool on the endpoints. [prometheus-metrics.txt](https://github.com/caddyserver/caddy/files/8301618/prometheus-metrics.txt) As seen in the example, all of the handler-specific metrics are pretty much the same for all handlers, even though in reality only the `respond` handler was requested. The handler metrics provide even less use if the web server hosts multiple domains, since requests from all domains get mixed up in the metrics. However, questionable metrics wouldn't be much of an issue if they were as fast as providing server-wide metrics, but they, of course, aren't, since they are getting updated multiple times until the request is finally answered. I've ran `pprof` while putting load using `h2load` to request one of the simple `respond` handlers, and it turned out that **73%** of the time spent during `caddyhttp.Server.primaryHandlerChain.ServeHTTP` was in the `metricsInstrumentedHandler` (only 30% of the time was spend by the actual `caddyhttp.StaticResponse.ServeHTTP`). Here's the profile: [profile.zip](https://github.com/caddyserver/caddy/files/8301688/profile.zip) ![image](https://user-images.githubusercontent.com/30644072/158941116-2f2ba661-d144-4c2d-97c8-7aaed062436a.png) I really think that metrics such as these should be server-wide where they make more sense and are quicker. https://github.com/nginxinc/nginx-prometheus-exporter could be seen as an example of similar Prometheus metrics commonly used for nginx.

I don’t think nginx emits these kinds of metrics by default, so to be on par with other servers (if you’re going to compare), probably will want to completely disable or rip out the metrics.

Test with different size responses too. Note that I have clocked Caddy at faster than nginx with over 100k req/sec on my commodity laptop hardware without any special tuning (or removing metrics, etc) – just vanilla config files for both servers. There are so many factors that these kinds of tests probably won’t actually tell you much. If anything, it’ll end up being, “Yeah, looks like Go is slightly slower than C because of garbage collection, so you trade off a slight edge in performance for memory safety to become impervious to all the scary exploits of C programs but are still fast enough for any real-world use case.” – i.e. things we already know.