Caddy EULA section 3.1 h - benchmarking info clarification?

Help
1

@matt just properly read the EULA https://github.com/mholt/caddy/blob/master/dist/EULA.txt and need clarification on section 3.1 h). The section stats

3.1 You SHALL NOT, and shall not allow any third party, to:

(h) publicly disseminate performance information or analysis (including, without limitation, benchmarks) from any source relating to the Software;

So if I understand that correctly, I am no longer allowed to publish and share performance benchmark testing of official Caddy binaries ? I can only share performance benchmarks against Caddy binaries I built myself from source ?

2

It’s a good question, thanks for asking it here.

That clause comes from fairly standard EULA boilerplate text. But I think it’s important. It requires extreme care and experience to draw conclusions from benchmarks, and to know precisely when and how they transfer or generalize. Especially when it comes to web servers, which are complex on so many levels, and depend on the system in so many dimensions.

I think the primary value of this clause is that it prevents benchmarks from being used improperly, or from causing the brand any harm from performance-related allegations which may or may not be true, and which may be impossible to verify.

As the EULA states near the beginning, the EULA does not apply to Caddy’s open source code.

1 Like
3

Maybe take a page from MariaDB and Percona folks from their MySQL benchmarking publications and comparison benchmarks they do for Oracle MySQL vs MariaDB MySQL vs Percona MySQL. They setup the testing parameters, tools and configurations for public consumption + educate end users as to how they derived those benchmark results so other users can replicate those benchmarks themselves.

This serves to

  1. Provide education to end users about suggested parameters and test benchmark tools to use
  2. Provide end users with benchmarks for analysis and more importantly for regression testing with each Caddy new release. End users might run into performance regression bugs for particular usage scenarios that Caddy devs didn’t test for which may come to light through sharing such benchmarks.
  3. Provide end users with the means to reproduce/retest themselves benchmarks based on Caddy documented suggestions and official benchmarks

I am sure other tech/devop centric companies are already doing their own internal testing and would contribute their know how or knowledge to such official Caddy documented benchmark testing publications.

I believe, publication of benchmarks should be encouraged not prohibited as all parties involved will learn and benefit from such - whether it’s improper or proper benchmark configuration/parameter testing or not. If it’s improper configuration, then the benchmark tester can be called out or corrected on it etc.

I know you can compile from source the Caddy binaries to bypass the EULA, but I am sure not everyone will read that EULA fully and might get caught out by this too.

2 Likes