Author’s note: This is a more intimate look into what it’s like to be a maintainer of popular open source software. This is written by me, personally, and not as a representative of the company. I’m going out on a limb here, so try to be nice, okay?
Over the past few days, I’ve been inundated with well over one thousand comments, tweets, emails, messages, and even phone calls about Wednesday’s announcement, ranging from support and encouragement to annoyance to vitriol, hate, and disgust.
Interestingly, most of the positive reactions came from individuals who have met in person or who are at least somewhat acquainted with me online. Whereas every one of the negative reactions stemmed from someone who has never met me or interacted with me (as far as I know). In other words, there was not a single person who responded negatively or unreasonably whom I knew or could recall prior interactions with. If you ever find yourself in a similar situation, look at this statistic. It should help you find a little relief.
The day after the announcement (on which I got about two notifications per minute, all day), I went with our research lab to a retreat in the mountains where I had no Internet connection. We socialized, talked about our research, and ate a lot of food. The break was good and gave me a chance to clear my head. I went to visit Cory after that, appropriately wearing my RethinkDB shirt, and we talked about how things were going.
Now I want to take a few moments and respond personally, candidly—and bluntly—to the reactions, both good and bad, address them directly, and set the record straight with my own testimony of how and what happened along with what we’re learning, knowing full well that most of the people who need to read this probably won’t. In any case, I suppose that if this post incites controversy from that crowd, it has succeeded in its purpose.
This post is only a reflection on what my experience has been, what I’m learning through it, and there are no announcements, changes, or new future plans here.
I posted the announcement early on Wednesday morning. Within 9 minutes after I tweeted the link, someone submitted it to Hacker news and it immediately rose to #2 on the front page, second only to the long-anticipated Sublime Text 3 announcement. Comments started rolling in, but I had class until almost noon. With me in class and Cory at his day job, it was impossible to coordinate any responses until much later that day.
I finished class before he finished work, but even by that time, it was too late. The Internet took it, twisted it, and ran with it; and like a storm surge, there was nothing I could have done to quell it. Having seen this kind of thing several times before, I knew how this would play out. I could respond to some comments, but ultimately you have to let the court of public opinion make a ruling and adjourn. This generally takes 1-2 days.
So at the end of the day I went to a church to detox, then spent time with a good friend to unwind. We went to the lake to get a nice picture at sunset.
I already started to feel better, and formulated my thoughts over the next couple days.
Forking is a normal part of open source, and it’s allowed by the Apache license, as long as they give attribution and state all changes (plus some other standard stuff). When one fork started a spin-off project on Wednesday, my cofounder Cory reminded them to remove the Caddy name and logo from the repo to avoid confusion with the Caddy brand. The spin-off was quick to change the name and leave attribution, and within minutes the issue was resolved and closed.
Unfortunately, the response to this by spectators has largely been less than professional, and I think this, along with some other reactions I discuss below, expose a glaring character flaw in our open source community.
There was no hostility in Cory’s comments nor in those of the project’s maintainer. Both Cory and the maintainer did the right things.
As has been seen many times before, toxicity festers in open source because it’s all too common for forks to ground their motivation in emnity towards other projects. This obviously isn’t healthy and we hope that the open source community will figure out a better way to deal with negative sentiment than by creating new communities rooted in it.
The Caddy-Sponsors Header
We removed the header. Although we did ask our sponsors about this well before we implemented it, we didn’t get any response. Add this to the list of things we’re learning. We’ll do better communicating with our sponsors in the future. Sorry. I don’t know what else to say about this.
(Is anyone offended that you can’t remove nginx’s Server self-promotional header without an extra module?)
One plugin author expressed concern that they would have to provide commercial support for our customers if our customers used their plugin. I apologize for not addressing this up front. The answer is no, the plugin authors have no extra obligations because of these changes. We’ll handle the support, and act as liaison if necessary. Thank you, plugin authors, for all you add to the community!
The EULA and Caddy Binaries
Along with the header, there was so much confusion and hate on this point, too. The header and the EULA really had nothing to do with each other. So much vitriol would have been spared if people with strong feelings about it had just read the EULA in full, which states:
The open source code of this Software is licensed under the terms of the Apache License Version 2.0 and not under this EULA.
So that’s it. Caddy is still open source. We go to extra effort to provide you with convenient binaries customized just for you, so the official binaries are licensed either for personal or commercial use under this EULA. Commercial licenses come with extra perks like private plugin hosting and, in the future, we hope to add other features for our customers related to deploying and installing Caddy. Please don’t hate on this. It’s not worth it. If you don’t want to be bound to the EULA, download and compile Caddy from source.
Enacting an EULA also exposed a lot of uses of Caddy we did not know about. Apparently companies are relying on our build server as a sort of production CI server for their deployments! And others are backing their paid services with Caddy, relying on its security, stability, and usability. We had no idea it was being used so professionally at this scale. It’s kind of freaky, honestly, not knowing where your side projects end up going or how much money and reputation is actually relying on them.
Yes, I know it was free before. But lemme be real with you: it won’t last like this. Making this into a business is a LOT of work, and I wouldn’t be doing it if I didn’t think there was value in it. I’m sorry for several things about this week, including how we handled communication with our sponsors, but I’m not apologizing that you can’t use the precompiled binaries for free in your company anymore.
The Caddy Build Server
To clarify, the Caddy build server was once posted online to see if we wanted to license it openly, but we closed it up in the interest of focusing the technical attention of our community and our limited development resources (mostly time) on Caddy itself. The build server is not generalizable, and only exists to serve the Caddy project. As such, we’re taking it under our wings to develop and maintain it as needed. If you find some old source code still online, be aware that no license file was added to the code, and we have not granted others any license to use it.
Reflections on: The Good
One phone call I received was from a friendly company I’ve had associations with, offering their support and congratulations for the bold move. It was much appreciated. I also have received dozens of tweets and DMs, mostly, from friends who saw the cesspool accumulating around my feet and offered their sympathies or encouragement. It sounded like some of them had been here before.
To those who took a side and were supportive, thank you—you didn’t have to, and it helped us weather the storm. From one human being struggling with the daily realities of life to another, thank you.
Of the positive feedback, a few had criticisms of the changes, and all of them were reasonable. But none of them could quite pin down what it was that made the changes wrong. To quote one, “But the whole affair has made me extremely uneasy for a variety of reasons that are really hard to put a finger on.” The crux of the opposition, even according to the most well-reasoned arguments, was missing.
The best criticism among the reasonable feedback was that a commercial license is out of reach of small startups that are bootstrapping or don’t have revenue yet. We agree. Cory and I are still looking into a good way to handle this, so for now, we encourage such startups to simply contact us with their needs and ask about special pricing.
Some feedback (read: opinions and advice) we received had good intentions, but wasn’t very well thought-through, or didn’t have the necessary perspective of “project owner” or “maintainer” to hold any water. Here, I address some of them candidly. These remarks are detached from authors and are just an aggregate of numerous comments received in just these two days.
“Why not offer a comprehensive support package?”
We tried this, nobody bought it.
“Why not offer an enterprise solution or consulting service instead?”
We’re too resource-constrained. We could probably do it, but that would stifle almost all Caddy development as well as my graduate schooling and research.
“So I have to pay to remove ads from my web server.”
This was one of the biggest misconceptions.
No; your company is required to pay for commercial use of official, precompiled Caddy binaries, not for the removal of the Caddy-Sponsors header. You could do that on your own because the source is still Apache licensed. (And we’ve already removed the header, anyway.) Further, our sponsors don’t pay us for advertising. They pay for its ongoing development, to help make the Web better, to support their customers who benefit from good web servers, and to keep Caddy free for you to use. The license frankly had nothing to do with the header.
“But then I have to build from source to get what I want.”
Yes… that’s the point. Welcome to open source.
I do find it ironic that the open source community is so irate about having to compile software from source to customize it the way they want.
“Limit the free features instead.”
I know this comment was given largely out of sympathy for the worldwide negative reaction. But let’s be real, people would also complain about a feature paywall. It’s simply too late; Caddy is already open source. Any change to try to close it down or hide features behind closed doors in an effort to make it sustainable and provide actual business value is infuriating or at least offensive to those who have been using something for free. Also, this ship sailed years ago after we made Caddy plugins so ubiquitous and normal and an every day part of using Caddy. To avoid a conflict of interest, locking down features would likely require locking down plugins, suffucating the ecosystem that Caddy is so well known for.
“Nobody uses plugins anyways.”
Since the end of April this year, 242,832 Caddy plugins have been downloaded in almost 100,000 builds. Over half of builds have at least one plugin included. Plugins are very popular because they do useful things, things that other web servers don’t or can’t do.
“Just take donations then!”
We did, and it was nice to have dinner taken care of every one in a while. But that is not a business model, and frankly is not sustainable.
“But ‘free’ users give back by contributing code and bug reports.”
And we love them for that. The reality is, though, that even pull requests are a maintenance burden. Up front, to review them; and later, to maintain them. We have to assume responsibility for what we merge. And we’ve merged a lot of code.
“Since Caddy’s not open source anymore…” or “I question Matt’s commitment to open source…”
Well, its still open source, under the same Apache license, sooooo… ¯\_(ツ)_/¯
I really don’t want to focus on this, but now it must be talked about. I’ve been bottling these thoughts up for years, so maybe I should just get them out after my experiences this week.
I think the worst thing I saw this week was twofold:
- The overwhelming illusion of entitlement
- Subtle emotional manipulation based on a misconception
Let’s talk about entitlement.
It is an illusion. Anyone reading this who feels like you or your company have some right to demand anything from someone’s volunteer efforts in ANY open source project, even small libraries, STOP IT. You are not being a leader. You’re doing a disservice to yourself and the community by acting this way.
This attitude is not always obvious. Phrases such as:
- “Now paying just restores what we had”
- “Glad I never used …”
- “… is getting more greedy”
- “it’s a bit of a burn to your users”
are often accurate indicators. So are using inflammatory terms like, in this case, “adware”. This is also generally true of comments that tend toward more polarizing positions than some middle ground, some compromise. In addition, using hyperbolic terms like “molest” and using generic, unjustified negatives like “That’s a bad move” or “Pointless, embarrassing” are a dead giveaway that your ego took a hit by someone who was/is giving you something for free.
The crowning motion of this illusion is to claim that dismissing such remarks is disingenuous since “the commentary is more valuable than original source of the discussion.”
(And yes, I pulled all these from the massive pool of responses from this week.)
Spectators, you aren’t off the hook either! If you read the comments and believe them without doing your own investigation to the source, you are responsible for contributing to this culture of entitlement. Having well-informed opinions is a skill. If you aren’t good at it yet, don’t read the comments. Choose low-profile articles to practice on. Read it, think about it, try to find any relevant, trustworthy cross-references. Then read the comments and see how they check out. You will protect yourself from tons of toxicity this way.
FOSS maintainers—this one’s for you, too. I believe that we are not entitled to be handed anything for our FOSS work. Consider every donation to be a gift as precious as it is. Be grateful for it. Nobody is required to compensate you for your work, especially if you use a typical open source license. If you want to be compensated, you probably cannot do it under MIT or Apache licenses. You’ll have to change licenses or find some other way to make it work.
This was the worst part of this experience, and is akin to abuse. Demands or demanding comments stem from the misconception that users are entitled to FOSS. While this manipulation also stems from a misconception, it’s a different one: that open source maintainers depend on, or need, their open source projects.
For most maintainers or project owners, this is FALSE!
You can’t dangle a maintainer’s open source project in front of them like a carrot and say, “You want this (to succeed), don’t you?”
Other forms this takes are:
- “In order to secure the future of Caddy…”
- “I’m not sure why anyone would buy software from you ever again…”
- “Final nail in the coffin for Caddy”
- “I know Caddy’s your baby, …”
- “Bye …” or “Have a nice day” (dismissively)
as well as any comment insinuating that the maintainer is reliant upon a project that is not profitable or sustainable. Here’s the brutal truth for 99.9% (* not an actual figure) of open source projects, folks: you (the user of an open source project) need and rely on the project more than the maintainers do. Do not make the mistake of thinking that maintainers are emotionally tied to their projects. Definitely don’t call it their ‘baby’.
So, you have it backwards. Remember, these changes are being made because Caddy is not sustainable as-is. I don’t make a living from it. Up till now, I’ve enjoyed working on it, so sure, I’d like to make a living from it when I finish grad school. I, along with most FOSS maintainers, have nothing to lose.
Remember that next time you think you can weaponize a project’s potential (or lack of) success against its owner or maintainers.
One Final Note in Closing
I just want to emphasize how great so many people have been. Several dozen individuals reached out over so many mediums to express their appreciation and support of the project, and even congratulations. They took time to write an email, make a phone call, or drop some tweets or DMs, simply out of respect and compassion. Thank you, for showing me the human side of an industry that is often dehumanized by long distances and screens and texting prose.
We expected some pushback as usual, but the extreme controversy this created was unforeseen. Sorry about that. Thank you to all who responded critically but at least reasoned out their comments before writing them.
Looking forward to what’s next. Hope you’ll be a part of it with me.