The Realities of Being a FOSS Maintainer


(Matt Holt) #1

Author’s note: This is a more intimate look into what it’s like to be a maintainer of popular open source software. This is written by me, personally, and not as a representative of the company. I’m going out on a limb here, so try to be nice, okay? :slight_smile:

Over the past few days, I’ve been inundated with well over one thousand comments, tweets, emails, messages, and even phone calls about Wednesday’s announcement, ranging from support and encouragement to annoyance to vitriol, hate, and disgust.

Interestingly, most of the positive reactions came from individuals who have met in person or who are at least somewhat acquainted with me online. Whereas every one of the negative reactions stemmed from someone who has never met me or interacted with me (as far as I know). In other words, there was not a single person who responded negatively or unreasonably whom I knew or could recall prior interactions with. If you ever find yourself in a similar situation, look at this statistic. It should help you find a little relief.

The day after the announcement (on which I got about two notifications per minute, all day), I went with our research lab to a retreat in the mountains where I had no Internet connection. We socialized, talked about our research, and ate a lot of food. The break was good and gave me a chance to clear my head. I went to visit Cory after that, appropriately wearing my RethinkDB shirt, and we talked about how things were going.

Now I want to take a few moments and respond personally, candidly—and bluntly—to the reactions, both good and bad, address them directly, and set the record straight with my own testimony of how and what happened along with what we’re learning, knowing full well that most of the people who need to read this probably won’t. In any case, I suppose that if this post incites controversy from that crowd, it has succeeded in its purpose.

This post is only a reflection on what my experience has been, what I’m learning through it, and there are no announcements, changes, or new future plans here.

Announcement Day

I posted the announcement early on Wednesday morning. Within 9 minutes after I tweeted the link, someone submitted it to Hacker news and it immediately rose to #2 on the front page, second only to the long-anticipated Sublime Text 3 announcement. Comments started rolling in, but I had class until almost noon. With me in class and Cory at his day job, it was impossible to coordinate any responses until much later that day.

I finished class before he finished work, but even by that time, it was too late. The Internet took it, twisted it, and ran with it; and like a storm surge, there was nothing I could have done to quell it. Having seen this kind of thing several times before, I knew how this would play out. I could respond to some comments, but ultimately you have to let the court of public opinion make a ruling and adjourn. This generally takes 1-2 days.

So at the end of the day I went to a church to detox, then spent time with a good friend to unwind. We went to the lake to get a nice picture at sunset.

I already started to feel better, and formulated my thoughts over the next couple days.

Forks

Forking is a normal part of open source, and it’s allowed by the Apache license, as long as they give attribution and state all changes (plus some other standard stuff). When one fork started a spin-off project on Wednesday, my cofounder Cory reminded them to remove the Caddy name and logo from the repo to avoid confusion with the Caddy brand. The spin-off was quick to change the name and leave attribution, and within minutes the issue was resolved and closed.

Unfortunately, the response to this by spectators has largely been less than professional, and I think this, along with some other reactions I discuss below, expose a glaring character flaw in our open source community.

There was no hostility in Cory’s comments nor in those of the project’s maintainer. Both Cory and the maintainer did the right things.

As has been seen many times before, toxicity festers in open source because it’s all too common for forks to ground their motivation in emnity towards other projects. This obviously isn’t healthy and we hope that the open source community will figure out a better way to deal with negative sentiment than by creating new communities rooted in it.

The Caddy-Sponsors Header

We removed the header. Although we did ask our sponsors about this well before we implemented it, we didn’t get any response. Add this to the list of things we’re learning. We’ll do better communicating with our sponsors in the future. Sorry. I don’t know what else to say about this.

(Is anyone offended that you can’t remove nginx’s Server self-promotional header without an extra module?)

Plugin Authors

One plugin author expressed concern that they would have to provide commercial support for our customers if our customers used their plugin. I apologize for not addressing this up front. The answer is no, the plugin authors have no extra obligations because of these changes. We’ll handle the support, and act as liaison if necessary. Thank you, plugin authors, for all you add to the community!

The EULA and Caddy Binaries

Along with the header, there was so much confusion and hate on this point, too. The header and the EULA really had nothing to do with each other. So much vitriol would have been spared if people with strong feelings about it had just read the EULA in full, which states:

The open source code of this Software is licensed under the terms of the Apache License Version 2.0 and not under this EULA.

So that’s it. Caddy is still open source. We go to extra effort to provide you with convenient binaries customized just for you, so the official binaries are licensed either for personal or commercial use under this EULA. Commercial licenses come with extra perks like private plugin hosting and, in the future, we hope to add other features for our customers related to deploying and installing Caddy. Please don’t hate on this. It’s not worth it. If you don’t want to be bound to the EULA, download and compile Caddy from source.

Enacting an EULA also exposed a lot of uses of Caddy we did not know about. Apparently companies are relying on our build server as a sort of production CI server for their deployments! And others are backing their paid services with Caddy, relying on its security, stability, and usability. We had no idea it was being used so professionally at this scale. It’s kind of freaky, honestly, not knowing where your side projects end up going or how much money and reputation is actually relying on them.

Yes, I know it was free before. But lemme be real with you: it won’t last like this. Making this into a business is a LOT of work, and I wouldn’t be doing it if I didn’t think there was value in it. I’m sorry for several things about this week, including how we handled communication with our sponsors, but I’m not apologizing that you can’t use the precompiled binaries for free in your company anymore.

The Caddy Build Server

To clarify, the Caddy build server was once posted online to see if we wanted to license it openly, but we closed it up in the interest of focusing the technical attention of our community and our limited development resources (mostly time) on Caddy itself. The build server is not generalizable, and only exists to serve the Caddy project. As such, we’re taking it under our wings to develop and maintain it as needed. If you find some old source code still online, be aware that no license file was added to the code, and we have not granted others any license to use it.

Reflections on: The Good

One phone call I received was from a friendly company I’ve had associations with, offering their support and congratulations for the bold move. It was much appreciated. I also have received dozens of tweets and DMs, mostly, from friends who saw the cesspool accumulating around my feet and offered their sympathies or encouragement. It sounded like some of them had been here before.

To those who took a side and were supportive, thank you—you didn’t have to, and it helped us weather the storm. From one human being struggling with the daily realities of life to another, thank you.

Of the positive feedback, a few had criticisms of the changes, and all of them were reasonable. But none of them could quite pin down what it was that made the changes wrong. To quote one, “But the whole affair has made me extremely uneasy for a variety of reasons that are really hard to put a finger on.” The crux of the opposition, even according to the most well-reasoned arguments, was missing.

The best criticism among the reasonable feedback was that a commercial license is out of reach of small startups that are bootstrapping or don’t have revenue yet. We agree. Cory and I are still looking into a good way to handle this, so for now, we encourage such startups to simply contact us with their needs and ask about special pricing.

The Bad

Some feedback (read: opinions and advice) we received had good intentions, but wasn’t very well thought-through, or didn’t have the necessary perspective of “project owner” or “maintainer” to hold any water. Here, I address some of them candidly. These remarks are detached from authors and are just an aggregate of numerous comments received in just these two days.

“Why not offer a comprehensive support package?”

We tried this, nobody bought it.

“Why not offer an enterprise solution or consulting service instead?”

We’re too resource-constrained. We could probably do it, but that would stifle almost all Caddy development as well as my graduate schooling and research.

“So I have to pay to remove ads from my web server.”

This was one of the biggest misconceptions.

No; your company is required to pay for commercial use of official, precompiled Caddy binaries, not for the removal of the Caddy-Sponsors header. You could do that on your own because the source is still Apache licensed. (And we’ve already removed the header, anyway.) Further, our sponsors don’t pay us for advertising. They pay for its ongoing development, to help make the Web better, to support their customers who benefit from good web servers, and to keep Caddy free for you to use. The license frankly had nothing to do with the header.

“But then I have to build from source to get what I want.”

Yes… that’s the point. Welcome to open source.

I do find it ironic that the open source community is so irate about having to compile software from source to customize it the way they want.

“Limit the free features instead.”

I know this comment was given largely out of sympathy for the worldwide negative reaction. But let’s be real, people would also complain about a feature paywall. It’s simply too late; Caddy is already open source. Any change to try to close it down or hide features behind closed doors in an effort to make it sustainable and provide actual business value is infuriating or at least offensive to those who have been using something for free. Also, this ship sailed years ago after we made Caddy plugins so ubiquitous and normal and an every day part of using Caddy. To avoid a conflict of interest, locking down features would likely require locking down plugins, suffucating the ecosystem that Caddy is so well known for.

“Nobody uses plugins anyways.”

Since the end of April this year, 242,832 Caddy plugins have been downloaded in almost 100,000 builds. Over half of builds have at least one plugin included. Plugins are very popular because they do useful things, things that other web servers don’t or can’t do.

“Just take donations then!”

We did, and it was nice to have dinner taken care of every one in a while. But that is not a business model, and frankly is not sustainable.

“But ‘free’ users give back by contributing code and bug reports.”

And we love them for that. The reality is, though, that even pull requests are a maintenance burden. Up front, to review them; and later, to maintain them. We have to assume responsibility for what we merge. And we’ve merged a lot of code.

“Since Caddy’s not open source anymore…” or “I question Matt’s commitment to open source…”

Well, its still open source, under the same Apache license, sooooo… ¯\_(ツ)_/¯

The Ugly

I really don’t want to focus on this, but now it must be talked about. I’ve been bottling these thoughts up for years, so maybe I should just get them out after my experiences this week.

I think the worst thing I saw this week was twofold:

  • The overwhelming illusion of entitlement
  • Subtle emotional manipulation based on a misconception

Entitlement

Let’s talk about entitlement.

It is an illusion. Anyone reading this who feels like you or your company have some right to demand anything from someone’s volunteer efforts in ANY open source project, even small libraries, STOP IT. You are not being a leader. You’re doing a disservice to yourself and the community by acting this way.

This attitude is not always obvious. Phrases such as:

  • “Now paying just restores what we had”
  • “Glad I never used …”
  • “… is getting more greedy”
  • “it’s a bit of a burn to your users”

are often accurate indicators. So are using inflammatory terms like, in this case, “adware”. This is also generally true of comments that tend toward more polarizing positions than some middle ground, some compromise. In addition, using hyperbolic terms like “molest” and using generic, unjustified negatives like “That’s a bad move” or “Pointless, embarrassing” are a dead giveaway that your ego took a hit by someone who was/is giving you something for free.

The crowning motion of this illusion is to claim that dismissing such remarks is disingenuous since “the commentary is more valuable than original source of the discussion.”

(And yes, I pulled all these from the massive pool of responses from this week.)

Spectators, you aren’t off the hook either! If you read the comments and believe them without doing your own investigation to the source, you are responsible for contributing to this culture of entitlement. Having well-informed opinions is a skill. If you aren’t good at it yet, don’t read the comments. Choose low-profile articles to practice on. Read it, think about it, try to find any relevant, trustworthy cross-references. Then read the comments and see how they check out. You will protect yourself from tons of toxicity this way.

FOSS maintainers—this one’s for you, too. I believe that we are not entitled to be handed anything for our FOSS work. Consider every donation to be a gift as precious as it is. Be grateful for it. Nobody is required to compensate you for your work, especially if you use a typical open source license. If you want to be compensated, you probably cannot do it under MIT or Apache licenses. You’ll have to change licenses or find some other way to make it work.

Emotional Manipulation

This was the worst part of this experience, and is akin to abuse. Demands or demanding comments stem from the misconception that users are entitled to FOSS. While this manipulation also stems from a misconception, it’s a different one: that open source maintainers depend on, or need, their open source projects.

For most maintainers or project owners, this is FALSE!

You can’t dangle a maintainer’s open source project in front of them like a carrot and say, “You want this (to succeed), don’t you?”

Other forms this takes are:

  • “In order to secure the future of Caddy…”
  • “I’m not sure why anyone would buy software from you ever again…”
  • “Final nail in the coffin for Caddy”
  • “I know Caddy’s your baby, …”
  • “Bye …” or “Have a nice day” (dismissively)

as well as any comment insinuating that the maintainer is reliant upon a project that is not profitable or sustainable. Here’s the brutal truth for 99.9% (* not an actual figure) of open source projects, folks: you (the user of an open source project) need and rely on the project more than the maintainers do. Do not make the mistake of thinking that maintainers are emotionally tied to their projects. Definitely don’t call it their ‘baby’.

So, you have it backwards. Remember, these changes are being made because Caddy is not sustainable as-is. I don’t make a living from it. Up till now, I’ve enjoyed working on it, so sure, I’d like to make a living from it when I finish grad school. I, along with most FOSS maintainers, have nothing to lose.

Remember that next time you think you can weaponize a project’s potential (or lack of) success against its owner or maintainers.

One Final Note in Closing

I just want to emphasize how great so many people have been. Several dozen individuals reached out over so many mediums to express their appreciation and support of the project, and even congratulations. They took time to write an email, make a phone call, or drop some tweets or DMs, simply out of respect and compassion. Thank you, for showing me the human side of an industry that is often dehumanized by long distances and screens and texting prose.

We expected some pushback as usual, but the extreme controversy this created was unforeseen. Sorry about that. Thank you to all who responded critically but at least reasoned out their comments before writing them.

Looking forward to what’s next. Hope you’ll be a part of it with me.


(Matt Holt) #2

(Stephen Paul Weber) #3

I’ve always loved libre-non-gratis models, and I hope this becomes more popular for other projects too :slight_smile:


(Shannon Robinson) #4

Makes perfect sense. I support you.


(John Chadwick) #5

Re: emotional manipulation. Effective or not, this is a chronic problem with the internet nowadays; it isn’t just open source, it’s providing services of any kind at no cost. People develop a bizarre mental model whereby the reason you provide this service is because you get something out of it that they’re in a position to take away, and if you aren’t careful you’ll be sunk by the internet mobs. It’s terrifying.

I support Caddy’s move toward commercial offerings, but I will say I’m relieved that header is gone. It didn’t feel like the right kind of limitation to me. I’m hoping for an open core sort of model eventually, because those extra features that are generally only useful to biz and Enterprise users help make a case to business people for buying licenses.

Anyways, best of luck. I definitely look forward to the future of Caddy.


(Schollz) #6

@matt Great write-up. “Emotional manipulation” is such a great term. The amount of people saying they were “thinking” about Caddy but disavowing just now seemed ridiculous to me. I find it highly unlikely that there were so many people, within the same few hours, that were actively looking for a web server and trying to choose between nginx and Caddy. (Also ridiculous to me is the notion that nginx is easy. I’ve used it for dozens of sites, with and without HTTPS. But once you have a dozen sites all the conf linking and the certificates drove me nuts. I switched to Caddy and I’ve never looked back.)

For the record, I was willing to have a content header with sponsors - it seems pretty trivial to me. There are lots of ways to limit Caddy for personal users, and I thought you chose a great implementation against some bad alternatives. There are two reasons I think its good for everyone to have a good division between “personal” and “corporate” use:

  1. If you, the FOSS maintainer, are compensated, everyone benefits. While your time spent coding Caddy is valuable, I think the frictionless delivery of new Caddy builds+plugins is amazing and I don’t want this feature to go away. Since I’m not actively supporting you financially to work on this, I can only hope that someone does so I can benefit from future updates and timely bug fixes. Thus, I encourage you to find a model that supports you, because that will also support all of us personal users!

  2. I trust more in Caddy if it has some sort of drawback for personal users. If you offered the exact same Caddy for corporate and personal use, I would believe this too good to be true (since Caddy is an amazing improvement over nginx imo). Things that seem too good to be true are often nefarious in hidden ways (for example, Facebook lets you post photos and add friends for free, but they can collect and sell your data). If you are transparent about what is different for personal users (which you were) then it makes me happy to have a drawback so I can be assured that something sketchy isn’t happening to monetize this in another way.

I believe most people wrote with the sentiment you articulated (i.e. self-entitlement, backwards ideology about OS, etc.). Please don’t worry about the hatemail - you don’t owe anything to anyone excepting your business clients. Your business is your business, and whatever you happen to let spill over in the open-source community should only be seen as a benefit, never a negative, to the universe of open-source code. Its unfortunate the angry (and vocal) people don’t see this.

Best of luck, I look forward to continue using Caddy (and all your great projects) in the future!


(Sebastian Mancke) #7

Hi @matt,

+1 from my side!

Open Source needs business models, otherwise the people behind it can not sustain.
I personally appreciate very much that you charge for the services and not try to make a closed ‘enterprise feature branch’.

Regards,
Sebastian


(Zigmas Slusnys) #8

Hey @matt! Really enjoyed the read. A lot of people could learn a lot from this. I’m happy for you guys you took this step forward and I hope your project prospers successfully in the future! :+1:t2:


(Bruno Lowagie) #9

I am the original developer of iText, an AGPL PDF library. In the past, I’ve been caught between vampires and zombies too. The vampires being those users who suck the life out of you by demanding that you fix their problem without having any intention to reward you for your work (people who don’t realize that free software doesn’t equal free consultancy; people writing things such as “but you are RESPONSIBLE for iText, you SHOULD fix MY problem”). The zombies being those open source zealots who claim that your brain is not yours, and that whatever you produce should be free and open source. It was a very toxic place to be in, and it almost killed iText, because it isn’t trivial at all to maintain an open source project. See https://opensource.stackexchange.com/questions/88/how-can-large-open-source-projects-be-monetized/210#210 to find out what I’ve tried. Fortunately, I succeeded in building a business for iText, and we can now afford to pay developers to further develop and maintain the project. We go to ISO meetings and we are in an association that develops new PDF standards. All of this wouldn’t have been possible if we didn’t start offering commercial licenses at some point. Always remember this: “Good engineers build great technology; great engineers also create a sustainable business model.”


(Daniel) #10

Thank you for this great post @matt, and giving us some insights into this matter.

Reading this post was eye-opening and helped me understand better what it means to maintain an open source project. I immediately recognized a “ugly” phrase that I have used recently in a caddy issue - I am sorry for that. I myself did not realize what exactly I was writing and what it caused - I was completely unaware of my distorted view of all this.

I am currently starting a FOSS project (we recently had our first tech preview release), so I will soon find myself in a similar position, thank you for letting me learn from your experience.

I wish you all the best for your project and the business plan you chose. Hopefully one day I will find myself among your paying customers.


(Michael Adams) #11

@matt As one of those “Limit the free features instead" folks; and a maintainer of /r/fork on Reddit; I hope you didn’t take all the criticism as entitlement / vampirism / etc. I’m in my mid-30s: I’m old enough to remember the BBS days, DOS & Windows shareware on floppy disks, and early FTP sites. As those smaller communities have given way to larger forums, the distance between creator and consumer isn’t so intimate anymore. Conversely, at least in the US, there is an alienation taking place with whom we work for: why pay the rest of us more, when we can rely on our families, credit cards, etc to offset wage increases, medical costs, and relocating to whatever the next job is? It’s an unsustainable situation; how are those of us supposed to pay you what you’re asking us to pay you; when most of us can’t even get paid what we need (let alone, want) to get by? And that’s not a fault of your own: that’s the socio-economic reality of right now. Whatever recruiters / business leaders are saying that IT is supposed to be paying such grand amounts to indicate all this money is out there: I wonder if too many of them are using drugs proscribed by your church.

At least in the past few weeks, my own boss has more or less allowed me to do some open-source contributing; on things that we’re finding useful for our own needs. It doesn’t pay the bills of the upstream folks like yourselves, but it’s the best I can give back to folks right now. I can’t speak for the other users; whether they’re confused, entitled, etc.

Best of luck figuring it all out. I believe I still owe you some promised documentation / write-up on a use case that you and your contributors enabled me to pursue.


(Kenny Stier) #12

Please tell me you aren’t trying to attack his religion…


(Michael Adams) #13

Nope, and I’m sorry it was taken that way. Mormonism proscribes even alcohol, tobacco, and caffeine: all regular vices in the professional world (guilty of the latter myself, and a tad of the first). So I’m pretty sure he’s had an earful or two from folks that are strung out, stressed out, and convey entitlement from the needs put upon them. I can’t recall how many folks post to places like Reddit, Stack Exchange, Experts Exchange; asking to fix whatever it is his/her boss demanded get done.

If there is any application of one’s beliefs/faith/etc in this at all, it’d be environmental. Utah isn’t Bay Area / Silicon Beach; with flush VCs, and ready access to vices and scenery. It is known for generating a fair amount of sales activity, from having to put one’s mission experience to practical use. I’ve believed for a good while now that software (and IT in general) is largely a function of where its made. You don’t get too many indie games outside of metro areas & Europe: too many other folks just getting by to even put in that time & money sink. You can find a ton of developer and technical support on forums dominated by Indian users: we outsourced a lot of IT there. Anti-virus developers cut their teeth on the Demoscene, and hacking Eastern European systems + smuggled/copied Western tech. And a ton of open-source code comes out of where-ever people do have time / resources / ability to contribute it.

So a guy working hard on code in Utah wants to make money? And he’s getting flak from folks on the outside for asking? I kinda get it.


(Connor Leech) #14

Good point about FOSS maintainers having nothing to loose and that it is not their baby. I like your perspective. I haven’t used Caddy before but mad props for doing what you’re doing. don’t let the haters bring you down (I have used PappaParse and it’s awesome so thanks for that!) :metal: :pray:


(Peter Passchier) #15

The downloaded Caddy binary is not libre. It comes with a EULA and usage restrictions in both the personal and the business edition.


(Dschissler) #16

I just think that you went with a bad model. When I first tried out Caddy some months ago the build service wouldn’t work when certain modules were selected. You fixed that quickly after I reported it. Then I quickly discovered that the systemd service file was broken since it would not reload. I was told to send Caddy a custom interrupt code instead of reloading and restarting as is expected on a service. Fortunately after insisting quite heavily I was able to convince a community member that it was in fact broken and he was then able to eventually get that update into Caddy. So I immediately had two breaking issues that would have caused less experienced developers to fall on their faces. Now just some months later you want to charge really quite a lot of money for this software and it will turn a lot of people off. IMO you should have allowed Caddy to seep into the community for at least many months more before trying to monetize it. Its too early, Caddy is too green and you chose the wrong model. Get better business advice.