I also read at caddy reddit that bad actors have been hammering the community forum and I wonder if they are attacking (for whatever reasons?) the socket/server that is used to build new binaries with “add-package” which might explain why the 400 and hangs. (should be a special new circle in hell for these bad actors and let’s stop calling them hackers they are simply criminals robbing us of our time and resources)
Anyway so I take a look at xcaddy again and see the latest release is now nov 2024 yet there are later commits so wondering if that is release is still good for latest caddy releases and if xcaddy is still being maintained to keep up with latest releases of caddy. GitHub - caddyserver/xcaddy: Build Caddy with plugins
Since I have caddy running on a number of servers (like 7+) using add-package was an easy way on the fly to upgrade caddy and each machine and get the plugins reloaded (since caddy add-package seems to use the latest caddy not the version on the machine which in effect it upgrades caddy while adding the package).
Maybe it would be best for me to build one binary without xcaddy and serve that for my 7+ servers to pull when they need an update. I have in the past used a git repo with my custom build then I can pull on each server to get the latest binary with packages. Only down side is having a relatively large binary committed in a repo.
Anyway can matt or francis comment on this. I just need a long term viable solution to keep caddy up to date with my packages included.
xcaddy has always been the most reliable and recommended way to build Caddy with plugins. It relies on Go’s and Google’s global cache and distributes the load around the world. The {add,remove}-package and upgrade commands are labelled experimental. Those commands call our build server, which is the one operating the Download page. The builder used on the download page never had promised SLA and was always good-to-have service and provided at best-effort.
While we had bad actors targeting the forum and the doc site, there are also oblivious users who simply didn’t consider the implications of the over-reliance on the server. One of the issues we noticed from the logs was the over-reliance of some users on our builder to the extent of automating against it. At least 3 people have an Ansible playbook requesting a build from our server every ~4 minutes. The situation in this particular paragraph is probably our fault for unclear messaging.
For those xcaddy commits, we can make a release. It just hasn’t been a priority, and we didn’t feel they’re critical.
tl;dr: These are my personal operational recommendations based on our current infra:
In CI/CD pipelines, build from source or download GitHub release artifacts
For infra that changes infrequently, use the apt repo.
Rapidly changing infra (building daily or shorter), build from source
Using plugins? xcaddy is always your best bet. Build your custom binary once, then distribute it across your infra.
The bottom line is try to avoid making our infra part of your ops.
Just a suggestion but maybe the xcaddy should always get a release (even if nothing has changed) when caddy does (just regular release) and it should match the caddy release version. Then for sure one would know that xcaddy has been “tested” with the latest caddy release and is current for it.
It can be a bit off putting to have a repo/builder that totally depends on the latest caddy be 5 months behind. If building xcaddy as part of the CI for regular caddy release (run any xcaddy tests and release) then hopefully it would not create a bigger burden for you and you/I would be absolutely sure that xcaddy is compatible with the latest release. A small thing but then again if you want us to stop using your build sever then make xcaddy attractive.
To that end can you consider having add-package accept user a custom url/endpoint? My idea is to run my own build server (is your build server open source? Is it really just a xcaddy “server”?). I totally like the idea of “add-package” as it allows flexible customization at the instance end. e.g. I have a repo for deployed instances where the deployed instance can specify the plugins needed for that instance in a file and it loads those (currently via your servers). I guess what I am saying is make it easy for the community/enterprises to run their own build server and then you can stop running your build server which clearly is a headache for all concerned and it is not really sustainable if caddy continues to grow.
Just want to say thanks again to Matt et.al. for caddy. It has been and will always be a critical part of my network infrastructure.
xcaddy doesn’t depend on Caddy at all. It’s just a wrapper over the go command.
Yeah, it is basically an HTTP server that decodes a request for a build and calls xcaddy functions. You could write a basic version of it in like 10 minutes. Ours is custom for our site.
Most of the bigcorp sponsors I work with do exactly this: they already have their own tooling/infrastructure around xcaddy.