Go 1.8 plugin model

Assuming it would work, the main downside I see is the app itself fighting the system package manager. Traditionally, the package manager would be responsible for updating caddy versions, but with this scheme, the app itself could do it.

Might be best if the customizer would still build on the same base version as is currently running (down to the commit), so apy/yum/whatever are still in charge of updating caddy itself.

So the plugins were dropped from darwin… We can move this conversation to 1.9 and beyond I guess

Yeah, it looks like this won’t be really good to pursue for a while yet.

@captncraig I do intend, eventually, for the Caddy API and web UI to be able to manage the binary itself, I think, including adding and removing plugins in a way similar to what you describe. When we get around to that I’d love to have some help implementing it. :slight_smile:

1 Like

Now Go 1.8 has been released, it might be useful to start looking into ways to use plugins. The fact that only Linux is currently supported will inevitably reduce the rate that this can have useful benefit for the time being. But don’t lose heart.

I haven’t yet tried building Caddy extensions using the new Go 1.8 plugin build mode (go build -buildmode=plugin), so perhaps this would be the first step to be evaluated. As a Linux user, this is reasonably easy for me.

The positive note is that Caddy plugins are already well designed to make them fit with Go 1.8. The extra that Go 1.8 offers is the ability to load them at run time, solving (or at least reducing) the packaging problem.

In Go 1.8, the vital restriction is that the build status of every plugin must be identical to the main server, i.e. all the deps (transitively) must have the same versions. This makes me think that we should not worry about the security issues mentioned in earlier posts: only properly-built plugins can be considered for execution and only a privileged user would have write access to these files. I.e. Caddy should not (and probably never) aim to support loading arbitrary plugins off the 'Net. The caddy binary and the plugin binaries would quite likely have the same file ownership and permissions. I think this security model would be complete and sufficient. Signing of downloads of Caddy and its plugins is a separate but related issue (necessary, but not an impediment to the use of plugins).

The key issue to tackle is the build process to make this possible. By having a single codebase of Caddy plus all its approved plugins, we would meet the consistent dependency requirement. To achieve this, the getcaddy.com server could have a new option to provide a zip package containing Caddy core and all the extensions as plugin binaries (Linux only, for the time being).

Just by loading 1.8 plugins, the existing func init() code would call func RegisterPlugin. No direct use of Go’s plugin api would be needed. So the loading work is easy.

But that’s not the whole story: the order of execution of middleware is important and plugins must have a way to insert themselves into the list (httpserver/plugin.go) in a suitable place. This would need an API change, e.g. add a Before string field to type Plugin or, equivalently, add a before string parameter to func RegisterPlugin.

One last thing needed would be a new --plugindir option or similar, with a sensible default.

One last thought: corporate users would probably appreciate an easier path to build Caddy + plugins directly from source. At the moment, the project encourages people to use getcaddy.com, which is fine for many users.

But some documented advice and perhaps some shell scripts to go get _x y z_ and go build / go install would be appreciated by many, I think.

Thanks Rick, I appreciate your thoughts.

This is solving a different problem, one which currently requires a pull request and for good reason. Absolute ordering is vital.

I agree; we’ll be working on something for this toward 1.0.

I totally agree that ordering is important, but I wasn’t suggesting that there needs to be anything that can confound the existing order.

Rather, a new plugin middleware would need to define its place within the existing order. That might in future be implemented via a small api change instead of via pull request.

There’s no particular urgency to change this yet, though.

I still see some glaring problems with plugins:

  1. Can’t be “unloaded”. That means if there was a plugin manager, it could not download updated plugins and run them without a hard process restart. That’s kind of a pain.
  2. Requires host app and plugin app to be built with the exact same version of all libraries involved, or it will not load. This means that you need the exact same go version, and the exact same version of the caddy packages the plugin references. Every update to caddy will break all plugins built for previous versions. I don’t see a tenable way to manage a ecosystem of plugins if they are so brittle.

My current thoughts are a plugin that downloads a new version of caddy from a build server with custom plugins installed, and restarts the process, hopefully gracefully. Is that new build server live yet matt?

Not yet :frowning2: You’ll know it when it’s live, believe me. The new website is taking longer than I thought… haven’t stretched these kind of creative muscles for years.

There are even more issues like requiring CGO_ENABLED=1 which rules out compatibility with Caddy’s build server.

I gave it a try and I have a POC project caddyplug and a demonstration video caddyplug - YouTube.

Thanks to Caddy’s extensibility, the plugin loader is a plugin as well hook.pluginloader.

I wish I’ve read your post earlier as I would’ve backed off. I already published the plugin before realising it is not compatible (CGO_ENABLED=1 required) and realising the need for same Go version.

@matt Feature Request: ability to disable plugins on dashboard. I had to put a notice to inform users not to download.

1 Like

That’s cool, Abiola!! What a neat demo. Thanks for making this. (I’ll work on a way to unlist plugins from the download page and/or their docs too. That’ll be at the top of my TODO list.)