Support for Go plugins

Plugins
1

What are Go plugins?

Go plugins are part of the language since 1.8 and allow dynamically loading go packages¹ similar to modules in Apache or nginx’s load_module. Functions and variables from these packages can then be used as usual.

What are the advantages?

Caddy has a large ecosystem of plugins for e.g. ACME challenges. However in many circumstances it is impossible or undesirable to use a custom compiled version of caddy. This might be the case when the use does not have control over the binary but only configuration+content or the binary is immutable by other means for example when using caddy distributed as a container² or snap³. And even if the user has the possibility to change the binary this might still be undesired which is why Fedora and its derivatives (RHEL) as well as Debian and its derivatives (Ubuntu) disable the respective subcommands4,5 (have not checked other distros). Using these officially distributed packages also has advantages regarding support and interoperability amongst other things. Security enforcements from SELinux or Apparmor also play into this.

Go plugins enable users to get these plugins separately and then tell caddy to load them on startup. This way ACME challenge providers and other plugins can be used even if they are not part of the original binary.

Alternatives considered

An alternative would be to bundle all plugins by default. Telegraf an agent for monitoring data does this for example to include hundreds of input, output and transformation plugins for data. The resulting binary is still very small. However this would either only include official caddy plugins or require some central information about trusted third-party plugins to include. Even then this still does not allow for self-made plugins (e.g. a custom libdns provider) to be used. Ultimately telegraf still bundles all plugins but also supports dynamically loading plugins from a directory (PR)

Possible disadvantages and limitations

A mismatch in signature of the exposed symbols can lead to panics. However this is no different from other panics caused by plugins today. Additional care also has to be taken to ensure that both the server executable and the plugin were compiled with the same compiler version. However in most cases this is not a problem. E.g. when compiling a plugin for Fedora’s Caddy package it is compiled with the same compiler version as is shipped with the distribution itself.

Plugins are also not supported on windows currently though this is not a problem as long as the current method of including plugins at compile time is not removed.

# First few links inside a code block due to "new user" limit
¹ https://pkg.go.dev/plugin
² https://hub.docker.com/_/caddy
³ https://snapcraft.io/caddy
⁴ https://src.fedoraproject.org/rpms/caddy/blob/28051782774a8e9f93d0cdf2b54e412e9dd32acc/f/0001-Disable-commands-that-can-alter-the-binary.patch
⁵ https://salsa.debian.org/go-team/packages/caddy/-/blob/015b8feab1d5dc91d84e8dd81756b4676c24c794/debian/patches/0004-Remove-dysfunct-commands.patch
2

Hi @septatrix – thanks for the thoughtful write-up!

Indeed, we have considered Go plugins before. It was a huge part of my deliberations with Caddy 2 in the early design days.

One of our philosophies of the Caddy project – and of Go projects generally – is simple compiling and deployment workflows, and extremely reliable/stable production setups. Unfortunately, Go’s plugin package goes against these grains.

I would urge any system administrator to have control over the binaries they execute; it’s one of the main gripes I have with popular Linux distro package management.

AFAIK the user can actually change the binary to a custom one:

But believe me, Debian/Ubuntu/Fedora rub me the wrong way which is why I’m looking at other distros for my dev machine. They are becoming about as restrictive as Macs.

One thing you haven’t addressed is how SELinux, Ubuntu, and other restrictive systems will make this easy or even allow it at all.

Even if these advantages would be compelling (which I’m not convinced of), it still doesn’t change the fact that the Go and Caddy philosophy is to compile at home, deploy easily, keep production stable. Once you have to start managing versions and dependencies in production, you might as well be deploying a C, Node or Python service.

Plus, when we start talking about Windows and having to support both modes of plugins, now we have even more complexity and ways to do things…

So, I do appreciate the consideration in this post. I think the arguments you’ve presented on both sides are valid. I’m just not convinced that the advantages are worth the costs, limitations, risks, etc.

3

While I agree with this there are also disadvantages to consider when using packages from external sources. They commonly lack accompanying files like firewall configuration, selinux profiles, systemd/logrotate/etc integration. Furthermore support contracts and backport policies do not expand to third party binaries.

Furthermore changing the binary can be troublesome at times like I explained when using containers or with the increasingly popular immutable distros like NixOS, Fedora Silverblue and apparently the next version of SUSE Linux Enterprise Server.

It could be done the same way as is currently the case for similar software like apache. There exists a httpd_modules_t which is applied by default to all respective files and allows loading/using them. The exact realization of such policies would be a task for the respective distros though they can probably copy everything from the likes of apache.

For the use case I primarily have in mind this would not be the case. This proposal is primarily to allow better extensibility of pre-built binaries. While this might not be the Caddy philosophy it is still employed by a vast amount of users. Individuals and small companies do not have the resources to compile, distribute and update their own versions of applications. Instead they refer to first party repositories or already provided containers which also get the job done. And even large corporations or enthusiastic individuals still see the advantage of dynamic plugins as can be seen with hashicorp: GitHub - hashicorp/go-plugin: Golang plugin system over RPC.

This complexity is almost near-zero. The current mechanism works by importing the plugin and letting init handle everything else. Adding support for go plugins would only mean adding a simple loop over a directory (or some other means to get all plugins which should be loaded) and opening them. The init method would still run as usual and take care of the rest. The simplicity of this can be seen very well with LoadExternalPlugins from telegraf. It simply has to be called at the start of the program et voila.

Also it is very easy to hide this support behind a build flag if that is desired.

4

With only ±10 lines it is already possible to load a plugin from a hardcoded path including error checking. I have been able to e.g. remove the github.com/caddyserver/caddy/v2/modules/standard from the main entrypoint and instead load it dynamically and everything shows up as expected in list-modules.

Currently there is a small issue with the transitive dependency on xxhash/v2 v2.1.1 however that is already solved upstream and can be circumvented.

Stability should not be a concern as Go performs a lot of checks during runtime to the point where building a plugin which fits is rather hard with manual setup. However for the intended use case (packages inside distribution repos containing caddy plugins) that is not a problem.

5

Do you mean that you’ve got a PoC working? Is it published anywhere?

6

Yes though due to Go being very strict with version matching doing this with the native modules from the same repo using the same go.sum was easier than it is for external plugins. Currently I have not published it as I try to simplify building plugins and ensure version matching however I will not be able to work on this in the next few weeks sadly.

1 Like
7

But in short these are the changes necessary to load e.g. the cloudflare-dns module…

In caddy itself (checkout of v2.5.2):

diff --git a/cmd/caddy/main.go b/cmd/caddy/main.go
index ee706d58..63136120 100644
--- a/cmd/caddy/main.go
+++ b/cmd/caddy/main.go
@@ -29,6 +29,9 @@
 package main
 
 import (
+	"plugin"
+	"fmt"
+
 	caddycmd "github.com/caddyserver/caddy/v2/cmd"
 
 	// plug in Caddy modules here
@@ -36,5 +39,9 @@ import (
 )
 
 func main() {
+	// This should loop through a (if possible configurable) directory
+	_, err := plugin.Open("<path-to-plugin>/main.so")
+	if err != nil {
+		fmt.Printf("error during plugin loading: %s", err)
+	}
 	caddycmd.Main()
 }

In caddy-dns/cloudflare:

diff --git a/go.mod b/go.mod
index 8686937..83fef36 100644
--- a/go.mod
+++ b/go.mod
@@ -3,6 +3,8 @@ module github.com/caddy-dns/cloudflare
 go 1.14
 
 require (
-	github.com/caddyserver/caddy/v2 v2.4.0
+	github.com/caddyserver/caddy/v2 v2.5.2
 	github.com/libdns/cloudflare v0.1.0
 )
+
+replace github.com/caddyserver/caddy/v2 => ../../caddy
diff --git a/plugin/main.go b/plugin/main.go
new file mode 100644
index 0000000..52292a7
--- /dev/null
+++ b/plugin/main.go
@@ -0,0 +1,3 @@
+package main
+
+import _ "github.com/caddy-dns/cloudflare"

Go plugins currently require a main package so I had to create a wrapper (cmd/go: Allow building non-main package with -buildmode=plugin · Issue #18124 · golang/go · GitHub). However as we only require the init hook this is trivial.
Also due to the strict version checks it is easiest to reference the local caddy directory (though not required). The plugin itself can be built using go build -v -buildmode=plugin plugin/main.go

1 Like
8

Thanks for sharing that!

This is technically interesting, but still probably not in-line with the direction we want to take the project right now. If you publish your code somewhere I’d be happy to link to it, maybe you could write a wiki post on our forum to document this knowledge. That way, people who are interested in using this approach can benefit from your effort!

9

What direction would that be?
You said before that one philosophies of the Caddy project is a simple deployment workflows.
Having to compile caddy yourself is definitely a lot more involved than using packages from a distributions official repositories.
Also as I have found out Go plugins are implemented such that they are extremely resilient and reliable with all the compatibility checks etc. which take place during loading.

The necessary changes for this to work are minimal and at most a few dozen lines.
Maintenance therefore should not be a concern.

My proposal would be to implement it with go flags such that it would default to being compiled in and xcaddy would then disable it unless a flag is passed.
Distributions could then use the Go plugin approach and those who want to compile in plugins directly are free to do so.
Again, both systems work the same way as they only require the init hook to be fired which makes this so easy.

10

This topic was automatically closed after 30 days. New replies are no longer allowed.