Having trouble with reproducable builds with caddy-dns providers

1. The problem I’m having:

I don’t have any issues with using Caddy, but rather with building it. I am building a caddy binary with xcaddy that includes most caddy-dns providers. The build is on freebsd using poudriere (building reproducable ports in a jail enviorment), in which xcaddy is executed.

Here is the caddy-custom port that is used: FreshPorts -- www/caddy-custom: Caddy (www/caddy) with customized plugins

Since Caddy needs network to build, anything that is pushed to master in one of the caddy-dns plugins can influence the whole build.

Recent example: Upgrade libdns dependencies · caddy-dns/cloudflare@e52afcd · GitHub

This commit makes this build fail: tools/config/24.1/make.conf at 367a04a51cb7587aea1a6e4c79192eb2d1ee6858 · opnsense/tools · GitHub

That happens because all caddy-dns providers that depend on libdns v0.2.1 can’t be built together with libdns v0.2.2. When I first ran into this issue, I found someone who could explain it: Getting a build error with xcaddy · Issue #6 · caddy-dns/desec · GitHub

Since Cloudflare has libdns v0.2.2 now, the build with it combined with other caddy-dns modules will fail.

I really want to know how to solve this issue and create a build that will be reproducable and won’t fail from one day to the next. Thanks for any hints.

2. Error messages and/or full log output:

Here is an example error that happens when libdns v0.2.1 and libdns v0.2.2 are mixed in the same Caddy build with caddy-dns providers. The error triggers on modules that use libdns v0.2.1 as soon as there is a single module in the build that requires libdns v0.2.2.

# github.com/libdns/desec
/root/go/pkg/mod/github.com/libdns/desec@v0.2.3/provider.go:486:14: cannot use int(prio) (value of type int) as uint value in struct literal

a. System environment:

Freebsd 13.2

I don’t think there’s any answer to this. The Go build tooling doesn’t really allow for that usecase.

I really can’t recommend making a build with all DNS plugins like that. It increases the risk surface so much, bloats the build. It would be better if you start from a vanilla build and allow users to customize to only add the specific pieces they actually need.

This is something we struggle with ourselves, sometimes a random dependency bump in Caddy itself causes a bunch of plugins to fail to build unexpectedly because of some transient dependency, out of nowhere.

Thanks for your answer. I understand what you mean.

I thought about compiling only the selected DNS Provider in on the fly (for example by triggering the module add command), but that would create other issues.

My usecase is that I offer these DNS Providers selectable in the OPNsense GUI with templating, taking care of the configuration for the user.

Since these FreeBSD builds depend on reproducability, so far it has been a real struggle since the build fails all the time when something changes.

It’s a real bummer these DNS Provider modules are so fragmented and not just one plugin module thats expanded with more providers over time. I have no real experience with golang, so I don’t understand why that approach was chosen.

I guess for now I have to fork everything and keep it at the current version that seems to work, and for the future maybe work with the module add feature (but those builds can also fail/hang/take a long time and its hard to give feedback what happened to the user). It also conflicts with the paket manager then.

Sorry if this sounds negative, I don’t mean it that way, I’m just a little frustrated.

Because each module might use a vendor-specific SDK, some of them are very bloated so it increases the binary size significantly, and it adds risk because you have code that can hit services you might not want it to if someone finds a way to escalate a vuln to change Caddy’s config or whatever. Just good practice to only include only what’s needed and nothing else.

And they’re all community maintained, we can’t vet that each plugin doesn’t have malicious code. We don’t have the time or resources to spend on maintaining/verifying them all. We can’t test them ourselves because we don’t have accounts for all the DNS providers (some are region-specific as well for example).

I understand. So I guess the way forward for me is to drop this current approach because it is literally unmaintainable and not a good practice.

I will drop these dns provider modules from the build, and inform users they have to execute a command in the shell to get their DNS Provider compiled in.

Then I don’t have the headache anymore about builds failing when the ports tree is build, and users have a safer binary since they only use the modules they need.

Thank you Francis. :slight_smile:

1 Like