I’ve gotten a lot of questions about DNS provider plugins for Caddy 2. These plugins are necessary to support the DNS Challenge which is needed to get wildcard certificates from Let’s Encrypt or to get certificates for servers that can’t be exposed to the Internet at all.
I know this feature is in high demand – and it’s always been a priority – but there’s a lot of friction to overcome before we can get this working well in Caddy 2. I ported a few provider plugins over to v2 but I quickly discovered that the upstream APIs and implementations are not sufficiently compatible with Caddy 2. To clarify, Caddy 2 already has full support for DNS challenge solver modules. The problem is that the implementations underlying those modules is not very compatible with the dynamic configuration requirements of Caddy 2.
After going down the rabbit hole a bit, I decided this was as good an opportunity as ever for some major structural and organizational changes in these critical upstream dependencies whereby we can improve not only the DNS providers, but also fix quite a few outstanding problems that hinder or complexify (is that a word?) Caddy and CertMagic today.
In order to support the DNS challenge in Caddy 2, the following work will be occurring, if the community will help (please?):
-
Fork go-acme/lego (
done) - Design new, generic DNS provider APIs ( almost done)
- Lego’s API can only add and remove special TXT records (Caddy 2 modules need to be able to do more)
- OctoDNS and StackExchange/dnscontrol can (mostly) only obliterate your whole zone and replace it
- This new API will allow for safe, incremental changes in general
- Move the DNS provider plugins into separate repos ( GitHub organizations created)
- Redesign and refactor go-acme/lego ( WIP – will be incremental)
- Port DNS providers to the new APIs ( update: done!)
- Lego’s existing 70+ DNS providers can be grandfathered in through an API very similar to what they already implement, so minimal code changes required to get up and running quickly, for now.
-
I will need the community’s help! It is tedious to do all 70+ myself. I will just do a few of the most popular ones as examples.All of them have been ported over in one fell swoop in lego-deprecated! -
Eventually, all DNS providers should be reimplemented using the new APIs.
needs your help!
As you can see, there is a lot of work to do. There will be many benefits as a result of this effort. We can probably produce something experimentally usable in a few months. Then it’ll be up to the community to help test it and upgrade the remaining DNS providers to use the new API. (Most can be grandfathered in with their near-current APIs very easily, but I still will need to crowdsource help for that.)
Please get involved to help accelerate the migration of all the plugins – thanks for your patience!
About the new DNS APIs
Currently, there are only a few DNS provider libraries implemented in pure Go.
-
go-acme/lego has 70+ implemented, but they can only set and clean up TXT records for the ACME challenges. Their API also lacks critical parameters such as context for cancellation, meaning that a lingering DNS request can stall config reloads. It also is not possible to configure different DNS provider instances using different credentials and/or env variables as defaults.
-
StackExchange/dnscontrol is like OctoDNS which is pretty cool: you just tell it what you want your zone file to look like, and the library will take care of the rest by figuring out which changes need to be made and then making them, across 24+ DNS providers! But of course, this approach essentially obliterates your whole zone and requires that you maintain the master list. Useful, but not always for Caddy modules…
The DNS interfaces I’m designing are small and idiomatic Go. The describe basic incremental operations such as:
- Append
- Set
- Delete
- Get
A DNS provider package can implement and reuse whichever of these methods it wants to (although, Append and Delete will be necessary for the ACME challenges). More advanced operations can be built fairly easily on top of these primitives. For example, OctoDNS-like behavior (replacing the whole zone) is doable with a Get, a bunch of Deletes, and a bunch of Appends. We can provide helper functions to attempt to make them as ACID as possible.
Provider packages will be organized across a couple different GitHub organizations: one in which the actual implementations live (each one having their own repository and Go module), and one in which each repo is a simple file that registers it as a Caddy module. Crucially, this structure allows us to endlessly add more DNS providers without bloating the code or dependency base. Current go.sum files are 1000+ lines and take a long time to clone down; this structure avoids that bloat and makes builds significantly faster and lighter-weight.
I’m looking forward to seeing this working, but in the long run it’ll only be possible with the community’s help.
Here’s to better DNS APIs!
I’ll leave it up to the community to do most others.