About DNS providers in Caddy 2

I’ve gotten a lot of questions about DNS provider plugins for Caddy 2. These plugins are necessary to support the DNS Challenge which is needed to get wildcard certificates from Let’s Encrypt or to get certificates for servers that can’t be exposed to the Internet at all.

I know this feature is in high demand – and it’s always been a priority – but there’s a lot of friction to overcome before we can get this working well in Caddy 2. I ported a few provider plugins over to v2 but I quickly discovered that the upstream APIs and implementations are not sufficiently compatible with Caddy 2. To clarify, Caddy 2 already has full support for DNS challenge solver modules. The problem is that the implementations underlying those modules is not very compatible with the dynamic configuration requirements of Caddy 2.

After going down the rabbit hole a bit, I decided this was as good an opportunity as ever for some major structural and organizational changes in these critical upstream dependencies whereby we can improve not only the DNS providers, but also fix quite a few outstanding problems that hinder or complexify (is that a word?) Caddy and CertMagic today.

In order to support the DNS challenge in Caddy 2, the following work will be occurring, if the community will help (please?):

  • Fork go-acme/lego (:white_check_mark: done)
  • Design new, generic DNS provider APIs (:white_check_mark: almost done)
    • Lego’s API can only add and remove special TXT records (Caddy 2 modules need to be able to do more)
    • OctoDNS and StackExchange/dnscontrol can (mostly) only obliterate your whole zone and replace it
    • This new API will allow for safe, incremental changes in general
  • Move the DNS provider plugins into separate repos (:white_check_mark: GitHub organizations created)
  • Redesign and refactor go-acme/lego (:white_check_mark: WIP – will be incremental)
  • Port DNS providers to the new APIs (:white_check_mark: update: done!)
    • Lego’s existing 70+ DNS providers can be grandfathered in through an API very similar to what they already implement, so minimal code changes required to get up and running quickly, for now.
    • I will need the community’s help! It is tedious to do all 70+ myself. I will just do a few of the most popular ones as examples. All of them have been ported over in one fell swoop in lego-deprecated!
    • Eventually, all DNS providers should be reimplemented using the new APIs. :point_left: needs your help!

As you can see, there is a lot of work to do. There will be many benefits as a result of this effort. We can probably produce something experimentally usable in a few months. Then it’ll be up to the community to help test it and upgrade the remaining DNS providers to use the new API. (Most can be grandfathered in with their near-current APIs very easily, but I still will need to crowdsource help for that.)

Please get involved to help accelerate the migration of all the plugins – thanks for your patience!

About the new DNS APIs

Currently, there are only a few DNS provider libraries implemented in pure Go.

  • go-acme/lego has 70+ implemented, but they can only set and clean up TXT records for the ACME challenges. Their API also lacks critical parameters such as context for cancellation, meaning that a lingering DNS request can stall config reloads. It also is not possible to configure different DNS provider instances using different credentials and/or env variables as defaults.

  • StackExchange/dnscontrol is like OctoDNS which is pretty cool: you just tell it what you want your zone file to look like, and the library will take care of the rest by figuring out which changes need to be made and then making them, across 24+ DNS providers! But of course, this approach essentially obliterates your whole zone and requires that you maintain the master list. Useful, but not always for Caddy modules…

The DNS interfaces I’m designing are small and idiomatic Go. The describe basic incremental operations such as:

  • Append
  • Set
  • Delete
  • Get

A DNS provider package can implement and reuse whichever of these methods it wants to (although, Append and Delete will be necessary for the ACME challenges). More advanced operations can be built fairly easily on top of these primitives. For example, OctoDNS-like behavior (replacing the whole zone) is doable with a Get, a bunch of Deletes, and a bunch of Appends. We can provide helper functions to attempt to make them as ACID as possible.

Provider packages will be organized across a couple different GitHub organizations: one in which the actual implementations live (each one having their own repository and Go module), and one in which each repo is a simple file that registers it as a Caddy module. Crucially, this structure allows us to endlessly add more DNS providers without bloating the code or dependency base. Current go.sum files are 1000+ lines and take a long time to clone down; this structure avoids that bloat and makes builds significantly faster and lighter-weight.

I’m looking forward to seeing this working, but in the long run it’ll only be possible with the community’s help.

Here’s to better DNS APIs!

4 Likes

Thank you for the very detailed response. So basically long of the short we should expect cloudflare api to be configured and be able to be used on caddy v2 sometime in the near future. Also the only reason i wanted to upgrade mainly to make a video tutorial to update the and kind of combine all the posts for emby/jellyfin which are not very therough for the average day joe. Nor would they understand the extent of the of what your talking about in the videos or posts but most people dont take the time to explain them. Mainly i wanted to make an updated video from caddy v1 to caddy v2 and implementing it in a Jellyfin Media Server as it said in here. https://jellyfin.org/docs/general/networking/caddy.html
What and how can i implement caddy v2 for the simplist user that will be easy to compose and install/configure on Windows 10 X64. I’m willing to change the file structure for the video and update it to a better more practical method that will hopefully stick for a longer time. Also i have another issue my domain seems to go down quite often. Like ill refresh and it will say 522 then refresh again or F5 and it says 522, ill F5 one more time and ill get the page to load is there any settings i can change in my Caddy v1 caddyfil or DNS settings i can change in Cloudflare to make sure that i dont get that 522 and its more consistant with loading. Again thanks again! The help is much appreciated. The information is very good to know as well.

Yep, mostly because I use Cloudflare for some of my domains, so, I need it. :grin: I’ll leave it up to the community to do most others.

I’m sure the community here will be happy to address your other questions in a separate thread! I see you’ve made one so I’ll leave it to that so we can keep this thread on-topic.

thank you for a quick response. I very much appreicate it as well as the information. Thank you!!