Complications with using `acme_dns` global option and DNS resolution

1. The problem I’m having:

I am using the acme_dns and cert_issuer global configuration options in my Caddyfile, but some of the domains I’m running Caddy for have different responses from my DHCP-provided DNS server (NextDNS) and don’t fall through to the correct nameserver. This is not an issue in general as the only network using this DNS is my local network, and there I don’t need TXT records or other things that internet services may need when using my domain. The domains exist on the regular internet-bound resolvers, the NextDNS configuration simply returns different values for these domains (local IP addresses instead of Cloudflare IP addresses which is what the internet-bound resolvers return).

The problem, concisely defined, is that when Caddy is trying to find the records it’s creating, it’s using my local DNS server, which isn’t returning these records (because it doesn’t know about them). If I were using the tls { dns { ... } } option for configuring this per-site, then I could supply the tls { resolvers ... } option, and this would work fine, but since I am using global config, I have no way to define this except switch the DNS resolver for the system running caddy entirely, but some of my reverse_proxy configurations rely on responses from my DHCP provided DNS resolver as well, so this won’t work without hardcoding IP addresses instead, which is undesirable.

Am I missing something and there is actually a way to configure the ACME-subsystems DNS resolver globally without changing the DNS resolver used for other things?

2. Error messages and/or full log output:

ERROR        tls.renew        could not get certificate from issuer        {"identifier": "mydomain.tld", "issuer": "acme.zerossl.com-v2-DV90", "error": "[mydomain.tld solving challenges: presenting for challenge: adding temporary record for zone \"_acme-challenge.mydomain.tld.\": expected 1 zone, got 0 for _acme-challenge.mydomain.tld. (order=https://acme.zerossl.com/v2/DV90/order/snipped) (ca=https://acme.zerossl.com/v2/DV90)"}

3. Caddy version:

v2.7.6 h1:w0NymbG2m9PcvKWsrXO6EEkY9Ru4FJK8uQbYcev1p3A=

4. How I installed and ran Caddy:

https://aur.archlinux.org/packages/caddy-cloudflare

a. System environment:

Arch Linux, systemd

b. Command:

/usr/bin/caddy run --config /etc/caddy/Caddyfile

c. Service/unit/compose file:

d. My complete Caddy config:

/etc/caddy/Caddyfile:

import /etc/caddy/conf.d/*

/etc/caddy/conf.d/01-global:

{
	admin off
	email my@email
	acme_dns cloudflare {env.CLOUDFLARE_API_TOKEN}
	cert_issuer zerossl {env.ZEROSSL_API_KEY}
	log {
		format console
	}
}

/etc/caddy/conf.d/10-snippets:

(cloudflare_dns) {
	tls {
		dns cloudflare {env.CLOUDFLARE_API_TOKEN}
		resolvers 1.1.1.1 1.0.0.1
	}
}
(common) {
	header Server "<My Name Here> in real-time"
}
(cors) {
	@options {
		method OPTIONS
	}
	header Access-Control-Allow-Origin "*"
	header Access-Control-Allow-Methods "GET, POST, OPTIONS, HEAD, DELETE"
	header Access-Control-Allow-Headers "Authorization, Origin, X-Requested-With, Content-Type, Accept"
	respond @options 200
}

/etc/caddy/conf.d/99-sites:

import /etc/caddy/sites-enabled/*

And then in sites-enabled I have 23 sites, but here’s one example that uses one of the relevant domains.

/etc/caddy/sites-enabled/personal:

mydomain.tld {
	import common
	header strict-transport-security "max-age=31556926; includeSubDomains; preload"
	redir https://mypunycodedomain.tld{uri}
}

mypunycodedomain.tld {
	import common
	reverse_proxy http://internal-server-1.mydomain.lan:14000 {
		header_down -server
		transport http
	}
	tls {
		protocols tls1.2 tls1.3
	}
}

Unfortunately there’s not. I do recommend just using the tls directive instead. More flexible overall.

Don’t set protocols, there’s no benefit to doing so. The default is already min tls1.2 → max tls1.3, so re-stating the defaults doesn’t do anything useful. In fact it might cause Caddy to disable some possible future tls1.4 by having it in your config, if you update Caddy without changing this.

You don’t need to specify transport http, that’s already the default.

1 Like

Yeah, I believe Caddy did allow tls1.1 (and 1.0) in the past, hence why I disabled it, but I could be wrong. I have Cloudflare “in front” of these domains as well (which is why the internal DNS is purposefully different than the internet-side records), this may have caused me to believe that my webserver was allowing TLS 1.1 connections when it was actually just the Cloudflare reverse-proxy.

But yeah I’ve been running Caddy since before the TLS 1.3 spec was finalised (probably started in 2016 or somewhere around then), I believe we were allowing older TLS versions at that time, but like I said I could be wrong.

Same with transport http, I’m sure the end server has probably been configured with self-signed HTTPS certs at one point, in which case I need a block containing:

transport http {
    tls_insecure_skip_verify
}

When I changed this I probably just removed tls_insecure_skip_verify when I should have removed the whole block. ¯\_(ツ)_/¯

Thanks for these pointers, I’m gonna have to do some configuration cleanup! :joy:

Regarding global vs site-specific, I have 41 sites enabled here @francislavoie, for the reasons we’re literally discussing now (configuration changes over time) I want to have as much of the configuration as possible be defined globally and then optionally overridden per-site.

I’d love to be able to specify more things globally, like headers, available TLS versions and cipher suites, matchers, etc.

Sidebar:

I currently have the following defined in quite a few of my sites:

@internal {
  remote_ip 10.0.0.0/8 fe80::/16 my:ipv6:prefix::/56
}

Which I then use as you’d expect (reverse_proxy @internal http://some-internal-server.domain.tld), but my IPv6 can change depending on the mood of my router, and then I have to change this in all my sites, rather than just at one place, so being able to define matchers globally would be very useful to me indeed.

Caddy v2 has had TLS 1.2 as a minimum since release. Before that in Caddy v0 though there was a time where 1.0 and 1.1 were possible to configure, but disabled by default. It’s been a very long time since it was disabled by default.

Yeah we added TLS 1.3 support in 2019.

That won’t happen – as soon as we add more ways to globally configure things, we’ll have users asking “but I want to ignore the global stuff for this one site” and then we’re expected to add a way to opt-out of these globally configured things. It’s no good. It’s bad design.

Instead, use snippets or named routes to deduplicate config.

1 Like