Troubles with DNS-01 challenge using DuckDNS

1. The problem I’m having:

I’m trying to achieve HTTPS using DuckDNS + DNS-01 challenge, but Caddy keeps reporting failures. I have moved on from testing with tls internal (by wiping the Caddy storage folder at /var/lib/caddy/.local/share) as reverse_proxy connectivity worked in that scenario.

2. Error messages and/or full log output:

resolvectl output (to highlight resolvers in use):

Global
       Protocols: -LLMNR -mDNS -DNSOverTLS DNSSEC=no/unsupported
resolv.conf mode: stub

Link 2 (ens18)
    Current Scopes: DNS
         Protocols: +DefaultRoute +LLMNR -mDNS -DNSOverTLS DNSSEC=no/unsupported
Current DNS Server: 8.8.8.8
       DNS Servers: 1.1.1.1 8.8.8.8

Link 3 (docker0)
Current Scopes: none
     Protocols: -DefaultRoute +LLMNR -mDNS -DNSOverTLS DNSSEC=no/unsupported

The key errors of interest (I believe) are the dial tcp timeout and/or read udp: connection refused errors, though I am unsure how to resolve these

Link due to character limit: cl1p.net - The internet clipboard

3. Caddy version:

v2.7.6 h1:w0NymbG2m9PcvKWsrXO6EEkY9Ru4FJK8uQbYcev1p3A=

4. How I installed and ran Caddy:

Built with xcaddy build --with github.com/caddy-dns/duckdns then replaced /usr/bin/caddy

a. System environment:

Run with systemd as service, on a Ubuntu Server 22.04 VM.
VM is on a Proxmox Host behind NAT. VM IP is 10.0.0.4 (mentioned in logs).

b. Command:

N/A

c. Service/unit/compose file:

d. My complete Caddy config:

# The Caddyfile is an easy way to configure your Caddy web server.
#
# Unless the file starts with a global options block, the first
# uncommented line is always the address of your site.
#
# To use your own domain name (with automatic HTTPS), first make
# sure your domain's A/AAAA DNS records are properly pointed to
# this machine's public IP, then replace ":80" below with your
# domain name.
{
        debug
        acme_dns duckdns {{TOKEN_REDACTED}}
        acme_ca https://acme-staging-v02.api.letsencrypt.org/directory
        email cert-ssl.tav91@8shield.net
}

(duck) {
        tls {
                dns duckdns {{TOKEN_REDACTED}}
        }
}

(acmecust) {
        tls {
                issuer acme https://acme-staging-v02.api.letsencrypt.org/directory {
                        resolvers 8.8.8.8 8.8.4.4
                        alt_http_port 4080
                        alt_tlsalpn_port 4443
                }
        }
}

zi0x.duckdns.org http://zi0x.duckdns.org {
        # Set this path to your site's directory.
        # root * /usr/share/caddy

        # Enable the static file server.
        # file_server
        respond "Hello world!"

        # Another common task is to set up a reverse proxy:
        # reverse_proxy localhost:8080

        # Or serve a PHP site through php-fpm:
        # php_fastcgi localhost:9000
}

homebox.zi0x.duckdns.org {
        reverse_proxy 10.0.0.6:3100
}

n8n.zi0x.duckdns.org {
        reverse_proxy 10.0.0.2:5678
}

n8n-2.zi0x.duckdns.org {
        reverse_proxy 10.0.0.11:5678
}

paperless.zi0x.duckdns.org {
        reverse_proxy 10.0.0.8:8000
}

portainer.zi0x.duckdns.org {
        reverse_proxy 10.0.0.9:9443
}

http://portainer.zi0x.duckdns.org {
        reverse_proxy 10.0.0.9:9000
}

nextcloud.zi0x.duckdns.org {
        reverse_proxy 10.0.0.10:11000
}

http://ut1.zi0x.duckdns.org {
        reverse_proxy 10.0.0.12:80
}

# Refer to the Caddy docs for more information:
# https://caddyserver.com/docs/caddyfile

5. Links to relevant resources:

Issue may be similar to this.

Where I got the idea to wipe caddy storage and re-do:

Building custom caddy:

Other possibly useful info

  • The IP assigned to my duckdns entry is that assigned to Caddy box (or rather, the Host to this VM) on my tail-net (tailscale).
  • Prior to changing to staging endpoint & setting up a custom email, I was able to obtain a cert after spinning up a new box + site block in Caddyfile and waiting an unknown-but-long amount of time.
    • This particular site block is now failing again after wiping Caddy storage.
    • Other site blocks were still failing though.

You should probably use these steps if you installed with the apt repo, because otherwise next time there’s an update, your custom binary will be wiped out.

From your logs:

dial tcp 3.97.58.28:53: i/o timeout

This means that Caddy wasn’t able to use your system’s DNS resolver for whatever reason.

You probably need to set resolvers for the TLS config to fix this.

Unfortunately you can’t set that via global options, so you’ll need to use your duck snippet and import that into each site (and remove acme_dns from global options).

	tls {
		dns duckdns {{TOKEN_REDACTED}}
		resolvers 8.8.8.8 8.8.4.4
	}

This is weird. You shouldn’t allow access to these via HTTP, since it’s insecure. If there’s any kind of login page, the password information would totally be in the clear if submitted over the internet.

Also, typically you have Caddy proxy to the HTTP port, not to the HTTPS port, regardless of how the user initially connected to the server. So HTTPS from the client to Caddy, then HTTP from Caddy to your app. Caddy’s job is to terminate TLS, so the connection between the client and Caddy is encrypted. At that point, there’s no need to re-encrypt the data to pass it to your app, because it’s already within your private network.

Tried this and still getting similar error.
Revised Caddyfile (scaling down services) to troubleshoot this issue:

{
	debug
	acme_ca https://acme-staging-v02.api.letsencrypt.org/directory
	email cert-ssl.tav91@8shield.net
}


(duck) {
	tls {
		dns duckdns {{REDACTED}}
		resolvers 8.8.8.8 8.8.4.4
	}
}

(acmecust) {
	tls {
		issuer acme https://acme-staging-v02.api.letsencrypt.org/directory {
			resolvers 8.8.8.8 8.8.4.4
			alt_http_port 4080
			alt_tlsalpn_port 4443
		}
	}
}	

zi0x.duckdns.org http://zi0x.duckdns.org {	
	# Set this path to your site's directory.
	# root * /usr/share/caddy

	# Enable the static file server.
	# file_server
	respond "Hello world!"

	# Another common task is to set up a reverse proxy:
	# reverse_proxy localhost:8080

	# Or serve a PHP site through php-fpm:
	# php_fastcgi localhost:9000
}

homebox.zi0x.duckdns.org {
	import duck
	reverse_proxy 10.0.0.6:3100
}

portainer.zi0x.duckdns.org {
	reverse_proxy 10.0.0.9:9443
}

http://portainer.zi0x.duckdns.org {
	reverse_proxy 10.0.0.9:9000
}

New log: cl1p.net - The internet clipboard

Thanks for the explanation for Caddy being the TLS terminator. At the moment I need portainer easily accessible and since HTTPS isn’t working for me right now, I think I should leave it this way. Also all these endpoints are accessible only within my VPN as the resolved IP isn’t public-facing.

That’s weird. I think some of DuckDNS’s DNS servers might be having trouble. Might just need to let Caddy keep retrying for a little while until it works.

After a long wait (almost 12hrs!!) it looks like I managed to get a staging cert for just homebox., so now I’m uncommenting the other endpoints and retrying. Going to be a long wait as those are also facing the same issue.

Meanwhile…is it possible to increase the timeout period (and if so, what’s the default timeout)?

EDIT: Further evidence to support that theory as I just saw paperless succeed after a couple failures. See below filtered log - timestamp difference being noteworthy:

Mar 03 13:18:03 revprox caddy[14584]: {"level":"info","ts":1709471883.2014885,"logger":"tls.obtain","msg":"obtaining certificate","identifier":"paperless.zi0x.duckdns.org"}
Mar 03 13:18:03 revprox caddy[14584]: {"level":"debug","ts":1709471883.2020526,"logger":"events","msg":"event","name":"cert_obtaining","id":"87b8017b-6144-491f-bf81-6cfbda2cfecf","origin":"tls","data":{"identifier":"paperless.zi0x.duckdns.org"}}
Mar 03 13:18:03 revprox caddy[14584]: {"level":"info","ts":1709471883.8790483,"logger":"tls.issuance.acme.acme_client","msg":"trying to solve challenge","identifier":"paperless.zi0x.duckdns.org","challenge_type":"dns-01","ca":"https://acme-staging-v02.api.letsencrypt.org/directory"}
Mar 03 13:18:04 revprox caddy[14584]: {"level":"debug","ts":1709471884.1303015,"logger":"tls.issuance.acme.acme_client","msg":"waiting for solver before continuing","identifier":"paperless.zi0x.duckdns.org","challenge_type":"dns-01"}
Mar 03 13:18:37 revprox caddy[14584]: {"level":"debug","ts":1709471917.3850908,"logger":"tls.issuance.acme.acme_client","msg":"done waiting for solver","identifier":"paperless.zi0x.duckdns.org","challenge_type":"dns-01"}
Mar 03 13:18:37 revprox caddy[14584]: {"level":"error","ts":1709471917.8557284,"logger":"tls.obtain","msg":"could not get certificate from issuer","identifier":"paperless.zi0x.duckdns.org","issuer":"acme-staging-v02.api.letsencrypt.org-directory","error":"[paperless.zi0x.duckdns.org] solving challenges: waiting for solver certmagic.solverWrapper to be ready: checking DNS propagation of \"_acme-challenge.paperless.zi0x.duckdns.org\": dial tcp 15.222.19.97:53: i/o timeout (order=https://acme-staging-v02.api.letsencrypt.org/acme/order/138723713/14988335173) (ca=https://acme-staging-v02.api.letsencrypt.org/directory)"}
Mar 03 13:18:37 revprox caddy[14584]: {"level":"debug","ts":1709471917.8558824,"logger":"events","msg":"event","name":"cert_failed","id":"b4ca1d54-0f90-4324-a037-5811294f85e6","origin":"tls","data":{"error":{},"identifier":"paperless.zi0x.duckdns.org","issuers":["acme-staging-v02.api.letsencrypt.org-directory"],"renewal":false}}
Mar 03 13:18:37 revprox caddy[14584]: {"level":"error","ts":1709471917.856004,"logger":"tls.obtain","msg":"will retry","error":"[paperless.zi0x.duckdns.org] Obtain: [paperless.zi0x.duckdns.org] solving challenges: waiting for solver certmagic.solverWrapper to be ready: checking DNS propagation of \"_acme-challenge.paperless.zi0x.duckdns.org\": dial tcp 15.222.19.97:53: i/o timeout (order=https://acme-staging-v02.api.letsencrypt.org/acme/order/138723713/14988335173) (ca=https://acme-staging-v02.api.letsencrypt.org/directory)","attempt":3,"retrying_in":120,"elapsed":278.92926686,"max_duration":2592000}
Mar 03 13:20:37 revprox caddy[14584]: {"level":"info","ts":1709472037.8571117,"logger":"tls.obtain","msg":"obtaining certificate","identifier":"paperless.zi0x.duckdns.org"}
Mar 03 13:20:37 revprox caddy[14584]: {"level":"debug","ts":1709472037.8576403,"logger":"events","msg":"event","name":"cert_obtaining","id":"9de1a4d8-4dac-4fc9-9637-a9d642175599","origin":"tls","data":{"identifier":"paperless.zi0x.duckdns.org"}}
Mar 03 13:20:38 revprox caddy[14584]: {"level":"info","ts":1709472038.518152,"logger":"tls.issuance.acme.acme_client","msg":"trying to solve challenge","identifier":"paperless.zi0x.duckdns.org","challenge_type":"dns-01","ca":"https://acme-staging-v02.api.letsencrypt.org/directory"}
Mar 03 13:20:38 revprox caddy[14584]: {"level":"debug","ts":1709472038.7701044,"logger":"tls.issuance.acme.acme_client","msg":"waiting for solver before continuing","identifier":"paperless.zi0x.duckdns.org","challenge_type":"dns-01"}
Mar 03 13:21:03 revprox caddy[14584]: {"level":"debug","ts":1709472063.52169,"logger":"tls.issuance.acme.acme_client","msg":"done waiting for solver","identifier":"paperless.zi0x.duckdns.org","challenge_type":"dns-01"}
Mar 03 13:21:05 revprox caddy[14584]: {"level":"error","ts":1709472065.5511808,"logger":"tls.obtain","msg":"could not get certificate from issuer","identifier":"paperless.zi0x.duckdns.org","issuer":"acme-staging-v02.api.letsencrypt.org-directory","error":"[paperless.zi0x.duckdns.org] solving challenges: waiting for solver certmagic.solverWrapper to be ready: checking DNS propagation of \"_acme-challenge.paperless.zi0x.duckdns.org\": dial tcp 15.223.106.16:53: i/o timeout (order=https://acme-staging-v02.api.letsencrypt.org/acme/order/138723713/14988382803) (ca=https://acme-staging-v02.api.letsencrypt.org/directory)"}
Mar 03 13:21:05 revprox caddy[14584]: {"level":"debug","ts":1709472065.5513434,"logger":"events","msg":"event","name":"cert_failed","id":"1b58c019-4522-482e-802a-7af56f997ccf","origin":"tls","data":{"error":{},"identifier":"paperless.zi0x.duckdns.org","issuers":["acme-staging-v02.api.letsencrypt.org-directory"],"renewal":false}}
Mar 03 13:21:05 revprox caddy[14584]: {"level":"error","ts":1709472065.5514565,"logger":"tls.obtain","msg":"will retry","error":"[paperless.zi0x.duckdns.org] Obtain: [paperless.zi0x.duckdns.org] solving challenges: waiting for solver certmagic.solverWrapper to be ready: checking DNS propagation of \"_acme-challenge.paperless.zi0x.duckdns.org\": dial tcp 15.223.106.16:53: i/o timeout (order=https://acme-staging-v02.api.letsencrypt.org/acme/order/138723713/14988382803) (ca=https://acme-staging-v02.api.letsencrypt.org/directory)","attempt":4,"retrying_in":300,"elapsed":426.624719207,"max_duration":2592000}
Mar 03 13:26:05 revprox caddy[14584]: {"level":"info","ts":1709472365.5786772,"logger":"tls.obtain","msg":"obtaining certificate","identifier":"paperless.zi0x.duckdns.org"}
Mar 03 13:26:05 revprox caddy[14584]: {"level":"debug","ts":1709472365.57915,"logger":"events","msg":"event","name":"cert_obtaining","id":"ad113bce-6bcb-4d32-8f6e-58411c6ea288","origin":"tls","data":{"identifier":"paperless.zi0x.duckdns.org"}}
Mar 03 13:26:06 revprox caddy[14584]: {"level":"info","ts":1709472366.2400987,"logger":"tls.issuance.acme.acme_client","msg":"trying to solve challenge","identifier":"paperless.zi0x.duckdns.org","challenge_type":"dns-01","ca":"https://acme-staging-v02.api.letsencrypt.org/directory"}
Mar 03 13:26:06 revprox caddy[14584]: {"level":"debug","ts":1709472366.4933376,"logger":"tls.issuance.acme.acme_client","msg":"waiting for solver before continuing","identifier":"paperless.zi0x.duckdns.org","challenge_type":"dns-01"}
Mar 03 13:26:11 revprox caddy[14584]: {"level":"debug","ts":1709472371.9537983,"logger":"tls.issuance.acme.acme_client","msg":"done waiting for solver","identifier":"paperless.zi0x.duckdns.org","challenge_type":"dns-01"}
Mar 03 13:26:12 revprox caddy[14584]: {"level":"debug","ts":1709472372.1705382,"logger":"tls.issuance.acme.acme_client","msg":"challenge accepted","identifier":"paperless.zi0x.duckdns.org","challenge_type":"dns-01"}
Mar 03 13:26:13 revprox caddy[14584]: {"level":"info","ts":1709472373.3400533,"logger":"tls.issuance.acme.acme_client","msg":"authorization finalized","identifier":"paperless.zi0x.duckdns.org","authz_status":"valid"}
Mar 03 13:26:17 revprox caddy[14584]: {"level":"info","ts":1709472377.2417629,"logger":"tls.obtain","msg":"certificate obtained successfully","identifier":"paperless.zi0x.duckdns.org"}
Mar 03 13:26:17 revprox caddy[14584]: {"level":"debug","ts":1709472377.242131,"logger":"events","msg":"event","name":"cert_obtained","id":"e62a818c-49e0-4171-a69d-2e6a37c5ae32","origin":"tls","data":{"certificate_path":"certificates/acme-staging-v02.api.letsencrypt.org-directory/paperless.zi0x.duckdns.org/paperless.zi0x.duckdns.org.crt","identifier":"paperless.zi0x.duckdns.org","issuer":"acme-staging-v02.api.letsencrypt.org-directory","metadata_path":"certificates/acme-staging-v02.api.letsencrypt.org-directory/paperless.zi0x.duckdns.org/paperless.zi0x.duckdns.org.json","private_key_path":"certificates/acme-staging-v02.api.letsencrypt.org-directory/paperless.zi0x.duckdns.org/paperless.zi0x.duckdns.org.key","renewal":false,"storage_path":"certificates/acme-staging-v02.api.letsencrypt.org-directory/paperless.zi0x.duckdns.org"}}
Mar 03 13:26:17 revprox caddy[14584]: {"level":"info","ts":1709472377.2423437,"logger":"tls.obtain","msg":"releasing lock","identifier":"paperless.zi0x.duckdns.org"}
Mar 03 13:26:18 revprox caddy[14584]: {"level":"debug","ts":1709472378.3645875,"logger":"tls.cache","msg":"added certificate to cache","subjects":["paperless.zi0x.duckdns.org"],"expiration":1717244773,"managed":true,"issuer_key":"acme-staging-v02.api.letsencrypt.org-directory","hash":"3c92579dcc5e8916c090e4dad4868804e01659424d54051362164c05559bc65d","cache_size":3,"cache_capacity":10000}
Mar 03 13:26:18 revprox caddy[14584]: {"level":"debug","ts":1709472378.3650982,"logger":"events","msg":"event","name":"cached_managed_cert","id":"76cc9a54-8faf-4292-81b8-59e058db6dca","origin":"tls","data":{"sans":["paperless.zi0x.duckdns.org"]}}

What exactly is happening when Caddy is “waiting for solver”? I can see where it failed it waited for ~28s but in the last success it only waited ~5s.

It’s not always that amount of time though: this instance for nextcloud took way longer to fail:

Mar 03 13:26:02 revprox caddy[14584]: {"level":"info","ts":1709472362.5413764,"logger":"tls.obtain","msg":"obtaining certificate","identifier":"nextcloud.zi0x.duckdns.org"}
Mar 03 13:26:02 revprox caddy[14584]: {"level":"debug","ts":1709472362.5418456,"logger":"events","msg":"event","name":"cert_obtaining","id":"7bd5448b-59cf-4dc5-b13f-10818b3b402f","origin":"tls","data":{"identifier":"nextcloud.zi0x.duckdns.org"}}
Mar 03 13:26:03 revprox caddy[14584]: {"level":"info","ts":1709472363.2084017,"logger":"tls.issuance.acme.acme_client","msg":"trying to solve challenge","identifier":"nextcloud.zi0x.duckdns.org","challenge_type":"dns-01","ca":"https://acme-staging-v02.api.letsencrypt.org/directory"}
Mar 03 13:26:03 revprox caddy[14584]: {"level":"debug","ts":1709472363.4638581,"logger":"tls.issuance.acme.acme_client","msg":"waiting for solver before continuing","identifier":"nextcloud.zi0x.duckdns.org","challenge_type":"dns-01"}
Mar 03 13:27:23 revprox caddy[14584]: {"level":"debug","ts":1709472443.603528,"logger":"tls.issuance.acme.acme_client","msg":"done waiting for solver","identifier":"nextcloud.zi0x.duckdns.org","challenge_type":"dns-01"}
Mar 03 13:27:24 revprox caddy[14584]: {"level":"error","ts":1709472444.0704942,"logger":"tls.obtain","msg":"could not get certificate from issuer","identifier":"nextcloud.zi0x.duckdns.org","issuer":"acme-staging-v02.api.letsencrypt.org-directory","error":"[nextcloud.zi0x.duckdns.org] solving challenges: waiting for solver certmagic.solverWrapper to be ready: checking DNS propagation of \"_acme-challenge.nextcloud.zi0x.duckdns.org\": dial tcp 99.79.16.64:53: i/o timeout (order=https://acme-staging-v02.api.letsencrypt.org/acme/order/138723713/14988484613) (ca=https://acme-staging-v02.api.letsencrypt.org/directory)"}
Mar 03 13:27:24 revprox caddy[14584]: {"level":"debug","ts":1709472444.0705192,"logger":"events","msg":"event","name":"cert_failed","id":"318daa46-2865-44de-ae81-3aecc24f5c7b","origin":"tls","data":{"error":{},"identifier":"nextcloud.zi0x.duckdns.org","issuers":["acme-staging-v02.api.letsencrypt.org-directory"],"renewal":false}}
Mar 03 13:27:24 revprox caddy[14584]: {"level":"error","ts":1709472444.0705514,"logger":"tls.obtain","msg":"will retry","error":"[nextcloud.zi0x.duckdns.org] Obtain: [nextcloud.zi0x.duckdns.org] solving challenges: waiting for solver certmagic.solverWrapper to be ready: checking DNS propagation of \"_acme-challenge.nextcloud.zi0x.duckdns.org\": dial tcp 99.79.16.64:53: i/o timeout (order=https://acme-staging-v02.api.letsencrypt.org/acme/order/138723713/14988484613) (ca=https://acme-staging-v02.api.letsencrypt.org/directory)","attempt":5,"retrying_in":600,"elapsed":805.144746561,"max_duration":2592000}

It’s waiting for DNS propagation, i.e. waiting until Caddy can make a DNS query to confirm that the TXT record is indeed visible. The relevant code is here certmagic/solvers.go at 8613f4a4444987b6226a2bedb980284d11dc1f44 · caddyserver/certmagic · GitHub

Propagation checks are optional, you can turn them off. It sometimes doesn’t work because your server’s perspective of DNS does not necessarily match ACME issuer’s perspectives, from their servers. But it’s an attempt at preventing pressure on ACME issuers by only asking them to continue once we’ve confirmed it’s ready to go for them to check the DNS TXT record.

It really looks like I just had to wait for an unknown amount of time? I added another site block today and it got a cert pretty quickly from the staging endpoint. Unsure if I should mark this as solved.

Here is a list of:

It’s more useful to look here, i.e. ones that actually integrate with Caddy:

https://github.com/orgs/caddy-dns/repositories

1 Like

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.