ACME DNS Challenging failing for sub-domain

michael.petricevic · June 23, 2023, 1:26am

1. The problem I’m having:

We are trying to change Caddy to use an ACME dns-01 challenge to get around an issue with the Cloudflare proxying that prevents the standard http-01 acme challenge from completing successfully. We are unable to get the dns-01 challenge working.

2. Error messages and/or full log output:

Jun 22 06:39:03 BSWEBCIWEB caddy[1424971]: {"level":"error","ts":1687415943.4292848,"logger":"tls.obtain","msg":"will retry","error":"[webci.blueshift.one] Obtain: [webci.blueshift.one] solving challenges: presenting for challenge: adding temporary record for zone \"webci.blueshift.one.\": expected 1 zone, got 0 for webci.blueshift.one. (order=https://acme.zerossl.com/v2/DV90/order/ryB5fU0hEWvzH_tRixB-dA) (ca=https://acme.zerossl.com/v2/DV90)","attempt":1,"retrying_in":60,"elapsed":6.162394446,"max_duration":2592000}
Jun 22 06:40:03 BSWEBCIWEB caddy[1424971]: {"level":"info","ts":1687416003.4296691,"logger":"tls.obtain","msg":"obtaining certificate","identifier":"webci.blueshift.one"}
Jun 22 06:40:05 BSWEBCIWEB caddy[1424971]: {"level":"info","ts":1687416005.001038,"logger":"http.acme_client","msg":"trying to solve challenge","identifier":"webci.blueshift.one","challenge_type":"dns-01","ca":"https://acme-staging-v02.api.letsencrypt.org/directory"}
Jun 22 06:40:05 BSWEBCIWEB caddy[1424971]: {"level":"error","ts":1687416005.3027284,"logger":"http.acme_client","msg":"cleaning up solver","identifier":"webci.blueshift.one","challenge_type":"dns-01","error":"no memory of presenting a DNS record for \"_acme-challenge.webci.blueshift.one\" (usually OK if presenting also failed)"}
Jun 22 06:40:05 BSWEBCIWEB caddy[1424971]: {"level":"error","ts":1687416005.5617292,"logger":"tls.obtain","msg":"could not get certificate from issuer","identifier":"webci.blueshift.one","issuer":"acme-v02.api.letsencrypt.org-directory","error":"[webci.blueshift.one] solving challenges: presenting for challenge: adding temporary record for zone \"webci.blueshift.one.\": expected 1 zone, got 0 for webci.blueshift.one. (order=https://acme-staging-v02.api.letsencrypt.org/acme/order/107764524/9375309454) (ca=https://acme-staging-v02.api.letsencrypt.org/directory)"}
Jun 22 06:40:06 BSWEBCIWEB caddy[1424971]: {"level":"info","ts":1687416006.724421,"logger":"http.acme_client","msg":"trying to solve challenge","identifier":"webci.blueshift.one","challenge_type":"dns-01","ca":"https://acme.zerossl.com/v2/DV90"}
Jun 22 06:40:06 BSWEBCIWEB caddy[1424971]: {"level":"error","ts":1687416006.9925613,"logger":"http.acme_client","msg":"cleaning up solver","identifier":"webci.blueshift.one","challenge_type":"dns-01","error":"no memory of presenting a DNS record for \"_acme-challenge.webci.blueshift.one\" (usually OK if presenting also failed)"}
Jun 22 06:40:07 BSWEBCIWEB caddy[1424971]: {"level":"error","ts":1687416007.3817232,"logger":"tls.obtain","msg":"could not get certificate from issuer","identifier":"webci.blueshift.one","issuer":"acme.zerossl.com-v2-DV90","error":"[webci.blueshift.one] solving challenges: presenting for challenge: adding temporary record for zone \"webci.blueshift.one.\": expected 1 zone, got 0 for webci.blueshift.one. (order=https://acme.zerossl.com/v2/DV90/order/VrD5wwdIxCOHIw1B0s-xGA) (ca=https://acme.zerossl.com/v2/DV90)"}
Jun 22 06:40:07 BSWEBCIWEB caddy[1424971]: {"level":"error","ts":1687416007.3818464,"logger":"tls.obtain","msg":"will retry","error":"[webci.blueshift.one] Obtain: [webci.blueshift.one] solving challenges: presenting for challenge: adding temporary record for zone \"webci.blueshift.one.\": expected 1 zone, got 0 for webci.blueshift.one. (order=https://acme.zerossl.com/v2/DV90/order/VrD5wwdIxCOHIw1B0s-xGA) (ca=https://acme.zerossl.com/v2/DV90)","attempt":2,"retrying_in":120,"elapsed":70.114956545,"max_duration":2592000}
Jun 22 06:42:07 BSWEBCIWEB caddy[1424971]: {"level":"info","ts":1687416127.3821473,"logger":"tls.obtain","msg":"obtaining certificate","identifier":"webci.blueshift.one"}
Jun 22 06:42:08 BSWEBCIWEB caddy[1424971]: {"level":"info","ts":1687416128.1655467,"logger":"http.acme_client","msg":"trying to solve challenge","identifier":"webci.blueshift.one","challenge_type":"dns-01","ca":"https://acme-staging-v02.api.letsencrypt.org/directory"}
Jun 22 06:42:08 BSWEBCIWEB caddy[1424971]: {"level":"error","ts":1687416128.44892,"logger":"http.acme_client","msg":"cleaning up solver","identifier":"webci.blueshift.one","challenge_type":"dns-01","error":"no memory of presenting a DNSrecord for \"_acme-challenge.webci.blueshift.one\" (usually OK if presenting also failed)"}
Jun 22 06:42:08 BSWEBCIWEB caddy[1424971]: {"level":"error","ts":1687416128.708666,"logger":"tls.obtain","msg":"could not get certificate from issuer","identifier":"webci.blueshift.one","issuer":"acme-v02.api.letsencrypt.org-directory","error":"[webci.blueshift.one] solving challenges: presenting for challenge: adding temporary record for zone \"webci.blueshift.one.\": expected 1 zone, got 0 for webci.blueshift.one. (order=https://acme-staging-v02.api.letsencrypt.org/acme/order/107764524/9375331704) (ca=https://acme-staging-v02.api.letsencrypt.org/directory)"}
Jun 22 06:42:11 BSWEBCIWEB caddy[1424971]: {"level":"info","ts":1687416131.2563155,"logger":"http.acme_client","msg":"trying to solve challenge","identifier":"webci.blueshift.one","challenge_type":"dns-01","ca":"https://acme.zerossl.com/v2/DV90"}
Jun 22 06:42:11 BSWEBCIWEB caddy[1424971]: {"level":"error","ts":1687416131.4966278,"logger":"http.acme_client","msg":"cleaning up solver","identifier":"webci.blueshift.one","challenge_type":"dns-01","error":"no memory of presenting a DNS record for \"_acme-challenge.webci.blueshift.one\" (usually OK if presenting also failed)"}
Jun 22 06:42:12 BSWEBCIWEB caddy[1424971]: {"level":"error","ts":1687416132.0291429,"logger":"tls.obtain","msg":"could not get certificate from issuer","identifier":"webci.blueshift.one","issuer":"acme.zerossl.com-v2-DV90","error":"[webci.blueshift.one] solving challenges: presenting for challenge: adding temporary record for zone \"webci.blueshift.one.\": expected 1 zone, got 0 for webci.blueshift.one. (order=https://acme.zerossl.com/v2/DV90/order/ZknktcxkhBjlXFnhITFZOw) (ca=https://acme.zerossl.com/v2/DV90)"}
Jun 22 06:42:12 BSWEBCIWEB caddy[1424971]: {"level":"error","ts":1687416132.0293689,"logger":"tls.obtain","msg":"will retry","error":"[webci.blueshift.one] Obtain: [webci.blueshift.one] solving challenges: presenting for challenge: adding temporary record for zone \"webci.blueshift.one.\": expected 1 zone, got 0 for webci.blueshift.one. (order=https://acme.zerossl.com/v2/DV90/order/ZknktcxkhBjlXFnhITFZOw) (ca=https://acme.zerossl.com/v2/DV90)","attempt":3,"retrying_in":120,"elapsed":194.762478949,"max_duration":2592000}
Jun 22 06:44:12 BSWEBCIWEB caddy[1424971]: {"level":"info","ts":1687416252.0297399,"logger":"tls.obtain","msg":"obtaining certificate","identifier":"webci.blueshift.one"}
Jun 22 06:44:12 BSWEBCIWEB caddy[1424971]: {"level":"info","ts":1687416252.8108354,"logger":"http.acme_client","msg":"trying to solve challenge","identifier":"webci.blueshift.one","challenge_type":"dns-01","ca":"https://acme-staging-v02.api.letsencrypt.org/directory"}
Jun 22 06:44:13 BSWEBCIWEB caddy[1424971]: {"level":"error","ts":1687416253.1005056,"logger":"http.acme_client","msg":"cleaning up solver","identifier":"webci.blueshift.one","challenge_type":"dns-01","error":"no memory of presenting a DNS record for \"_acme-challenge.webci.blueshift.one\" (usually OK if presenting also failed)"}
Jun 22 06:44:13 BSWEBCIWEB caddy[1424971]: {"level":"error","ts":1687416253.360279,"logger":"tls.obtain","msg":"could not get certificate from issuer","identifier":"webci.blueshift.one","issuer":"acme-v02.api.letsencrypt.org-directory","error":"[webci.blueshift.one] solving challenges: presenting for challenge: adding temporary record for zone \"webci.blueshift.one.\": expected 1 zone, got 0 for webci.blueshift.one. (order=https://acme-staging-v02.api.letsencrypt.org/acme/order/107764524/9375352794) (ca=https://acme-staging-v02.api.letsencrypt.org/directory)"}
Jun 22 06:44:16 BSWEBCIWEB caddy[1424971]: {"level":"info","ts":1687416256.4304647,"logger":"http.acme_client","msg":"trying to solve challenge","identifier":"webci.blueshift.one","challenge_type":"dns-01","ca":"https://acme.zerossl.com/v2/DV90"}
Jun 22 06:44:16 BSWEBCIWEB caddy[1424971]: {"level":"error","ts":1687416256.69554,"logger":"http.acme_client","msg":"cleaning up solver","identifier":"webci.blueshift.one","challenge_type":"dns-01","error":"no memory of presenting a DNSrecord for \"_acme-challenge.webci.blueshift.one\" (usually OK if presenting also failed)"}
Jun 22 06:44:17 BSWEBCIWEB caddy[1424971]: {"level":"error","ts":1687416257.106103,"logger":"tls.obtain","msg":"could not get certificate from issuer","identifier":"webci.blueshift.one","issuer":"acme.zerossl.com-v2-DV90","error":"[webci.blueshift.one] solving challenges: presenting for challenge: adding temporary record for zone \"webci.blueshift.one.\": expected 1 zone, got 0 for webci.blueshift.one. (order=https://acme.zerossl.com/v2/DV90/order/mGFlKG1ilLyetA-6zUR2Ew) (ca=https://acme.zerossl.com/v2/DV90)"}
Jun 22 06:44:17 BSWEBCIWEB caddy[1424971]: {"level":"error","ts":1687416257.106177,"logger":"tls.obtain","msg":"will retry","error":"[webci.blueshift.one] Obtain: [webci.blueshift.one] solving challenges: presenting for challenge: adding temporary record for zone \"webci.blueshift.one.\": expected 1 zone, got 0 for webci.blueshift.one. (order=https://acme.zerossl.com/v2/DV90/order/mGFlKG1ilLyetA-6zUR2Ew) (ca=https://acme.zerossl.com/v2/DV90)","attempt":4,"retrying_in":300,"elapsed":319.839286953,"max_duration":2592000}

3. Caddy version:

v2.6.4

4. How I installed and ran Caddy:

We install Caddy with a custom binary (includes Cloudflare dns provider) as a service on our Ubuntu server.

a. System environment:

Ubuntu 22.04 LTS

c. Service/unit/compose file:

# caddy.service
#
# For using Caddy with a config file.
#
# Make sure the ExecStart and ExecReload commands are correct
# for your installation.
#
# See https://caddyserver.com/docs/install for instructions.
#
# WARNING: This service does not use the --resume flag, so if you
# use the API to make changes, they will be overwritten by the
# Caddyfile next time the service is restarted. If you intend to
# use Caddy's API to configure it, add the --resume flag to the
# `caddy run` command or use the caddy-api.service file instead.

[Unit]
Description=Caddy
Documentation=https://caddyserver.com/docs/
After=network.target network-online.target
Requires=network-online.target

[Service]
Type=notify
User=caddy
Group=caddy
ExecStart=/usr/local/bin/caddy run --environ --config /etc/caddy/Caddyfile
ExecReload=/usr/local/bin/caddy reload --config /etc/caddy/Caddyfile --force
TimeoutStopSec=5s
LimitNOFILE=1048576
LimitNPROC=512
PrivateTmp=true
ProtectSystem=full
AmbientCapabilities=CAP_NET_ADMIN CAP_NET_BIND_SERVICE

[Install]
WantedBy=multi-user.target

d. My complete Caddy config:

Caddyfile is as follows:

import /etc/caddy/global-config.caddy
import /etc/caddy/sites-enabled/*.caddy

webci.blueshift.one.caddy under /sites-enabled/ is as follows:

webci.blueshift.one {
        # <webci-beta-webui>
        handle /beta/one {
                reverse_proxy {
                        to localhost:50810
                        transport http {
                                tls_trusted_ca_certs /etc/caddy/certificates/internal-ca/ca.pem
                        }
                        # Create a matcher for html content
                        @htmlContent {
                                header Content-Type text/html
                        }
                        # Handle html responses by cloning the response and Cache-Control to no-store.
                        # This is because WebUI html must be authenticated, we don't want it cached.
                        handle_response @htmlContent {
                                copy_response
                                copy_response_headers
                                header Cache-Control no-store
                        }
                }
        }
        handle /beta/one/* {
                reverse_proxy {
                        to localhost:50810
                        transport http {
                                tls_trusted_ca_certs /etc/caddy/certificates/internal-ca/ca.pem
                        }
                        # Create a matcher for html content
                        @htmlContent {
                                header Content-Type text/html
                        }
                        # Handle html responses by cloning the response and Cache-Control to no-store.
                        # This is because WebUI html must be authenticated, we don't want it cached.
                        handle_response @htmlContent {
                                copy_response
                                copy_response_headers
                                header Cache-Control no-store
                        }
                }
        }
        # </webci-beta-webui>
        # <RedirectionRule1> ENV=Beta&SERVICE=WebUI
        redir / https://webci.blueshift.one/beta/one permanent
        # </RedirectionRule1>
        # <RedirectionRulebeta-Deeplink>
        @HkjDtWA9 path_regexp ywhNArYc ^/beta/deeplink/(.+)$
        redir @HkjDtWA9 https://webciconnect.blueshift.one/EricomXml/accesssso/accessnowsso_oauth.htm?appName=webci%20-%20MilkyWay%20(Beta)%20Deeplinked&groupName=Beta&param1=deeplink={http.regexp.ywhNArYc.1} permanent
        # </RedirectionRulebeta-Deeplink>
        # <RedirectionRulebetaEnvRedirect>
        redir /beta https://webci.blueshift.one/beta/one permanent
        # </RedirectionRulebetaEnvRedirect>
        # <webci-beta-odataapi>
        handle /beta/odata {
                reverse_proxy {
                        to localhost:50811
                        transport http {
                                tls_trusted_ca_certs /etc/caddy/certificates/internal-ca/ca.pem
                        }
                }
        }
        handle /beta/odata/* {
                reverse_proxy {
                        to localhost:50811
                        transport http {
                                tls_trusted_ca_certs /etc/caddy/certificates/internal-ca/ca.pem
                        }
                }
        }
        # </webci-beta-odataapi>
        # <webci-beta-webui-tls>
        # </webci-beta-webui-tls>
        # <webci-beta-odataapi-tls>
        # </webci-beta-odataapi-tls>
}

global-config.caddy is as follows:

{
  acme_dns cloudflare [REDACTED]
}

5. Links to relevant resources:

Have been following instructions from: How to use Caddy with Cloudflare's SSL settings

francislavoie · June 23, 2023, 7:44am

FYI if you turn off strict mode in Cloudflare, it’ll allow HTTP requests through to Caddy instead of having Cloudflare redirect HTTP requests to HTTPS itself. Caddy can do the redirects anyway. That way HTTP-01 should work.

This is the same issue as error getting certificate for sub-subdomain · Issue #13 · caddy-dns/cloudflare · GitHub

I think the problem is that your local DNS resolver isn’t able to properly resolve that domain. The typical workaround is to override the DNS resolvers in your config to not use your local DNS resolver.

You can do this by using the tls directive instead of the acme_dns global option:

tls {
	dns cloudflare <api-key>
	resolvers 1.1.1.1
}

system · July 23, 2023, 7:44am

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.