DNS Challenge with DuckDNS inside podman

1. The problem I’m having:

I’m trying to run caddy built with duckdns support. The problem is that it is unable to finish the dns challenge inside a podman container.

It works when I run ./caddy run locally but not inside podman.

2. Error messages and/or full log output:

caddy log:

INF ts=1705359573.732452 msg=using adjacent Caddyfile
INF ts=1705359573.733568 logger=admin msg=admin endpoint started address=localhost:2019 enforce_origin=false origins=["//localhost:2019","//[::1]:2019","//127.0.0.1:2019"]
INF ts=1705359573.7337308 logger=tls.cache.maintenance msg=started background certificate maintenance cache=0xc00082ba00
INF ts=1705359573.7337382 logger=http.auto_https msg=server is listening only on the HTTPS port but has no TLS connection policies; adding one to enable TLS server_name=srv0 https_port=443
INF ts=1705359573.7337463 logger=http.auto_https msg=enabling automatic HTTP->HTTPS redirects server_name=srv0
INF ts=1705359573.7338672 logger=http msg=enabling HTTP/3 listener addr=:443
INF ts=1705359573.7339706 logger=http.log msg=server running name=srv0 protocols=["h1","h2","h3"]
INF ts=1705359573.7339835 logger=http.log msg=server running name=remaining_auto_https_redirects protocols=["h1","h2","h3"]
INF ts=1705359573.7339852 logger=http msg=enabling automatic TLS certificate management domains=["*****.duckdns.org"]
INF ts=1705359573.734161 msg=autosaved config (load with --resume flag) file=/root/.config/caddy/autosave.json
INF ts=1705359573.734175 msg=serving initial configuration
INF ts=1705359573.7344208 logger=tls.obtain msg=acquiring lock identifier=*****.duckdns.org
WRN ts=1705359573.7405355 logger=tls msg=storage cleaning happened too recently; skipping for now storage=FileStorage:/root/.local/share/caddy instance=f2e23162-1e61-466d-b076-825ffad70972 try_again=1705445973.7405338 try_again_in=86399.999999616
INF ts=1705359573.7406144 logger=tls msg=finished cleaning storage units
INF ts=1705359582.7368224 msg=[INFO][FileStorage:/root/.local/share/caddy] Lock for 'issue_cert_*****.duckdns.org' is stale (created: 2024-01-15 22:58:06.662203783 +0000 UTC, last update: 2024-01-15 22:59:31.933551932 +0000 UTC); removing then retrying: /root/.local/share/caddy/locks/issue_cert_*****.duckdns.org.lock
INF ts=1705359582.7523756 logger=tls.obtain msg=lock acquired identifier=*****.duckdns.org
INF ts=1705359582.752801 logger=tls.obtain msg=obtaining certificate identifier=*****.duckdns.org
INF ts=1705359582.7539809 logger=tls.issuance.acme msg=waiting on internal rate limiter identifiers=["*****.duckdns.org"] ca=https://acme-v02.api.letsencrypt.org/directory account=*****@*****.com
INF ts=1705359582.7540123 logger=tls.issuance.acme msg=done waiting on internal rate limiter identifiers=["*****.duckdns.org"] ca=https://acme-v02.api.letsencrypt.org/directory account=*****@*****.com
INF ts=1705359584.4194448 logger=tls.issuance.acme.acme_client msg=trying to solve challenge identifier=*****.duckdns.org challenge_type=dns-01 ca=https://acme-v02.api.letsencrypt.org/directory
ERR ts=1705359591.8046212 logger=tls.obtain msg=could not get certificate from issuer identifier=*****.duckdns.org issuer=acme-v02.api.letsencrypt.org-directory error=[*****.duckdns.org] solving challenges: waiting for solver certmagic.solverWrapper to be ready: checking DNS propagation of "_acme-challenge.*****.duckdns.org": read udp 10.88.0.45:46834->15.223.21.81:53: read: connection refused (order=https://acme-v02.api.letsencrypt.org/acme/order/1518427076/236967714786) (ca=https://acme-v02.api.letsencrypt.org/directory)
INF ts=1705359591.8051624 logger=tls.issuance.zerossl msg=waiting on internal rate limiter identifiers=["*****.duckdns.org"] ca=https://acme.zerossl.com/v2/DV90 account=*****@*****.com
INF ts=1705359591.8051877 logger=tls.issuance.zerossl msg=done waiting on internal rate limiter identifiers=["*****.duckdns.org"] ca=https://acme.zerossl.com/v2/DV90 account=*****@*****.com
INF ts=1705359598.8851979 logger=tls.issuance.zerossl.acme_client msg=trying to solve challenge identifier=*****.duckdns.org challenge_type=dns-01 ca=https://acme.zerossl.com/v2/DV90

3. Caddy version:

I’m using caddy v2.7.6 h1:w0NymbG2m9PcvKWsrXO6EEkY9Ru4FJK8uQbYcev1p3A=

4. How I installed and ran Caddy:

I wrote a dockerfile and built with: podman build -t mycaddy .

FROM docker.io/golang:alpine as builder

RUN go install github.com/caddyserver/xcaddy/cmd/xcaddy@latest
RUN xcaddy build --with github.com/caddy-dns/duckdns

FROM docker.io/alpine:latest
COPY --from=builder /go/caddy /caddy

CMD ["/caddy", "run"]

a. System environment:

I’m currenlty working on my gentoo desktop with podman, portainer but I plan to move all of this stuff to a proxmox vm.
As a firewall I’m using ufw and I’ve also disabled it to see if it causes any issue but it doesn’t seem so.

b. Command:

podman run -d --replace \
    --name caddy \
    -p 80:80 \
    -p 443:443 \
    -v ./www:/www \
    -v ./Caddyfile:/Caddyfile \
    mycaddy

c. Service/unit/compose file:

d. My complete Caddy config:

caddyfile:

{
	#debug	
	http_port 80
	https_port 443
	email ******@******.com
}

******.duckdns.org
file_server browse
root * ./www

tls {
	dns duckdns ******
}

5. Links to relevant resources:

none

Why not use our builder image? See the docs on https://hub.docker.com/_/caddy, specifically the section “Adding custom Caddy modules”

You’re missing a bunch of default setup we do in our official images (some Alpine packages installed, Caddy storage paths setup).

There’s some issues with how you’re running this – you’re not persisting Caddy’s data storage directory, which is important to ensure you don’t lose your certificates when the container is recreated.

Remove these lines, they don’t do anything.

This is weird. Looks like your container wasn’t able to get a response from your DNS resolver at 15.223.21.81. Is that your ISP’s DNS resolver?

You might need to configure resolvers in your Caddy tls config to some other DNS server like 1.1.1.1 (Cloudflare’s DNS).

Hi Francis,
thanks for your response.
Yeah It’s all still rough I’had already refined the Dockerfile to use the builder image and other small things.

I’m still having issues.

INF ts=1705394388.6209052 msg=using provided configuration config_file=/etc/caddy/Caddyfile config_adapter=caddyfile
INF ts=1705394388.6218362 logger=admin msg=admin endpoint started address=localhost:2019 enforce_origin=false origins=["//localhost:2019","//[::1]:2019","//127.0.0.1:2019"]
INF ts=1705394388.6219456 logger=http.auto_https msg=server is listening only on the HTTPS port but has no TLS connection policies; adding one to enable TLS server_name=srv0 https_port=443
INF ts=1705394388.6219509 logger=http.auto_https msg=enabling automatic HTTP->HTTPS redirects server_name=srv0
DBG ts=1705394388.6219623 logger=http.auto_https msg=adjusted config tls={"automation":{"policies":[{"subjects":["*****.duckdns.org"]},{}]}} http={"servers":{"remaining_auto_https_redirects":{"listen":[":80"],"routes":[{},{}]},"srv0":{"listen":[":443"],"routes":[{"handle":[{"handler":"subroute","routes":[{"handle":[{"handler":"vars","root":"/www"},{"browse":{},"handler":"file_server","hide":["/etc/caddy/Caddyfile"]}]}]}],"terminal":true}],"tls_connection_policies":[{}],"automatic_https":{}}}}
INF ts=1705394388.621983 logger=tls.cache.maintenance msg=started background certificate maintenance cache=0xc000127100
INF ts=1705394388.6220593 logger=http msg=enabling HTTP/3 listener addr=:443
DBG ts=1705394388.6221666 logger=http msg=starting server loop address=[::]:443 tls=true http3=true
INF ts=1705394388.6221752 logger=http.log msg=server running name=srv0 protocols=["h1","h2","h3"]
DBG ts=1705394388.6221888 logger=http msg=starting server loop address=[::]:80 tls=false http3=false
INF ts=1705394388.6221917 logger=http.log msg=server running name=remaining_auto_https_redirects protocols=["h1","h2","h3"]
INF ts=1705394388.622193 logger=http msg=enabling automatic TLS certificate management domains=["*****.duckdns.org"]
INF ts=1705394388.6223388 msg=autosaved config (load with --resume flag) file=/config/caddy/autosave.json
INF ts=1705394388.6223414 msg=serving initial configuration
INF ts=1705394388.6223998 logger=tls.obtain msg=acquiring lock identifier=*****.duckdns.org

{"level":"warn","ts":1705394388.6286087,"logger":"tls","msg":"storage cleaning happened too recently; skipping for now","storage":"FileStorage:/data/caddy","instance":"1e426a44-dca5-45d1-b2c3-7f4d0742ecef","try_again":1705480788.6286068,"try_again_in":86399.999999607}
INF ts=1705394388.6286645 logger=tls msg=finished cleaning storage units
INF ts=1705394388.634508 logger=tls.obtain msg=lock acquired identifier=*****.duckdns.org
INF ts=1705394388.6349092 logger=tls.obtain msg=obtaining certificate identifier=*****.duckdns.org
DBG ts=1705394388.6350422 logger=events msg=event name=cert_obtaining id=55988b0e-60af-4276-b887-086bea57d8c8 origin=tls data={"identifier":"*****.duckdns.org"}
DBG ts=1705394388.6365952 logger=tls.obtain msg=trying issuer 1/2 issuer=acme-v02.api.letsencrypt.org-directory
INF ts=1705394388.6368206 logger=tls.issuance.acme msg=waiting on internal rate limiter identifiers=["*****.duckdns.org"] ca=https://acme-v02.api.letsencrypt.org/directory account=*****@*****.com
INF ts=1705394388.6368318 logger=tls.issuance.acme msg=done waiting on internal rate limiter identifiers=["*****.duckdns.org"] ca=https://acme-v02.api.letsencrypt.org/directory account=*****@*****.com
DBG ts=1705394389.139512 logger=tls.issuance.acme.acme_client msg=http request method=GET url=https://acme-v02.api.letsencrypt.org/directory headers={"User-Agent":["Caddy/2.7.6 CertMagic acmez (linux; amd64)"]} response_headers={"Cache-Control":["public, max-age=0, no-cache"],"Content-Length":["752"],"Content-Type":["application/json"],"Date":["Tue, 16 Jan 2024 08:39:49 GMT"],"Server":["nginx"],"Strict-Transport-Security":["max-age=604800"],"X-Frame-Options":["DENY"]} status_code=200
DBG ts=1705394389.291042 logger=tls.issuance.acme.acme_client msg=http request method=HEAD url=https://acme-v02.api.letsencrypt.org/acme/new-nonce headers={"User-Agent":["Caddy/2.7.6 CertMagic acmez (linux; amd64)"]} response_headers={"Cache-Control":["public, max-age=0, no-cache"],"Date":["Tue, 16 Jan 2024 08:39:49 GMT"],"Link":["<https://acme-v02.api.letsencrypt.org/directory>;rel=\"index\""],"Replay-Nonce":["cBLV7b5isUOakpoAg3lXeHAhkzDLKrUMj3YvFCDvtQiqaLZ8HlE"],"Server":["nginx"],"Strict-Transport-Security":["max-age=604800"],"X-Frame-Options":["DENY"]} status_code=200
DBG ts=1705394389.6688604 logger=tls.issuance.acme.acme_client msg=http request method=POST url=https://acme-v02.api.letsencrypt.org/acme/new-order headers={"Content-Type":["application/jose+json"],"User-Agent":["Caddy/2.7.6 CertMagic acmez (linux; amd64)"]} response_headers={"Boulder-Requester":["1519018246"],"Cache-Control":["public, max-age=0, no-cache"],"Content-Length":["343"],"Content-Type":["application/json"],"Date":["Tue, 16 Jan 2024 08:39:49 GMT"],"Link":["<https://acme-v02.api.letsencrypt.org/directory>;rel=\"index\""],"Location":["https://acme-v02.api.letsencrypt.org/acme/order/1519018246/237071005576"],"Replay-Nonce":["wFJtXv4d5NdBPY-ggWKQstcK0wGaXVyzo92yAQLpX14RsT3joxQ"],"Server":["nginx"],"Strict-Transport-Security":["max-age=604800"],"X-Frame-Options":["DENY"]} status_code=201
DBG ts=1705394389.825626 logger=tls.issuance.acme.acme_client msg=http request method=POST url=https://acme-v02.api.letsencrypt.org/acme/authz-v3/304740693186 headers={"Content-Type":["application/jose+json"],"User-Agent":["Caddy/2.7.6 CertMagic acmez (linux; amd64)"]} response_headers={"Boulder-Requester":["1519018246"],"Cache-Control":["public, max-age=0, no-cache"],"Content-Length":["801"],"Content-Type":["application/json"],"Date":["Tue, 16 Jan 2024 08:39:49 GMT"],"Link":["<https://acme-v02.api.letsencrypt.org/directory>;rel=\"index\""],"Replay-Nonce":["cBLV7b5i5Ts6MJ4s8trYgRgo3SqworRyhNhGpMtRm2cNyV3D8uM"],"Server":["nginx"],"Strict-Transport-Security":["max-age=604800"],"X-Frame-Options":["DENY"]} status_code=200
INF ts=1705394389.826035 logger=tls.issuance.acme.acme_client msg=trying to solve challenge identifier=*****.duckdns.org challenge_type=dns-01 ca=https://acme-v02.api.letsencrypt.org/directory
DBG ts=1705394393.7865484 logger=tls.issuance.acme.acme_client msg=waiting for solver before continuing identifier=*****.duckdns.org challenge_type=dns-01
DBG ts=1705394410.6390145 logger=tls.issuance.acme.acme_client msg=done waiting for solver identifier=*****.duckdns.org challenge_type=dns-01
DBG ts=1705394410.9641414 logger=tls.issuance.acme.acme_client msg=http request method=POST url=https://acme-v02.api.letsencrypt.org/acme/authz-v3/304740693186 headers={"Content-Type":["application/jose+json"],"User-Agent":["Caddy/2.7.6 CertMagic acmez (linux; amd64)"]} response_headers={"Boulder-Requester":["1519018246"],"Cache-Control":["public, max-age=0, no-cache"],"Content-Length":["805"],"Content-Type":["application/json"],"Date":["Tue, 16 Jan 2024 08:40:10 GMT"],"Link":["<https://acme-v02.api.letsencrypt.org/directory>;rel=\"index\""],"Replay-Nonce":["wFJtXv4d9AqgJLzUd15yBQdoCHQDsMfRS_Ksb6UZm0nIj6Clc2Q"],"Server":["nginx"],"Strict-Transport-Security":["max-age=604800"],"X-Frame-Options":["DENY"]} status_code=200
ERR ts=1705394410.9643714 logger=tls.obtain msg=could not get certificate from issuer identifier=*****.duckdns.org issuer=acme-v02.api.letsencrypt.org-directory error=[*****.duckdns.org] solving challenges: waiting for solver certmagic.solverWrapper to be ready: checking DNS propagation of "_acme-challenge.*****.duckdns.org": dial udp: lookup ns3.duckdns.org. on 1.1.1.1:53: no such host (order=https://acme-v02.api.letsencrypt.org/acme/order/1519018246/237071005576) (ca=https://acme-v02.api.letsencrypt.org/directory)
DBG ts=1705394410.9644015 logger=tls.obtain msg=trying issuer 2/2 issuer=acme.zerossl.com-v2-DV90
INF ts=1705394410.9649627 logger=tls.issuance.zerossl msg=waiting on internal rate limiter identifiers=["*****.duckdns.org"] ca=https://acme.zerossl.com/v2/DV90 account=*****@*****.com
INF ts=1705394410.9650488 logger=tls.issuance.zerossl msg=done waiting on internal rate limiter identifiers=["*****.duckdns.org"] ca=https://acme.zerossl.com/v2/DV90 account=*****@*****.com
DBG ts=1705394411.235994 logger=tls.issuance.zerossl.acme_client msg=http request method=GET url=https://acme.zerossl.com/v2/DV90 headers={"User-Agent":["Caddy/2.7.6 CertMagic acmez (linux; amd64)"]} response_headers={"Access-Control-Allow-Origin":["*"],"Content-Length":["645"],"Content-Type":["application/json"],"Date":["Tue, 16 Jan 2024 08:40:11 GMT"],"Server":["nginx"],"Strict-Transport-Security":["max-age=15724800; includeSubDomains"]} status_code=200
DBG ts=1705394413.8743181 logger=tls.issuance.zerossl.acme_client msg=http request method=HEAD url=https://acme.zerossl.com/v2/DV90/newNonce headers={"User-Agent":["Caddy/2.7.6 CertMagic acmez (linux; amd64)"]} response_headers={"Access-Control-Allow-Origin":["*"],"Cache-Control":["max-age=0, no-cache, no-store"],"Content-Type":["application/octet-stream"],"Date":["Tue, 16 Jan 2024 08:40:13 GMT"],"Link":["<https://acme.zerossl.com/v2/DV90>;rel=\"index\""],"Replay-Nonce":["hWscuT7IBwKiq09iUvBgdDG-OEmbkgto4R2M7NkBScU"],"Server":["nginx"],"Strict-Transport-Security":["max-age=15724800; includeSubDomains"]} status_code=200
DBG ts=1705394415.175307 logger=tls.issuance.zerossl.acme_client msg=http request method=POST url=https://acme.zerossl.com/v2/DV90/newOrder headers={"Content-Type":["application/jose+json"],"User-Agent":["Caddy/2.7.6 CertMagic acmez (linux; amd64)"]} response_headers={"Access-Control-Allow-Origin":["*"],"Cache-Control":["max-age=0, no-cache, no-store"],"Content-Length":["279"],"Content-Type":["application/json"],"Date":["Tue, 16 Jan 2024 08:40:15 GMT"],"Location":["https://acme.zerossl.com/v2/DV90/order/2NS6k_0nyafFf11xX-bBVw"],"Replay-Nonce":["45NLcqglV53SDDBaUEnMJPJANiqqEiCzsQMITXg5Z74"],"Server":["nginx"],"Strict-Transport-Security":["max-age=15724800; includeSubDomains"]} status_code=201
DBG ts=1705394416.9916315 logger=tls.issuance.zerossl.acme_client msg=http request method=POST url=https://acme.zerossl.com/v2/DV90/authz/zdJhVUy_8zgmeLIfOZVbyg headers={"Content-Type":["application/jose+json"],"User-Agent":["Caddy/2.7.6 CertMagic acmez (linux; amd64)"]} response_headers={"Access-Control-Allow-Origin":["*"],"Cache-Control":["max-age=0, no-cache, no-store"],"Content-Length":["447"],"Content-Type":["application/json"],"Date":["Tue, 16 Jan 2024 08:40:16 GMT"],"Link":["<https://acme.zerossl.com/v2/DV90>;rel=\"index\""],"Replay-Nonce":["54Y7I7xdlDPEQPVLITMJpzSpeigj5wgeKxoq35qwNQs"],"Retry-After":["5"],"Server":["nginx"],"Strict-Transport-Security":["max-age=15724800; includeSubDomains"]} status_code=200
INF ts=1705394416.9918294 logger=tls.issuance.zerossl.acme_client msg=trying to solve challenge identifier=*****.duckdns.org challenge_type=dns-01 ca=https://acme.zerossl.com/v2/DV90
DBG ts=1705394417.1247382 logger=tls.issuance.zerossl.acme_client msg=waiting for solver before continuing identifier=*****.duckdns.org challenge_type=dns-01
DBG ts=1705394442.2014103 logger=tls.issuance.zerossl.acme_client msg=done waiting for solver identifier=*****.duckdns.org challenge_type=dns-01
DBG ts=1705394442.5642166 logger=tls.issuance.zerossl.acme_client msg=http request method=POST url=https://acme.zerossl.com/v2/DV90/authz/zdJhVUy_8zgmeLIfOZVbyg headers={"Content-Type":["application/jose+json"],"User-Agent":["Caddy/2.7.6 CertMagic acmez (linux; amd64)"]} response_headers={"Access-Control-Allow-Origin":["*"],"Cache-Control":["max-age=0, no-cache, no-store"],"Content-Length":["129"],"Content-Type":["application/json"],"Date":["Tue, 16 Jan 2024 08:40:42 GMT"],"Link":["<https://acme.zerossl.com/v2/DV90>;rel=\"index\""],"Replay-Nonce":["XuF-nwR_hlQoYVplakqKm_FtAhUD5SSVuvEhq38L6iI"],"Retry-After":["86400"],"Server":["nginx"],"Strict-Transport-Security":["max-age=15724800; includeSubDomains"]} status_code=200

:"error","ts":1705394442.5644646,"logger":"tls.obtain","msg":"could not get certificate from issuer","identifier":"*****.duckdns.org","issuer":"acme.zerossl.com-v2-DV90","error":"[*****.duckdns.org] solving challenges: waiting for solver certmagic.solverWrapper to be ready: checking DNS propagation of \"_acme-challenge.*****.duckdns.org\": dial tcp: lookup ns3.duckdns.org. on 1.1.1.1:53: no such host (order=https://acme.zerossl.com/v2/DV90/order/2NS6k_0nyafFf11xX-bBVw) (ca=https://acme.zerossl.com/v2/DV90)"}
DBG ts=1705394442.564525 logger=events msg=event name=cert_failed id=dfc3056c-337a-41a7-a537-13307c7c7eda origin=tls data={"error":{},"identifier":"*****.duckdns.org","issuers":["acme-v02.api.letsencrypt.org-directory","acme.zerossl.com-v2-DV90"],"renewal":false}
ERR ts=1705394442.56466 logger=tls.obtain msg=will retry error=[*****.duckdns.org] Obtain: [*****.duckdns.org] solving challenges: waiting for solver certmagic.solverWrapper to be ready: checking DNS propagation of "_acme-challenge.*****.duckdns.org": dial tcp: lookup ns3.duckdns.org. on 1.1.1.1:53: no such host (order=https://acme.zerossl.com/v2/DV90/order/2NS6k_0nyafFf11xX-bBVw) (ca=https://acme.zerossl.com/v2/DV90) attempt=1 retrying_in=60 elapsed=53.930139181 max_duration=2592000

Other updated conf/commands:

#caddyfile
{
	debug
	email ******@******.com
}

******.duckdns.org
file_server browse
root * /www

tls {
	resolvers 8.8.8.8:53
	dns duckdns aaaaaaa
}
FROM docker.io/caddy:builder as builder

RUN xcaddy build --with github.com/caddy-dns/duckdns

FROM docker.io/caddy:latest

COPY --from=builder /usr/bin/caddy /usr/bin/caddy
sudo podman run -d --replace \
    --name caddy \
    -p 80:80 -p 443:443 -p 443:443/udp \
    -v $PWD/data:/data \
    -v $PWD/config:/config \
    -v $PWD/www:/www \
    -v $PWD/Caddyfile:/etc/caddy/Caddyfile \
    mycaddy

I’m actually using cloudflare 1.1.1.1 in my isp router.

Something about your system is messed up. This reports just fine for me:

dig @1.1.1.1 ns3.duckdns.org 

Anyway you can turn off propagation checks (the shotgun approach) with propagation_timeout -1 in your tls config. tls (Caddyfile directive) — Caddy Documentation

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.