Sudden issues with Caddy Container (DNS and ACME challenge)

Below config used to work flawlessly 2 months ago.

Okay so I downloaded the Caddy module for Duckdns for Linux AMD 64 from website.

I use Duckdns for giving https to my local ip 192.168.1.197 with domain: adguardcad.duckdns.org
And my API key for DuckDNS is token01-ford-apli1-lane-8c21055d2331

Now I use caddy for doing it, where my CaddyFile is

adguardcad.duckdns.org:443 {
  

  # Use the ACME DNS-01 challenge to get a cert for the configured domain.
  tls {
    dns duckdns token01-ford-apli1-lane-8c21055d2331
  }

  # This setting may have compatibility issues with some browsers
  # (e.g., attachment downloading on Firefox). Try disabling this
  # if you encounter issues.
  encode gzip
  
  reverse_proxy adguardhome:80
  reverse_proxy /notifications/hub/ adguardhome:3012

  # Proxy everything else to Rocket
}

Now my code for Caddy Docker container is

docker run -d \
  --name adguardcaddy \
  --restart unless-stopped \
  --ip 192.168.1.197 \
  --dns 1.1.1.1 \
  --network="net" \
  -p 80:80 \
  -p 443:443 \
  -v /drive1t/dockerconf/caddy/caddy:/usr/bin/caddy \
  -v /drive1t/dockerconf/caddy/Caddyfile:/etc/caddy/Caddyfile:ro \
  -v /drive1t/dockerconf/caddy/caddy-config:/config \
  -v /drive1t/dockerconf/caddy/caddy-data:/data \
  -e DOMAIN=adguardcad.duckdns.org \
  -e EMAIL=home@xyz.com \
  -e TOKEN=token01-ford-apli1-lane-8c21055d2331 \
  -e LOG_FILE=/data/access.log \
  -e DUCKDNS_DOMAIN=https://adguardcad.duckdns.org \
  -e DUCKDNS_TOKEN=token01-ford-apli1-lane-8c21055d2331 \
  caddy:latest

But I get error in container logs:

INF ts=1714099311.3780174 logger=tls.obtain msg=obtaining certificate identifier=adguardcad.duckdns.org
INF ts=1714099312.7734187 logger=tls.issuance.acme msg=waiting on internal rate limiter identifiers=[“adguardcad.duckdns.org”] ca=https://acme-v02.api.letsencrypt.org/directory account=
INF ts=1714099312.7734914 logger=tls.issuance.acme msg=done waiting on internal rate limiter identifiers=[“adguardcad.duckdns.org”] ca=https://acme-v02.api.letsencrypt.org/directory account=
INF ts=1714099313.3487773 logger=tls.issuance.acme.acme_client msg=trying to solve challenge identifier=adguardcad.duckdns.org challenge_type=dns-01 ca=https://acme-v02.api.letsencrypt.org/directory
ERR ts=1714099360.3287015 logger=tls.obtain msg=could not get certificate from issuer identifier=adguardcad.duckdns.org issuer=acme-v02.api.letsencrypt.org-directory error=[adguardcad.duckdns.org] solving challenges: waiting for solver certmagic.solverWrapper to be ready: checking DNS propagation of “_acme-challenge.adguardcad.duckdns.org”: dial tcp 3.97.51.116:53: i/o timeout (order=https://acme-v02.api.letsencrypt.org/acme/order/1693662117/264145700467) (ca=https://acme-v02.api.letsencrypt.org/directory)
WRN ts=1714099360.3292382 logger=tls.issuance.zerossl msg=missing email address for ZeroSSL; it is strongly recommended to set one for next time
INF ts=1714099361.5142632 logger=tls.issuance.zerossl msg=generated EAB credentials key_id=NYlXS6Fv5Pl9UuKzcH5JXA
INF ts=1714099363.5542395 logger=tls.issuance.zerossl msg=waiting on internal rate limiter identifiers=[“adguardcad.duckdns.org”] ca=https://acme.zerossl.com/v2/DV90 account=
INF ts=1714099363.5543056 logger=tls.issuance.zerossl msg=done waiting on internal rate limiter identifiers=[“adguardcad.duckdns.org”] ca=https://acme.zerossl.com/v2/DV90 account=
INF ts=1714099364.2849708 logger=tls.issuance.zerossl.acme_client msg=trying to solve challenge identifier=adguardcad.duckdns.org challenge_type=dns-01 ca=https://acme.zerossl.com/v2/DV90
ERR ts=1714099392.3516834 logger=tls.obtain msg=could not get certificate from issuer identifier=adguardcad.duckdns.org issuer=acme.zerossl.com-v2-DV90 error=[adguardcad.duckdns.org] solving challenges: waiting for solver certmagic.solverWrapper to be ready: checking DNS propagation of “_acme-challenge.adguardcad.duckdns.org”: dial tcp 35.183.157.249:53: i/o timeout (order=https://acme.zerossl.com/v2/DV90/order/-mGHWGkMGrmqEIIl5eKjwQ) (ca=https://acme.zerossl.com/v2/DV90)
ERR ts=1714099392.3518286 logger=tls.obtain msg=will retry error=[adguardcad.duckdns.org] Obtain: [adguardcad.duckdns.org] solving challenges: waiting for solver certmagic.solverWrapper to be ready: checking DNS propagation of “_acme-challenge.adguardcad.duckdns.org”: dial tcp 35.183.157.249:53: i/o timeout (order=https://acme.zerossl.com/v2/DV90/order/-mGHWGkMGrmqEIIl5eKjwQ) (ca=https://acme.zerossl.com/v2/DV90) attempt=1 retrying_in=60 elapsed=80.97429205 max_duration=2592000

I have also tried ping 1.1.1.1 from inside the container and it works:

root@omv:~# docker exec -it adguardcaddy /bin/sh
/srv # ping 1.1.1.1
PING 1.1.1.1 (1.1.1.1): 56 data bytes
64 bytes from 1.1.1.1: seq=0 ttl=51 time=14.364 ms
64 bytes from 1.1.1.1: seq=1 ttl=51 time=14.050 ms
64 bytes from 1.1.1.1: seq=2 ttl=51 time=14.132 ms
64 bytes from 1.1.1.1: seq=4 ttl=51 time=14.134 ms
64 bytes from 1.1.1.1: seq=5 ttl=51 time=14.052 ms
64 bytes from 1.1.1.1: seq=6 ttl=51 time=13.622 ms
64 bytes from 1.1.1.1: seq=7 ttl=51 time=13.499 ms
64 bytes from 1.1.1.1: seq=9 ttl=51 time=13.405 ms

Hi @fedonr,

Let’s Encrypt added more remote perspectives from different geos, on their community forum there have be several instances of firewalls blocking the new geos, and therefor failing now when in the past they were successful.

2 Likes

Another option is consider using the DNS-01 challenge of the Challenge Types - Let's Encrypt.
Also consider using a DNS providers who easily integrate with Let's Encrypt DNS validation - Issuance Tech - Let's Encrypt Community Support.

1 Like

But for me it is now failing in both Let’s Encrypt and ZeroSSL, and I with my limited knowledge is trying to solve this for 3 days now without any luck

1 Like

As part of the DNS challenge issuance, Caddy does “propagation checks”, i.e. it tries to do DNS queries to verify that the TXT record was correctly written to your DNS.

It seems like when trying to do that, the DNS queries Caddy is making are failing. I assume 35.183.157.249 is your ISP’s DNS server (DNS is performed on port 53).

You can try to configure resolvers 1.1.1.1 in your tls config to tell Caddy to use a different DNS server (e.g. Cloudflare’s).

2 Likes

Tried this, didn’t seem to work. I get the same error logs

tls {
    resolvers 1.1.1.1:53
    dns duckdns token01-ford-apli1-lane-8c21055d2331
  }

Show your logs.

2 Likes

@fedonr I strongly suggest getting new (or regenerate) and different DuckDNS API Tokens!

And this too
image

Never post Private Keys, Credentials, nor secret API keys.
Editing them out is nice, but they are still loose in the wild for bad actors to get and use.

Now Public Keys are just that; anyone can and should be able to see them, so posting them is OK.

1 Like

Still facing the issues, I tried to do some trial and error. Where I did factory reset on my router and it worked for the first time, so I kept on it for 3 days trying to delete certs and restart container every 5 hours after deleting caddy data and config files. It’s working 1 out of 4 times in a day.

INF ts=xxx logger=tls.issuance.zerossl.acme_client msg=trying to solve challenge identifier=gharnas.duckdns.org challenge_type=dns-01 ca=https://acme.zerossl.com/v2/DV90

ERR ts=xxx logger=tls.issuance.zerossl.acme_client msg=cleaning up solver identifier=*.gharnas.duckdns.org challenge_type=dns-01 error=no memory of presenting a DNS record for "_acme-challenge.gharnas.duckdns.org" (usually OK if presenting also failed)

ERR ts=xxx logger=tls.issuance.zerossl.acme_client msg=cleaning up solver identifier=gharnas.duckdns.org challenge_type=dns-01 error=no memory of presenting a DNS record for "_acme-challenge.gharnas.duckdns.org" (usually OK if presenting also failed)

ERR ts=xxx logger=tls.obtain msg=could not get certificate from issuer identifier=*.gharnas.duckdns.org issuer=acme.zerossl.com-v2-DV90 error=[*.gharnas.duckdns.org] solving challenges: presenting for challenge: could not determine zone for domain "_acme-challenge.gharnas.duckdns.org": unexpected response code 'SERVFAIL' for gharnas.duckdns.org. (order=https://acme.zerossl.com/v2/DV90/order/KqujzNEER2dkJKK5RpFzvg) (ca=https://acme.zerossl.com/v2/DV90)

ERR ts=xxx logger=tls.obtain msg=will retry error=[*.gharnas.duckdns.org] Obtain: [*.gharnas.duckdns.org] solving challenges: presenting for challenge: could not determine zone for domain "_acme-challenge.gharnas.duckdns.org": unexpected response code 'SERVFAIL' for gharnas.duckdns.org. (order=https://acme.zerossl.com/v2/DV90/order/KqujzNEER2dkJKK5RpFzvg) (ca=https://acme.zerossl.com/v2/DV90) attempt=1 retrying_in=60 elapsed=10.854302968 max_duration=2592000

ERR ts=xxx logger=tls.obtain msg=could not get certificate from issuer identifier=gharnas.duckdns.org issuer=acme.zerossl.com-v2-DV90 error=[gharnas.duckdns.org] solving challenges: presenting for challenge: could not determine zone for domain "_acme-challenge.gharnas.duckdns.org": unexpected response code 'SERVFAIL' for _acme-challenge.gharnas.duckdns.org. (order=https://acme.zerossl.com/v2/DV90/order/ODv6ymYqrO5Z9lkO9-MRCg) (ca=https://acme.zerossl.com/v2/DV90)

ERR ts=xxx logger=tls.obtain msg=will retry error=[gharnas.duckdns.org] Obtain: [gharnas.duckdns.org] solving challenges: presenting for challenge: could not determine zone for domain "_acme-challenge.gharnas.duckdns.org": unexpected response code 'SERVFAIL' for _acme-challenge.gharnas.duckdns.org. (order=https://acme.zerossl.com/v2/DV90/order/ODv6ymYqrO5Z9lkO9-MRCg) (ca=https://acme.zerossl.com/v2/DV90) attempt=1 retrying_in=60 elapsed=10.85496946 max_duration=2592000

Thank you so much. I would keep that in mind

1 Like

Be careful, you might start hitting rate limits.

This just looks like DuckDNS having a short amount of downtime:

Caddy tries to renew when the cert has 30 days remaining of lifetime (of its total 90 days), and it continually retries. So intermittent errors like this are fine.

1 Like

Thank you, but I realized this issue when I URLs stopped working completely. That is when I posted this thread after 2 to 3 days of it.

That’s when I witnessed such errors in logs.

Can this be the case?

Jio block tcp and udp requests for dns

Jio, a major Indian ISP, has been observed to employ various methods to block certain websites and services. According to the search results, Jio is using a technique called SNI (Server Name Indication) inspection to block websites for its users. This method involves inspecting the SNI field in the TLS protocol to determine the intended destination of a connection and block it if necessary.

Additionally, Jio has also been reported to block UDP ports required for Cloudflare Warp, a service that provides a secure and fast way to access the internet. This blocking of UDP ports is likely done to prevent users from accessing certain websites or services that Jio may deem inappropriate.

It’s also worth noting that Jio has been known to change its public IP addresses frequently to maintain efficiency and manage its network resources. This can cause issues for users who rely on static IP addresses or have specific configurations that rely on a specific IP address.

In summary, Jio has been observed to block TCP and UDP requests for DNS, as well as block specific ports and services, in order to manage its network and block certain websites or services.

The errors you showed just now are different than the ones you had before. It’s not the same problem.

1 Like

Okay, so I did try once again and this time I kept on swapping resolvers.

It seems that Clouldflare is failing each time (out of 3), Google works everytime, and ControlD fails 2 out of 3 times.

I keep goggle Dns as of now and would test it for 3 more days and would update here.

1 Like

Hey, thanks you so much for all your help.

It is resolved now, as I had to change the resolver to google 8.8.8.8

As cloudflare and controld didn’t work.

So, for the solution was resetting router and change resolver to Google.

Initially all google, cloudflare and controld but after resetting the router Google started working.

Still I do have a question, why suddenly cloudflare stopped working it worked for more than a year for me. And why do some resolvers work while others don’t?

1 Like

Well you suggested it could have to do with your ISP. If that’s the case and they’re blocking Cloudflare for whatever reason, that would be an explanation.

1 Like