Could not get certificate from issuer due to DNS timeout

Help
1

1. The problem I’m having:

The requests from Caddy to get certificates stopped working suddenly. I do not recollect making any changes to my setup recently except updating Ubuntu and rebooting my Raspberry Pi 4.

I connected to the docker container and I verified using curl that the domains are reachable and I’m able to get a response. I don’t quite understand why caddy is having issues connecting to the DNS server. Appreciate any help to fix this problem.

/etc/caddy # curl -v https://acme-staging-v02.api.letsencrypt.org/directory
* Host acme-staging-v02.api.letsencrypt.org:443 was resolved.
* IPv6: 2606:4700:60:0:f41b:d4fe:4325:6026
* IPv4: 172.65.46.172
*   Trying [2606:4700:60:0:f41b:d4fe:4325:6026]:443...
* Connected to acme-staging-v02.api.letsencrypt.org (2606:4700:60:0:f41b:d4fe:4325:6026) port 443
* ALPN: curl offers h2,http/1.1
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
*  CAfile: /etc/ssl/certs/ca-certificates.crt
*  CApath: /etc/ssl/certs
* TLSv1.3 (IN), TLS handshake, Server hello (2):
* TLSv1.3 (IN), TLS handshake, Encrypted Extensions (8):
* TLSv1.3 (IN), TLS handshake, Certificate (11):
* TLSv1.3 (IN), TLS handshake, CERT verify (15):
* TLSv1.3 (IN), TLS handshake, Finished (20):
* TLSv1.3 (OUT), TLS change cipher, Change cipher spec (1):
* TLSv1.3 (OUT), TLS handshake, Finished (20):
* SSL connection using TLSv1.3 / TLS_AES_256_GCM_SHA384 / X25519 / RSASSA-PSS
* ALPN: server accepted h2
* Server certificate:
*  subject: CN=acme-staging-v02.api.letsencrypt.org
*  start date: Dec 27 10:10:47 2023 GMT
*  expire date: Mar 26 10:10:46 2024 GMT
*  subjectAltName: host "acme-staging-v02.api.letsencrypt.org" matched cert's "acme-staging-v02.api.letsencrypt.org"
*  issuer: C=US; O=Let's Encrypt; CN=R3
*  SSL certificate verify ok.
*   Certificate level 0: Public key type RSA (2048/112 Bits/secBits), signed using sha256WithRSAEncryption
*   Certificate level 1: Public key type RSA (2048/112 Bits/secBits), signed using sha256WithRSAEncryption
*   Certificate level 2: Public key type RSA (4096/152 Bits/secBits), signed using sha256WithRSAEncryption
* using HTTP/2
* [HTTP/2] [1] OPENED stream for https://acme-staging-v02.api.letsencrypt.org/directory
* [HTTP/2] [1] [:method: GET]
* [HTTP/2] [1] [:scheme: https]
* [HTTP/2] [1] [:authority: acme-staging-v02.api.letsencrypt.org]
* [HTTP/2] [1] [:path: /directory]
* [HTTP/2] [1] [user-agent: curl/8.5.0]
* [HTTP/2] [1] [accept: */*]
> GET /directory HTTP/2
> Host: acme-staging-v02.api.letsencrypt.org
> User-Agent: curl/8.5.0
> Accept: */*
> 
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
* old SSL session ID is stale, removing
< HTTP/2 200 
< server: nginx
< date: Sat, 17 Feb 2024 21:05:17 GMT
< content-type: application/json
< content-length: 826
< cache-control: public, max-age=0, no-cache
< x-frame-options: DENY
< strict-transport-security: max-age=604800
< 
{
  "G4wLOYJydaQ": "https://community.letsencrypt.org/t/adding-random-entries-to-the-directory/33417",
  "keyChange": "https://acme-staging-v02.api.letsencrypt.org/acme/key-change",
  "meta": {
    "caaIdentities": [
      "letsencrypt.org"
    ],
    "termsOfService": "https://letsencrypt.org/documents/LE-SA-v1.3-September-21-2022.pdf",
    "website": "https://letsencrypt.org/docs/staging-environment/"
  },
  "newAccount": "https://acme-staging-v02.api.letsencrypt.org/acme/new-acct",
  "newNonce": "https://acme-staging-v02.api.letsencrypt.org/acme/new-nonce",
  "newOrder": "https://acme-staging-v02.api.letsencrypt.org/acme/new-order",
  "renewalInfo": "https://acme-staging-v02.api.letsencrypt.org/draft-ietf-acme-ari-02/renewalInfo/",
  "revokeCert": "https://acme-staging-v02.api.letsencrypt.org/acme/revoke-cert"
* Connection #0 to host acme-staging-v02.api.letsencrypt.org left intact
}

/etc/caddy # curl -v https://acme.zerossl.com/v2/DV90
* Host acme.zerossl.com:443 was resolved.
* IPv6: (none)
* IPv4: 91.199.212.80
*   Trying 91.199.212.80:443...
* Connected to acme.zerossl.com (91.199.212.80) port 443
* ALPN: curl offers h2,http/1.1
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
*  CAfile: /etc/ssl/certs/ca-certificates.crt
*  CApath: /etc/ssl/certs
* TLSv1.3 (IN), TLS handshake, Server hello (2):
* TLSv1.2 (IN), TLS handshake, Certificate (11):
* TLSv1.2 (IN), TLS handshake, Server key exchange (12):
* TLSv1.2 (IN), TLS handshake, Server finished (14):
* TLSv1.2 (OUT), TLS handshake, Client key exchange (16):
* TLSv1.2 (OUT), TLS change cipher, Change cipher spec (1):
* TLSv1.2 (OUT), TLS handshake, Finished (20):
* TLSv1.2 (IN), TLS handshake, Finished (20):
* SSL connection using TLSv1.2 / ECDHE-RSA-CHACHA20-POLY1305 / X25519 / RSASSA-PSS
* ALPN: server accepted h2
* Server certificate:
*  subject: C=GB; ST=Manchester; O=Sectigo Limited; CN=acme.zerossl.com
*  start date: Jan 26 00:00:00 2024 GMT
*  expire date: Apr 25 23:59:59 2024 GMT
*  subjectAltName: host "acme.zerossl.com" matched cert's "acme.zerossl.com"
*  issuer: C=GB; ST=Greater Manchester; L=Salford; O=Sectigo Limited; CN=Sectigo RSA Organization Validation Secure Server CA
*  SSL certificate verify ok.
*   Certificate level 0: Public key type RSA (4096/152 Bits/secBits), signed using sha256WithRSAEncryption
*   Certificate level 1: Public key type RSA (2048/112 Bits/secBits), signed using sha384WithRSAEncryption
*   Certificate level 2: Public key type RSA (4096/152 Bits/secBits), signed using sha384WithRSAEncryption
* using HTTP/2
* [HTTP/2] [1] OPENED stream for https://acme.zerossl.com/v2/DV90
* [HTTP/2] [1] [:method: GET]
* [HTTP/2] [1] [:scheme: https]
* [HTTP/2] [1] [:authority: acme.zerossl.com]
* [HTTP/2] [1] [:path: /v2/DV90]
* [HTTP/2] [1] [user-agent: curl/8.5.0]
* [HTTP/2] [1] [accept: */*]
> GET /v2/DV90 HTTP/2
> Host: acme.zerossl.com
> User-Agent: curl/8.5.0
> Accept: */*
> 
< HTTP/2 200 
< server: nginx
< date: Sat, 17 Feb 2024 21:06:57 GMT
< content-type: application/json
< content-length: 645
< access-control-allow-origin: *
< strict-transport-security: max-age=15724800; includeSubDomains
< 
{
  "newNonce": "https://acme.zerossl.com/v2/DV90/newNonce",
  "newAccount": "https://acme.zerossl.com/v2/DV90/newAccount",
  "newOrder": "https://acme.zerossl.com/v2/DV90/newOrder",
  "revokeCert": "https://acme.zerossl.com/v2/DV90/revokeCert",
  "keyChange": "https://acme.zerossl.com/v2/DV90/keyChange",
  "meta": {
    "termsOfService": "https://secure.trust-provider.com/repository/docs/Legacy/20230516_Certificate_Subscriber_Agreement_v_2_6_click.pdf",
    "website": "https://zerossl.com",
    "caaIdentities": ["sectigo.com", "trust-provider.com", "usertrust.com", "comodoca.com", "comodo.com"],
    "externalAccountRequired": true
  }
* Connection #0 to host acme.zerossl.com left intact

2. Error messages and/or full log output:

INF ts=1708201167.5861866 logger=tls.obtain msg=obtaining certificate identifier=bwtechhideout.duckdns.org

WRN ts=1708201177.607031 logger=http.acme_client msg=HTTP request failed; retrying url=https://acme-staging-v02.api.letsencrypt.org/directory error=performing request: Get "https://acme-staging-v02.api.letsencrypt.org/directory": dial tcp: lookup acme-staging-v02.api.letsencrypt.org on 1.1.1.1:53: read udp 192.168.1.22:47631->1.1.1.1:53: i/o timeout

WRN ts=1708201187.8615015 logger=http.acme_client msg=HTTP request failed; retrying url=https://acme-staging-v02.api.letsencrypt.org/directory error=performing request: Get "https://acme-staging-v02.api.letsencrypt.org/directory": dial tcp: lookup acme-staging-v02.api.letsencrypt.org on 1.1.1.1:53: read udp 192.168.1.22:56622->1.1.1.1:53: i/o timeout

WRN ts=1708201198.1158926 logger=http.acme_client msg=HTTP request failed; retrying url=https://acme-staging-v02.api.letsencrypt.org/directory error=performing request: Get "https://acme-staging-v02.api.letsencrypt.org/directory": dial tcp: lookup acme-staging-v02.api.letsencrypt.org on 1.1.1.1:53: read udp 192.168.1.22:36649->1.1.1.1:53: i/o timeout

ERR ts=1708201198.116072 logger=tls.obtain msg=could not get certificate from issuer identifier=bwtechhideout.duckdns.org issuer=acme-v02.api.letsencrypt.org-directory error=[bwtechhideout.duckdns.org] creating new order: provisioning client: performing request: Get "https://acme-staging-v02.api.letsencrypt.org/directory": dial tcp: lookup acme-staging-v02.api.letsencrypt.org on 1.1.1.1:53: read udp 192.168.1.22:36649->1.1.1.1:53: i/o timeout (ca=https://acme-staging-v02.api.letsencrypt.org/directory)

WRN ts=1708201208.121339 logger=http.acme_client msg=HTTP request failed; retrying url=https://acme.zerossl.com/v2/DV90 error=performing request: Get "https://acme.zerossl.com/v2/DV90": dial tcp: lookup acme.zerossl.com on 1.1.1.1:53: read udp 192.168.1.22:37054->1.1.1.1:53: i/o timeout

WRN ts=1708201218.375631 logger=http.acme_client msg=HTTP request failed; retrying url=https://acme.zerossl.com/v2/DV90 error=performing request: Get "https://acme.zerossl.com/v2/DV90": dial tcp: lookup acme.zerossl.com on 1.1.1.1:53: read udp 192.168.1.22:55081->1.1.1.1:53: i/o timeout

WRN ts=1708201228.6300025 logger=http.acme_client msg=HTTP request failed; retrying url=https://acme.zerossl.com/v2/DV90 error=performing request: Get "https://acme.zerossl.com/v2/DV90": dial tcp: lookup acme.zerossl.com on 1.1.1.1:53: read udp 192.168.1.22:43947->1.1.1.1:53: i/o timeout

ERR ts=1708201228.6302085 logger=tls.obtain msg=could not get certificate from issuer identifier=bwtechhideout.duckdns.org issuer=acme.zerossl.com-v2-DV90 error=[bwtechhideout.duckdns.org] creating new order: provisioning client: performing request: Get "https://acme.zerossl.com/v2/DV90": dial tcp: lookup acme.zerossl.com on 1.1.1.1:53: read udp 192.168.1.22:43947->1.1.1.1:53: i/o timeout (ca=https://acme.zerossl.com/v2/DV90)

ERR ts=1708201228.6303654 logger=tls.obtain msg=will retry error=[bwtechhideout.duckdns.org] Obtain: [bwtechhideout.duckdns.org] creating new order: provisioning client: performing request: Get "https://acme.zerossl.com/v2/DV90": dial tcp: lookup acme.zerossl.com on 1.1.1.1:53: read udp 192.168.1.22:43947->1.1.1.1:53: i/o timeout (ca=https://acme.zerossl.com/v2/DV90) attempt=66 retrying_in=21600 elapsed=954520.23355187 max_duration=2592000

3. Caddy version:

caddy version
v2.7.6 h1:w0NymbG2m9PcvKWsrXO6EEkY9Ru4FJK8uQbYcev1p3A=

4. How I installed and ran Caddy:

a. System environment:

Docker on Ubuntu 22.04

Distributor ID: Ubuntu
Description:    Ubuntu 22.04.3 LTS
Release:        22.04
Codename:       jammy

b. Command:

docker compose up

c. Service/unit/compose file:

  caddy:
    image: caddy:alpine
    restart: unless-stopped
    container_name: caddy
    volumes:
      - /media/kiran/Kiran/docker-config/caddy/Caddyfile:/etc/caddy/Caddyfile
      - /media/kiran/Kiran/docker-config/caddy/data:/data
      - /media/kiran/Kiran/docker-config/caddy/config:/config
      - /media/kiran/Kiran/docker-config/caddy/certs:/certs
      - /media/kiran/Kiran/docker-config/caddy//sites:/srv
    network_mode: "host"

d. My complete Caddy config:

https://nxttechhideout.duckdns.org:443 {
    header Strict-Transport-Security max-age=31536000;
    reverse_proxy localhost:11000
}

https://bwtechhideout.duckdns.org:443 {
#    header Strict-Transport-Security max-age=31536000;
    reverse_proxy localhost:7080
}

5. Links to relevant resources:

2

I have a temporary workaround now by copying the IP address of these domains to /etc/hosts file and I’m able to get certificates now. But, I still don’t understand why Caddy is having issues connecting to the DNS server when curl does not have a problem.

3

:man_shrugging:

It must be something weird with your Docker setup. Caddy just uses the system’s DNS resolver.