Resolver using localhost instead of /etc/resolv.conf

1. The problem I’m having:

When trying to get a certificate via acme from LE using dns challenge, caddy queries localhost instead of the DNS set in /etc/resolv.conf. As localhost has no process listening on port 53, this fails.

I tried to force the use of cgo using export GODEBUG=netdns=cgo+1 but got
go package net: cgo resolver not supported; using Go's DNS resolver

I then tried to get debug info from the go resolver using export GODEBUG=netdns=go+1. But all I got was go package net: GODEBUG setting forcing use of Go's resolver and no other debug info.

2. Error messages and/or full log output:

2024/05/02 15:07:44.587 INFO    using adjacent Caddyfile
2024/05/02 15:07:44.589 INFO    admin   admin endpoint started  {"address": "localhost:2019", "enforce_origin": false, "origins": ["//localhost:2019", "//[::1]:2019", "//127.0.0.1:2019"]}
2024/05/02 15:07:44.589 INFO    http.auto_https server is listening only on the HTTPS port but has no TLS connection policies; adding one to enable TLS {"server_name": "srv0", "https_port": 443}
2024/05/02 15:07:44.590 INFO    http.auto_https enabling automatic HTTP->HTTPS redirects        {"server_name": "srv0"}
2024/05/02 15:07:44.590 INFO    tls.cache.maintenance   started background certificate maintenance      {"cache": "0xc000381380"}
2024/05/02 15:07:44.590 INFO    http.log        server running  {"name": "remaining_auto_https_redirects", "protocols": ["h1", "h2", "h3"]}
2024/05/02 15:07:44.590 INFO    http    enabling HTTP/3 listener        {"addr": ":443"}
2024/05/02 15:07:44.590 INFO    failed to sufficiently increase receive buffer size (was: 208 kiB, wanted: 2048 kiB, got: 416 kiB). See https://github.com/quic-go/quic-go/wiki/UDP-Buffer-Sizes for details.
2024/05/02 15:07:44.590 INFO    http.log        server running  {"name": "srv0", "protocols": ["h1", "h2", "h3"]}
2024/05/02 15:07:44.590 INFO    http    enabling automatic TLS certificate management   {"domains": ["server.tld.com"]}
2024/05/02 15:07:44.590 INFO    autosaved config (load with --resume flag)      {"file": "/root/.config/caddy/autosave.json"}
2024/05/02 15:07:44.590 INFO    serving initial configuration
2024/05/02 15:07:44.591 INFO    tls.obtain      acquiring lock  {"identifier": "server.tld.com"}
2024/05/02 15:07:44.592 WARN    tls     storage cleaning happened too recently; skipping for now        {"storage": "FileStorage:/root/.local/share/caddy", "instance": "1ccda99d-cd0c-42e2-8eac-dfb0144bfcda", "try_again": "2024/05/03 15:07:44.592", "try_again_in": 86399.999999359}
2024/05/02 15:07:44.592 INFO    tls     finished cleaning storage units
2024/05/02 15:07:44.593 INFO    tls.obtain      lock acquired   {"identifier": "server.tld.com"}
2024/05/02 15:07:44.593 INFO    tls.obtain      obtaining certificate   {"identifier": "server.tld.com"}
2024/05/02 15:07:44.593 INFO    tls.issuance.acme       waiting on internal rate limiter        {"identifiers": ["server.tld.com"], "ca": "https://acme-staging-v02.api.letsencrypt.org/directory", "account": "letsencrypt@tld.com"}
2024/05/02 15:07:44.593 INFO    tls.issuance.acme       done waiting on internal rate limiter   {"identifiers": ["server.tld.com"], "ca": "https://acme-staging-v02.api.letsencrypt.org/directory", "account": "letsencrypt@tld.com"}
2024/05/02 15:07:45.588 INFO    tls.issuance.acme.acme_client   trying to solve challenge       {"identifier": "server.tld.com", "challenge_type": "dns-01", "ca": "https://acme-staging-v02.api.letsencrypt.org/directory"}
2024/05/02 15:07:57.904 ERROR   tls.issuance.acme.acme_client   cleaning up solver      {"identifier": "server.tld.com", "challenge_type": "dns-01", "error": "deleting temporary record for name \"_acme-challenge.server.tld.com\" in zone \"tld.com.\": (2200) Authentication error. Reason: (INVALID) The confirmation code is invalid."}
-->2024/05/02 15:07:58.080 ERROR   tls.obtain      could not get certificate from issuer   {"identifier": "server.tld.com", "issuer": "acme-staging-v02.api.letsencrypt.org-directory", "error": "[server.tld.com] solving challenges: waiting for solver certmagic.solverWrapper to be ready: checking DNS propagation of \"_acme-challenge.server.tld.com\": read udp 127.0.0.1:52185->127.0.0.1:53: read: connection refused (order=https://acme-staging-v02.api.letsencrypt.org/acme/order/146529714/16276960954) (ca=https://acme-staging-v02.api.letsencrypt.org/directory)"}
2024/05/02 15:07:58.080 ERROR   tls.obtain      will retry      {"error": "[server.tld.com] Obtain: [server.tld.com] solving challenges: waiting for solver certmagic.solverWrapper to be ready: checking DNS propagation of \"_acme-challenge.server.tld.com\": read udp 127.0.0.1:52185->127.0.0.1:53: read: connection refused (order=https://acme-staging-v02.api.letsencrypt.org/acme/order/146529714/16276960954) (ca=https://acme-staging-v02.api.letsencrypt.org/directory)", "attempt": 1, "retrying_in": 60, "elapsed": 13.487398367, "max_duration": 2592000}
^C2024/05/02 15:08:00.019       INFO    shutting down   {"signal": "SIGINT"}
2024/05/02 15:08:00.019 WARN    exiting; byeee!! 👋     {"signal": "SIGINT"}
2024/05/02 15:08:00.019 INFO    http    servers shutting down with eternal grace period
2024/05/02 15:08:00.020 INFO    tls.obtain      releasing lock  {"identifier": "server.tld.com"}
2024/05/02 15:08:00.020 ERROR   tls.obtain      unable to unlock        {"identifier": "server.tld.com", "lock_key": "issue_cert_server.tld.com", "error": "remove /root/.local/share/caddy/locks/issue_cert_server.tld.com.lock: no such file or directory"}
2024/05/02 15:08:00.020 ERROR   tls     job failed      {"error": "server.tld.com: obtaining certificate: context canceled"}
2024/05/02 15:08:00.020 INFO    admin   stopped previous server {"address": "localhost:2019"}
2024/05/02 15:08:00.020 INFO    shutdown complete       {"signal": "SIGINT", "exit_code": 0}

I marked the line showing the error with arrow, here’s a copy. Note the
read udp 127.0.0.1:52185->127.0.0.1:53: read: connection refused

2024/05/02 15:07:58.080 ERROR   tls.obtain      could not get certificate from issuer   {"identifier": "server.tld.com", "issuer": "acme-staging-v02.api.letsencrypt.org-directory", "error": "[server.tld.com] solving challenges: waiting for solver certmagic.solverWrapper to be ready: checking DNS propagation of \"_acme-challenge.server.tld.com\": read udp 127.0.0.1:52185->127.0.0.1:53: read: connection refused (order=https://acme-staging-v02.api.letsencrypt.org/acme/order/146529714/16276960954) (ca=https://acme-staging-v02.api.letsencrypt.org/directory)"}

3. Caddy version:

v2.7.6

4. How I installed and ran Caddy:

I selected the inwx plugin and downloaded the caddy binary from Download Caddy.

a. System environment:

Debian 12.5, 64bit, no docker

b. Command:

caddy run

c. Service/unit/compose file:

Still evaluating caddy, no unit file.

d. My complete Caddy config:

{
        acme_ca https://acme-staging-v02.api.letsencrypt.org/directory
        email letsencrypt@tld.com
        acme_dns inwx {
                username redacted
                password redacted
                shared_secret redacted
        }
}

server.tld.com:443 {
        reverse_proxy :8080
}

5. Links to relevant resources:

Well, systemd-resolved runs a DNS server at 127.0.0.1:53 which caches DNS queries and otherwise routes through your resolv.conf. That’s how it’s meant to work, anyway.

For that, you would need to build Caddy with CGO_ENABLED=1. We disable CGO by default to have portable builds.

1 Like

I don’t use

My system doesn’t run systemd-resolved and - as I mentioned above - no process is listening on that port. Other ideas why only caddy ignores my /etc/resolv.conf?

Both Caddy and Go parse the /etc/resolv.conf, see:

https://cs.opensource.google/go/go/+/master:src/net/dnsclient_unix.go;l=368?q=dnsReadConfig&sq=&ss=go%2Fgo

(I’m on the phone. Please excuse the under-specific link).

Can you share the content of your /etc/resolv.conf?

1 Like