Windows/Linux Amd64 validates ACME DNS but Raspberry OS armv7l does not

1. Caddy version (caddy version):

  • Win - v2.4.1 => D:\Code\kmpm\dns-loopia\experiment\caddy
  • Linux - v2.4.1 => /srv/caddy-loopia-propagation/caddy

2. How I run Caddy:

I run a custom caddy, on 2 different machines. One is Windows and the other is a Rapberry Pi with Raspperry OS.
On RaspOS i run as root.

bin/caddy run

The binary is compiled the same way with the same options using the same ref of my caddy branch.
https://github.com/kmpm/caddy@caddyfile-propagation

xcaddy build \
		--output bin/caddy \
		--with github.com/caddy-dns/loopia \
		--with github.com/caddyserver/caddy/v2=./caddy

a. System environment:

  • Windows 10 Pro, amd64
  • RaspberryOS, Raspberry Pi 4

Windows

> go version
go version go1.16.3 windows/amd64

> go env
set GO111MODULE=
set GOARCH=amd64
set GOBIN=
set GOCACHE=D:\Code\kmpm\dns-loopia\experiment\var\go-build
set GOENV=D:\Code\kmpm\dns-loopia\experiment\var\go
set GOEXE=.exe
set GOFLAGS=
set GOHOSTARCH=amd64
set GOHOSTOS=windows
set GOINSECURE=
set GOMODCACHE=D:\Code\kmpm\dns-loopia\experiment\var\pkg\mod
set GONOPROXY=
set GONOSUMDB=
set GOOS=windows
set GOPATH=C:\Users\peter\go
set GOPRIVATE=
set GOPROXY=https://proxy.golang.org,direct
set GOROOT=C:\Program Files\Go
set GOSUMDB=sum.golang.org
set GOTMPDIR=
set GOTOOLDIR=C:\Program Files\Go\pkg\tool\windows_amd64
set GOVCS=
set GOVERSION=go1.16.3
set GCCGO=gccgo
set AR=ar
set CC=gcc
set CXX=g++
set CGO_ENABLED=1
set GOMOD=NUL
set CGO_CFLAGS=-g -O2
set CGO_CPPFLAGS=
set CGO_CXXFLAGS=-g -O2
set CGO_FFLAGS=-g -O2
set CGO_LDFLAGS=-g -O2
set PKG_CONFIG=pkg-config
set GOGCCFLAGS=-m64 -mthreads -fmessage-length=0 -fdebug-prefix-map=C:\Users\peter\AppData\Local\Temp\go-build3294507345=/tmp/go-build -gno-record-gcc-switches

RaspOS

$ go version
go version go1.16.4 linux/arm

$ go env
GO111MODULE=""
GOARCH="arm"
GOBIN=""
GOCACHE="/srv/caddy-loopia-propagation/var/go-build"
GOENV="/srv/caddy-loopia-propagation/var/go"
GOEXE=""
GOFLAGS=""
GOHOSTARCH="arm"
GOHOSTOS="linux"
GOINSECURE=""
GOMODCACHE="/srv/caddy-loopia-propagation/var/pkg/mod"
GONOPROXY=""
GONOSUMDB=""
GOOS="linux"
GOPATH="/root/go"
GOPRIVATE=""
GOPROXY="https://proxy.golang.org,direct"
GOROOT="/usr/local/go"
GOSUMDB="sum.golang.org"
GOTMPDIR=""
GOTOOLDIR="/usr/local/go/pkg/tool/linux_arm"
GOVCS=""
GOVERSION="go1.16.4"
GCCGO="gccgo"
GOARM="6"
AR="ar"
CC="gcc"
CXX="g++"
CGO_ENABLED="1"
GOMOD="/dev/null"
CGO_CFLAGS="-g -O2"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-g -O2"
CGO_FFLAGS="-g -O2"
CGO_LDFLAGS="-g -O2"
PKG_CONFIG="pkg-config"
GOGCCFLAGS="-fPIC -marm -pthread -fmessage-length=0 -fdebug-prefix-map=/tmp/go-build1608304632=/tmp/go-build -gno-record-gcc-switches"

b. Command:

bin/caddy run

c. Service/unit/compose file:

Not used.

d. My complete Caddyfile or JSON config:

{
    debug
}
faasd1.lcl.kapi.se
reverse_proxy 127.0.0.1:8080
tls {
    issuer acme {
        email "<redacted email>"
        propagation_timeout "15m0s"
        dns loopia {
            username "<redacteduser>@loopiaapi"
            password "<redactedpassword>"
        }
        resolvers ns1.loopia.se
    }
}

3. The problem I’m having:

On Windows caddy starts and creates a certificate without any issue, and it’s often done in <5 minutes.
On RaspOS it waits for the full propagation_timeout duration and fails.

When looking at the logs I noticed that they don’t produce the same type of output.
In Windows the first thing after “trying to solve challenge” is a POST to https://acme-v02.api.letsencrypt.org/acme/chall-v3/xxxxxxxxx.
The first thing after “trying to solve challenge” in RaspOS is much more delayed and is a POST to https://acme-v02.api.letsencrypt.org/acme/authz-v3/xxxxxxxxx.

I have not been able to get it working on RaspOS no matter what and that differens, that chall-v3 is never called in RaspOS, seemed really strange.

Can anyone help me out, please? Ideas, theories or just wild ramblings, anything is accepted :slight_smile:

4. Error messages and/or full log output:

Windows

> bin/caddy.exe run
2021/05/26 11:22:46.747 INFO    using adjacent Caddyfile
2021/05/26 11:22:46.748 WARN    input is not formatted with 'caddy fmt' {"adapter": "caddyfile", "file": "Caddyfile", "line": 2}
2021/05/26 11:22:46.756 INFO    admin   admin endpoint started  {"address": "tcp/localhost:2019", "enforce_origin": false, "origins": ["localhost:2019", "[::1]:2019", "127.0.0.1:2019"]}
2021/05/26 11:22:46.757 INFO    tls.cache.maintenance   started background certificate maintenance      {"cache": "0xc0003b58f0"}
2021/05/26 11:22:46.757 INFO    http    server is listening only on the HTTPS port but has no TLS connection policies; adding one to enable TLS {"server_name": "srv0", "https_port": 443}
2021/05/26 11:22:46.758 INFO    http    enabling automatic HTTP->HTTPS redirects        {"server_name": "srv0"}
2021/05/26 11:22:46.758 INFO    tls     cleaning storage unit   {"description": "FileStorage:C:\\Users\\peter\\AppData\\Roaming\\Caddy"}
2021/05/26 11:22:46.758 DEBUG   http    starting server loop    {"address": "[::]:443", "http3": false, "tls": true}
2021/05/26 11:22:46.759 DEBUG   http    starting server loop    {"address": "[::]:80", "http3": false, "tls": false}
2021/05/26 11:22:46.759 INFO    http    enabling automatic TLS certificate management   {"domains": ["faasd1.lcl.kapi.se"]}
2021/05/26 11:22:46.761 INFO    autosaved config (load with --resume flag)      {"file": "C:\\Users\\peter\\AppData\\Roaming\\Caddy\\autosave.json"}
2021/05/26 11:22:46.761 INFO    serving initial configuration
2021/05/26 11:22:46.761 INFO    tls     finished cleaning storage units
2021/05/26 11:22:46.762 INFO    tls.obtain      acquiring lock  {"identifier": "faasd1.lcl.kapi.se"}
2021/05/26 11:22:46.836 INFO    tls.obtain      lock acquired   {"identifier": "faasd1.lcl.kapi.se"}
2021/05/26 11:22:47.609 DEBUG   tls.issuance.acme.acme_client   http request    {"method": "GET", "url": "https://acme-v02.api.letsencrypt.org/directory", "headers": {"User-Agent":["Caddy/2.4.1 CertMagic acmez (windows; amd64)"]}, "status_code": 200, "response_headers": {"Cache-Control":["public, max-age=0, no-cache"],"Content-Length":["658"],"Content-Type":["application/json"],"Date":["Wed, 26 May 2021 11:22:49 GMT"],"Server":["nginx"],"Strict-Transport-Security":["max-age=604800"],"X-Frame-Options":["DENY"]}}
2021/05/26 11:22:47.759 DEBUG   tls.issuance.acme.acme_client   http request    {"method": "HEAD", "url": "https://acme-v02.api.letsencrypt.org/acme/new-nonce", "headers": {"User-Agent":["Caddy/2.4.1 CertMagic acmez (windows; amd64)"]}, "status_code": 200, "response_headers": {"Cache-Control":["public, max-age=0, no-cache"],"Date":["Wed, 26 May 2021 11:22:49 GMT"],"Link":["<https://acme-v02.api.letsencrypt.org/directory>;rel=\"index\""],"Replay-Nonce":["0004uHRhtf0Th83g96oRzTK_B5IBgGBGJtEunHFjGfmbago"],"Server":["nginx"],"Strict-Transport-Security":["max-age=604800"],"X-Frame-Options":["DENY"]}}
2021/05/26 11:22:47.973 DEBUG   tls.issuance.acme.acme_client   http request    {"method": "POST", "url": "https://acme-v02.api.letsencrypt.org/acme/new-acct", "headers": {"Content-Type":["application/jose+json"],"User-Agent":["Caddy/2.4.1 CertMagic acmez (windows; amd64)"]}, "status_code": 201, "response_headers": {"Boulder-Requester":["124875463"],"Cache-Control":["public, max-age=0, no-cache"],"Content-Length":["331"],"Content-Type":["application/json"],"Date":["Wed, 26 May 2021 11:22:49 GMT"],"Link":["<https://acme-v02.api.letsencrypt.org/directory>;rel=\"index\"","<https://letsencrypt.org/documents/LE-SA-v1.2-November-15-2017.pdf>;rel=\"terms-of-service\""],"Location":["https://acme-v02.api.letsencrypt.org/acme/acct/124875463"],"Replay-Nonce":["0003L-GiwsSlOeIL4bjEcm3hC9zuQhcPS59Spyp84EWJy14"],"Server":["nginx"],"Strict-Transport-Security":["max-age=604800"],"X-Frame-Options":["DENY"]}}
2021/05/26 11:22:47.980 INFO    tls.issuance.acme       waiting on internal rate limiter        {"identifiers": ["faasd1.lcl.kapi.se"]}
2021/05/26 11:22:47.980 INFO    tls.issuance.acme       done waiting on internal rate limiter   {"identifiers": ["faasd1.lcl.kapi.se"]}
2021/05/26 11:22:48.269 DEBUG   tls.issuance.acme.acme_client   http request    {"method": "POST", "url": "https://acme-v02.api.letsencrypt.org/acme/new-order", "headers": {"Content-Type":["application/jose+json"],"User-Agent":["Caddy/2.4.1 CertMagic acmez (windows; amd64)"]}, "status_code": 201, "response_headers": {"Boulder-Requester":["124875463"],"Cache-Control":["public, max-age=0, no-cache"],"Content-Length":["340"],"Content-Type":["application/json"],"Date":["Wed, 26 May 2021 11:22:50 GMT"],"Link":["<https://acme-v02.api.letsencrypt.org/directory>;rel=\"index\""],"Location":["https://acme-v02.api.letsencrypt.org/acme/order/124875463/9964176239"],"Replay-Nonce":["00037nSxPHpVlN7BYfiOjrC4fZzBVGAeolDQXIx-9K2Nb4M"],"Server":["nginx"],"Strict-Transport-Security":["max-age=604800"],"X-Frame-Options":["DENY"]}}
2021/05/26 11:22:48.457 DEBUG   tls.issuance.acme.acme_client   http request    {"method": "POST", "url": "https://acme-v02.api.letsencrypt.org/acme/authz-v3/13451017962", "headers": {"Content-Type":["application/jose+json"],"User-Agent":["Caddy/2.4.1 CertMagic acmez (windows; amd64)"]}, "status_code": 200, "response_headers": {"Boulder-Requester":["124875463"],"Cache-Control":["public, max-age=0, no-cache"],"Content-Length":["799"],"Content-Type":["application/json"],"Date":["Wed, 26 May 2021 11:22:50 GMT"],"Link":["<https://acme-v02.api.letsencrypt.org/directory>;rel=\"index\""],"Replay-Nonce":["0003NqfSGO2Ws5Qr8aLpRqXIgcMAPP1CVmczwk7rz-jsKWE"],"Server":["nginx"],"Strict-Transport-Security":["max-age=604800"],"X-Frame-Options":["DENY"]}}
2021/05/26 11:22:48.458 INFO    tls.issuance.acme.acme_client   trying to solve challenge       {"identifier": "faasd1.lcl.kapi.se", "challenge_type": "dns-01", "ca": "https://acme-v02.api.letsencrypt.org/directory"}

2021/05/26 11:25:40.778 DEBUG   tls.issuance.acme.acme_client   http request    {"method": "POST", "url": "https://acme-v02.api.letsencrypt.org/acme/chall-v3/13451017962/ODTgJA", "headers": {"Content-Type":["application/jose+json"],"User-Agent":["Caddy/2.4.1 CertMagic acmez (windows; amd64)"]}, "status_code": 400, "response_headers": {"Boulder-Requester":["124875463"],"Cache-Control":["public, max-age=0, no-cache"],"Content-Length":["173"],"Content-Type":["application/problem+json"],"Date":["Wed, 26 May 2021 11:25:42 GMT"],"Link":["<https://acme-v02.api.letsencrypt.org/directory>;rel=\"index\""],"Replay-Nonce":["0004Cx21hvTRNKDMV0OWepOxhDAtBkN4LXceHe2zEiy6AKE"],"Server":["nginx"]}}
2021/05/26 11:25:40.779 DEBUG   tls.issuance.acme.acme_client   server rejected our nonce; retrying     {"detail": "JWS has an invalid anti-replay nonce: \"0003NqfSGO2Ws5Qr8aLpRqXIgcMAPP1CVmczwk7rz-jsKWE\"", "error": "HTTP 400 urn:ietf:params:acme:error:badNonce - JWS has an invalid anti-replay nonce: \"0003NqfSGO2Ws5Qr8aLpRqXIgcMAPP1CVmczwk7rz-jsKWE\""}
2021/05/26 11:25:41.221 DEBUG   tls.issuance.acme.acme_client   http request    {"method": "POST", "url": "https://acme-v02.api.letsencrypt.org/acme/chall-v3/13451017962/ODTgJA", "headers": {"Content-Type":["application/jose+json"],"User-Agent":["Caddy/2.4.1 CertMagic acmez (windows; amd64)"]}, "status_code": 200, "response_headers": {"Boulder-Requester":["124875463"],"Cache-Control":["public, max-age=0, no-cache"],"Content-Length":["185"],"Content-Type":["application/json"],"Date":["Wed, 26 May 2021 11:25:43 GMT"],"Link":["<https://acme-v02.api.letsencrypt.org/directory>;rel=\"index\"","<https://acme-v02.api.letsencrypt.org/acme/authz-v3/13451017962>;rel=\"up\""],"Location":["https://acme-v02.api.letsencrypt.org/acme/chall-v3/13451017962/ODTgJA"],"Replay-Nonce":["0003Nl9VKrTktGxhisZ_9WlM3xkmrHnSnIQbs8Xo0LwqPVo"],"Server":["nginx"],"Strict-Transport-Security":["max-age=604800"],"X-Frame-Options":["DENY"]}}
2021/05/26 11:25:41.222 DEBUG   tls.issuance.acme.acme_client   challenge accepted      {"identifier": "faasd1.lcl.kapi.se", "challenge_type": "dns-01"}
2021/05/26 11:25:41.659 DEBUG   tls.issuance.acme.acme_client   http request    {"method": "POST", "url": "https://acme-v02.api.letsencrypt.org/acme/authz-v3/13451017962", "headers": {"Content-Type":["application/jose+json"],"User-Agent":["Caddy/2.4.1 CertMagic acmez (windows; amd64)"]}, "status_code": 200, "response_headers": {"Boulder-Requester":["124875463"],"Cache-Control":["public, max-age=0, no-cache"],"Content-Length":["799"],"Content-Type":["application/json"],"Date":["Wed, 26 May 2021 11:25:43 GMT"],"Link":["<https://acme-v02.api.letsencrypt.org/directory>;rel=\"index\""],"Replay-Nonce":["0004lNNyXqGMO0waaOD5kluPy4i3MXkdYiBkQkEodIt9alI"],"Server":["nginx"],"Strict-Transport-Security":["max-age=604800"],"X-Frame-Options":["DENY"]}}
2021/05/26 11:25:42.108 DEBUG   tls.issuance.acme.acme_client   http request    {"method": "POST", "url": "https://acme-v02.api.letsencrypt.org/acme/authz-v3/13451017962", "headers": {"Content-Type":["application/jose+json"],"User-Agent":["Caddy/2.4.1 CertMagic acmez (windows; amd64)"]}, "status_code": 200, "response_headers": {"Boulder-Requester":["124875463"],"Cache-Control":["public, max-age=0, no-cache"],"Content-Length":["799"],"Content-Type":["application/json"],"Date":["Wed, 26 May 2021 11:25:43 GMT"],"Link":["<https://acme-v02.api.letsencrypt.org/directory>;rel=\"index\""],"Replay-Nonce":["0003TJ6MNG0Vi65OmVSVqkw4c-8nA85e9cMcdqlZcPhCQGM"],"Server":["nginx"],"Strict-Transport-Security":["max-age=604800"],"X-Frame-Options":["DENY"]}}
2021/05/26 11:25:42.548 DEBUG   tls.issuance.acme.acme_client   http request    {"method": "POST", "url": "https://acme-v02.api.letsencrypt.org/acme/authz-v3/13451017962", "headers": {"Content-Type":["application/jose+json"],"User-Agent":["Caddy/2.4.1 CertMagic acmez (windows; amd64)"]}, "status_code": 200, "response_headers": {"Boulder-Requester":["124875463"],"Cache-Control":["public, max-age=0, no-cache"],"Content-Length":["510"],"Content-Type":["application/json"],"Date":["Wed, 26 May 2021 11:25:44 GMT"],"Link":["<https://acme-v02.api.letsencrypt.org/directory>;rel=\"index\""],"Replay-Nonce":["0003KyafI3GttXdzI6fxI8zkleGgxcBLjGeCGuJgEBKK4N4"],"Server":["nginx"],"Strict-Transport-Security":["max-age=604800"],"X-Frame-Options":["DENY"]}}
2021/05/26 11:25:56.008 INFO    tls.issuance.acme.acme_client   validations succeeded; finalizing order {"order": "https://acme-v02.api.letsencrypt.org/acme/order/124875463/9964176239"}
2021/05/26 11:25:56.663 DEBUG   tls.issuance.acme.acme_client   http request    {"method": "POST", "url": "https://acme-v02.api.letsencrypt.org/acme/finalize/124875463/9964176239", "headers": {"Content-Type":["application/jose+json"],"User-Agent":["Caddy/2.4.1 CertMagic acmez (windows; amd64)"]}, "status_code": 200, "response_headers": {"Boulder-Requester":["124875463"],"Cache-Control":["public, max-age=0, no-cache"],"Content-Length":["442"],"Content-Type":["application/json"],"Date":["Wed, 26 May 2021 11:25:58 GMT"],"Link":["<https://acme-v02.api.letsencrypt.org/directory>;rel=\"index\""],"Location":["https://acme-v02.api.letsencrypt.org/acme/order/124875463/9964176239"],"Replay-Nonce":["00046SXxfLYOhn7wgxH3nnFHsfdQQkFyeUq_UTOOZQ2xLgQ"],"Server":["nginx"],"Strict-Transport-Security":["max-age=604800"],"X-Frame-Options":["DENY"]}}
2021/05/26 11:25:56.840 DEBUG   tls.issuance.acme.acme_client   http request    {"method": "POST", "url": "https://acme-v02.api.letsencrypt.org/acme/cert/03619fd2b3a142142f3af8865dbbe48aadf0", "headers": {"Content-Type":["application/jose+json"],"User-Agent":["Caddy/2.4.1 CertMagic acmez (windows; amd64)"]}, "status_code": 200, "response_headers": {"Cache-Control":["public, max-age=0, no-cache"],"Content-Length":["5329"],"Content-Type":["application/pem-certificate-chain"],"Date":["Wed, 26 May 2021 11:25:58 GMT"],"Link":["<https://acme-v02.api.letsencrypt.org/directory>;rel=\"index\"","<https://acme-v02.api.letsencrypt.org/acme/cert/03619fd2b3a142142f3af8865dbbe48aadf0/1>;rel=\"alternate\""],"Replay-Nonce":["0003zzrvQ1jM3Xx1WQbmTJt8UbzxSPZ-TLHAPCOkg_oM2iA"],"Server":["nginx"],"Strict-Transport-Security":["max-age=604800"],"X-Frame-Options":["DENY"]}}
2021/05/26 11:25:57.026 DEBUG   tls.issuance.acme.acme_client   http request    {"method": "POST", "url": "https://acme-v02.api.letsencrypt.org/acme/cert/03619fd2b3a142142f3af8865dbbe48aadf0/1", "headers": {"Content-Type":["application/jose+json"],"User-Agent":["Caddy/2.4.1 CertMagic acmez (windows; amd64)"]}, "status_code": 200, "response_headers": {"Cache-Control":["public, max-age=0, no-cache"],"Content-Length":["3405"],"Content-Type":["application/pem-certificate-chain"],"Date":["Wed, 26 May 2021 11:25:58 GMT"],"Link":["<https://acme-v02.api.letsencrypt.org/directory>;rel=\"index\"","<https://acme-v02.api.letsencrypt.org/acme/cert/03619fd2b3a142142f3af8865dbbe48aadf0/0>;rel=\"alternate\""],"Replay-Nonce":["0004MrFGa5GUqrSDx-HGQXNnos3zuDik44ss9EZNXIaUwWU"],"Server":["nginx"],"Strict-Transport-Security":["max-age=604800"],"X-Frame-Options":["DENY"]}}
2021/05/26 11:25:57.027 INFO    tls.issuance.acme.acme_client   successfully downloaded available certificate chains    {"count": 2, "first_url": "https://acme-v02.api.letsencrypt.org/acme/cert/03619fd2b3a142142f3af8865dbbe48aadf0"}
2021/05/26 11:25:57.032 INFO    tls.obtain      certificate obtained successfully       {"identifier": "faasd1.lcl.kapi.se"}
2021/05/26 11:25:57.033 INFO    tls.obtain      releasing lock  {"identifier": "faasd1.lcl.kapi.se"}
2021/05/26 11:26:01.679 INFO    shutting down   {"signal": "SIGINT"}
2021/05/26 11:26:01.679 WARN    exiting; byeee!! 👋     {"signal": "SIGINT"}

RaspOS

$ bin/caddy run
2021/05/26 11:32:32.909 INFO    using adjacent Caddyfile
2021/05/26 11:32:32.912 WARN    input is not formatted with 'caddy fmt' {"adapter": "caddyfile", "file": "Caddyfile", "line": 2}
2021/05/26 11:32:32.914 INFO    admin   admin endpoint started  {"address": "tcp/localhost:2019", "enforce_origin": false, "origins": ["localhost:2019", "[::1]:2019", "127.0.0.1:2019"]}
2021/05/26 11:32:32.915 INFO    tls.cache.maintenance   started background certificate maintenance      {"cache": "0x28ba4b0"}
2021/05/26 11:32:32.915 INFO    http    server is listening only on the HTTPS port but has no TLS connection policies; adding one to enable TLS {"server_name": "srv0", "https_port": 443}
2021/05/26 11:32:32.917 INFO    http    enabling automatic HTTP->HTTPS redirects        {"server_name": "srv0"}
2021/05/26 11:32:32.918 INFO    tls     cleaning storage unit   {"description": "FileStorage:/root/.local/share/caddy"}
2021/05/26 11:32:32.918 INFO    tls     finished cleaning storage units
2021/05/26 11:32:32.919 DEBUG   http    starting server loop    {"address": "[::]:443", "http3": false, "tls": true}
2021/05/26 11:32:32.920 DEBUG   http    starting server loop    {"address": "[::]:80", "http3": false, "tls": false}
2021/05/26 11:32:32.920 INFO    http    enabling automatic TLS certificate management   {"domains": ["faasd1.lcl.kapi.se"]}
2021/05/26 11:32:32.921 INFO    autosaved config (load with --resume flag)      {"file": "/root/.config/caddy/autosave.json"}
2021/05/26 11:32:32.922 INFO    serving initial configuration
2021/05/26 11:32:32.922 INFO    tls.obtain      acquiring lock  {"identifier": "faasd1.lcl.kapi.se"}
2021/05/26 11:32:32.930 INFO    tls.obtain      lock acquired   {"identifier": "faasd1.lcl.kapi.se"}
2021/05/26 11:32:33.752 DEBUG   tls.issuance.acme.acme_client   http request    {"method": "GET", "url": "https://acme-v02.api.letsencrypt.org/directory", "headers": {"User-Agent":["Caddy/2.4.1 CertMagic acmez (linux; arm)"]}, "status_code": 200, "response_headers": {"Cache-Control":["public, max-age=0, no-cache"],"Content-Length":["658"],"Content-Type":["application/json"],"Date":["Wed, 26 May 2021 11:32:33 GMT"],"Server":["nginx"],"Strict-Transport-Security":["max-age=604800"],"X-Frame-Options":["DENY"]}}
2021/05/26 11:32:33.890 DEBUG   tls.issuance.acme.acme_client   http request    {"method": "HEAD", "url": "https://acme-v02.api.letsencrypt.org/acme/new-nonce", "headers": {"User-Agent":["Caddy/2.4.1 CertMagic acmez (linux; arm)"]}, "status_code": 200, "response_headers": {"Cache-Control":["public, max-age=0, no-cache"],"Date":["Wed, 26 May 2021 11:32:33 GMT"],"Link":["<https://acme-v02.api.letsencrypt.org/directory>;rel=\"index\""],"Replay-Nonce":["0103F-XX1bzcw-Gko1jUoAR3kaIllDsWp3sEtpWyFx-vfPY"],"Server":["nginx"],"Strict-Transport-Security":["max-age=604800"],"X-Frame-Options":["DENY"]}}
2021/05/26 11:32:34.092 DEBUG   tls.issuance.acme.acme_client   http request    {"method": "POST", "url": "https://acme-v02.api.letsencrypt.org/acme/new-acct", "headers": {"Content-Type":["application/jose+json"],"User-Agent":["Caddy/2.4.1 CertMagic acmez (linux; arm)"]}, "status_code": 201, "response_headers": {"Boulder-Requester":["124876308"],"Cache-Control":["public, max-age=0, no-cache"],"Content-Length":["331"],"Content-Type":["application/json"],"Date":["Wed, 26 May 2021 11:32:34 GMT"],"Link":["<https://acme-v02.api.letsencrypt.org/directory>;rel=\"index\"","<https://letsencrypt.org/documents/LE-SA-v1.2-November-15-2017.pdf>;rel=\"terms-of-service\""],"Location":["https://acme-v02.api.letsencrypt.org/acme/acct/124876308"],"Replay-Nonce":["0103NbUNoB4QIHoZNcyXEzvpfMuQ06Xtm1Bo6Ie88Ha0RKo"],"Server":["nginx"],"Strict-Transport-Security":["max-age=604800"],"X-Frame-Options":["DENY"]}}
2021/05/26 11:32:34.094 INFO    tls.issuance.acme       waiting on internal rate limiter        {"identifiers": ["faasd1.lcl.kapi.se"]}
2021/05/26 11:32:34.095 INFO    tls.issuance.acme       done waiting on internal rate limiter   {"identifiers": ["faasd1.lcl.kapi.se"]}
2021/05/26 11:32:34.259 DEBUG   tls.issuance.acme.acme_client   http request    {"method": "POST", "url": "https://acme-v02.api.letsencrypt.org/acme/new-order", "headers": {"Content-Type":["application/jose+json"],"User-Agent":["Caddy/2.4.1 CertMagic acmez (linux; arm)"]}, "status_code": 201, "response_headers": {"Boulder-Requester":["124876308"],"Cache-Control":["public, max-age=0, no-cache"],"Content-Length":["340"],"Content-Type":["application/json"],"Date":["Wed, 26 May 2021 11:32:34 GMT"],"Link":["<https://acme-v02.api.letsencrypt.org/directory>;rel=\"index\""],"Location":["https://acme-v02.api.letsencrypt.org/acme/order/124876308/9964321423"],"Replay-Nonce":["0104v7r096kis1thX2mWReqER-gMXtN8PUow867sjFBL5FE"],"Server":["nginx"],"Strict-Transport-Security":["max-age=604800"],"X-Frame-Options":["DENY"]}}
2021/05/26 11:32:34.406 DEBUG   tls.issuance.acme.acme_client   http request    {"method": "POST", "url": "https://acme-v02.api.letsencrypt.org/acme/authz-v3/13451192785", "headers": {"Content-Type":["application/jose+json"],"User-Agent":["Caddy/2.4.1 CertMagic acmez (linux; arm)"]}, "status_code": 200, "response_headers": {"Boulder-Requester":["124876308"],"Cache-Control":["public, max-age=0, no-cache"],"Content-Length":["799"],"Content-Type":["application/json"],"Date":["Wed, 26 May 2021 11:32:34 GMT"],"Link":["<https://acme-v02.api.letsencrypt.org/directory>;rel=\"index\""],"Replay-Nonce":["0104zazfdDHFRoPgek9JqXGQCVF9b9NJoKVc-M6dAdJIPtA"],"Server":["nginx"],"Strict-Transport-Security":["max-age=604800"],"X-Frame-Options":["DENY"]}}
2021/05/26 11:32:34.412 DEBUG   tls.issuance.acme.acme_client   no solver configured    {"challenge_type": "http-01"}
2021/05/26 11:32:34.414 INFO    tls.issuance.acme.acme_client   trying to solve challenge       {"identifier": "faasd1.lcl.kapi.se", "challenge_type": "dns-01", "ca": "https://acme-v02.api.letsencrypt.org/directory"}

2021/05/26 11:47:43.318 DEBUG   tls.issuance.acme.acme_client   http request    {"method": "POST", "url": "https://acme-v02.api.letsencrypt.org/acme/authz-v3/13451192785", "headers": {"Content-Type":["application/jose+json"],"User-Agent":["Caddy/2.4.1 CertMagic acmez (linux; arm)"]}, "status_code": 400, "response_headers": {"Boulder-Requester":["124876308"],"Cache-Control":["public, max-age=0, no-cache"],"Content-Length":["173"],"Content-Type":["application/problem+json"],"Date":["Wed, 26 May 2021 11:47:43 GMT"],"Link":["<https://acme-v02.api.letsencrypt.org/directory>;rel=\"index\""],"Replay-Nonce":["01034jaAZKge-lEITZLTNtGgVYv_ynHc7TT0dyydMS5uRu4"],"Server":["nginx"]}}
2021/05/26 11:47:43.319 DEBUG   tls.issuance.acme.acme_client   server rejected our nonce; retrying     {"detail": "JWS has an invalid anti-replay nonce: \"0104zazfdDHFRoPgek9JqXGQCVF9b9NJoKVc-M6dAdJIPtA\"", "error": "HTTP 400 urn:ietf:params:acme:error:badNonce - JWS has an invalid anti-replay nonce: \"0104zazfdDHFRoPgek9JqXGQCVF9b9NJoKVc-M6dAdJIPtA\""}
2021/05/26 11:47:43.722 DEBUG   tls.issuance.acme.acme_client   http request    {"method": "POST", "url": "https://acme-v02.api.letsencrypt.org/acme/authz-v3/13451192785", "headers": {"Content-Type":["application/jose+json"],"User-Agent":["Caddy/2.4.1 CertMagic acmez (linux; arm)"]}, "status_code": 200, "response_headers": {"Boulder-Requester":["124876308"],"Cache-Control":["public, max-age=0, no-cache"],"Content-Length":["803"],"Content-Type":["application/json"],"Date":["Wed, 26 May 2021 11:47:43 GMT"],"Link":["<https://acme-v02.api.letsencrypt.org/directory>;rel=\"index\""],"Replay-Nonce":["0103G8LHKH0frHR1yr4JkplsmZ9G7f70zpfhPEFFiKYhyIk"],"Server":["nginx"],"Strict-Transport-Security":["max-age=604800"],"X-Frame-Options":["DENY"]}}
2021/05/26 11:47:43.722 ERROR   tls.obtain      will retry      {"error": "[faasd1.lcl.kapi.se] Obtain: [faasd1.lcl.kapi.se] solving challenges: waiting for solver certmagic.solverWrapper to be ready: timed out waiting for record to fully propagate; verify DNS provider configuration is correct - last error: <nil> (order=https://acme-v02.api.letsencrypt.org/acme/order/124876308/9964321423) (ca=https://acme-v02.api.letsencrypt.org/directory)", "attempt": 1, "retrying_in": 60, "elapsed": 910.791086928, "max_duration": 2592000}
2021/05/26 11:48:08.527 INFO    shutting down   {"signal": "SIGINT"}
2021/05/26 11:48:08.527 WARN    exiting; byeee!! 👋     {"signal": "SIGINT"}
2021/05/26 11:48:08.530 INFO    tls.cache.maintenance   stopped background certificate maintenance      {"cache": "0x28ba4b0"}
2021/05/26 11:48:08.531 INFO    tls.obtain      releasing lock  {"identifier": "faasd1.lcl.kapi.se"}
2021/05/26 11:48:08.532 ERROR   tls.obtain      unable to unlock        {"identifier": "faasd1.lcl.kapi.se", "lock_key": "issue_cert_faasd1.lcl.kapi.se", "error": "remove /root/.local/share/caddy/locks/issue_cert_faasd1.lcl.kapi.se.lock: no such file or directory"}
2021/05/26 11:48:08.533 ERROR   tls     job failed      {"error": "faasd1.lcl.kapi.se: obtaining certificate: context canceled"}
2021/05/26 11:48:08.533 INFO    admin   stopped previous server {"address": "tcp/localhost:2019"}
2021/05/26 11:48:08.534 INFO    shutdown complete       {"signal": "SIGINT", "exit_code": 0}

5. What I already tried:

  • recompiled on both environments with clean GOENV, GOCACHE, GOMODCACHE
  • validated git hashes on the caddy version used.
  • looked for anything strange in the build logs
  • tried different hostnames
  • validated that the DNS reports a updated _acme-challenge TXT record on all occations using dig

6. Links to relevant resources:

I just ran the exact same code in a fresh github codespace and it worked flawlessly but still no luck on RaspOS.

$ bin/caddy version
v2.4.1 => /workspaces/caddy-loopia-propagation/caddy

$ uname -a
Linux codespaces_2516f6 5.4.0-1047-azure #49~18.04.1-Ubuntu SMP Thu Apr 22 21:28:54 UTC 2021 
x86_64 x86_64 x86_64 GNU/Linux

$ go env
GO111MODULE=""
GOARCH="amd64"
GOBIN=""
GOCACHE="/home/codespace/.cache/go-build"
GOENV="/home/codespace/.config/go/env"
GOEXE=""
GOFLAGS=""
GOHOSTARCH="amd64"
GOHOSTOS="linux"
GOINSECURE=""
GOMODCACHE="/go/pkg/mod"
GONOPROXY=""
GONOSUMDB=""
GOOS="linux"
GOPATH="/go"
GOPRIVATE=""
GOPROXY="https://proxy.golang.org,direct"
GOROOT="/usr/local/go"
GOSUMDB="sum.golang.org"
GOTMPDIR=""
GOTOOLDIR="/usr/local/go/pkg/tool/linux_amd64"
GOVCS=""
GOVERSION="go1.16.4"
GCCGO="gccgo"
AR="ar"
CC="gcc"
CXX="g++"
CGO_ENABLED="1"
GOMOD="/dev/null"
CGO_CFLAGS="-g -O2"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-g -O2"
CGO_FFLAGS="-g -O2"
CGO_LDFLAGS="-g -O2"
PKG_CONFIG="pkg-config"
GOGCCFLAGS="-fPIC -m64 -pthread -fmessage-length=0 -fdebug-prefix-map=/tmp/go-build2083110074=/tmp/go-build -gno-record-gcc-switches"
...
2021/05/26 12:13:25.371 INFO    tls.issuance.acme.acme_client   trying to solve challenge       {"identifier": "faasd1.lcl.kapi.se", "challenge_type": "dns-01", "ca": "https://acme-v02.api.letsencrypt.org/directory"}

2021/05/26 12:21:15.092 DEBUG   tls.issuance.acme.acme_client   http request    {"method": "POST", "url": "https://acme-v02.api.letsencrypt.org/acme/chall-v3/13451940309/Pby2Zg", "headers": {"Content-Type":["application/jose+json"],"User-Agent":["Caddy/2.4.1 CertMagic acmez (linux; amd64)"]}, "status_code": 400, "response_headers": {"Boulder-Requester":["124879968"],"Cache-Control":["public, max-age=0, no-cache"],"Content-Length":["173"],"Content-Type":["application/problem+json"],"Date":["Wed, 26 May 2021 12:21:15 GMT"],"Link":["<https://acme-v02.api.letsencrypt.org/directory>;rel=\"index\""],"Replay-Nonce":["0103h2JO7636QEqVEtnMVYvj9Iv_JjknFuJxMwiYMazeSCs"],"Server":["nginx"]}}
2021/05/26 12:21:15.092 DEBUG   tls.issuance.acme.acme_client   server rejected our nonce; retrying     {"detail": "JWS has an invalid anti-replay nonce: \"0103VZxxSbC3hLex1DZJA3WDGh0UjEAioNd-MM3WqPeiuyA\"", "error": "HTTP 400 urn:ietf:params:acme:error:badNonce - JWS has an invalid anti-replay nonce: \"0103VZxxSbC3hLex1DZJA3WDGh0UjEAioNd-MM3WqPeiuyA\""}
2021/05/26 12:21:15.482 DEBUG   tls.issuance.acme.acme_client   http request    {"method": "POST", "url": "https://acme-v02.api.letsencrypt.org/acme/chall-v3/13451940309/Pby2Zg", "headers": {"Content-Type":["application/jose+json"],"User-Agent":["Caddy/2.4.1 CertMagic acmez (linux; amd64)"]}, "status_code": 200, "response_headers": {"Boulder-Requester":["124879968"],"Cache-Control":["public, max-age=0, no-cache"],"Content-Length":["185"],"Content-Type":["application/json"],"Date":["Wed, 26 May 2021 12:21:15 GMT"],"Link":["<https://acme-v02.api.letsencrypt.org/directory>;rel=\"index\"","<https://acme-v02.api.letsencrypt.org/acme/authz-v3/13451940309>;rel=\"up\""],"Location":["https://acme-v02.api.letsencrypt.org/acme/chall-v3/13451940309/Pby2Zg"],"Replay-Nonce":["0103CvrRe8UG6d2pUqLg8NdoBl2WaxSTVl9DspjVd6NkjaI"],"Server":["nginx"],"Strict-Transport-Security":["max-age=604800"],"X-Frame-Options":["DENY"]}}
2021/05/26 12:21:15.483 DEBUG   tls.issuance.acme.acme_client   challenge accepted      {"identifier": "faasd1.lcl.kapi.se", "challenge_type": "dns-01"}
2021/05/26 12:21:15.860 DEBUG   tls.issuance.acme.acme_client   http request    {"method": "POST", "url": "https://acme-v02.api.letsencrypt.org/acme/authz-v3/13451940309", "headers": {"Content-Type":["application/jose+json"],"User-Agent":["Caddy/2.4.1 CertMagic acmez (linux; amd64)"]}, "status_code": 200, "response_headers": {"Boulder-Requester":["124879968"],"Cache-Control":["public, max-age=0, no-cache"],"Content-Length":["799"],"Content-Type":["application/json"],"Date":["Wed, 26 May 2021 12:21:15 GMT"],"Link":["<https://acme-v02.api.letsencrypt.org/directory>;rel=\"index\""],"Replay-Nonce":["0104cyLZ7Xx8lChf9zR3igA80-8jwpG2iaJfjc28UM5GU50"],"Server":["nginx"],"Strict-Transport-Security":["max-age=604800"],"X-Frame-Options":["DENY"]}}
2021/05/26 12:21:16.238 DEBUG   tls.issuance.acme.acme_client   http request    {"method": "POST", "url": "https://acme-v02.api.letsencrypt.org/acme/authz-v3/13451940309", "headers": {"Content-Type":["application/jose+json"],"User-Agent":["Caddy/2.4.1 CertMagic acmez (linux; amd64)"]}, "status_code": 200, "response_headers": {"Boulder-Requester":["124879968"],"Cache-Control":["public, max-age=0, no-cache"],"Content-Length":["799"],"Content-Type":["application/json"],"Date":["Wed, 26 May 2021 12:21:16 GMT"],"Link":["<https://acme-v02.api.letsencrypt.org/directory>;rel=\"index\""],"Replay-Nonce":["0104heoZWTJI66EHSOuvwmMIrxE9SWUiGn2LTEPmsW2ApP4"],"Server":["nginx"],"Strict-Transport-Security":["max-age=604800"],"X-Frame-Options":["DENY"]}}
2021/05/26 12:21:16.616 DEBUG   tls.issuance.acme.acme_client   http request    {"method": "POST", "url": "https://acme-v02.api.letsencrypt.org/acme/authz-v3/13451940309", "headers": {"Content-Type":["application/jose+json"],"User-Agent":["Caddy/2.4.1 CertMagic acmez (linux; amd64)"]}, "status_code": 200, "response_headers": {"Boulder-Requester":["124879968"],"Cache-Control":["public, max-age=0, no-cache"],"Content-Length":["510"],"Content-Type":["application/json"],"Date":["Wed, 26 May 2021 12:21:16 GMT"],"Link":["<https://acme-v02.api.letsencrypt.org/directory>;rel=\"index\""],"Replay-Nonce":["0103TR88u6Jl5iGE1niMZDhnFpcsqxYrWZB-7GDOFtE1Wws"],"Server":["nginx"],"Strict-Transport-Security":["max-age=604800"],"X-Frame-Options":["DENY"]}}
2021/05/26 12:21:23.303 INFO    tls.issuance.acme.acme_client   validations succeeded; finalizing order {"order": "https://acme-v02.api.letsencrypt.org/acme/order/124879968/9964935014"}
2021/05/26 12:21:23.577 DEBUG   tls.issuance.acme.acme_client   http request    {"method": "POST", "url": "https://acme-v02.api.letsencrypt.org/acme/finalize/124879968/9964935014", "headers": {"Content-Type":["application/jose+json"],"User-Agent":["Caddy/2.4.1 CertMagic acmez (linux; amd64)"]}, "status_code": 200, "response_headers": {"Boulder-Requester":["124879968"],"Cache-Control":["public, max-age=0, no-cache"],"Content-Length":["442"],"Content-Type":["application/json"],"Date":["Wed, 26 May 2021 12:21:23 GMT"],"Link":["<https://acme-v02.api.letsencrypt.org/directory>;rel=\"index\""],"Location":["https://acme-v02.api.letsencrypt.org/acme/order/124879968/9964935014"],"Replay-Nonce":["0103J4zXwPYYCMjW9OhPdNylBF7O6le9QNFs4w_yXCiQmOs"],"Server":["nginx"],"Strict-Transport-Security":["max-age=604800"],"X-Frame-Options":["DENY"]}}
2021/05/26 12:21:23.706 DEBUG   tls.issuance.acme.acme_client   http request    {"method": "POST", "url": "https://acme-v02.api.letsencrypt.org/acme/cert/04ef754599c458c37708849715f63d9e32e2", "headers": {"Content-Type":["application/jose+json"],"User-Agent":["Caddy/2.4.1 CertMagic acmez (linux; amd64)"]}, "status_code": 200, "response_headers": {"Cache-Control":["public, max-age=0, no-cache"],"Content-Length":["5329"],"Content-Type":["application/pem-certificate-chain"],"Date":["Wed, 26 May 2021 12:21:23 GMT"],"Link":["<https://acme-v02.api.letsencrypt.org/directory>;rel=\"index\"","<https://acme-v02.api.letsencrypt.org/acme/cert/04ef754599c458c37708849715f63d9e32e2/1>;rel=\"alternate\""],"Replay-Nonce":["0103ZNZHIFdfmD6CCQb0FjU_AX71jkAloSLZTYJAkBuh0zg"],"Server":["nginx"],"Strict-Transport-Security":["max-age=604800"],"X-Frame-Options":["DENY"]}}
2021/05/26 12:21:23.841 DEBUG   tls.issuance.acme.acme_client   http request    {"method": "POST", "url": "https://acme-v02.api.letsencrypt.org/acme/cert/04ef754599c458c37708849715f63d9e32e2/1", "headers": {"Content-Type":["application/jose+json"],"User-Agent":["Caddy/2.4.1 CertMagic acmez (linux; amd64)"]}, "status_code": 200, "response_headers": {"Cache-Control":["public, max-age=0, no-cache"],"Content-Length":["3405"],"Content-Type":["application/pem-certificate-chain"],"Date":["Wed, 26 May 2021 12:21:23 GMT"],"Link":["<https://acme-v02.api.letsencrypt.org/directory>;rel=\"index\"","<https://acme-v02.api.letsencrypt.org/acme/cert/04ef754599c458c37708849715f63d9e32e2/0>;rel=\"alternate\""],"Replay-Nonce":["0104BeCkgFiEeASAsp9XAv4kRGdP4v2oGCKAKmRMVKqBaTI"],"Server":["nginx"],"Strict-Transport-Security":["max-age=604800"],"X-Frame-Options":["DENY"]}}
2021/05/26 12:21:23.841 INFO    tls.issuance.acme.acme_client   successfully downloaded available certificate chains    {"count": 2, "first_url": "https://acme-v02.api.letsencrypt.org/acme/cert/04ef754599c458c37708849715f63d9e32e2"}
2021/05/26 12:21:23.841 INFO    tls.obtain      certificate obtained successfully       {"identifier": "faasd1.lcl.kapi.se"}
2021/05/26 12:21:23.841 INFO    tls.obtain      releasing lock  {"identifier": "faasd1.lcl.kapi.se"}
^C2021/05/26 12:21:33.896       INFO    shutting down   {"signal": "SIGINT"}
2021/05/26 12:21:33.896 WARN    exiting; byeee!! 👋     {"signal": "SIGINT"}
2021/05/26 12:21:33.899 INFO    tls.cache.maintenance   stopped background certificate maintenance      {"cache": "0xc000381420"}
2021/05/26 12:21:33.901 INFO    admin   stopped previous server {"address": "tcp/localhost:2019"}
2021/05/26 12:21:33.901 INFO    shutdown complete       {"signal": "SIGINT", "exit_code": 0}

/cc @matt, since he’d have a better idea

Not sure at a glance – will need to look into it more, but it (the different API requests) might have something to do with the fact that challenge authorizations are cached and reused for some amount of time?

Then again, this:

timed out waiting for record to fully propagate

suggests a local DNS issue to the RaspOS machine.

So, on RaspOS the resolver config doesn’t get used, could happen I guess. The challenge is fully resolved since I can see it using both ns1.loopia.se (the registrars DNS) as well as 8.8.8.8.
If it uses only the internal os resolver then there will be issues with split DNS but I thought the resolver config in the Caddyfile would take care of that. Easily tested though.

Ok. Moving the raspberry pi to another network (that isn’t affected by split DNS), then it worked immediately.

How come it doesn’t use the configured resolvers that is external? How can I verify what DNS that gets used, tcpdump or is there something I can do to show it in the logs of Caddy?

Does ns1.loopia.se resolve to what you expect? I think usually I see people configure IP addresses there, not DNS names.

Is there any way of blocking certmagic from using resovle.conf on Linux.
We are using split-head DNS for the zone in question, lcl.kapi.se, and even though I clearly specify an external resolver it ends up using the internal DNS.

This is partly because how our registrar handles subdomains (not in a good way, not presenting a SOA).
But…
On windows certmagic never falls back to using the host DNS since there is no /etc/resolve.confand it works even with this funky subdomain because I defined an external resolver or the defaultNameservers defined in certmagic.
But on Linux it falls back to resolve.conf and by that finds our internal DNS which has a SOA for lcl.kapi.se and uses that internal DNS which doesn’t have the _acme-challenge record.

I tested this by just using the custom + defaultNameservers in recursiveNameserver function of dnsutil.go in certmagic and by that never picking up whatever resolve.conf has.

So, is there a way to stop this fallback? It works from Windows inside our network but not from anything that has a /etc/resolve.conf that points to our internal DNS unless I use the hacked certmagic.

1 Like

I just made a small change to certmagic but I don’t know if this is anything you would consider to accept in a pull request.
Edit: Prioritize the configured resolvers by kmpm · Pull Request #131 · caddyserver/certmagic · GitHub

I borrowed the !important concept from css.

// Changed in certmagic dnsutil.go

// isImportant is used to check the custom nameservers for
// a suffix '!important'. If found, the suffix is removed from 
// the records and the bool result is set to true
func isImportant(custom []string) ([]string, bool) {
	important := false
	list := make([]string, len(custom))
	for i, v := range custom {
		if strings.HasSuffix(v, "!important") {
			important = true
			list[i] = strings.TrimSuffix(v, "!important")
		} else {
			list[i] = v
		}
	}
	return list, important
}

// recursiveNameservers are used to pre-check DNS propagation. It
// prepends user-configured nameservers (custom) to the defaults
// obtained from resolv.conf and defaultNameservers and ensures
// that all server addresses have a port value.
// If any of the custom servers are declared as `!important` then 
// only the custom servers are used.
func recursiveNameservers(custom []string) []string {
	var servers []string
	if list, important := isImportant(custom); important {
		servers = list
	} else {
		servers = append(custom, systemOrDefaultNameservers(defaultResolvConf, defaultNameservers)...)
	}
	populateNameserverPorts(servers)
	return servers
}

This topic was automatically closed after 30 days. New replies are no longer allowed.