1. Caddy version (caddy version):
CADDY_VERSION=v2.2.0 xcaddy build --with github.com/caddy-dns/gandi
2. How I run Caddy:
# caddy.service
[Unit]
Description=Caddy
Documentation=https://caddyserver.com/docs/
After=network.target network-online.target
Requires=network-online.target
[Service]
User=caddy
Group=caddy
ExecStart=/usr/bin/caddy run --envfile /etc/caddy/Caddyfile.env --config /etc/caddy/Caddyfile
ExecReload=/usr/bin/caddy reload --config /etc/caddy/Caddyfile
TimeoutStopSec=5s
LimitNOFILE=1048576
LimitNPROC=512
PrivateTmp=true
ProtectSystem=full
AmbientCapabilities=CAP_NET_BIND_SERVICE
[Install]
WantedBy=multi-user.target
a. System environment:
- Running on a Scaleway instance
- Ubuntu bionic
- Go 1.15.2
d. My complete Caddyfile or JSON config:
# /etc/caddy/Caddyfile
{
# Enable Debug mode
debug
# Disable admin console
admin off
# Default email for tls
email contact@skynewz.dev
acme_ca https://acme-v02.api.letsencrypt.org/directory
}
:80 {
header -Server
}
cloud.skynewz.dev, *.cloud.skynewz.dev {
# https://caddyserver.com/docs/caddyfile/directives/push
push
# https://caddyserver.com/docs/caddyfile/directives/encode
encode zstd gzip
# https://caddyserver.com/docs/caddyfile/directives/metrics
metrics /metrics
# https://caddyserver.com/docs/caddyfile/directives/tls
tls {
dns gandi {env.GANDI_API_TOKEN}
}
# https://caddyserver.com/docs/caddyfile/directives/header
header {
# Hide "Server: Caddy"
-Server
# prevent attacks such as Cross Site Scripting (XSS)
Content-Security-Policy default-src 'self' cloud.skynewz.dev, *.cloud.skynewz.dev
# enable the cross-site scripting (XSS) filter built into modern web browsers
X-XSS-Protection 1; mode=block
# ensures the connection cannot be establish through an insecure HTTP connection
Strict-Transport-Security max-age=31536000; includeSubDomains; preload
# clickjacking protection
X-Frame-Options DENY
# provides clickjacking protection. Disable iframes
X-Frame-Options: SAMEORIGIN
# disable clients from sniffing the media type
X-Content-Type-Options nosniff
# keep referrer data off of HTTP connections
Referrer-Policy no-referrer-when-downgrade
}
# https://caddyserver.com/docs/caddyfile/directives/respond
# Replace backends health checks and provide one for this LB
# respond /health 200
# https://caddyserver.com/docs/caddyfile/directives/log
log {
output stdout
format console
}
# https://caddyserver.com/docs/caddyfile/directives/reverse_proxy
reverse_proxy * {
# Specify backend here
to 10.70.12.85:30438
to 10.69.102.65:30438
lb_policy round_robin
lb_try_duration 1s
lb_try_interval 250ms
# health_path /health # Backend health check path
# health_port 80 # Default same as backend port
# health_interval 10s
# health_timeout 2s
# health_status 200
# health_body "OK"
fail_duration 2s
max_fails 2
unhealthy_status 5xx
unhealthy_latency 10s
unhealthy_request_count 10
}
}
# /etc/caddy/Caddyfile.env
GANDI_API_TOKEN=[redacted]
3. The problem I’m having:
My acme TXT records are properly set in the related Gandi DNS zones. But obtaining the certificate always timeout.
4. Error messages and/or full log output:
@ 10800 IN SOA ns1.gandi.net. hostmaster.gandi.net. 1602253807 10800 3600 604800 10800
*.cloud 300 IN CNAME cloud.skynewz.dev.
@ 300 IN A 185.199.108.153
@ 300 IN A 185.199.109.153
@ 300 IN A 185.199.110.153
@ 300 IN A 185.199.111.153
@ 10800 IN MX 10 spool.mail.gandi.net.
@ 10800 IN MX 50 fb.mail.gandi.net.
@ 300 IN TXT "v=spf1 include:_mailcust.gandi.net ?all"
_acme-challenge.cloud.skynewz.dev 10800 IN TXT "1MXf_ZXP2fsqcL5aoHRp7lEKlFKKfF80nRrqQ_U9KXI"
_acme-challenge.cloud.skynewz.dev 10800 IN TXT "41fqxzBfjOf60IdI8IwIEX_3re4aEYpJj_1lGBCuu6s"
_acme-challenge.cloud.skynewz.dev 10800 IN TXT "41fqxzBfjOf60IdI8IwIEX_3re4aEYpJj_1lGBCuu6s"
_acme-challenge.cloud.skynewz.dev 10800 IN TXT "CazgFDn_IrmDw0KXVqs4Kl5-Vv8MKvUwKT_YQzBoz0o"
_acme-challenge.cloud.skynewz.dev 10800 IN TXT "KYCFYVONLOBSUoDgM6KZp55POKlZxouvu7WqxR-EfSo"
_acme-challenge.cloud.skynewz.dev 10800 IN TXT "VtcQKWe7NBIqe2aBvR5Wy6NBrgP3Q8yIS5yRCmIZ1TI"
_acme-challenge.cloud.skynewz.dev 10800 IN TXT "VtcQKWe7NBIqe2aBvR5Wy6NBrgP3Q8yIS5yRCmIZ1TI"
_acme-challenge.cloud.skynewz.dev 10800 IN TXT "VtcQKWe7NBIqe2aBvR5Wy6NBrgP3Q8yIS5yRCmIZ1TI"
_acme-challenge.cloud.skynewz.dev 10800 IN TXT "VtcQKWe7NBIqe2aBvR5Wy6NBrgP3Q8yIS5yRCmIZ1TI"
_acme-challenge.cloud.skynewz.dev 10800 IN TXT "fafuT42wKuL1AjHUOtDuuH0jABkwmdRk1lFl_O9qKmc"
_acme-challenge.cloud.skynewz.dev 10800 IN TXT "gswLJvKEqFcuyDgUriNoE_hlgf71USFm4AePzj-NHJ4"
_acme-challenge.cloud.skynewz.dev 10800 IN TXT "hVzS2nnXTErK6Xv8D4xGO15q96V2OO5uJT41d4i2Tro"
_acme-challenge.cloud.skynewz.dev 10800 IN TXT "slAdKF1G9qWnPknE9Tua1aVg9yux9JWB-ObtBGfx4Uc"
_acme-challenge.cloud.skynewz.dev 10800 IN TXT "tTmFZ_um62V4jNCnirwWB533pq-esRsOCxfDmWvi1As"
cloud 300 IN A 51.15.212.58
Oct 09 14:42:05 caddy systemd[1]: Started Caddy.
Oct 09 14:42:05 caddy caddy[5517]: {"level":"info","ts":1602254525.1453805,"msg":"using provided configuration","config_file":"/etc/caddy/Caddyfile","config_adapter":""}
Oct 09 14:42:05 caddy caddy[5517]: {"level":"info","ts":1602254525.1503983,"logger":"admin","msg":"admin endpoint started","address":"tcp/localhost:2019","enforce_origin":false,"origins":["localhost:2019","[::1]:2019","127.0.0.1:2019"]}
Oct 09 14:42:05 caddy caddy[5517]: {"level":"info","ts":1602254525.151538,"logger":"tls.cache.maintenance","msg":"started background certificate maintenance","cache":"0xc000338690"}
Oct 09 14:42:05 caddy caddy[5517]: {"level":"info","ts":1602254525.1533282,"logger":"http","msg":"server is listening only on the HTTPS port but has no TLS connection policies; adding one to enable TLS","server_name":"srv0","https_port":4
Oct 09 14:42:05 caddy caddy[5517]: {"level":"info","ts":1602254525.1533804,"logger":"http","msg":"enabling automatic HTTP->HTTPS redirects","server_name":"srv0"}
Oct 09 14:42:05 caddy caddy[5517]: {"level":"info","ts":1602254525.153854,"logger":"http","msg":"enabling automatic TLS certificate management","domains":["cloud.skynewz.dev"]}
Oct 09 14:42:05 caddy caddy[5517]: {"level":"info","ts":1602254525.1541166,"msg":"autosaved config","file":"/var/lib/caddy/.config/caddy/autosave.json"}
Oct 09 14:42:05 caddy caddy[5517]: {"level":"info","ts":1602254525.154134,"msg":"serving initial configuration"}
Oct 09 14:42:05 caddy caddy[5517]: {"level":"info","ts":1602254525.1555028,"logger":"tls","msg":"cleaned up storage units"}
Oct 09 14:42:05 caddy caddy[5517]: {"level":"info","ts":1602254525.1563451,"logger":"tls.obtain","msg":"acquiring lock","identifier":"cloud.skynewz.dev"}
Oct 09 14:42:05 caddy caddy[5517]: {"level":"info","ts":1602254525.1569538,"logger":"tls.obtain","msg":"lock acquired","identifier":"cloud.skynewz.dev"}
Oct 09 14:42:05 caddy caddy[5517]: {"level":"info","ts":1602254525.1688776,"logger":"tls.issuance.acme","msg":"waiting on internal rate limiter","identifiers":["cloud.skynewz.dev"]}
Oct 09 14:42:05 caddy caddy[5517]: {"level":"info","ts":1602254525.169309,"logger":"tls.issuance.acme","msg":"done waiting on internal rate limiter","identifiers":["cloud.skynewz.dev"]}
Oct 09 14:42:06 caddy caddy[5517]: Oct 09 14:42:06 caddy caddy[5517]: {"level":"info","ts":1602254526.2846875,"logger":"tls.issuance.acme.acme_client","msg":"trying to solve challenge","identifier":"cloud.skynewz.dev","challenge_type":"dns-01","ca":"https://acme-v02.api.letsencrypt.org/directory"}
Oct 09 14:44:08 caddy caddy[5517]: {"level":"error","ts":1602254648.374959,"logger":"tls.obtain","msg":"will retry","error":"[cloud.skynewz.dev] Obtain: [cloud.skynewz.dev] solving challenges: waiting for solver *certmagic.DNS01Solver to be ready: timed out waiting for record to fully propagate; verify DNS provider configuration is correct - last error: <nil> (order=https://acme-v02.api.letsencrypt.org/acme/order/98841065/5602236743) (ca=https://acme-v02.api.letsencrypt.org/directory)","attempt":1,"retrying_in":60,"elapsed":123.217807825,"max_duration":2592000}
Oct 09 14:45:11 caddy caddy[5517]: {"level":"info","ts":1602254711.0881233,"logger":"tls.issuance.acme.acme_client","msg":"trying to solve challenge","identifier":"cloud.skynewz.dev","challenge_type":"dns-01","ca":"https://acme-staging-v02.api.letsencrypt.org/directory"}
…
Looping on this 😞
5. What I already tried:
- Same issue with basic Caddyfile
cloud.skynewz.dev
tls contact@skynewz.dev {
dns gandi {env.GANDI_API_TOKEN}
}
respond "Hello, world!"
- Set
resolv.confto1.1.1.1/1.0.01anddig
root@caddy:~# dig cloud.skynewz.dev
; <<>> DiG 9.11.3-1ubuntu1.13-Ubuntu <<>> cloud.skynewz.dev
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 35385
;; flags: qr rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 1
;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 1232
;; QUESTION SECTION:
;cloud.skynewz.dev. IN A
;; ANSWER SECTION:
cloud.skynewz.dev. 189 IN A 51.15.139.15
;; Query time: 3 msec
;; SERVER: 1.1.1.1#53(1.1.1.1)
;; WHEN: Fri Oct 09 20:59:03 UTC 2020
;; MSG SIZE rcvd: 62
root@caddy:~# dig foo.cloud.skynewz.dev
; <<>> DiG 9.11.3-1ubuntu1.13-Ubuntu <<>> foo.cloud.skynewz.dev
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 39890
;; flags: qr rd ra; QUERY: 1, ANSWER: 2, AUTHORITY: 0, ADDITIONAL: 1
;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 1232
;; QUESTION SECTION:
;foo.cloud.skynewz.dev. IN A
;; ANSWER SECTION:
foo.cloud.skynewz.dev. 300 IN CNAME cloud.skynewz.dev.
cloud.skynewz.dev. 300 IN A 51.15.139.15
;; Query time: 88 msec
;; SERVER: 1.1.1.1#53(1.1.1.1)
;; WHEN: Fri Oct 09 21:01:33 UTC 2020
;; MSG SIZE rcvd: 80
- Tried on my laptop, also have this issue
2020/10/10 14:50:51.407 INFO using adjacent Caddyfile
2020/10/10 14:50:51.412 INFO admin admin endpoint started {"address": "tcp/localhost:2019", "enforce_origin": false, "origins": ["localhost:2019", "[::1]:2019", "127.0.0.1:2019"]}
2020/10/10 14:50:51.413 INFO tls.cache.maintenance started background certificate maintenance {"cache": "0xc0003b5c00"}
2020/10/10 14:50:51.414 INFO http server is listening only on the HTTPS port but has no TLS connection policies; adding one to enable TLS {"server_name": "srv0", "https_port": 443}
2020/10/10 14:50:51.414 INFO http enabling automatic HTTP->HTTPS redirects {"server_name": "srv0"}
2020/10/10 14:50:51.415 INFO http enabling automatic TLS certificate management {"domains": ["cloud.skynewz.dev", "*.cloud.skynewz.dev"]}
2020/10/10 14:50:51.416 INFO autosaved config {"file": "/Users/quentin/Library/Application Support/Caddy/autosave.json"}
2020/10/10 14:50:51.416 INFO serving initial configuration
2020/10/10 14:50:51.416 INFO tls.obtain acquiring lock {"identifier": "*.cloud.skynewz.dev"}
2020/10/10 14:50:51.416 INFO tls.obtain acquiring lock {"identifier": "cloud.skynewz.dev"}
2020/10/10 14:50:51.418 INFO tls.obtain lock acquired {"identifier": "*.cloud.skynewz.dev"}
2020/10/10 14:50:51.418 INFO tls.obtain lock acquired {"identifier": "cloud.skynewz.dev"}
2020/10/10 14:50:51.420 INFO tls cleaned up storage units
2020/10/10 14:50:57.504 INFO tls.issuance.acme waiting on internal rate limiter {"identifiers": ["*.cloud.skynewz.dev"]}
2020/10/10 14:50:57.504 INFO tls.issuance.acme done waiting on internal rate limiter {"identifiers": ["*.cloud.skynewz.dev"]}
2020/10/10 14:50:57.838 INFO tls.issuance.acme.acme_client trying to solve challenge {"identifier": "*.cloud.skynewz.dev", "challenge_type": "dns-01", "ca": "https://acme-v02.api.letsencrypt.org/directory"}
2020/10/10 14:51:02.967 INFO tls.issuance.acme waiting on internal rate limiter {"identifiers": ["cloud.skynewz.dev"]}
2020/10/10 14:51:02.967 INFO tls.issuance.acme done waiting on internal rate limiter {"identifiers": ["cloud.skynewz.dev"]}
2020/10/10 14:51:05.479 INFO tls.issuance.acme.acme_client trying to solve challenge {"identifier": "cloud.skynewz.dev", "challenge_type": "dns-01", "ca": "https://acme-v02.api.letsencrypt.org/directory"}
2020/10/10 14:53:00.456 ERROR tls.obtain will retry {"error": "[*.cloud.skynewz.dev] Obtain: [*.cloud.skynewz.dev] solving challenges: waiting for solver *certmagic.DNS01Solver to be ready: timed out waiting for record to fully propagate; verify DNS provider configuration is correct - last error: <nil> (order=https://acme-v02.api.letsencrypt.org/acme/order/98930981/5619072497) (ca=https://acme-v02.api.letsencrypt.org/directory)", "attempt": 1, "retrying_in": 60, "elapsed": 129.039618791, "max_duration": 2592000}
- Try on laptop with another root domain
hello.lemairepro.fr, same issue…
6. Links to relevant resources:
Github issue : [Gandi] Unable to obtain certificates HTTPS certificates · Issue #3787 · caddyserver/caddy · GitHub