1. Caddy version (caddy version
): v2.5.1
2. How I run Caddy: Through docker container compiled with cloudflare module, available at GitHub Packages & Docker Hub. Dockerfile & source repo available here.
a. System environment:
Docker container. Host OS & System: Ubuntu 20.04 LTS.
b. Command:
docker compose up -d
docker container logs -t -f caddy
c. Service/unit/compose file:
version: '3'
services:
caddy:
image: 'roxonic/caddy:2.5.1'
container_name: caddy
environment:
CF_API_EMAIL: "mail@example.com"
CF_API_TOKEN: "Secret_Token"
volumes:
- ./Caddyfile:/etc/caddy/Caddyfile
- ./sites:/srv
- ./data:/data
- ./config:/config
- /etc/localtime:/etc/localtime:ro
- /etc/timezone:/etc/timezone:ro
networks:
dockernet-bridge:
ipv4_address: 172.20.11.1
dockernet-ipvlan:
ipv4_address: 172.20.1.1
dns:
- 172.20.11.0
restart: always
networks:
dockernet-bridge:
external: true
dockernet-ipvlan:
external: true
d. My complete Caddyfile or JSON config:
{
email {env.CF_API_EMAIL}
acme_dns cloudflare {env.CF_API_TOKEN}
servers :443 {
protocol {
experimental_http3
}
}
}
*.roxonic.net {
@smokeping host smokeping.roxonic.net
handle @smokeping {
reverse_proxy 172.20.1.8:80
}
# Fallback for otherwise unhandled domains
handle {
abort
}
}
3. The problem I’m having:
Caddy is failing to obtain a certificate from either LetsEncrypt or ZeroSSL. Here are some interesting things that I’ve observed:
-
Caddy is able to access my zone through the API credentials successfully. How I know this? Because right after starting the caddy container, I can see that my token was used by caddy on the API tokens page as it shows Last used: a few seconds ago.
-
Caddy can also add the acme TXT record just fine. Whenever I start the container, I can see multiple logs showing the DNS record was created, and then 3 to 4 seconds later it gets deleted. This can be seen from the audit logs section of cloudflare dashboard (under manage account).
4. Error messages and/or full log output:
2022-05-31T11:43:39.190113919Z {"level":"warn","ts":1653997419.1900136,"msg":"Caddyfile input is not formatted; run the 'caddy fmt' command to fix inconsistencies","adapter":"caddyfile","file":"/etc/caddy/Caddyfile","line":13}
2022-05-31T11:43:39.191012960Z {"level":"info","ts":1653997419.1909132,"logger":"admin","msg":"admin endpoint started","address":"tcp/localhost:2019","enforce_origin":false,"origins":["//localhost:2019","//[::1]:2019","//127.0.0.1:2019"]}
2022-05-31T11:43:39.191257244Z {"level":"info","ts":1653997419.191199,"logger":"http","msg":"server is listening only on the HTTPS port but has no TLS connection policies; adding one to enable TLS","server_name":"srv0","https_port":443}
2022-05-31T11:43:39.191322483Z {"level":"info","ts":1653997419.1911993,"logger":"tls.cache.maintenance","msg":"started background certificate maintenance","cache":"0xc0003a20e0"}
2022-05-31T11:43:39.191334695Z {"level":"info","ts":1653997419.1912217,"logger":"http","msg":"enabling automatic HTTP->HTTPS redirects","server_name":"srv0"}
2022-05-31T11:43:39.191732906Z {"level":"info","ts":1653997419.191681,"logger":"http","msg":"enabling experimental HTTP/3 listener","addr":":443"}
2022-05-31T11:43:39.191789161Z {"level":"info","ts":1653997419.1917357,"logger":"tls","msg":"cleaning storage unit","description":"FileStorage:/data/caddy"}
2022-05-31T11:43:39.191798462Z {"level":"info","ts":1653997419.19176,"logger":"tls","msg":"finished cleaning storage units"}
2022-05-31T11:43:39.191840749Z {"level":"info","ts":1653997419.1917984,"msg":"failed to sufficiently increase receive buffer size (was: 208 kiB, wanted: 2048 kiB, got: 416 kiB). See https://github.com/lucas-clemente/quic-go/wiki/UDP-Receive-Buffer-Size for details."}
2022-05-31T11:43:39.191974927Z {"level":"debug","ts":1653997419.1919231,"logger":"http","msg":"starting server loop","address":"[::]:443","http3":true,"tls":true}
2022-05-31T11:43:39.191992710Z {"level":"debug","ts":1653997419.1919606,"logger":"http","msg":"starting server loop","address":"[::]:80","http3":false,"tls":false}
2022-05-31T11:43:39.192028445Z {"level":"info","ts":1653997419.1919963,"logger":"http","msg":"enabling automatic TLS certificate management","domains":["*.roxonic.net"]}
2022-05-31T11:43:39.192300908Z {"level":"info","ts":1653997419.1922388,"msg":"autosaved config (load with --resume flag)","file":"/config/caddy/autosave.json"}
2022-05-31T11:43:39.192309967Z {"level":"info","ts":1653997419.1922507,"msg":"serving initial configuration"}
2022-05-31T11:43:39.192481558Z {"level":"info","ts":1653997419.1924295,"logger":"tls.obtain","msg":"acquiring lock","identifier":"*.roxonic.net"}
2022-05-31T11:43:39.199012587Z {"level":"info","ts":1653997419.1988788,"logger":"tls.obtain","msg":"lock acquired","identifier":"*.roxonic.net"}
2022-05-31T11:43:39.199575930Z {"level":"debug","ts":1653997419.1994972,"logger":"tls.obtain","msg":"trying issuer 1/2","issuer":"acme-v02.api.letsencrypt.org-directory"}
2022-05-31T11:43:40.244836416Z {"level":"debug","ts":1653997420.244686,"logger":"tls.issuance.acme.acme_client","msg":"http request","method":"GET","url":"https://acme-v02.api.letsencrypt.org/directory","headers":{"User-Agent":["Caddy/2.5.1 CertMagic acmez (linux; amd64)"]},"response_headers":{"Cache-Control":["public, max-age=0, no-cache"],"Content-Length":["658"],"Content-Type":["application/json"],"Date":["Tue, 31 May 2022 11:43:40 GMT"],"Server":["nginx"],"Strict-Transport-Security":["max-age=604800"],"X-Frame-Options":["DENY"]},"status_code":200}
2022-05-31T11:43:40.528443536Z {"level":"debug","ts":1653997420.528327,"logger":"tls.issuance.acme.acme_client","msg":"http request","method":"HEAD","url":"https://acme-v02.api.letsencrypt.org/acme/new-nonce","headers":{"User-Agent":["Caddy/2.5.1 CertMagic acmez (linux; amd64)"]},"response_headers":{"Cache-Control":["public, max-age=0, no-cache"],"Date":["Tue, 31 May 2022 11:43:40 GMT"],"Link":["<https://acme-v02.api.letsencrypt.org/directory>;rel=\"index\""],"Replay-Nonce":["0002AcrDuc95t6wHLfqhueFzQZBSR8W-uT48j8GRBG1-8CQ"],"Server":["nginx"],"Strict-Transport-Security":["max-age=604800"],"X-Frame-Options":["DENY"]},"status_code":200}
2022-05-31T11:43:40.836530370Z {"level":"debug","ts":1653997420.8364105,"logger":"tls.issuance.acme.acme_client","msg":"http request","method":"POST","url":"https://acme-v02.api.letsencrypt.org/acme/new-acct","headers":{"Content-Type":["application/jose+json"],"User-Agent":["Caddy/2.5.1 CertMagic acmez (linux; amd64)"]},"response_headers":{"Boulder-Requester":["567235036"],"Cache-Control":["public, max-age=0, no-cache"],"Content-Length":["320"],"Content-Type":["application/json"],"Date":["Tue, 31 May 2022 11:43:40 GMT"],"Link":["<https://acme-v02.api.letsencrypt.org/directory>;rel=\"index\"","<https://letsencrypt.org/documents/LE-SA-v1.2-November-15-2017.pdf>;rel=\"terms-of-service\""],"Location":["https://acme-v02.api.letsencrypt.org/acme/acct/567235036"],"Replay-Nonce":["00011HTG9vt8TghmTfne3wcgVZ98olJb-jA73Sd1Zqh3zR0"],"Server":["nginx"],"Strict-Transport-Security":["max-age=604800"],"X-Frame-Options":["DENY"]},"status_code":201}
2022-05-31T11:43:40.837103714Z {"level":"info","ts":1653997420.8370197,"logger":"tls.issuance.acme","msg":"waiting on internal rate limiter","identifiers":["*.roxonic.net"],"ca":"https://acme-v02.api.letsencrypt.org/directory","account":"mail@example.com"}
2022-05-31T11:43:40.837122879Z {"level":"info","ts":1653997420.8370402,"logger":"tls.issuance.acme","msg":"done waiting on internal rate limiter","identifiers":["*.roxonic.net"],"ca":"https://acme-v02.api.letsencrypt.org/directory","account":"mail@example.com"}
2022-05-31T11:43:41.195216553Z {"level":"debug","ts":1653997421.1951027,"logger":"tls.issuance.acme.acme_client","msg":"http request","method":"POST","url":"https://acme-v02.api.letsencrypt.org/acme/new-order","headers":{"Content-Type":["application/jose+json"],"User-Agent":["Caddy/2.5.1 CertMagic acmez (linux; amd64)"]},"response_headers":{"Boulder-Requester":["567235036"],"Cache-Control":["public, max-age=0, no-cache"],"Content-Length":["337"],"Content-Type":["application/json"],"Date":["Tue, 31 May 2022 11:43:41 GMT"],"Link":["<https://acme-v02.api.letsencrypt.org/directory>;rel=\"index\""],"Location":["https://acme-v02.api.letsencrypt.org/acme/order/567235036/93524766186"],"Replay-Nonce":["0002n0s6X5AZJHSr9oYsycrJ-_cGmTdJxHQKyI0-Ipj_IQg"],"Server":["nginx"],"Strict-Transport-Security":["max-age=604800"],"X-Frame-Options":["DENY"]},"status_code":201}
2022-05-31T11:43:41.485706631Z {"level":"debug","ts":1653997421.4855986,"logger":"tls.issuance.acme.acme_client","msg":"http request","method":"POST","url":"https://acme-v02.api.letsencrypt.org/acme/authz-v3/114523941506","headers":{"Content-Type":["application/jose+json"],"User-Agent":["Caddy/2.5.1 CertMagic acmez (linux; amd64)"]},"response_headers":{"Boulder-Requester":["567235036"],"Cache-Control":["public, max-age=0, no-cache"],"Content-Length":["385"],"Content-Type":["application/json"],"Date":["Tue, 31 May 2022 11:43:41 GMT"],"Link":["<https://acme-v02.api.letsencrypt.org/directory>;rel=\"index\""],"Replay-Nonce":["0002WPZaqdstvxO8CWciVxB_oJehtQGgMZGOw4ddrKYE23Q"],"Server":["nginx"],"Strict-Transport-Security":["max-age=604800"],"X-Frame-Options":["DENY"]},"status_code":200}
2022-05-31T11:43:41.485824142Z {"level":"info","ts":1653997421.4857597,"logger":"tls.issuance.acme.acme_client","msg":"trying to solve challenge","identifier":"*.roxonic.net","challenge_type":"dns-01","ca":"https://acme-v02.api.letsencrypt.org/directory"}
2022-05-31T11:43:47.625503974Z {"level":"debug","ts":1653997427.625386,"logger":"tls.issuance.acme.acme_client","msg":"http request","method":"POST","url":"https://acme-v02.api.letsencrypt.org/acme/authz-v3/114523941506","headers":{"Content-Type":["application/jose+json"],"User-Agent":["Caddy/2.5.1 CertMagic acmez (linux; amd64)"]},"response_headers":{"Boulder-Requester":["567235036"],"Cache-Control":["public, max-age=0, no-cache"],"Content-Length":["389"],"Content-Type":["application/json"],"Date":["Tue, 31 May 2022 11:43:47 GMT"],"Link":["<https://acme-v02.api.letsencrypt.org/directory>;rel=\"index\""],"Replay-Nonce":["0001SHUIya9BtQ70m2Ur5BxdiPuuvlBMLwXy_-p6Wx0FbnI"],"Server":["nginx"],"Strict-Transport-Security":["max-age=604800"],"X-Frame-Options":["DENY"]},"status_code":200}
2022-05-31T11:43:47.625570173Z {"level":"error","ts":1653997427.6255155,"logger":"tls.obtain","msg":"could not get certificate from issuer","identifier":"*.roxonic.net","issuer":"acme-v02.api.letsencrypt.org-directory","error":"[*.roxonic.net] solving challenges: waiting for solver certmagic.solverWrapper to be ready: checking DNS propagation of _acme-challenge.roxonic.net: NS igor.ns.cloudflare.com. returned SERVFAIL for _acme-challenge.roxonic.net. (order=https://acme-v02.api.letsencrypt.org/acme/order/567235036/93524766186) (ca=https://acme-v02.api.letsencrypt.org/directory)"}
2022-05-31T11:43:47.625586367Z {"level":"debug","ts":1653997427.6255298,"logger":"tls.obtain","msg":"trying issuer 2/2","issuer":"acme.zerossl.com-v2-DV90"}
2022-05-31T11:43:49.289439376Z {"level":"info","ts":1653997429.28935,"logger":"tls.issuance.zerossl","msg":"generated EAB credentials","key_id":"N6sUFdX2A0VE7tGq-8zVPg"}
2022-05-31T11:43:59.076210034Z {"level":"debug","ts":1653997439.0761042,"logger":"tls.issuance.acme.acme_client","msg":"http request","method":"GET","url":"https://acme.zerossl.com/v2/DV90","headers":{"User-Agent":["Caddy/2.5.1 CertMagic acmez (linux; amd64)"]},"response_headers":{"Access-Control-Allow-Origin":["*"],"Cache-Control":["max-age=-1"],"Content-Length":["645"],"Content-Type":["application/json"],"Date":["Tue, 31 May 2022 11:43:59 GMT"],"Server":["nginx"],"Strict-Transport-Security":["max-age=15552000"]},"status_code":200}
2022-05-31T11:44:05.780184408Z {"level":"debug","ts":1653997445.7800715,"logger":"tls.issuance.acme.acme_client","msg":"http request","method":"HEAD","url":"https://acme.zerossl.com/v2/DV90/newNonce","headers":{"User-Agent":["Caddy/2.5.1 CertMagic acmez (linux; amd64)"]},"response_headers":{"Access-Control-Allow-Origin":["*"],"Cache-Control":["max-age=-1"],"Content-Type":["application/octet-stream"],"Date":["Tue, 31 May 2022 11:44:05 GMT"],"Link":["<https://acme.zerossl.com/v2/DV90>;rel=\"index\""],"Replay-Nonce":["3o4s--8J8jRf3rM5mT61aTdkcgGKuLOZt4iZeMr5AeE"],"Server":["nginx"],"Strict-Transport-Security":["max-age=15552000"]},"status_code":200}
2022-05-31T11:44:11.826933407Z {"level":"debug","ts":1653997451.8268108,"logger":"tls.issuance.acme.acme_client","msg":"http request","method":"POST","url":"https://acme.zerossl.com/v2/DV90/newAccount","headers":{"Content-Type":["application/jose+json"],"User-Agent":["Caddy/2.5.1 CertMagic acmez (linux; amd64)"]},"response_headers":{"Access-Control-Allow-Origin":["*"],"Cache-Control":["max-age=0, no-cache, no-store","max-age=-1"],"Content-Length":["579"],"Content-Type":["application/json"],"Date":["Tue, 31 May 2022 11:44:11 GMT"],"Location":["https://acme.zerossl.com/v2/DV90/account/N6sUFdX2A0VE7tGq-8zVPg"],"Replay-Nonce":["EvTDzJWHwP3Mrj7r_YxH5KRgmnfsQzdAa0nLWmKA7CY"],"Server":["nginx"],"Status":[""],"Strict-Transport-Security":["max-age=15552000"]},"status_code":201}
2022-05-31T11:44:11.827590962Z {"level":"info","ts":1653997451.8272316,"logger":"tls.issuance.acme","msg":"waiting on internal rate limiter","identifiers":["*.roxonic.net"],"ca":"https://acme.zerossl.com/v2/DV90","account":"mail@example.com"}
2022-05-31T11:44:11.827616438Z {"level":"info","ts":1653997451.8272593,"logger":"tls.issuance.acme","msg":"done waiting on internal rate limiter","identifiers":["*.roxonic.net"],"ca":"https://acme.zerossl.com/v2/DV90","account":"mail@example.com"}
2022-05-31T11:44:20.837596359Z {"level":"debug","ts":1653997460.837503,"logger":"tls.issuance.acme.acme_client","msg":"http request","method":"POST","url":"https://acme.zerossl.com/v2/DV90/newOrder","headers":{"Content-Type":["application/jose+json"],"User-Agent":["Caddy/2.5.1 CertMagic acmez (linux; amd64)"]},"response_headers":{"Access-Control-Allow-Origin":["*"],"Cache-Control":["max-age=0, no-cache, no-store","max-age=-1"],"Content-Length":["275"],"Content-Type":["application/json"],"Date":["Tue, 31 May 2022 11:44:20 GMT"],"Location":["https://acme.zerossl.com/v2/DV90/order/C53EccWndJRT94tT0tz_JA"],"Replay-Nonce":["jXxSLoif0-SX-oLo_xAHb6PKhaSPs4HVaav-WG-D_00"],"Server":["nginx"],"Status":[""],"Strict-Transport-Security":["max-age=15552000"]},"status_code":201}
2022-05-31T11:44:21.426272549Z {"level":"debug","ts":1653997461.4261749,"logger":"tls.issuance.acme.acme_client","msg":"http request","method":"POST","url":"https://acme.zerossl.com/v2/DV90/authz/gLL_U19Nyd5m1Xxis_gHpw","headers":{"Content-Type":["application/jose+json"],"User-Agent":["Caddy/2.5.1 CertMagic acmez (linux; amd64)"]},"response_headers":{"Access-Control-Allow-Origin":["*"],"Cache-Control":["max-age=-1"],"Content-Length":["295"],"Content-Type":["application/json"],"Date":["Tue, 31 May 2022 11:44:21 GMT"],"Link":["<https://acme.zerossl.com/v2/DV90>;rel=\"index\""],"Replay-Nonce":["SQb2-Q8qiPjofGHFs8d8NWHweib7AmtPW9hPrKPlJ_s"],"Retry-After":["5"],"Server":["nginx"],"Strict-Transport-Security":["max-age=15552000"]},"status_code":200}
2022-05-31T11:44:21.426346695Z {"level":"info","ts":1653997461.4262722,"logger":"tls.issuance.acme.acme_client","msg":"trying to solve challenge","identifier":"*.roxonic.net","challenge_type":"dns-01","ca":"https://acme.zerossl.com/v2/DV90"}
2022-05-31T11:44:33.371488816Z {"level":"debug","ts":1653997473.3713396,"logger":"tls.issuance.acme.acme_client","msg":"http request","method":"POST","url":"https://acme.zerossl.com/v2/DV90/authz/gLL_U19Nyd5m1Xxis_gHpw","headers":{"Content-Type":["application/jose+json"],"User-Agent":["Caddy/2.5.1 CertMagic acmez (linux; amd64)"]},"response_headers":{"Access-Control-Allow-Origin":["*"],"Cache-Control":["max-age=-1"],"Content-Length":["139"],"Content-Type":["application/json"],"Date":["Tue, 31 May 2022 11:44:33 GMT"],"Link":["<https://acme.zerossl.com/v2/DV90>;rel=\"index\""],"Replay-Nonce":["UnSwRBIv7REgecP2grHMuyEQvOx-muwbG_4jeEkgJEg"],"Retry-After":["5"],"Server":["nginx"],"Strict-Transport-Security":["max-age=15552000"]},"status_code":200}
2022-05-31T11:44:33.371523860Z {"level":"error","ts":1653997473.371437,"logger":"tls.obtain","msg":"could not get certificate from issuer","identifier":"*.roxonic.net","issuer":"acme.zerossl.com-v2-DV90","error":"[*.roxonic.net] solving challenges: waiting for solver certmagic.solverWrapper to be ready: checking DNS propagation of _acme-challenge.roxonic.net: NS igor.ns.cloudflare.com. returned SERVFAIL for _acme-challenge.roxonic.net. (order=https://acme.zerossl.com/v2/DV90/order/C53EccWndJRT94tT0tz_JA) (ca=https://acme.zerossl.com/v2/DV90)"}
2022-05-31T11:44:33.371551876Z {"level":"error","ts":1653997473.3714733,"logger":"tls.obtain","msg":"will retry","error":"[*.roxonic.net] Obtain: [*.roxonic.net] solving challenges: waiting for solver certmagic.solverWrapper to be ready: checking DNS propagation of _acme-challenge.roxonic.net: NS igor.ns.cloudflare.com. returned SERVFAIL for _acme-challenge.roxonic.net. (order=https://acme.zerossl.com/v2/DV90/order/C53EccWndJRT94tT0tz_JA) (ca=https://acme.zerossl.com/v2/DV90)","attempt":1,"retrying_in":60,"elapsed":54.172561157,"max_duration":2592000}
2022-05-31T11:44:43.695975432Z {"level":"info","ts":1653997483.695863,"msg":"shutting down apps, then terminating","signal":"SIGTERM"}
2022-05-31T11:44:43.696007229Z {"level":"warn","ts":1653997483.6958883,"msg":"exiting; byeee!! 👋","signal":"SIGTERM"}
2022-05-31T11:44:43.698325860Z {"level":"info","ts":1653997483.6982532,"logger":"tls.cache.maintenance","msg":"stopped background certificate maintenance","cache":"0xc0003a20e0"}
2022-05-31T11:44:43.698341086Z {"level":"info","ts":1653997483.6982796,"logger":"tls.obtain","msg":"releasing lock","identifier":"*.roxonic.net"}
2022-05-31T11:44:43.698407829Z {"level":"error","ts":1653997483.6983466,"msg":"unable to clean up lock in storage backend","signal":"SIGTERM","storage":"FileStorage:/data/caddy","lock_key":"issue_cert_*.roxonic.net","error":"remove /data/caddy/locks/issue_cert_wildcard_.roxonic.net.lock: no such file or directory"}
2022-05-31T11:44:43.698417656Z {"level":"error","ts":1653997483.6983783,"logger":"tls","msg":"job failed","error":"*.roxonic.net: obtaining certificate: context canceled"}
2022-05-31T11:44:43.699554974Z {"level":"info","ts":1653997483.6995027,"logger":"admin","msg":"stopped previous server","address":"tcp/localhost:2019"}
2022-05-31T11:44:43.699566323Z {"level":"info","ts":1653997483.6995122,"msg":"shutdown complete","signal":"SIGTERM","exit_code":0}
5. What I already tried:
Don’t know what I’m missing but I’ve tried a lot, starting from recreating token to rebuilding the image, trying with other domains (*.roxonic.com), replacing environment variable in caddyfile with actual values, changing DNS resolver for the container manually… none worked so far.