Error creating new host via API

1. Output of caddy version:

v2.5.1 h1:bAWwslD1jNeCzDa+jDCNwb8M3UJ2tPa8UZFFzPVmGKs=

2. How I run Caddy:

I using Docker image running into Kubernetes cluster hosted in GCP.

a. System environment:

Docker in Kubernetes cluster using a default dockerfile. The storage is hosted in cloud storage using a volume mounted in caddy deployment… The domain are pointing to the Caddy IP, so the communication is allow between caddy and my hosting in order to manage the certificates.

b. Command:

CMD ["caddy", "run", "--resume"]

c. Service/unit/compose file:

deployment.yaml:

apiVersion: apps/v1
kind: Deployment
metadata:
  annotations:
  labels:
    deployment: caddy
  name: caddy-deployment
  namespace: default
spec:
  progressDeadlineSeconds: 600
  replicas: 1
  template:
    spec:
      automountServiceAccountToken: true
      containers:
      - env:
        - name: XDG_CONFIG_HOME
          value: /data/config
        image: eu.gcr.io/caddy/caddy:313bf44304d07e5c53a3d6ced0dd5f
        imagePullPolicy: IfNotPresent
        name: caddy
        volumeMounts:
        - mountPath: /data
          name: caddy-volume    
      volumes:
      - name: caddy-volume
        persistentVolumeClaim:
          claimName: csi-gcs-pvc

service.yaml:

apiVersion: v1
kind: Service
metadata:
  finalizers:
  - service.kubernetes.io/load-balancer-cleanup
  name: td-load-balancer
  namespace: default
spec:
  clusterIP: x.x.x.x
  clusterIPs:
  - x.x.x.x
  ports:
  - name: http
    nodePort: 31930
    port: 80
    protocol: TCP
    targetPort: 80
  - name: https
    nodePort: 32746
    port: 443
    protocol: TCP
    targetPort: 443
  - name: api
    nodePort: 31462
    port: 2019
    protocol: TCP
    targetPort: 2019
  selector:
    deployment: caddy
  type: LoadBalancer

d. My complete Caddy config:

{
  "admin": {
    "listen": "0.0.0.0:2019"
  },
  "apps": {
    "http": {
      "http_port": 80,
      "https_port": 443,
      "servers": {
        "srv0": {
          "listen": [
            "0.0.0.0:443"
          ],
          "routes": [
            {
              "handle": [
                {
                  "handler": "subroute",
                  "routes": [
                    {
                      "handle": [
                        {
                          "handler": "reverse_proxy",
                          "upstreams": [
                            {
                              "dial": "frontend-service:3000"
                            }
                          ]
                        }
                      ]
                    }
                  ]
                }
              ],
              "match": [
                {
                  "host": [
                    "hachedigital.net",
                    "www.hachedigital.net",
                    "1.hachedigital.net",
                    "2.hachedigital.net",
                    "3.hachedigital.net",
                    "4.hachedigital.net",
                    "5.hachedigital.net",
                    "6.hachedigital.net",
                    "7.hachedigital.net",
                    "8.hachedigital.net",
                    "9.hachedigital.net",
                    "10.hachedigital.net"
                  ]
                }
              ],
              "terminal": true
            }
          ]
        }
      }
    }
  },
  "logging": {
    "logs": {
      "default": {
        "level": "DEBUG"
      }
    }
  }
}

3. The problem I’m having:

All of these hosts were added via API with the next command:

curl -XPOST http://api-caddy-dev.hachedigital.net:2019/config/apps/http/servers/srv0/routes/0/match/0/host -H "Content-Type: application/json" -d '"5.hachedigital.net"'

The first 4 or 5 calls, working well, but suddenly the next calls stop working receiving this error:

{"error":"loading new config: http app module: start: finalizing automatic HTTPS: managing certificates for [7.hachedigital.net 9.hachedigital.net 3.hachedigital.net 6.hachedigital.net 11.hachedigital.net 2.hachedigital.net 4.hachedigital.net  1.hachedigital.net hachedigital.com www.domainmine.net hachedigital.net 5.hachedigital.net 8.hachedigital.net 10.domainmine.net domainmine.net 10.hachedigital.com]: automate: manage [7.hachedigital.net 9.hachedigital.net 3.hachedigital.net 6.hachedigital.net 11.hachedigital.net 2.hachedigital.net 4.hachedigital.net  1.domainmine.net hachedigital.com www.hachedigital.net hachedigital.net 5.hachedigital.net 8.hachedigital.net 10.hachedigital.net hachedigital.net 10.hachedigital.com]: 4.hachedigital.net: caching certificate: read /data/caddy/certificates/acme-v02.api.letsencrypt.org-directory/4.hachedigital.net/4.hachedigital.net.crt: operation canceled"}

If I wait some seconds and I try again, suddenly it works again and if I check the config I can se the host added, but the certificate of most of then are not created, and I see in the logs the next messages:

{"level":"debug","ts":1662986574.8595495,"logger":"http.stdlib","msg":"http: TLS handshake error from 10.1.2.205:22149: no certificate available for '7.domainmine.net'"}
{"level":"error","ts":1662986787.284492,"logger":"tls","msg":"job failed","error":"3.domainmine.net: obtaining certificate: context canceled"}

Sometimes when I wait like 20-30 mins or restarting the caddy pod, the creation of the certificate works properly. But I think there is some problem somewhere because the behavior is unexpected.

4. Error messages and/or full log output:

I just try to create new hosts and the first curl doesn’t work , and the next 5 it works. But as you can see the certificate generation is failing.

{"level":"error","ts":1663004856.8218925,"logger":"tls.issuance.acme.acme_client","msg":"validating authorization","identifier":"18.hachedigital.net","problem":{"type":"urn:ietf:params:acme:error:dns","title":"","detail":"DNS problem: NXDOMAIN looking up A for 18.hachedigital.net - check that a DNS record exists for this domain; DNS problem: NXDOMAIN looking up AAAA for 18.hachedigital.net - check that a DNS record exists for this domain","instance":"","subproblems":[]},"order":"https://acme-v02.api.letsencrypt.org/acme/order/728093807/124878024847","attempt":2,"max_attempts":3}
{"level":"error","ts":1663004856.8220396,"logger":"tls.obtain","msg":"could not get certificate from issuer","identifier":"18.hachedigital.net","issuer":"acme-v02.api.letsencrypt.org-directory","error":"HTTP 400 urn:ietf:params:acme:error:dns - DNS problem: NXDOMAIN looking up A for 18.hachedigital.net - check that a DNS record exists for this domain; DNS problem: NXDOMAIN looking up AAAA for 18.hachedigital.net - check that a DNS record exists for this domain"}
{"level":"debug","ts":1663004856.822155,"logger":"tls.obtain","msg":"trying issuer 2/2","issuer":"acme.zerossl.com-v2-DV90"}
{"level":"warn","ts":1663004856.8486001,"logger":"tls.issuance.zerossl","msg":"missing email address for ZeroSSL; it is strongly recommended to set one for next time"}
{"level":"info","ts":1663004857.1070814,"logger":"tls.issuance.zerossl","msg":"generated EAB credentials","key_id":"Un1DTf8GXFBbzHQlsyLVsQ"}
{"level":"error","ts":1663004857.8462055,"logger":"tls","msg":"job failed","error":"13.hachedigital.net: obtaining certificate: unable to acquire lock 'issue_cert_13.hachedigital.net': creating lock file: open /data/caddy/locks/issue_cert_13.hachedigital.net.lock: operation canceled"}
{"level":"error","ts":1663004858.0800753,"logger":"tls","msg":"job failed","error":"15.hachedigital.net: obtaining certificate: unable to acquire lock 'issue_cert_15.hachedigital.net': accessing lock file: open /data/caddy/locks/issue_cert_15.hachedigital.net.lock: operation canceled"}
{"level":"info","ts":1663004859.037914,"logger":"admin.api","msg":"received request","method":"POST","host":"api-caddy-dev.topdoctors.es:2019","uri":"/config/apps/http/servers/srv0/routes/0/match/0/host","remote_ip":"10.1.2.205","remote_port":"5819","headers":{"Accept":["*/*"],"Content-Length":["21"],"Content-Type":["application/json"],"User-Agent":["curl/7.81.0"]}}
{"level":"info","ts":1663004859.0388248,"logger":"admin","msg":"admin endpoint started","address":"tcp/0.0.0.0:2019","enforce_origin":false,"origins":["//0.0.0.0:2019"]}
{"level":"warn","ts":1663004859.0390475,"logger":"admin","msg":"admin endpoint on open interface; host checking disabled","address":"tcp/0.0.0.0:2019"}
{"level":"info","ts":1663004859.039507,"logger":"tls.cache.maintenance","msg":"started background certificate maintenance","cache":"0xc00049ca10"}
{"level":"info","ts":1663004859.0396595,"logger":"http","msg":"server is listening only on the HTTPS port but has no TLS connection policies; adding one to enable TLS","server_name":"srv0","https_port":443}
{"level":"info","ts":1663004859.039823,"logger":"http","msg":"enabling automatic HTTP->HTTPS redirects","server_name":"srv0"}
{"level":"debug","ts":1663004859.040214,"logger":"http","msg":"starting server loop","address":"[::]:443","http3":false,"tls":true}
{"level":"debug","ts":1663004859.040312,"logger":"http","msg":"starting server loop","address":"[::]:80","http3":false,"tls":false}
{"level":"info","ts":1663004859.0403857,"logger":"http","msg":"enabling automatic TLS certificate management","domains":["19.hachedigital.net","18.hachedigital.net","13.hachedigital.net","10.hachedigital.net","7.hachedigital.net","www.hachedigital.net","15.hachedigital.net","12.hachedigital.net","4.hachedigital.net","6.hachedigital.net","5.hachedigital.net","9.hachedigital.net","10.doctorenpruebas.com","doctorantoniofernandez.com","hachedigital.net","100.doctorenpruebas.com","2.hachedigital.net","3.hachedigital.net","8.hachedigital.net","17.hachedigital.net","1.hachedigital.net","23hachedigital.net"]}
{"level":"info","ts":1663004859.1377804,"logger":"tls.cache.maintenance","msg":"stopped background certificate maintenance","cache":"0xc00049ca10"}
{"level":"error","ts":1663004859.1379566,"logger":"admin.api","msg":"request error","error":"loading new config: http app module: start: finalizing automatic HTTPS: managing certificates for [19.hachedigital.net 18.hachedigital.net 13.hachedigital.net 10.hachedigital.net 7.hachedigital.net www.hachedigital.net 15.hachedigital.net 12.hachedigital.net 4.hachedigital.net 6.hachedigital.net 5.hachedigital.net 9.hachedigital.net 10.doctorenpruebas.com doctorantoniofernandez.com hachedigital.net 100.doctorenpruebas.com 2.hachedigital.net 3.hachedigital.net 8.hachedigital.net 17.hachedigital.net 1.hachedigital.net 23hachedigital.net]: automate: manage [19.hachedigital.net 18.hachedigital.net 13.hachedigital.net 10.hachedigital.net 7.hachedigital.net www.hachedigital.net 15.hachedigital.net 12.hachedigital.net 4.hachedigital.net 6.hachedigital.net 5.hachedigital.net 9.hachedigital.net 10.doctorenpruebas.com doctorantoniofernandez.com hachedigital.net 100.doctorenpruebas.com 2.hachedigital.net 3.hachedigital.net 8.hachedigital.net 17.hachedigital.net 1.hachedigital.net 23hachedigital.net]: 18.hachedigital.net: caching certificate: open /data/caddy/certificates/acme.zerossl.com-v2-dv90/18.hachedigital.net/18.hachedigital.net.key: operation canceled","status_code":500}
{"level":"info","ts":1663004859.1774104,"logger":"admin","msg":"stopped previous server","address":"tcp/0.0.0.0:2019"}
{"level":"error","ts":1663004859.8429208,"logger":"tls","msg":"job failed","error":"19.hachedigital.net: obtaining certificate: failed storage check: open /data/caddy/rw_test_3191312775412548101: operation canceled - storage is probably misconfigured"}
{"level":"info","ts":1663004861.844687,"msg":"[ERROR] Keeping lock file fresh: open /data/caddy/locks/issue_cert_18.hachedigital.net.lock: operation canceled - terminating lock maintenance (lockfile: /data/caddy/locks/issue_cert_18.hachedigital.net.lock)"}
{"level":"debug","ts":1663004864.6286128,"logger":"tls.issuance.acme.acme_client","msg":"http request","method":"HEAD","url":"https://acme.zerossl.com/v2/DV90/newNonce","headers":{"User-Agent":["Caddy/2.5.1 CertMagic acmez (linux; amd64)"]},"response_headers":{"Access-Control-Allow-Origin":["*"],"Cache-Control":["max-age=-1"],"Content-Type":["application/octet-stream"],"Date":["Mon, 12 Sep 2022 17:47:44 GMT"],"Link":["<https://acme.zerossl.com/v2/DV90>;rel=\"index\""],"Replay-Nonce":["0-aGWN8ntOg1caBHuKG-9o85x7FCAKXy6VZFtrXrlnY"],"Server":["nginx"],"Strict-Transport-Security":["max-age=15552000"]},"status_code":200}
{"level":"info","ts":1663004865.3422885,"msg":"[ERROR] Keeping lock file fresh: sync /data/caddy/locks/issue_cert_13.hachedigital.net.lock: operation canceled - terminating lock maintenance (lockfile: /data/caddy/locks/issue_cert_13.hachedigital.net.lock)"}
{"level":"info","ts":1663004865.370495,"msg":"[ERROR] Keeping lock file fresh: sync /data/caddy/locks/issue_cert_17.hachedigital.net.lock: operation canceled - terminating lock maintenance (lockfile: /data/caddy/locks/issue_cert_17.hachedigital.net.lock)"}

5. What I already tried:

I’ve tried to redeploy all the caddy, with a clean config and removing al the data.

There’s quite a few issues there.

curl -XPOST

Remove -XPOST: Unnecessary use of curl -X | daniel.haxx.se

"http_port": 80, "https_port": 443,

Remove those parts of your config, they are the defaults.

"listen": "0.0.0.0:2019"

Be careful with this. You say “the communication is allow between caddy and my hosting in order to manage the certificates.” but if you’re doing this you need to make sure it’s not world-writeable! To access the admin API remotely, you should set up secure remote management in your config instead.

Additionally, the logs are inconsistent with your config. I understand you’re adding domains via API but your config shows a small handful of domains under hachedigital.net and your logs are showing errors for other domains, like www.domainmine.net and 7.domainmine.net (which do not exist btw; NXDOMAIN – did you redact this too? or misconfigure your DNS?) and 100.doctorenpruebas.com.

Your storage backend seems to be slower than your API requests. Are you adding 1 domain at a time? Why?

Let’s Encrypt has a rate limit of 50 subdomains per registered domain – and it looks like you are trying to get certs for 100+ subdomains of the same registered domain. Caddy will fallback to using ZeroSSL, but you should probably just manage a single wildcard certificate instead.

If you would post your full logs please, as requested (now for the 3rd time), that would give us a clearer picture.

Hello,

Thanks for reply @matt. Sorry I copied the logs from the other post where I hid my domain name.

I add one by one because I have a batch proccess to migrate all the host from other platform ony by one reading a CSV. Do you think is better to pass directly the autosave.json with all the hosts added instead do a lot of ‘curls’?

About the security, I have reading your documentation but I dont find any clear doc explaining how to apply any kind of auth to all endpoint. Could you send me some link about it? I’d appreciate it.

In this case I’m using subdomains because is a test, but in our production environment we will use totally different domains, so the wildcald is not valid for this case…

I attach my full log here after add 10 new hosts via API. (one by one). It only work 4/10… It is curious because the first call not worked, and the second one yes… It is random :confused:

{"level":"error","ts":1663008234.0219235,"logger":"tls","msg":"job failed","error":"17.hachedigital.net: obtaining certificate: unable to acquire lock 'issue_cert_17.hachedigital.net': accessing lock file: open /data/caddy/locks/issue_cert_17.hachedigital.net.lock: operation canceled"}
{"level":"debug","ts":1663008234.0338342,"logger":"tls.cache","msg":"added certificate to cache","subjects":["4.hachedigital.net"],"expiration":1670755200,"managed":true,"issuer_key":"acme-v02.api.letsencrypt.org-directory","hash":"312adebb66f0a30b90ac14bf905217f4695b2251c2aab24f3e1896bfee1f1ed2","cache_size":11,"cache_capacity":10000}
{"level":"info","ts":1663008234.2106519,"msg":"[INFO][FileStorage:/data/caddy] Lock for 'issue_cert_19.hachedigital.net' is stale (created: 2022-09-12 17:47:33.549746506 +0000 UTC, last update: 2022-09-12 17:48:00.715176981 +0000 UTC); removing then retrying: /data/caddy/locks/issue_cert_19.hachedigital.net.lock"}
{"level":"debug","ts":1663008234.212636,"logger":"tls","msg":"loading managed certificate","domain":"6.hachedigital.net","expiration":1670755390,"issuer_key":"acme-v02.api.letsencrypt.org-directory","storage":"FileStorage:/data/caddy"}
{"level":"debug","ts":1663008234.2264025,"logger":"tls.cache","msg":"added certificate to cache","subjects":["6.hachedigital.net"],"expiration":1670755390,"managed":true,"issuer_key":"acme-v02.api.letsencrypt.org-directory","hash":"3101f787ea8fa826641b04038d65f07734b7e39a5b1ce8059ab85b3255ddfd3e","cache_size":12,"cache_capacity":10000}
{"level":"error","ts":1663008234.2600574,"logger":"tls.issuance.acme.acme_client","msg":"challenge failed","identifier":"340.hachedigital.net","challenge_type":"tls-alpn-01","problem":{"type":"urn:ietf:params:acme:error:dns","title":"","detail":"DNS problem: NXDOMAIN looking up A for 340.hachedigital.net - check that a DNS record exists for this domain; DNS problem: NXDOMAIN looking up AAAA for 340.hachedigital.net - check that a DNS record exists for this domain","instance":"","subproblems":[]}}
{"level":"error","ts":1663008234.260111,"logger":"tls.issuance.acme.acme_client","msg":"validating authorization","identifier":"340.hachedigital.net","problem":{"type":"urn:ietf:params:acme:error:dns","title":"","detail":"DNS problem: NXDOMAIN looking up A for 340.hachedigital.net - check that a DNS record exists for this domain; DNS problem: NXDOMAIN looking up AAAA for 340.hachedigital.net - check that a DNS record exists for this domain","instance":"","subproblems":[]},"order":"https://acme-v02.api.letsencrypt.org/acme/order/728093807/124889448667","attempt":1,"max_attempts":3}
{"level":"info","ts":1663008234.2628145,"logger":"tls.obtain","msg":"acquiring lock","identifier":"13.hachedigital.net"}
{"level":"info","ts":1663008234.3993378,"logger":"tls.cache.maintenance","msg":"stopped background certificate maintenance","cache":"0xc000166d20"}
{"level":"error","ts":1663008234.3994503,"logger":"admin.api","msg":"request error","error":"loading new config: http app module: start: finalizing automatic HTTPS: managing certificates for [23hachedigital.net 7.hachedigital.net 5.hachedigital.net 8.hachedigital.net 38.hachedigital.net hachedigital.net 15.hachedigital.net 32.hachedigital.net 340.hachedigital.net 13.hachedigital.net 35.hachedigital.net www.hachedigital.net 18.hachedigital.net  9.hachedigital.net 40.hachedigital.net 36.hachedigital.net 10.hachedigital.net 1.hachedigital.net 4.hachedigital.net 6.hachedigital.net 17.hachedigital.net  3.hachedigital.net 19.hachedigital.net 2.hachedigital.net]: automate: manage [23hachedigital.net 7.hachedigital.net 5.hachedigital.net 8.hachedigital.net 38.hachedigital.net hachedigital.net 15.hachedigital.net 32.hachedigital.net 340.hachedigital.net 13.hachedigital.net 35.hachedigital.net www.hachedigital.net 18.hachedigital.net 10.doctorenpruebas.com 9.hachedigital.net 40.hachedigital.net 36.hachedigital.net 10.hachedigital.net 1.hachedigital.net 4.hachedigital.net 6.hachedigital.net 17.hachedigital.net 100.doctorenpruebas.com 3.hachedigital.net 19.hachedigital.net 2.hachedigital.net]: 32.hachedigital.net caching certificate: open /data/caddy/certificates/acme-v02.api.letsencrypt.org-directory/32.hachedigital.net/32.hachedigital.net: operation canceled","status_code":500}
{"level":"error","ts":1663008234.399674,"logger":"tls","msg":"job failed","error":"38.hachedigital.net: obtaining certificate: unable to acquire lock 'issue_cert_38.hachedigital.net': context canceled"}
{"level":"error","ts":1663008234.3996909,"logger":"tls","msg":"job failed","error":"340.hachedigital.net: obtaining certificate: unable to acquire lock 'issue_cert_340.hachedigital.net': context canceled"}
{"level":"error","ts":1663008234.399699,"logger":"tls","msg":"job failed","error":"32.hachedigital.net: obtaining certificate: unable to acquire lock 'issue_cert_32.hachedigital.net': context canceled"}
{"level":"error","ts":1663008234.4430587,"logger":"tls","msg":"job failed","error":"13.hachedigital.net: obtaining certificate: unable to acquire lock 'issue_cert_13.hachedigital.net': context canceled"}
{"level":"info","ts":1663008234.4648838,"msg":"[INFO][FileStorage:/data/caddy] Lock for 'issue_cert_19.hachedigital.net' is stale (created: 2022-09-12 17:47:33.549746506 +0000 UTC, last update: 2022-09-12 17:48:00.715176981 +0000 UTC); removing then retrying: /data/caddy/locks/issue_cert_19.hachedigital.net.lock"}
{"level":"info","ts":1663008234.510643,"logger":"tls.obtain","msg":"acquiring lock","identifier":"18.hachedigital.net"}
{"level":"info","ts":1663008234.6856496,"msg":"[INFO][FileStorage:/data/caddy] Lock for 'issue_cert_18.hachedigital.net' is stale (created: 2022-09-12 17:47:31.237228165 +0000 UTC, last update: 2022-09-12 17:47:36.628711032 +0000 UTC); removing then retrying: /data/caddy/locks/issue_cert_18.hachedigital.net.lock"}
{"level":"info","ts":1663008234.7008991,"msg":"[INFO][FileStorage:/data/caddy] Lock for 'issue_cert_19.hachedigital.net' is stale (created: 2022-09-12 17:47:33.549746506 +0000 UTC, last update: 2022-09-12 17:48:00.715176981 +0000 UTC); removing then retrying: /data/caddy/locks/issue_cert_19.hachedigital.net.lock"}
{"level":"info","ts":1663008234.7226942,"logger":"tls.obtain","msg":"acquiring lock","identifier":"36.hachedigital.net"}
{"level":"info","ts":1663008234.883141,"logger":"admin","msg":"stopped previous server","address":"tcp/0.0.0.0:2019"}
{"level":"error","ts":1663008234.9339762,"logger":"tls","msg":"job failed","error":"35.hachedigital.net: obtaining certificate: failed storage check: open /data/caddy/rw_test_1024236072950214952: operation canceled - storage is probably misconfigured"}
{"level":"info","ts":1663008234.9889033,"msg":"[INFO][FileStorage:/data/caddy] Lock for 'issue_cert_36.hachedigital.net' is stale (created: 2022-09-12 18:43:18.179811002 +0000 UTC, last update: 2022-09-12 18:43:23.56032323 +0000 UTC); removing then retrying: /data/caddy/locks/issue_cert_36.hachedigital.net.lock"}
{"level":"info","ts":1663008235.100951,"msg":"[INFO][FileStorage:/data/caddy] Lock for 'issue_cert_19.hachedigital.net' is stale (created: 2022-09-12 17:47:33.549746506 +0000 UTC, last update: 2022-09-12 17:48:00.715176981 +0000 UTC); removing then retrying: /data/caddy/locks/issue_cert_19.hachedigital.net.lock"}
{"level":"info","ts":1663008235.1650612,"logger":"tls.obtain","msg":"acquiring lock","identifier":"40.hachedigital.net"}
{"level":"error","ts":1663008235.2101107,"logger":"tls","msg":"job failed","error":"18.hachedigital.net: obtaining certificate: unable to acquire lock 'issue_cert_18.hachedigital.net': accessing lock file: open /data/caddy/locks/issue_cert_18.hachedigital.net.lock: operation canceled"}
{"level":"info","ts":1663008235.616998,"msg":"[INFO][FileStorage:/data/caddy] Lock for 'issue_cert_36.hachedigital.net' is stale (created: 2022-09-12 18:43:18.179811002 +0000 UTC, last update: 2022-09-12 18:43:23.56032323 +0000 UTC); removing then retrying: /data/caddy/locks/issue_cert_36.hachedigital.net.lock"}
{"level":"debug","ts":1663008235.7005174,"logger":"tls.issuance.acme.acme_client","msg":"http request","method":"POST","url":"https://acme-v02.api.letsencrypt.org/acme/new-order","headers":{"Content-Type":["application/jose+json"],"User-Agent":["Caddy/2.5.1 CertMagic acmez (linux; amd64)"]},"response_headers":{"Boulder-Requester":["728093807"],"Cache-Control":["public, max-age=0, no-cache"],"Content-Length":["345"],"Content-Type":["application/json"],"Date":["Mon, 12 Sep 2022 18:43:55 GMT"],"Link":["<https://acme-v02.api.letsencrypt.org/directory>;rel=\"index\""],"Location":["https://acme-v02.api.letsencrypt.org/acme/order/728093807/124889457477"],"Replay-Nonce":["0102tDJmSzgQTWTnlilaK1uEG90cKPlIv1m1Ehcozlksdtg"],"Server":["nginx"],"Strict-Transport-Security":["max-age=604800"],"X-Frame-Options":["DENY"]},"status_code":201}
{"level":"info","ts":1663008235.7838118,"logger":"tls.obtain","msg":"acquiring lock","identifier":"17.hachedigital.net"}
{"level":"info","ts":1663008235.7959516,"logger":"tls.obtain","msg":"lock acquired","identifier":"40.hachedigital.net"}
{"level":"info","ts":1663008235.796126,"logger":"tls.obtain","msg":"releasing lock","identifier":"40.hachedigital.net"}
{"level":"error","ts":1663008235.8552158,"logger":"tls","msg":"job failed","error":"19.hachedigital.net: obtaining certificate: unable to acquire lock 'issue_cert_19.hachedigital.net': accessing lock file: open /data/caddy/locks/issue_cert_19.hachedigital.net.lock: operation canceled"}
{"level":"debug","ts":1663008235.8599265,"logger":"tls.issuance.acme.acme_client","msg":"http request","method":"POST","url":"https://acme-v02.api.letsencrypt.org/acme/authz-v3/152648768797","headers":{"Content-Type":["application/jose+json"],"User-Agent":["Caddy/2.5.1 CertMagic acmez (linux; amd64)"]},"response_headers":{"Boulder-Requester":["728093807"],"Cache-Control":["public, max-age=0, no-cache"],"Content-Length":["804"],"Content-Type":["application/json"],"Date":["Mon, 12 Sep 2022 18:43:55 GMT"],"Link":["<https://acme-v02.api.letsencrypt.org/directory>;rel=\"index\""],"Replay-Nonce":["0102mDLXOC7gSLgV83I1Q4X8ousPirrajafOBMzHeCjodq0"],"Server":["nginx"],"Strict-Transport-Security":["max-age=604800"],"X-Frame-Options":["DENY"]},"status_code":200}
{"level":"debug","ts":1663008235.860216,"logger":"tls.issuance.acme.acme_client","msg":"no solver configured","challenge_type":"dns-01"}
{"level":"info","ts":1663008235.860237,"logger":"tls.issuance.acme.acme_client","msg":"trying to solve challenge","identifier":"340.hachedigital.net","challenge_type":"http-01","ca":"https://acme-v02.api.letsencrypt.org/directory"}
{"level":"error","ts":1663008235.8899689,"logger":"tls","msg":"job failed","error":"17.hachedigital.net: obtaining certificate: unable to acquire lock 'issue_cert_17.hachedigital.net': decoding lockfile contents: read /data/caddy/locks/issue_cert_17.hachedigital.net.lock: operation canceled"}
{"level":"error","ts":1663008236.0835989,"logger":"tls","msg":"job failed","error":"40.hachedigital.net: obtaining certificate: context canceled"}
{"level":"info","ts":1663008236.1442323,"msg":"[INFO][FileStorage:/data/caddy] Lock for 'issue_cert_36.hachedigital.net' is stale (created: 2022-09-12 18:43:18.179811002 +0000 UTC, last update: 2022-09-12 18:43:23.56032323 +0000 UTC); removing then retrying: /data/caddy/locks/issue_cert_36.hachedigital.net.lock"}
{"level":"debug","ts":1663008236.224374,"logger":"tls.issuance.acme.acme_client","msg":"http request","method":"POST","url":"https://acme-v02.api.letsencrypt.org/acme/chall-v3/152648768797/6ArW8A","headers":{"Content-Type":["application/jose+json"],"User-Agent":["Caddy/2.5.1 CertMagic acmez (linux; amd64)"]},"response_headers":{"Boulder-Requester":["728093807"],"Cache-Control":["public, max-age=0, no-cache"],"Content-Length":["187"],"Content-Type":["application/json"],"Date":["Mon, 12 Sep 2022 18:43:56 GMT"],"Link":["<https://acme-v02.api.letsencrypt.org/directory>;rel=\"index\"","<https://acme-v02.api.letsencrypt.org/acme/authz-v3/152648768797>;rel=\"up\""],"Location":["https://acme-v02.api.letsencrypt.org/acme/chall-v3/152648768797/6ArW8A"],"Replay-Nonce":["0102MN4GoeKV2aIFPgv_J9vZrG37G5cSUuNDsDT2eHya82Q"],"Server":["nginx"],"Strict-Transport-Security":["max-age=604800"],"X-Frame-Options":["DENY"]},"status_code":200}
{"level":"debug","ts":1663008236.224481,"logger":"tls.issuance.acme.acme_client","msg":"challenge accepted","identifier":"340.hachedigital.net","challenge_type":"http-01"}
{"level":"info","ts":1663008236.3655853,"msg":"[INFO][FileStorage:/data/caddy] Lock for 'issue_cert_36.hachedigital.net' is stale (created: 2022-09-12 18:43:18.179811002 +0000 UTC, last update: 2022-09-12 18:43:23.56032323 +0000 UTC); removing then retrying: /data/caddy/locks/issue_cert_36.hachedigital.net.lock"}
{"level":"error","ts":1663008236.492895,"logger":"tls","msg":"job failed","error":"36.hachedigital.net: obtaining certificate: unable to acquire lock 'issue_cert_36.hachedigital.net': creating lock file: open /data/caddy/locks/issue_cert_36.hachedigital.net.lock: operation canceled"}
{"level":"debug","ts":1663008236.6333919,"logger":"tls.issuance.acme.acme_client","msg":"http request","method":"POST","url":"https://acme-v02.api.letsencrypt.org/acme/authz-v3/152648768797","headers":{"Content-Type":["application/jose+json"],"User-Agent":["Caddy/2.5.1 CertMagic acmez (linux; amd64)"]},"response_headers":{"Boulder-Requester":["728093807"],"Cache-Control":["public, max-age=0, no-cache"],"Content-Length":["759"],"Content-Type":["application/json"],"Date":["Mon, 12 Sep 2022 18:43:56 GMT"],"Link":["<https://acme-v02.api.letsencrypt.org/directory>;rel=\"index\""],"Replay-Nonce":["0101Vyb8BMRAZa1RCSpMO4_yT3dJUXDPPSsLVVvDBGFsw_Y"],"Server":["nginx"],"Strict-Transport-Security":["max-age=604800"],"X-Frame-Options":["DENY"]},"status_code":200}
{"level":"error","ts":1663008236.682713,"logger":"tls.issuance.acme.acme_client","msg":"challenge failed","identifier":"340.hachedigital.net","challenge_type":"http-01","problem":{"type":"urn:ietf:params:acme:error:dns","title":"","detail":"DNS problem: NXDOMAIN looking up A for 340.hachedigital.net - check that a DNS record exists for this domain; DNS problem: NXDOMAIN looking up AAAA for 340.hachedigital.net - check that a DNS record exists for this domain","instance":"","subproblems":[]}}
{"level":"error","ts":1663008236.6827931,"logger":"tls.issuance.acme.acme_client","msg":"validating authorization","identifier":"340.hachedigital.net","problem":{"type":"urn:ietf:params:acme:error:dns","title":"","detail":"DNS problem: NXDOMAIN looking up A for 340.hachedigital.net - check that a DNS record exists for this domain; DNS problem: NXDOMAIN looking up AAAA for 340.hachedigital.net - check that a DNS record exists for this domain","instance":"","subproblems":[]},"order":"https://acme-v02.api.letsencrypt.org/acme/order/728093807/124889457477","attempt":2,"max_attempts":3}
{"level":"error","ts":1663008236.6828368,"logger":"tls.obtain","msg":"could not get certificate from issuer","identifier":"340.hachedigital.net","issuer":"acme-v02.api.letsencrypt.org-directory","error":"HTTP 400 urn:ietf:params:acme:error:dns - DNS problem: NXDOMAIN looking up A for 340.hachedigital.net - check that a DNS record exists for this domain; DNS problem: NXDOMAIN looking up AAAA for 340.hachedigital.net - check that a DNS record exists for this domain"}
{"level":"debug","ts":1663008236.6828535,"logger":"tls.obtain","msg":"trying issuer 2/2","issuer":"acme.zerossl.com-v2-DV90"}
{"level":"warn","ts":1663008236.7172105,"logger":"tls.issuance.zerossl","msg":"missing email address for ZeroSSL; it is strongly recommended to set one for next time"}
{"level":"info","ts":1663008236.9760365,"logger":"tls.issuance.zerossl","msg":"generated EAB credentials","key_id":"2XrwaSU8cZmiU4Q2GTq41A"}
{"level":"debug","ts":1663008239.0198808,"logger":"tls.issuance.acme.acme_client","msg":"http request","method":"HEAD","url":"https://acme.zerossl.com/v2/DV90/newNonce","headers":{"User-Agent":["Caddy/2.5.1 CertMagic acmez (linux; amd64)"]},"response_headers":{"Access-Control-Allow-Origin":["*"],"Cache-Control":["max-age=-1"],"Content-Type":["application/octet-stream"],"Date":["Mon, 12 Sep 2022 18:43:59 GMT"],"Link":["<https://acme.zerossl.com/v2/DV90>;rel=\"index\""],"Replay-Nonce":["4M28zpviRY7hqXlIpt9XjEjRop8wENMHZWVfDIyrn_0"],"Server":["nginx"],"Strict-Transport-Security":["max-age=15552000"]},"status_code":200}
{"level":"debug","ts":1663008242.853677,"logger":"tls.issuance.acme.acme_client","msg":"http request","method":"HEAD","url":"https://acme.zerossl.com/v2/DV90/newNonce","headers":{"User-Agent":["Caddy/2.5.1 CertMagic acmez (linux; amd64)"]},"response_headers":{"Access-Control-Allow-Origin":["*"],"Cache-Control":["max-age=-1"],"Content-Type":["application/octet-stream"],"Date":["Mon, 12 Sep 2022 18:44:02 GMT"],"Link":["<https://acme.zerossl.com/v2/DV90>;rel=\"index\""],"Replay-Nonce":["v30QvDF_SY9x9UgW9JtE8cSq6hFnMkDWQt3EM4hzRzA"],"Server":["nginx"],"Strict-Transport-Security":["max-age=15552000"]},"status_code":200}
{"level":"debug","ts":1663008245.156858,"logger":"tls.issuance.acme.acme_client","msg":"http request","method":"POST","url":"https://acme.zerossl.com/v2/DV90/newAccount","headers":{"Content-Type":["application/jose+json"],"User-Agent":["Caddy/2.5.1 CertMagic acmez (linux; amd64)"]},"response_headers":{"Access-Control-Allow-Origin":["*"],"Cache-Control":["max-age=0, no-cache, no-store","max-age=-1"],"Content-Length":["579"],"Content-Type":["application/json"],"Date":["Mon, 12 Sep 2022 18:44:05 GMT"],"Link":["<https://acme.zerossl.com/v2/DV90>;rel=\"index\""],"Location":["https://acme.zerossl.com/v2/DV90/account/5G-eMblBeLq7XkqYRAI70A"],"Replay-Nonce":["ytr5JPvNj6BgYSxjZPgjkgm6u5THB1Uu3GNgLpRgdSE"],"Server":["nginx"],"Status":[""],"Strict-Transport-Security":["max-age=15552000"]},"status_code":201}
{"level":"info","ts":1663008245.6711755,"logger":"tls.issuance.acme","msg":"waiting on internal rate limiter","identifiers":["35.hachedigital.net"],"ca":"https://acme.zerossl.com/v2/DV90","account":""}
{"level":"info","ts":1663008245.671227,"logger":"tls.issuance.acme","msg":"done waiting on internal rate limiter","identifiers":["35.hachedigital.net"],"ca":"https://acme.zerossl.com/v2/DV90","account":""}
{"level":"debug","ts":1663008249.6260316,"logger":"tls.issuance.acme.acme_client","msg":"http request","method":"POST","url":"https://acme.zerossl.com/v2/DV90/newAccount","headers":{"Content-Type":["application/jose+json"],"User-Agent":["Caddy/2.5.1 CertMagic acmez (linux; amd64)"]},"response_headers":{"Access-Control-Allow-Origin":["*"],"Cache-Control":["max-age=0, no-cache, no-store","max-age=-1"],"Content-Length":["579"],"Content-Type":["application/json"],"Date":["Mon, 12 Sep 2022 18:44:09 GMT"],"Link":["<https://acme.zerossl.com/v2/DV90>;rel=\"index\""],"Location":["https://acme.zerossl.com/v2/DV90/account/2XrwaSU8cZmiU4Q2GTq41A"],"Replay-Nonce":["_M-IpXD2-lduKvslWJJCWjYqoWXysMJbnHmdserklhc"],"Server":["nginx"],"Status":[""],"Strict-Transport-Security":["max-age=15552000"]},"status_code":201}
{"level":"info","ts":1663008250.092957,"logger":"tls.issuance.acme","msg":"waiting on internal rate limiter","identifiers":["340.hachedigital.net"],"ca":"https://acme.zerossl.com/v2/DV90","account":""}
{"level":"info","ts":1663008250.0930078,"logger":"tls.issuance.acme","msg":"done waiting on internal rate limiter","identifiers":["340.hachedigital.net"],"ca":"https://acme.zerossl.com/v2/DV90","account":""}
{"level":"debug","ts":1663008257.0705953,"logger":"tls.issuance.acme.acme_client","msg":"http request","method":"POST","url":"https://acme.zerossl.com/v2/DV90/newOrder","headers":{"Content-Type":["application/jose+json"],"User-Agent":["Caddy/2.5.1 CertMagic acmez (linux; amd64)"]},"response_headers":{"Access-Control-Allow-Origin":["*"],"Cache-Control":["max-age=0, no-cache, no-store","max-age=-1"],"Content-Length":["282"],"Content-Type":["application/json"],"Date":["Mon, 12 Sep 2022 18:44:17 GMT"],"Location":["https://acme.zerossl.com/v2/DV90/order/UXVUTLjBVPBIDz_-IK-nqg"],"Replay-Nonce":["HZMz1FA_h8FNOuBiuEmcxprw4Ttek-uhcZZSMN5pS0w"],"Server":["nginx"],"Status":[""],"Strict-Transport-Security":["max-age=15552000"]},"status_code":201}
{"level":"debug","ts":1663008259.0150948,"logger":"tls.issuance.acme.acme_client","msg":"http request","method":"POST","url":"https://acme.zerossl.com/v2/DV90/newOrder","headers":{"Content-Type":["application/jose+json"],"User-Agent":["Caddy/2.5.1 CertMagic acmez (linux; amd64)"]},"response_headers":{"Access-Control-Allow-Origin":["*"],"Cache-Control":["max-age=0, no-cache, no-store","max-age=-1"],"Content-Length":["281"],"Content-Type":["application/json"],"Date":["Mon, 12 Sep 2022 18:44:19 GMT"],"Location":["https://acme.zerossl.com/v2/DV90/order/r543KBm6HrPY6o4OpRYDAw"],"Replay-Nonce":["GTt3ouLl7eCseetS3aXMXrZYFyCh9YlYjPaxmdI57xo"],"Server":["nginx"],"Status":[""],"Strict-Transport-Security":["max-age=15552000"]},"status_code":201}
{"level":"debug","ts":1663008263.815419,"logger":"tls.issuance.acme.acme_client","msg":"http request","method":"POST","url":"https://acme.zerossl.com/v2/DV90/authz/eoH2Mjx2fS76haE3Kke2Hw","headers":{"Content-Type":["application/jose+json"],"User-Agent":["Caddy/2.5.1 CertMagic acmez (linux; amd64)"]},"response_headers":{"Access-Control-Allow-Origin":["*"],"Cache-Control":["max-age=-1"],"Content-Length":["450"],"Content-Type":["application/json"],"Date":["Mon, 12 Sep 2022 18:44:23 GMT"],"Link":["<https://acme.zerossl.com/v2/DV90>;rel=\"index\""],"Replay-Nonce":["ShtIdXcuDKo_g0axZCoUt63QEbijbPTuA0nCn3vEuRc"],"Retry-After":["5"],"Server":["nginx"],"Strict-Transport-Security":["max-age=15552000"]},"status_code":200}
{"level":"debug","ts":1663008263.8156385,"logger":"tls.issuance.acme.acme_client","msg":"no solver configured","challenge_type":"dns-01"}
{"level":"info","ts":1663008263.8161776,"logger":"tls.issuance.acme.acme_client","msg":"trying to solve challenge","identifier":"340.hachedigital.net","challenge_type":"http-01","ca":"https://acme.zerossl.com/v2/DV90"}
{"level":"info","ts":1663008265.4444993,"msg":"[ERROR] Keeping lock file fresh: truncate /data/caddy/locks/issue_cert_35.hachedigital.net.lock: operation canceled - terminating lock maintenance (lockfile: /data/caddy/locks/issue_cert_35.hachedigital.net.lock)"}
{"level":"info","ts":1663008266.396085,"msg":"[ERROR] Keeping lock file fresh: truncate /data/caddy/locks/issue_cert_40.hachedigital.net.lock: operation canceled - terminating lock maintenance (lockfile: /data/caddy/locks/issue_cert_40.hachedigital.net.lock)"}
{"level":"info","ts":1663008268.776918,"msg":"[ERROR] Keeping lock file fresh: truncate /data/caddy/locks/issue_cert_340.hachedigital.net.lock: operation canceled - terminating lock maintenance (lockfile: /data/caddy/locks/issue_cert_340.hachedigital.net.lock)"}
{"level":"debug","ts":1663008270.411104,"logger":"tls.issuance.acme.acme_client","msg":"http request","method":"POST","url":"https://acme.zerossl.com/v2/DV90/chall/BMFW78JoOrXH7VayKZHsUQ","headers":{"Content-Type":["application/jose+json"],"User-Agent":["Caddy/2.5.1 CertMagic acmez (linux; amd64)"]},"response_headers":{"Access-Control-Allow-Origin":["*"],"Cache-Control":["max-age=-1"],"Content-Length":["164"],"Content-Type":["application/json"],"Date":["Mon, 12 Sep 2022 18:44:30 GMT"],"Link":["<https://acme.zerossl.com/v2/DV90>;rel=\"index\"","<https://acme.zerossl.com/v2/DV90/authz/eoH2Mjx2fS76haE3Kke2Hw>;rel=\"up\""],"Replay-Nonce":["4EYNXwoj9uy_jYIUEK-bC1PxSSK_2QK4yR9Z1nU3qP0"],"Retry-After":["10"],"Server":["nginx"],"Strict-Transport-Security":["max-age=15552000"]},"status_code":200}
{"level":"debug","ts":1663008270.411219,"logger":"tls.issuance.acme.acme_client","msg":"challenge accepted","identifier":"340.hachedigital.net","challenge_type":"http-01"}
{"level":"debug","ts":1663008271.2123134,"logger":"tls.issuance.acme.acme_client","msg":"http request","method":"POST","url":"https://acme.zerossl.com/v2/DV90/authz/bmXvktiLyQrmXerYGWRxjw","headers":{"Content-Type":["application/jose+json"],"User-Agent":["Caddy/2.5.1 CertMagic acmez (linux; amd64)"]},"response_headers":{"Access-Control-Allow-Origin":["*"],"Cache-Control":["max-age=-1"],"Content-Length":["449"],"Content-Type":["application/json"],"Date":["Mon, 12 Sep 2022 18:44:31 GMT"],"Link":["<https://acme.zerossl.com/v2/DV90>;rel=\"index\""],"Replay-Nonce":["R1MiNjXnJzjzvOxVVFRZGNsetEgzRoR--Uij_A_j2Yo"],"Retry-After":["5"],"Server":["nginx"],"Strict-Transport-Security":["max-age=15552000"]},"status_code":200}
{"level":"debug","ts":1663008271.2126725,"logger":"tls.issuance.acme.acme_client","msg":"no solver configured","challenge_type":"dns-01"}
{"level":"info","ts":1663008271.2127964,"logger":"tls.issuance.acme.acme_client","msg":"trying to solve challenge","identifier":"35.hachedigital.net","challenge_type":"http-01","ca":"https://acme.zerossl.com/v2/DV90"}

And I add my current config:

{
  "admin": {
    "listen": "0.0.0.0:2019"
  },
  "apps": {
    "http": {
      "http_port": 80,
      "https_port": 443,
      "servers": {
        "srv0": {
          "listen": [
            "0.0.0.0:443"
          ],
          "routes": [
            {
              "handle": [
                {
                  "handler": "subroute",
                  "routes": [
                    {
                      "handle": [
                        {
                          "handler": "reverse_proxy",
                          "upstreams": [
                            {
                              "dial": "frontend-service:3000"
                            }
                          ]
                        }
                      ]
                    }
                  ]
                }
              ],
              "match": [
                {
                  "host": [
                    "hachedigital.net",
                    "www.hachedigital.net",
                    "1.hachedigital.net",
                    "2.hachedigital.net",
                    "23hachedigital.net",
                    "3.hachedigital.net",
                    "4.hachedigital.net",
                    "5.hachedigital.net",
                    "6.hachedigital.net",
                    "7.hachedigital.net",
                    "8.hachedigital.net",
                    "9.hachedigital.net",
                    "10.hachedigital.net",
                    "13.hachedigital.net",
                    "15.hachedigital.net",
                    "17.hachedigital.net",
                    "18.hachedigital.net",
                    "19.hachedigital.net",
                    "32.hachedigital.net",
                    "35.hachedigital.net",
                    "36.hachedigital.net",
                    "38.hachedigital.net",
                    "340.hachedigital.net"
                  ]
                }
              ],
              "terminal": true
            }
          ]
        }
      }
    }
  },
  "logging": {
    "logs": {
      "default": {
        "level": "DEBUG"
      }
    }
  }

I just realize that we are receiving this error in the log:

storage is probably misconfigured"

Yes. Config reloads are lightweight, but they clean up after themselves, so pending cert operations will be canceled so as to not leak resources.

All the “operation canceled” errors are likely due to your config reloads interrupting storage accesses. Storage seems to be pretty slow. You said it’s over a network?

Here ya go: admin/remote part of the JSON config:

Will you control those domain names? Or are they owned and managed by your customers?

Hello again,

Yes, I’m using a volume in caddy deployment mounteb by google cloud storage (it is like AWS S3 service). It is the only way we have found to create persistent storage using Kubernetes in Google cloud. We cannot use normal disk because the disk driver used by GCP not allow multiattach, so when the caddy pod is restarted, we lost the volume… This kind of storage we are using right now is something like a NFS, using Google Cloud Storage buckets.

New test I just try. I have clean all the data from bucket and I have restarted the pod. This is like a clean installation. Then I have modifed the autosave.json with this data:

{
	"admin": {
		"listen": "0.0.0.0:2019"
	},
	"apps": {
		"http": {
			"http_port": 80,
			"https_port": 443,
			"servers": {
				"srv0": {
					"listen": ["0.0.0.0:443"],
					"routes": [{
						"handle": [{
							"handler": "subroute",
							"routes": [{
								"handle": [{
									"handler": "reverse_proxy",
									"upstreams": [{
										"dial": "frontend-service:3000"
									}]
								}]
							}]
						}],
						"match": [{
							"host": ["hachedigital.net", "1.hachedigital.net", "2.hachedigital.net", "3.hachedigital.net", "4.hachedigital.net", "5.hachedigital.net", "6.hachedigital.net", "7.hachedigital.net", "8.hachedigital.net", "9.hachedigital.net", "10.hachedigital.net"]
						}],
						"terminal": true
					}]
				}
			}
		}
	},
	"logging": {
		"logs": {
			"default": {
				"level": "DEBUG"
			}
		}
	}
}

Then I execute the next curl. The first 3 times I got the same error I was receiving executing the curl one by one.

curl -XPOST http://api-caddy-dev.topdoctors.es:2019/load -H "Content-Type: application/json" -d@autosave.json

{"error":"loading new config: http app module: start: finalizing automatic HTTPS: managing certificates for [7.hachedigital.net 9.hachedigital.net 3.hachedigital.net 6.hachedigital.net 2.hachedigital.net 4.hachedigital.net  1.hachedigital.net hachedigital.com 5.hachedigital.net 8.hachedigital.net 10.domainmine.net 10.hachedigital.com]: automate: manage [7.hachedigital.net 9.hachedigital.net 3.hachedigital.net 6.hachedigital.net  2.hachedigital.net 4.hachedigital.net hachedigital.com www.hachedigital.net hachedigital.net 5.hachedigital.net 8.hachedigital.net 10.hachedigital.net hachedigital.net 10.hachedigital.com]: 4.hachedigital.net: caching certificate: read /data/caddy/certificates/acme-v02.api.letsencrypt.org-directory/4.hachedigital.net/4.hachedigital.net.crt: operation canceled"}

And suddenly the fourth time the command works, and then I checked the logs in order to verify the certificate creation and Caddy only create some of then as you can see in the logs (I have had to cut the log input because I was overcoming the body limit.

["1.hachedigital.net","2.hachedigital.net","5.hachedigital.net","10.hachedigital.net","hachedigital.net","9.hachedigital.net","4.hachedigital.net","6.hachedigital.net","3.hachedigital.net","7.hachedigital.net","8.hachedigital.net"]}
{"level":"info","ts":1663056615.627493,"logger":"tls.cache.maintenance","msg":"started background certificate maintenance","cache":"0xc0004d67e0"}
{"level":"error","ts":1663056616.3435411,"logger":"tls","msg":"job failed","error":"2.hachedigital.net: obtaining certificate: failed storage check: close /data/caddy/rw_test_2345786471577943998: operation canceled - storage is probably misconfigured"}
{"level":"error","ts":1663056616.6922307,"logger":"tls","msg":"deleting test key from storage","key":"rw_test_7217349486913215582","error":"open /data/caddy/rw_test_7217349486913215582: operation canceled"}
{"level":"error","ts":1663056616.6923597,"logger":"tls","msg":"job failed","error":"1.hachedigital.net: obtaining certificate: failed storage check: open /data/caddy/rw_test_7217349486913215582: operation canceled - storage is probably misconfigured"}
{"level":"info","ts":1663056617.2977443,"logger":"tls.obtain","msg":"acquiring lock","identifier":"5.hachedigital.net"}
{"level":"info","ts":1663056617.6429179,"logger":"tls.obtain","msg":"acquiring lock","identifier":"10.hachedigital.net"}
{"level":"info","ts":1663056617.8344915,"logger":"tls.obtain","msg":"lock acquired","identifier":"5.hachedigital.net"}
{"level":"info","ts":1663056618.1294773,"logger":"tls.obtain","msg":"lock acquired","identifier":"10.hachedigital.net"}
{"level":"info","ts":1663056618.5684779,"logger":"tls.obtain","msg":"acquiring lock","identifier":"hachedigital.net"}
{"level":"debug","ts":1663056618.5947864,"logger":"tls.obtain","msg":"trying issuer 1/2","issuer":"acme-v02.api.letsencrypt.org-directory"}
{"level":"error","ts":1663056618.8566482,"logger":"tls","msg":"job failed","error":"4.hachedigital.net: obtaining certificate: failed storage check: close /data/caddy/rw_test_9055767639255195990: operation canceled - storage is probably misconfigured"}
{"level":"info","ts":1663056619.0409782,"logger":"tls.obtain","msg":"lock acquired","identifier":"hachedigital.net"}
{"level":"info","ts":1663056619.103426,"logger":"tls.obtain","msg":"acquiring lock","identifier":"9.hachedigital.net"}
{"level":"debug","ts":1663056619.2149763,"logger":"tls.obtain","msg":"trying issuer 1/2","issuer":"acme-v02.api.letsencrypt.org-directory"}
{"level":"info","ts":1663056619.673278,"logger":"tls.obtain","msg":"lock acquired","identifier":"9.hachedigital.net"}
{"level":"debug","ts":1663056619.9218223,"logger":"tls.obtain","msg":"trying issuer 1/2","issuer":"acme-v02.api.letsencrypt.org-directory"}
{"level":"info","ts":1663056620.1899335,"logger":"tls.obtain","msg":"acquiring lock","identifier":"6.hachedigital.net"}
{"level":"debug","ts":1663056620.2193222,"logger":"http.stdlib","msg":"http: Accept error: accept tcp [::]:443: i/o timeout; retrying in 5ms"}
{"level":"info","ts":1663056620.2216654,"logger":"tls.cache.maintenance","msg":"stopped background certificate maintenance","cache":"0xc0004d6230"}
{"level":"debug","ts":1663056620.2484803,"logger":"tls.issuance.acme.acme_client","msg":"http request","method":"GET","url":"https://acme-v02.api.letsencrypt.org/directory","headers":{"User-Agent":["Caddy/2.5.1 CertMagic acmez (linux; amd64)"]},"response_headers":{"Cache-Control":["public, max-age=0, no-cache"],"Content-Length":["672"],"Content-Type":["application/json"],"Date":["Tue, 13 Sep 2022 08:10:20 GMT"],"Server":["nginx"],"Strict-Transport-Security":["max-age=604800"],"X-Frame-Options":["DENY"]},"status_code":200}
{"level":"debug","ts":1663056620.3751333,"logger":"tls.issuance.acme.acme_client","msg":"http request","method":"HEAD","url":"https://acme-v02.api.letsencrypt.org/acme/new-nonce","headers":{"User-Agent":["Caddy/2.5.1 CertMagic acmez (linux; amd64)"]},"response_headers":{"Cache-Control":["public, max-age=0, no-cache"],"Date":["Tue, 13 Sep 2022 08:10:20 GMT"],"Link":["<https://acme-v02.api.letsencrypt.org/directory>;rel=\"index\""],"Replay-Nonce":["0102h1WfAy97R1KPqdpQAut2-3KE8KIFmxezxK-GVefyubA"],"Server":["nginx"],"Strict-Transport-Security":["max-age=604800"],"X-Frame-Options":["DENY"]},"status_code":200}
{"level":"debug","ts":1663056620.5600238,"logger":"tls.issuance.acme.acme_client","msg":"http request","method":"POST","url":"https://acme-v02.api.letsencrypt.org/acme/new-acct","headers":{"Content-Type":["application/jose+json"],"User-Agent":["Caddy/2.5.1 CertMagic acmez (linux; amd64)"]},"response_headers":{"Boulder-Requester":["729455207"],"Cache-Control":["public, max-age=0, no-cache"],"Content-Length":["268"],"Content-Type":["application/json"],"Date":["Tue, 13 Sep 2022 08:10:20 GMT"],"Link":["<https://acme-v02.api.letsencrypt.org/directory>;rel=\"index\"","<https://letsencrypt.org/documents/LE-SA-v1.2-November-15-2017-w-v1.3-notice.pdf>;rel=\"terms-of-service\""],"Location":["https://acme-v02.api.letsencrypt.org/acme/acct/729455207"],"Replay-Nonce":["0101itbY5mhEgvta8oesfnVolAT9TwBG9_X65J-QNl3asRI"],"Server":["nginx"],"Strict-Transport-Security":["max-age=604800"],"X-Frame-Options":["DENY"]},"status_code":201}
{"level":"debug","ts":1663056620.6636422,"logger":"tls.issuance.acme.acme_client","msg":"http request","method":"HEAD","url":"https://acme-v02.api.letsencrypt.org/acme/new-nonce","headers":{"User-Agent":["Caddy/2.5.1 CertMagic acmez (linux; amd64)"]},"response_headers":{"Cache-Control":["public, max-age=0, no-cache"],"Date":["Tue, 13 Sep 2022 08:10:20 GMT"],"Link":["<https://acme-v02.api.letsencrypt.org/directory>;rel=\"index\""],"Replay-Nonce":["0101hWkwPZVM3ldJoLQbnAubNiEzDWRt4ZF-Bgb8-FDpAAs"],"Server":["nginx"],"Strict-Transport-Security":["max-age=604800"],"X-Frame-Options":["DENY"]},"status_code":200}
{"level":"info","ts":1663056620.727509,"msg":"autosaved config (load with --resume flag)","file":"/data/config/caddy/autosave.json"}
{"level":"info","ts":1663056620.7276533,"logger":"admin.api","msg":"load complete"}
{"level":"info","ts":1663056620.796221,"logger":"tls.obtain","msg":"lock acquired","identifier":"6.hachedigital.net"}
{"level":"info","ts":1663056620.8388932,"logger":"tls.obtain","msg":"acquiring lock","identifier":"3.hachedigital.net"}
{"level":"error","ts":1663056620.8393214,"logger":"tls.obtain","msg":"will retry","error":"loading existing private key for reuse with issuer acme-v02.api.letsencrypt.org-directory: open /data/caddy/certificates/acme-v02.api.letsencrypt.org-directory/9.hachedigital.net/9.hachedigital.net.key: operation canceled","attempt":1,"retrying_in":60,"elapsed":1.165994383,"max_duration":2592000}
{"level":"debug","ts":1663056620.8769279,"logger":"tls.issuance.acme.acme_client","msg":"http request","method":"POST","url":"https://acme-v02.api.letsencrypt.org/acme/new-acct","headers":{"Content-Type":["application/jose+json"],"User-Agent":["Caddy/2.5.1 CertMagic acmez (linux; amd64)"]},"response_headers":{"Boulder-Requester":["729455227"],"Cache-Control":["public, max-age=0, no-cache"],"Content-Length":["268"],"Content-Type":["application/json"],"Date":["Tue, 13 Sep 2022 08:10:20 GMT"],"Link":["<https://acme-v02.api.letsencrypt.org/directory>;rel=\"index\"","<https://letsencrypt.org/documents/LE-SA-v1.2-November-15-2017-w-v1.3-notice.pdf>;rel=\"terms-of-service\""],"Location":["https://acme-v02.api.letsencrypt.org/acme/acct/729455227"],"Replay-Nonce":["0101eYaSE2BachFi5ZA-bamH0LWDghMMmT0E0GTCyYyZQ8k"],"Server":["nginx"],"Strict-Transport-Security":["max-age=604800"],"X-Frame-Options":["DENY"]},"status_code":201}
{"level":"info","ts":1663056620.9223764,"logger":"admin","msg":"stopped previous server","address":"tcp/0.0.0.0:2019"}
{"level":"info","ts":1663056621.1954315,"logger":"tls.obtain","msg":"acquiring lock","identifier":"7.hachedigital.net"}
{"level":"debug","ts":1663056621.2883375,"logger":"tls.issuance.acme.acme_client","msg":"http request","method":"HEAD","url":"https://acme-v02.api.letsencrypt.org/acme/new-nonce","headers":{"User-Agent":["Caddy/2.5.1 CertMagic acmez (linux; amd64)"]},"response_headers":{"Cache-Control":["public, max-age=0, no-cache"],"Date":["Tue, 13 Sep 2022 08:10:21 GMT"],"Link":["<https://acme-v02.api.letsencrypt.org/directory>;rel=\"index\""],"Replay-Nonce":["0102jM3aV2WL-WDkAEK7AjKvEepa5w42hSFlqkE7KpJsh7Y"],"Server":["nginx"],"Strict-Transport-Security":["max-age=604800"],"X-Frame-Options":["DENY"]},"status_code":200}
{"level":"info","ts":1663056621.3325696,"logger":"tls.obtain","msg":"lock acquired","identifier":"3.hachedigital.net"}
{"level":"debug","ts":1663056621.5219607,"logger":"tls.issuance.acme.acme_client","msg":"http request","method":"POST","url":"https://acme-v02.api.letsencrypt.org/acme/new-acct","headers":{"Content-Type":["application/jose+json"],"User-Agent":["Caddy/2.5.1 CertMagic acmez (linux; amd64)"]},"response_headers":{"Boulder-Requester":["729455237"],"Cache-Control":["public, max-age=0, no-cache"],"Content-Length":["268"],"Content-Type":["application/json"],"Date":["Tue, 13 Sep 2022 08:10:21 GMT"],"Link":["<https://acme-v02.api.letsencrypt.org/directory>;rel=\"index\"","<https://letsencrypt.org/documents/LE-SA-v1.2-November-15-2017-w-v1.3-notice.pdf>;rel=\"terms-of-service\""],"Location":["https://acme-v02.api.letsencrypt.org/acme/acct/729455237"],"Replay-Nonce":["0102db3ZzDcEn2_Uukvo7kP-k77TVMfpvIuFJJBFND9D5E8"],"Server":["nginx"],"Strict-Transport-Security":["max-age=604800"],"X-Frame-Options":["DENY"]},"status_code":201}
{"level":"debug","ts":1663056621.6798892,"logger":"tls.obtain","msg":"trying issuer 1/2","issuer":"acme-v02.api.letsencrypt.org-directory"}
{"level":"info","ts":1663056621.6930716,"logger":"tls.obtain","msg":"lock acquired","identifier":"7.hachedigital.net"}
{"level":"info","ts":1663056622.2015085,"logger":"tls.obtain","msg":"acquiring lock","identifier":"8.hachedigital.net"}
{"level":"error","ts":1663056622.244327,"logger":"tls.obtain","msg":"will retry","error":"loading existing private key for reuse with issuer acme.zerossl.com-v2-DV90: open /data/caddy/certificates/acme.zerossl.com-v2-dv90/7.hachedigital.net/7.hachedigital.net.key: operation canceled","attempt":1,"retrying_in":60,"elapsed":0.549400612,"max_duration":2592000}
{"level":"debug","ts":1663056622.5399861,"logger":"tls.issuance.acme.acme_client","msg":"http request","method":"HEAD","url":"https://acme-v02.api.letsencrypt.org/acme/new-nonce","headers":{"User-Agent":["Caddy/2.5.1 CertMagic acmez (linux; amd64)"]},"response_headers":{"Cache-Control":["public, max-age=0, no-cache"],"Date":["Tue, 13 Sep 2022 08:10:22 GMT"],"Link":["<https://acme-v02.api.letsencrypt.org/directory>;rel=\"index\""],"Replay-Nonce":["0001LEGsx8MVC4Pp53lCd8OMtmZ6Y0InJ7wa31U1IUefwoY"],"Server":["nginx"],"Strict-Transport-Security":["max-age=604800"],"X-Frame-Options":["DENY"]},"status_code":200}
{"level":"debug","ts":1663056622.6956375,"logger":"tls.issuance.acme.acme_client","msg":"http request","method":"POST","url":"https://acme-v02.api.letsencrypt.org/acme/new-acct","headers":{"Content-Type":["application/jose+json"],"User-Agent":["Caddy/2.5.1 CertMagic acmez (linux; amd64)"]},"response_headers":{"Boulder-Requester":["729455287"],"Cache-Control":["public, max-age=0, no-cache"],"Content-Length":["268"],"Content-Type":["application/json"],"Date":["Tue, 13 Sep 2022 08:10:22 GMT"],"Link":["<https://acme-v02.api.letsencrypt.org/directory>;rel=\"index\"","<https://letsencrypt.org/documents/LE-SA-v1.2-November-15-2017-w-v1.3-notice.pdf>;rel=\"terms-of-service\""],"Location":["https://acme-v02.api.letsencrypt.org/acme/acct/729455287"],"Replay-Nonce":["0001PBXoBY9ml35MxBSoszu1aeefanQuybBeSSrKl5M5wgE"],"Server":["nginx"],"Strict-Transport-Security":["max-age=604800"],"X-Frame-Options":["DENY"]},"status_code":201}
{"level":"error","ts":1663056622.737557,"logger":"tls.obtain","msg":"could not get certificate from issuer","identifier":"5.hachedigital.net","issuer":"acme-v02.api.letsencrypt.org-directory","error":"could not save account []: open /data/caddy/acme/acme-v02.api.letsencrypt.org-directory/users/default/default.json: operation canceled"}
{"level":"warn","ts":1663056622.814721,"logger":"tls.issuance.zerossl","msg":"missing email address for ZeroSSL; it is strongly recommended to set one for next time"}
{"level":"info","ts":1663056623.2947507,"logger":"tls.issuance.acme","msg":"done waiting on internal rate limiter","identifiers":["10.hachedigital.net"],"ca":"https://acme-v02.api.letsencrypt.org/directory","account":""}
{"level":"info","ts":1663056623.3754025,"logger":"tls.issuance.zerossl","msg":"generated EAB credentials","key_id":"Z4zOKNzcs9hPmQWIw0rd0Q"}
{"level":"info","ts":1663056623.4918823,"logger":"tls.issuance.acme","msg":"waiting on internal rate limiter","identifiers":["hachedigital.net"],"ca":"https://acme-v02.api.letsencrypt.org/directory","account":""}
{"level":"info","ts":1663056623.4919376,"logger":"tls.issuance.acme","msg":"done waiting on internal rate limiter","identifiers":["hachedigital.net"],"ca":"https://acme-v02.api.letsencrypt.org/directory","account":""}
{"level":"error","ts":1663056623.5044358,"logger":"tls.obtain","msg":"could not get certificate from issuer","identifier":"10.hachedigital.net","issuer":"acme-v02.api.letsencrypt.org-directory","error":"HTTP 429 urn:ietf:params:acme:error:rateLimited - Error creating new order :: too many certificates (5) already issued for this exact set of domains in the last 168 hours: 10.hachedigital.net, retry after 2022-09-13T21:01:46Z: see https://letsencrypt.org/docs/duplicate-certificate-limit/"}
{"level":"debug","ts":1663056623.5044906,"logger":"tls.obtain","msg":"trying issuer 2/2","issuer":"acme.zerossl.com-v2-DV90"}
{"level":"warn","ts":1663056623.5555704,"logger":"tls.issuance.zerossl","msg":"missing email address for ZeroSSL; it is strongly recommended to set one for next time"}
{"level":"debug","ts":1663056623.8127441,"logger":"tls.issuance.acme.acme_client","msg":"http request","method":"POST","url":"https://acme-v02.api.letsencrypt.org/acme/new-order","headers":{"Content-Type":["application/jose+json"],"User-Agent":["Caddy/2.5.1 CertMagic acmez (linux; amd64)"]},"response_headers":{"Boulder-Requester":["729455237"],"Cache-Control":["public, max-age=0, no-cache"],"Content-Length":["341"],"Content-Type":["application/json"],"Date":["Tue, 13 Sep 2022 08:10:23 GMT"],"Link":["<https://acme-v02.api.letsencrypt.org/directory>;rel=\"index\""],"Location":["https://acme-v02.api.letsencrypt.org/acme/order/729455237/125056837367"],"Replay-Nonce":["0102ntiaSlM5G2_cB-9u_7VEYXAk9E6xdGN5b-9yXSww0ZE"],"Server":["nginx"],"Strict-Transport-Security":["max-age=604800"],"X-Frame-Options":["DENY"]},"status_code":201}
{"level":"info","ts":1663056623.852528,"logger":"tls.issuance.zerossl","msg":"generated EAB credentials","key_id":"jU133Z4Y_G2LaTfva2CekQ"}
{"level":"debug","ts":1663056623.9741,"logger":"tls.issuance.acme.acme_client","msg":"http request","method":"POST","url":"https://acme-v02.api.letsencrypt.org/acme/authz-v3/152854636887","headers":{"Content-Type":["application/jose+json"],"User-Agent":["Caddy/2.5.1 CertMagic acmez (linux; amd64)"]},"response_headers":{"Boulder-Requester":["729455237"],"Cache-Control":["public, max-age=0, no-cache"],"Content-Length":["800"],"Content-Type":["application/json"],"Date":["Tue, 13 Sep 2022 08:10:23 GMT"],"Link":["<https://acme-v02.api.letsencrypt.org/directory>;rel=\"index\""],"Replay-Nonce":["0101JY_IYUObRB3o1s55kpDblIqTwYDaXFOm1YcAIs4r4es"],"Server":["nginx"],"Strict-Transport-Security":["max-age=604800"],"X-Frame-Options":["DENY"]},"status_code":200}
{"level":"debug","ts":1663056623.9748473,"logger":"tls.issuance.acme.acme_client","msg":"no solver configured","challenge_type":"dns-01"}
{"level":"info","ts":1663056623.9750416,"logger":"tls.issuance.acme.acme_client","msg":"trying to solve challenge","identifier":"hachedigital.net","challenge_type":"http-01","ca":"https://acme-v02.api.letsencrypt.org/directory"}
{"level":"info","ts":1663056624.151617,"logger":"tls.issuance.acme","msg":"waiting on internal rate limiter","identifiers":["6.hachedigital.net"],"ca":"https://acme-v02.api.letsencrypt.org/directory","account":""}
{"level":"info","ts":1663056624.1516657,"logger":"tls.issuance.acme","msg":"done waiting on internal rate limiter","identifiers":["6.hachedigital.net"],"ca":"https://acme-v02.api.letsencrypt.org/directory","account":""}
{"level":"info","ts":1663056624.189281,"logger":"tls.issuance.acme","msg":"waiting on internal rate limiter","identifiers":["3.hachedigital.net"],"ca":"https://acme-v02.api.letsencrypt.org/directory","account":""}
{"level":"info","ts":1663056624.1893454,"logger":"tls.issuance.acme","msg":"done waiting on internal rate limiter","identifiers":["3.hachedigital.net"],"ca":"https://acme-v02.api.letsencrypt.org/directory","account":""}
{"level":"info","ts":1663056624.2153091,"logger":"tls.issuance.acme","msg":"waiting on internal rate limiter","identifiers":["8.hachedigital.net"],"ca":"https://acme-v02.api.letsencrypt.org/directory","account":""}
{"level":"info","ts":1663056624.2153666,"logger":"tls.issuance.acme","msg":"done waiting on internal rate limiter","identifiers":["8.hachedigital.net"],"ca":"https://acme-v02.api.letsencrypt.org/directory","account":""}
{"level":"debug","ts":1663056624.3291736,"logger":"tls.issuance.acme.acme_client","msg":"http request","method":"POST","url":"https://acme-v02.api.letsencrypt.org/acme/new-order","headers":{"Content-Type":["application/jose+json"],"User-Agent":["Caddy/2.5.1 CertMagic acmez (linux; amd64)"]},"response_headers":{"Boulder-Requester":["729455287"],"Cache-Control":["public, max-age=0, no-cache"],"Content-Length":["343"],"Content-Type":["application/json"],"Date":["Tue, 13 Sep 2022 08:10:24 GMT"],"Link":["<https://acme-v02.api.letsencrypt.org/directory>;rel=\"index\""],"Location":["https://acme-v02.api.letsencrypt.org/acme/order/729455287/125056839507"],"Replay-Nonce":["0002JA1LTwk8itnMYkFvdGrPquzw7sbnYpzwsJ37I86iq24"],"Server":["nginx"],"Strict-Transport-Security":["max-age=604800"],"X-Frame-Options":["DENY"]},"status_code":201}
{"level":"debug","ts":1663056624.4745715,"logger":"tls.issuance.acme.acme_client","msg":"http request","method":"POST","url":"https://acme-v02.api.letsencrypt.org/acme/authz-v3/152854639307","headers":{"Content-Type":["application/jose+json"],"User-Agent":["Caddy/2.5.1 CertMagic acmez (linux; amd64)"]},"response_headers":{"Boulder-Requester":["729455287"],"Cache-Control":["public, max-age=0, no-cache"],"Content-Length":["802"],"Content-Type":["application/json"],"Date":["Tue, 13 Sep 2022 08:10:24 GMT"],"Link":["<https://acme-v02.api.letsencrypt.org/directory>;rel=\"index\""],"Replay-Nonce":["0002tKVQFD2Gfu180A364Cy9v5BvpHWqvcvqJx_-brEAFXc"],"Server":["nginx"],"Strict-Transport-Security":["max-age=604800"],"X-Frame-Options":["DENY"]},"status_code":200}
{"level":"debug","ts":1663056624.4749,"logger":"tls.issuance.acme.acme_client","msg":"no solver configured","challenge_type":"dns-01"}
{"level":"info","ts":1663056624.4749258,"logger":"tls.issuance.acme.acme_client","msg":"trying to solve challenge","identifier":"6.hachedigital.net","challenge_type":"http-01","ca":"https://acme-v02.api.letsencrypt.org/directory"}
{"level":"debug","ts":1663056624.768603,"logger":"tls.issuance.acme.acme_client","msg":"http request","method":"POST","url":"https://acme-v02.api.letsencrypt.org/acme/chall-v3/152854636887/q60tmQ","headers":{"Content-Type":["application/jose+json"],"User-Agent":["Caddy/2.5.1 CertMagic acmez (linux; amd64)"]},"response_headers":{"Boulder-Requester":["729455237"],"Cache-Control":["public, max-age=0, no-cache"],"Content-Length":["187"],"Content-Type":["application/json"],"Date":["Tue, 13 Sep 2022 08:10:24 GMT"],"Link":["<https://acme-v02.api.letsencrypt.org/directory>;rel=\"index\"","<https://acme-v02.api.letsencrypt.org/acme/authz-v3/152854636887>;rel=\"up\""],"Location":["https://acme-v02.api.letsencrypt.org/acme/chall-v3/152854636887/q60tmQ"],"Replay-Nonce":["0101oCi7YfgoyWiWkQgDUuWUxemDBtYe6AUMN85rZLaBxiA"],"Server":["nginx"],"Strict-Transport-Security":["max-age=604800"],"X-Frame-Options":["DENY"]},"status_code":200}
{"level":"debug","ts":1663056624.7687318,"logger":"tls.issuance.acme.acme_client","msg":"challenge accepted","identifier":"hachedigital.net","challenge_type":"http-01"}
{"level":"info","ts":1663056624.9096093,"logger":"tls.issuance.acme","msg":"served key authentication","identifier":"hachedigital.net","challenge":"http-01","remote":"10.1.2.209:61396","distributed":false}
{"level":"info","ts":1663056625.0354965,"logger":"tls.issuance.acme","msg":"served key authentication","identifier":"hachedigital.net","challenge":"http-01","remote":"10.20.11.1:4547","distributed":false}
{"level":"info","ts":1663056625.0690525,"logger":"tls.issuance.acme","msg":"served key authentication","identifier":"hachedigital.net","challenge":"http-01","remote":"10.1.2.207:4278","distributed":false}
{"level":"debug","ts":1663056625.074502,"logger":"tls.issuance.acme.acme_client","msg":"http request","method":"POST","url":"https://acme-v02.api.letsencrypt.org/acme/chall-v3/152854639307/M9TW8Q","headers":{"Content-Type":["application/jose+json"],"User-Agent":["Caddy/2.5.1 CertMagic acmez (linux; amd64)"]},"response_headers":{"Boulder-Requester":["729455287"],"Cache-Control":["public, max-age=0, no-cache"],"Content-Length":["187"],"Content-Type":["application/json"],"Date":["Tue, 13 Sep 2022 08:10:25 GMT"],"Link":["<https://acme-v02.api.letsencrypt.org/directory>;rel=\"index\"","<https://acme-v02.api.letsencrypt.org/acme/authz-v3/152854639307>;rel=\"up\""],"Location":["https://acme-v02.api.letsencrypt.org/acme/chall-v3/152854639307/M9TW8Q"],"Replay-Nonce":["0001Ouh4lt6p0QuuW2v_h0XaS9y0iJyYF_7sdDeXKliDTYU"],"Server":["nginx"],"Strict-Transport-Security":["max-age=604800"],"X-Frame-Options":["DENY"]},"status_code":200}
{"level":"debug","ts":1663056625.0746148,"logger":"tls.issuance.acme.acme_client","msg":"challenge accepted","identifier":"6.hachedigital.net","challenge_type":"http-01"}
{"level":"debug","ts":1663056625.1664946,"logger":"tls.issuance.acme.acme_client","msg":"http request","method":"POST","url":"https://acme-v02.api.letsencrypt.org/acme/authz-v3/152854636887","headers":{"Content-Type":["application/jose+json"],"User-Agent":["Caddy/2.5.1 CertMagic acmez (linux; amd64)"]},"response_headers":{"Boulder-Requester":["729455237"],"Cache-Control":["public, max-age=0, no-cache"],"Content-Length":["800"],"Content-Type":["application/json"],"Date":["Tue, 13 Sep 2022 08:10:25 GMT"],"Link":["<https://acme-v02.api.letsencrypt.org/directory>;rel=\"index\""],"Replay-Nonce":["0101CVqVOyc5G7RYgQdOF-CYig4lBvV0dD6QnnjDMCa-D9I"],"Server":["nginx"],"Strict-Transport-Security":["max-age=604800"],"X-Frame-Options":["DENY"]},"status_code":200}
{"level":"info","ts":1663056625.3200924,"logger":"tls.issuance.acme","msg":"served key authentication","identifier":"hachedigital.net","challenge":"http-01","remote":"10.1.2.209:42508","distributed":false}
{"level":"info","ts":1663056625.3350475,"logger":"tls.issuance.acme","msg":"served key authentication","identifier":"6.hachedigital.net","challenge":"http-01","remote":"10.20.11.1:14622","distributed":false}
{"level":"debug","ts":1663056625.4677505,"logger":"tls.issuance.acme.acme_client","msg":"http request","method":"POST","url":"https://acme-v02.api.letsencrypt.org/acme/authz-v3/152854639307","headers":{"Content-Type":["application/jose+json"],"User-Agent":["Caddy/2.5.1 CertMagic acmez (linux; amd64)"]},"response_headers":{"Boulder-Requester":["729455287"],"Cache-Control":["public, max-age=0, no-cache"],"Content-Length":["802"],"Content-Type":["application/json"],"Date":["Tue, 13 Sep 2022 08:10:25 GMT"],"Link":["<https://acme-v02.api.letsencrypt.org/directory>;rel=\"index\""],"Replay-Nonce":["00013yw2j1rg_cjuUXIvUdBbR-3CrNvEj10oRdVFM_BNn-o"],"Server":["nginx"],"Strict-Transport-Security":["max-age=604800"],"X-Frame-Options":["DENY"]},"status_code":200}
{"level":"info","ts":1663056625.550031,"logger":"tls.issuance.acme","msg":"served key authentication","identifier":"6.hachedigital.net","challenge":"http-01","remote":"10.1.2.209:10790","distributed":false}
{"level":"info","ts":1663056625.5589337,"logger":"tls.issuance.acme","msg":"served key authentication","identifier":"6.hachedigital.net","challenge":"http-01","remote":"10.1.2.207:32483","distributed":false}
{"level":"debug","ts":1663056625.5641637,"logger":"tls.issuance.acme.acme_client","msg":"http request","method":"POST","url":"https://acme-v02.api.letsencrypt.org/acme/authz-v3/152854636887","headers":{"Content-Type":["application/jose+json"],"User-Agent":["Caddy/2.5.1 CertMagic acmez (linux; amd64)"]},"response_headers":{"Boulder-Requester":["729455237"],"Cache-Control":["public, max-age=0, no-cache"],"Content-Length":["761"],"Content-Type":["application/json"],"Date":["Tue, 13 Sep 2022 08:10:25 GMT"],"Link":["<https://acme-v02.api.letsencrypt.org/directory>;rel=\"index\""],"Replay-Nonce":["0101ffE_0Shshf4Kr5z3Q22yCKRTrzx7Q9uttFwyzAXo48Y"],"Server":["nginx"],"Strict-Transport-Security":["max-age=604800"],"X-Frame-Options":["DENY"]},"status_code":200}
{"level":"info","ts":1663056625.8058574,"logger":"tls.issuance.acme.acme_client","msg":"validations succeeded; finalizing order","order":"https://acme-v02.api.letsencrypt.org/acme/order/729455237/125056837367"}
{"level":"debug","ts":1663056625.862454,"logger":"tls.issuance.acme.acme_client","msg":"http request","method":"POST","url":"https://acme-v02.api.letsencrypt.org/acme/authz-v3/152854639307","headers":{"Content-Type":["application/jose+json"],"User-Agent":["Caddy/2.5.1 CertMagic acmez (linux; amd64)"]},"response_headers":{"Boulder-Requester":["729455287"],"Cache-Control":["public, max-age=0, no-cache"],"Content-Length":["767"],"Content-Type":["application/json"],"Date":["Tue, 13 Sep 2022 08:10:25 GMT"],"Link":["<https://acme-v02.api.letsencrypt.org/directory>;rel=\"index\""],"Replay-Nonce":
{"level":"info","ts":1663056627.2162027,"logger":"tls.issuance.acme.acme_client","msg":"successfully downloaded available certificate chains","count":2,"first_url":"https://acme-v02.api.letsencrypt.org/acme/cert/030b9afcdb666e29529f7c7aa6366f41bb64"}
{"level":"error","ts":1663056627.5372224,"logger":"tls.obtain","msg":"will retry","error":"[6.hachedigital.net] Obtain: saving assets: mkdir /data/caddy/certificates/acme-v02.api.letsencrypt.org-directory/6.hachedigital.net: operation canceled","attempt":1,"retrying_in":60,"elapsed":6.740961155,"max_duration":2592000}
{"level":"error","ts":1663056628.180641,"logger":"tls.obtain","msg":"will retry","error":"[hachedigital.net] Obtain: saving assets: open /data/caddy/certificates/acme-v02.api.letsencrypt.org-directory/hachedigital.net/hachedigital.net.crt: operation canceled","attempt":1,"retrying_in":60,"elapsed":9.139617547,"max_duration":2592000}
{"level":"info","ts":1663056628.5626104,"msg":"[ERROR] Keeping lock file fresh: sync /data/caddy/locks/issue_cert_5.hachedigital.net.lock: operation canceled - terminating lock maintenance (lockfile: /data/caddy/locks/issue_cert_5.hachedigital.net.lock)"}

And this is the content of the Cloud Storage bucket, where you can see only a few certificate created and some of this directory are empty.

image

Thanks for helping @matt
Regards

Can you upload the full logs as a file attachment (either here or on some other site, maybe even google drive or similar)? I’m curious what is causing those ‘operation canceled’ errors if you’re not reloading the config right before that. If you are, then that’s what’s causing it probably, but for it to happen so often and consistently makes me wonder if your storage backend is just way too slow. Like, open /data/caddy/rw_test_7217349486913215582: operation canceled – that open op should only take a millisecond at most.

Hello Matt, I think is solved.

I have changed the path where the volumes were creating using the Caddy module:

I have tried create 50 new host and all worked perfectly! I think it was the storage volume…

Thanks for your help.
Keep in touch by mail!

Regards

1 Like

Excellent. Glad you got it working!