Hello again,
Yes, I’m using a volume in caddy deployment mounteb by google cloud storage (it is like AWS S3 service). It is the only way we have found to create persistent storage using Kubernetes in Google cloud. We cannot use normal disk because the disk driver used by GCP not allow multiattach, so when the caddy pod is restarted, we lost the volume… This kind of storage we are using right now is something like a NFS, using Google Cloud Storage buckets.
New test I just try. I have clean all the data from bucket and I have restarted the pod. This is like a clean installation. Then I have modifed the autosave.json with this data:
{
"admin": {
"listen": "0.0.0.0:2019"
},
"apps": {
"http": {
"http_port": 80,
"https_port": 443,
"servers": {
"srv0": {
"listen": ["0.0.0.0:443"],
"routes": [{
"handle": [{
"handler": "subroute",
"routes": [{
"handle": [{
"handler": "reverse_proxy",
"upstreams": [{
"dial": "frontend-service:3000"
}]
}]
}]
}],
"match": [{
"host": ["hachedigital.net", "1.hachedigital.net", "2.hachedigital.net", "3.hachedigital.net", "4.hachedigital.net", "5.hachedigital.net", "6.hachedigital.net", "7.hachedigital.net", "8.hachedigital.net", "9.hachedigital.net", "10.hachedigital.net"]
}],
"terminal": true
}]
}
}
}
},
"logging": {
"logs": {
"default": {
"level": "DEBUG"
}
}
}
}
Then I execute the next curl. The first 3 times I got the same error I was receiving executing the curl one by one.
curl -XPOST http://api-caddy-dev.topdoctors.es:2019/load -H "Content-Type: application/json" -d@autosave.json
{"error":"loading new config: http app module: start: finalizing automatic HTTPS: managing certificates for [7.hachedigital.net 9.hachedigital.net 3.hachedigital.net 6.hachedigital.net 2.hachedigital.net 4.hachedigital.net 1.hachedigital.net hachedigital.com 5.hachedigital.net 8.hachedigital.net 10.domainmine.net 10.hachedigital.com]: automate: manage [7.hachedigital.net 9.hachedigital.net 3.hachedigital.net 6.hachedigital.net 2.hachedigital.net 4.hachedigital.net hachedigital.com www.hachedigital.net hachedigital.net 5.hachedigital.net 8.hachedigital.net 10.hachedigital.net hachedigital.net 10.hachedigital.com]: 4.hachedigital.net: caching certificate: read /data/caddy/certificates/acme-v02.api.letsencrypt.org-directory/4.hachedigital.net/4.hachedigital.net.crt: operation canceled"}
And suddenly the fourth time the command works, and then I checked the logs in order to verify the certificate creation and Caddy only create some of then as you can see in the logs (I have had to cut the log input because I was overcoming the body limit.
["1.hachedigital.net","2.hachedigital.net","5.hachedigital.net","10.hachedigital.net","hachedigital.net","9.hachedigital.net","4.hachedigital.net","6.hachedigital.net","3.hachedigital.net","7.hachedigital.net","8.hachedigital.net"]}
{"level":"info","ts":1663056615.627493,"logger":"tls.cache.maintenance","msg":"started background certificate maintenance","cache":"0xc0004d67e0"}
{"level":"error","ts":1663056616.3435411,"logger":"tls","msg":"job failed","error":"2.hachedigital.net: obtaining certificate: failed storage check: close /data/caddy/rw_test_2345786471577943998: operation canceled - storage is probably misconfigured"}
{"level":"error","ts":1663056616.6922307,"logger":"tls","msg":"deleting test key from storage","key":"rw_test_7217349486913215582","error":"open /data/caddy/rw_test_7217349486913215582: operation canceled"}
{"level":"error","ts":1663056616.6923597,"logger":"tls","msg":"job failed","error":"1.hachedigital.net: obtaining certificate: failed storage check: open /data/caddy/rw_test_7217349486913215582: operation canceled - storage is probably misconfigured"}
{"level":"info","ts":1663056617.2977443,"logger":"tls.obtain","msg":"acquiring lock","identifier":"5.hachedigital.net"}
{"level":"info","ts":1663056617.6429179,"logger":"tls.obtain","msg":"acquiring lock","identifier":"10.hachedigital.net"}
{"level":"info","ts":1663056617.8344915,"logger":"tls.obtain","msg":"lock acquired","identifier":"5.hachedigital.net"}
{"level":"info","ts":1663056618.1294773,"logger":"tls.obtain","msg":"lock acquired","identifier":"10.hachedigital.net"}
{"level":"info","ts":1663056618.5684779,"logger":"tls.obtain","msg":"acquiring lock","identifier":"hachedigital.net"}
{"level":"debug","ts":1663056618.5947864,"logger":"tls.obtain","msg":"trying issuer 1/2","issuer":"acme-v02.api.letsencrypt.org-directory"}
{"level":"error","ts":1663056618.8566482,"logger":"tls","msg":"job failed","error":"4.hachedigital.net: obtaining certificate: failed storage check: close /data/caddy/rw_test_9055767639255195990: operation canceled - storage is probably misconfigured"}
{"level":"info","ts":1663056619.0409782,"logger":"tls.obtain","msg":"lock acquired","identifier":"hachedigital.net"}
{"level":"info","ts":1663056619.103426,"logger":"tls.obtain","msg":"acquiring lock","identifier":"9.hachedigital.net"}
{"level":"debug","ts":1663056619.2149763,"logger":"tls.obtain","msg":"trying issuer 1/2","issuer":"acme-v02.api.letsencrypt.org-directory"}
{"level":"info","ts":1663056619.673278,"logger":"tls.obtain","msg":"lock acquired","identifier":"9.hachedigital.net"}
{"level":"debug","ts":1663056619.9218223,"logger":"tls.obtain","msg":"trying issuer 1/2","issuer":"acme-v02.api.letsencrypt.org-directory"}
{"level":"info","ts":1663056620.1899335,"logger":"tls.obtain","msg":"acquiring lock","identifier":"6.hachedigital.net"}
{"level":"debug","ts":1663056620.2193222,"logger":"http.stdlib","msg":"http: Accept error: accept tcp [::]:443: i/o timeout; retrying in 5ms"}
{"level":"info","ts":1663056620.2216654,"logger":"tls.cache.maintenance","msg":"stopped background certificate maintenance","cache":"0xc0004d6230"}
{"level":"debug","ts":1663056620.2484803,"logger":"tls.issuance.acme.acme_client","msg":"http request","method":"GET","url":"https://acme-v02.api.letsencrypt.org/directory","headers":{"User-Agent":["Caddy/2.5.1 CertMagic acmez (linux; amd64)"]},"response_headers":{"Cache-Control":["public, max-age=0, no-cache"],"Content-Length":["672"],"Content-Type":["application/json"],"Date":["Tue, 13 Sep 2022 08:10:20 GMT"],"Server":["nginx"],"Strict-Transport-Security":["max-age=604800"],"X-Frame-Options":["DENY"]},"status_code":200}
{"level":"debug","ts":1663056620.3751333,"logger":"tls.issuance.acme.acme_client","msg":"http request","method":"HEAD","url":"https://acme-v02.api.letsencrypt.org/acme/new-nonce","headers":{"User-Agent":["Caddy/2.5.1 CertMagic acmez (linux; amd64)"]},"response_headers":{"Cache-Control":["public, max-age=0, no-cache"],"Date":["Tue, 13 Sep 2022 08:10:20 GMT"],"Link":["<https://acme-v02.api.letsencrypt.org/directory>;rel=\"index\""],"Replay-Nonce":["0102h1WfAy97R1KPqdpQAut2-3KE8KIFmxezxK-GVefyubA"],"Server":["nginx"],"Strict-Transport-Security":["max-age=604800"],"X-Frame-Options":["DENY"]},"status_code":200}
{"level":"debug","ts":1663056620.5600238,"logger":"tls.issuance.acme.acme_client","msg":"http request","method":"POST","url":"https://acme-v02.api.letsencrypt.org/acme/new-acct","headers":{"Content-Type":["application/jose+json"],"User-Agent":["Caddy/2.5.1 CertMagic acmez (linux; amd64)"]},"response_headers":{"Boulder-Requester":["729455207"],"Cache-Control":["public, max-age=0, no-cache"],"Content-Length":["268"],"Content-Type":["application/json"],"Date":["Tue, 13 Sep 2022 08:10:20 GMT"],"Link":["<https://acme-v02.api.letsencrypt.org/directory>;rel=\"index\"","<https://letsencrypt.org/documents/LE-SA-v1.2-November-15-2017-w-v1.3-notice.pdf>;rel=\"terms-of-service\""],"Location":["https://acme-v02.api.letsencrypt.org/acme/acct/729455207"],"Replay-Nonce":["0101itbY5mhEgvta8oesfnVolAT9TwBG9_X65J-QNl3asRI"],"Server":["nginx"],"Strict-Transport-Security":["max-age=604800"],"X-Frame-Options":["DENY"]},"status_code":201}
{"level":"debug","ts":1663056620.6636422,"logger":"tls.issuance.acme.acme_client","msg":"http request","method":"HEAD","url":"https://acme-v02.api.letsencrypt.org/acme/new-nonce","headers":{"User-Agent":["Caddy/2.5.1 CertMagic acmez (linux; amd64)"]},"response_headers":{"Cache-Control":["public, max-age=0, no-cache"],"Date":["Tue, 13 Sep 2022 08:10:20 GMT"],"Link":["<https://acme-v02.api.letsencrypt.org/directory>;rel=\"index\""],"Replay-Nonce":["0101hWkwPZVM3ldJoLQbnAubNiEzDWRt4ZF-Bgb8-FDpAAs"],"Server":["nginx"],"Strict-Transport-Security":["max-age=604800"],"X-Frame-Options":["DENY"]},"status_code":200}
{"level":"info","ts":1663056620.727509,"msg":"autosaved config (load with --resume flag)","file":"/data/config/caddy/autosave.json"}
{"level":"info","ts":1663056620.7276533,"logger":"admin.api","msg":"load complete"}
{"level":"info","ts":1663056620.796221,"logger":"tls.obtain","msg":"lock acquired","identifier":"6.hachedigital.net"}
{"level":"info","ts":1663056620.8388932,"logger":"tls.obtain","msg":"acquiring lock","identifier":"3.hachedigital.net"}
{"level":"error","ts":1663056620.8393214,"logger":"tls.obtain","msg":"will retry","error":"loading existing private key for reuse with issuer acme-v02.api.letsencrypt.org-directory: open /data/caddy/certificates/acme-v02.api.letsencrypt.org-directory/9.hachedigital.net/9.hachedigital.net.key: operation canceled","attempt":1,"retrying_in":60,"elapsed":1.165994383,"max_duration":2592000}
{"level":"debug","ts":1663056620.8769279,"logger":"tls.issuance.acme.acme_client","msg":"http request","method":"POST","url":"https://acme-v02.api.letsencrypt.org/acme/new-acct","headers":{"Content-Type":["application/jose+json"],"User-Agent":["Caddy/2.5.1 CertMagic acmez (linux; amd64)"]},"response_headers":{"Boulder-Requester":["729455227"],"Cache-Control":["public, max-age=0, no-cache"],"Content-Length":["268"],"Content-Type":["application/json"],"Date":["Tue, 13 Sep 2022 08:10:20 GMT"],"Link":["<https://acme-v02.api.letsencrypt.org/directory>;rel=\"index\"","<https://letsencrypt.org/documents/LE-SA-v1.2-November-15-2017-w-v1.3-notice.pdf>;rel=\"terms-of-service\""],"Location":["https://acme-v02.api.letsencrypt.org/acme/acct/729455227"],"Replay-Nonce":["0101eYaSE2BachFi5ZA-bamH0LWDghMMmT0E0GTCyYyZQ8k"],"Server":["nginx"],"Strict-Transport-Security":["max-age=604800"],"X-Frame-Options":["DENY"]},"status_code":201}
{"level":"info","ts":1663056620.9223764,"logger":"admin","msg":"stopped previous server","address":"tcp/0.0.0.0:2019"}
{"level":"info","ts":1663056621.1954315,"logger":"tls.obtain","msg":"acquiring lock","identifier":"7.hachedigital.net"}
{"level":"debug","ts":1663056621.2883375,"logger":"tls.issuance.acme.acme_client","msg":"http request","method":"HEAD","url":"https://acme-v02.api.letsencrypt.org/acme/new-nonce","headers":{"User-Agent":["Caddy/2.5.1 CertMagic acmez (linux; amd64)"]},"response_headers":{"Cache-Control":["public, max-age=0, no-cache"],"Date":["Tue, 13 Sep 2022 08:10:21 GMT"],"Link":["<https://acme-v02.api.letsencrypt.org/directory>;rel=\"index\""],"Replay-Nonce":["0102jM3aV2WL-WDkAEK7AjKvEepa5w42hSFlqkE7KpJsh7Y"],"Server":["nginx"],"Strict-Transport-Security":["max-age=604800"],"X-Frame-Options":["DENY"]},"status_code":200}
{"level":"info","ts":1663056621.3325696,"logger":"tls.obtain","msg":"lock acquired","identifier":"3.hachedigital.net"}
{"level":"debug","ts":1663056621.5219607,"logger":"tls.issuance.acme.acme_client","msg":"http request","method":"POST","url":"https://acme-v02.api.letsencrypt.org/acme/new-acct","headers":{"Content-Type":["application/jose+json"],"User-Agent":["Caddy/2.5.1 CertMagic acmez (linux; amd64)"]},"response_headers":{"Boulder-Requester":["729455237"],"Cache-Control":["public, max-age=0, no-cache"],"Content-Length":["268"],"Content-Type":["application/json"],"Date":["Tue, 13 Sep 2022 08:10:21 GMT"],"Link":["<https://acme-v02.api.letsencrypt.org/directory>;rel=\"index\"","<https://letsencrypt.org/documents/LE-SA-v1.2-November-15-2017-w-v1.3-notice.pdf>;rel=\"terms-of-service\""],"Location":["https://acme-v02.api.letsencrypt.org/acme/acct/729455237"],"Replay-Nonce":["0102db3ZzDcEn2_Uukvo7kP-k77TVMfpvIuFJJBFND9D5E8"],"Server":["nginx"],"Strict-Transport-Security":["max-age=604800"],"X-Frame-Options":["DENY"]},"status_code":201}
{"level":"debug","ts":1663056621.6798892,"logger":"tls.obtain","msg":"trying issuer 1/2","issuer":"acme-v02.api.letsencrypt.org-directory"}
{"level":"info","ts":1663056621.6930716,"logger":"tls.obtain","msg":"lock acquired","identifier":"7.hachedigital.net"}
{"level":"info","ts":1663056622.2015085,"logger":"tls.obtain","msg":"acquiring lock","identifier":"8.hachedigital.net"}
{"level":"error","ts":1663056622.244327,"logger":"tls.obtain","msg":"will retry","error":"loading existing private key for reuse with issuer acme.zerossl.com-v2-DV90: open /data/caddy/certificates/acme.zerossl.com-v2-dv90/7.hachedigital.net/7.hachedigital.net.key: operation canceled","attempt":1,"retrying_in":60,"elapsed":0.549400612,"max_duration":2592000}
{"level":"debug","ts":1663056622.5399861,"logger":"tls.issuance.acme.acme_client","msg":"http request","method":"HEAD","url":"https://acme-v02.api.letsencrypt.org/acme/new-nonce","headers":{"User-Agent":["Caddy/2.5.1 CertMagic acmez (linux; amd64)"]},"response_headers":{"Cache-Control":["public, max-age=0, no-cache"],"Date":["Tue, 13 Sep 2022 08:10:22 GMT"],"Link":["<https://acme-v02.api.letsencrypt.org/directory>;rel=\"index\""],"Replay-Nonce":["0001LEGsx8MVC4Pp53lCd8OMtmZ6Y0InJ7wa31U1IUefwoY"],"Server":["nginx"],"Strict-Transport-Security":["max-age=604800"],"X-Frame-Options":["DENY"]},"status_code":200}
{"level":"debug","ts":1663056622.6956375,"logger":"tls.issuance.acme.acme_client","msg":"http request","method":"POST","url":"https://acme-v02.api.letsencrypt.org/acme/new-acct","headers":{"Content-Type":["application/jose+json"],"User-Agent":["Caddy/2.5.1 CertMagic acmez (linux; amd64)"]},"response_headers":{"Boulder-Requester":["729455287"],"Cache-Control":["public, max-age=0, no-cache"],"Content-Length":["268"],"Content-Type":["application/json"],"Date":["Tue, 13 Sep 2022 08:10:22 GMT"],"Link":["<https://acme-v02.api.letsencrypt.org/directory>;rel=\"index\"","<https://letsencrypt.org/documents/LE-SA-v1.2-November-15-2017-w-v1.3-notice.pdf>;rel=\"terms-of-service\""],"Location":["https://acme-v02.api.letsencrypt.org/acme/acct/729455287"],"Replay-Nonce":["0001PBXoBY9ml35MxBSoszu1aeefanQuybBeSSrKl5M5wgE"],"Server":["nginx"],"Strict-Transport-Security":["max-age=604800"],"X-Frame-Options":["DENY"]},"status_code":201}
{"level":"error","ts":1663056622.737557,"logger":"tls.obtain","msg":"could not get certificate from issuer","identifier":"5.hachedigital.net","issuer":"acme-v02.api.letsencrypt.org-directory","error":"could not save account []: open /data/caddy/acme/acme-v02.api.letsencrypt.org-directory/users/default/default.json: operation canceled"}
{"level":"warn","ts":1663056622.814721,"logger":"tls.issuance.zerossl","msg":"missing email address for ZeroSSL; it is strongly recommended to set one for next time"}
{"level":"info","ts":1663056623.2947507,"logger":"tls.issuance.acme","msg":"done waiting on internal rate limiter","identifiers":["10.hachedigital.net"],"ca":"https://acme-v02.api.letsencrypt.org/directory","account":""}
{"level":"info","ts":1663056623.3754025,"logger":"tls.issuance.zerossl","msg":"generated EAB credentials","key_id":"Z4zOKNzcs9hPmQWIw0rd0Q"}
{"level":"info","ts":1663056623.4918823,"logger":"tls.issuance.acme","msg":"waiting on internal rate limiter","identifiers":["hachedigital.net"],"ca":"https://acme-v02.api.letsencrypt.org/directory","account":""}
{"level":"info","ts":1663056623.4919376,"logger":"tls.issuance.acme","msg":"done waiting on internal rate limiter","identifiers":["hachedigital.net"],"ca":"https://acme-v02.api.letsencrypt.org/directory","account":""}
{"level":"error","ts":1663056623.5044358,"logger":"tls.obtain","msg":"could not get certificate from issuer","identifier":"10.hachedigital.net","issuer":"acme-v02.api.letsencrypt.org-directory","error":"HTTP 429 urn:ietf:params:acme:error:rateLimited - Error creating new order :: too many certificates (5) already issued for this exact set of domains in the last 168 hours: 10.hachedigital.net, retry after 2022-09-13T21:01:46Z: see https://letsencrypt.org/docs/duplicate-certificate-limit/"}
{"level":"debug","ts":1663056623.5044906,"logger":"tls.obtain","msg":"trying issuer 2/2","issuer":"acme.zerossl.com-v2-DV90"}
{"level":"warn","ts":1663056623.5555704,"logger":"tls.issuance.zerossl","msg":"missing email address for ZeroSSL; it is strongly recommended to set one for next time"}
{"level":"debug","ts":1663056623.8127441,"logger":"tls.issuance.acme.acme_client","msg":"http request","method":"POST","url":"https://acme-v02.api.letsencrypt.org/acme/new-order","headers":{"Content-Type":["application/jose+json"],"User-Agent":["Caddy/2.5.1 CertMagic acmez (linux; amd64)"]},"response_headers":{"Boulder-Requester":["729455237"],"Cache-Control":["public, max-age=0, no-cache"],"Content-Length":["341"],"Content-Type":["application/json"],"Date":["Tue, 13 Sep 2022 08:10:23 GMT"],"Link":["<https://acme-v02.api.letsencrypt.org/directory>;rel=\"index\""],"Location":["https://acme-v02.api.letsencrypt.org/acme/order/729455237/125056837367"],"Replay-Nonce":["0102ntiaSlM5G2_cB-9u_7VEYXAk9E6xdGN5b-9yXSww0ZE"],"Server":["nginx"],"Strict-Transport-Security":["max-age=604800"],"X-Frame-Options":["DENY"]},"status_code":201}
{"level":"info","ts":1663056623.852528,"logger":"tls.issuance.zerossl","msg":"generated EAB credentials","key_id":"jU133Z4Y_G2LaTfva2CekQ"}
{"level":"debug","ts":1663056623.9741,"logger":"tls.issuance.acme.acme_client","msg":"http request","method":"POST","url":"https://acme-v02.api.letsencrypt.org/acme/authz-v3/152854636887","headers":{"Content-Type":["application/jose+json"],"User-Agent":["Caddy/2.5.1 CertMagic acmez (linux; amd64)"]},"response_headers":{"Boulder-Requester":["729455237"],"Cache-Control":["public, max-age=0, no-cache"],"Content-Length":["800"],"Content-Type":["application/json"],"Date":["Tue, 13 Sep 2022 08:10:23 GMT"],"Link":["<https://acme-v02.api.letsencrypt.org/directory>;rel=\"index\""],"Replay-Nonce":["0101JY_IYUObRB3o1s55kpDblIqTwYDaXFOm1YcAIs4r4es"],"Server":["nginx"],"Strict-Transport-Security":["max-age=604800"],"X-Frame-Options":["DENY"]},"status_code":200}
{"level":"debug","ts":1663056623.9748473,"logger":"tls.issuance.acme.acme_client","msg":"no solver configured","challenge_type":"dns-01"}
{"level":"info","ts":1663056623.9750416,"logger":"tls.issuance.acme.acme_client","msg":"trying to solve challenge","identifier":"hachedigital.net","challenge_type":"http-01","ca":"https://acme-v02.api.letsencrypt.org/directory"}
{"level":"info","ts":1663056624.151617,"logger":"tls.issuance.acme","msg":"waiting on internal rate limiter","identifiers":["6.hachedigital.net"],"ca":"https://acme-v02.api.letsencrypt.org/directory","account":""}
{"level":"info","ts":1663056624.1516657,"logger":"tls.issuance.acme","msg":"done waiting on internal rate limiter","identifiers":["6.hachedigital.net"],"ca":"https://acme-v02.api.letsencrypt.org/directory","account":""}
{"level":"info","ts":1663056624.189281,"logger":"tls.issuance.acme","msg":"waiting on internal rate limiter","identifiers":["3.hachedigital.net"],"ca":"https://acme-v02.api.letsencrypt.org/directory","account":""}
{"level":"info","ts":1663056624.1893454,"logger":"tls.issuance.acme","msg":"done waiting on internal rate limiter","identifiers":["3.hachedigital.net"],"ca":"https://acme-v02.api.letsencrypt.org/directory","account":""}
{"level":"info","ts":1663056624.2153091,"logger":"tls.issuance.acme","msg":"waiting on internal rate limiter","identifiers":["8.hachedigital.net"],"ca":"https://acme-v02.api.letsencrypt.org/directory","account":""}
{"level":"info","ts":1663056624.2153666,"logger":"tls.issuance.acme","msg":"done waiting on internal rate limiter","identifiers":["8.hachedigital.net"],"ca":"https://acme-v02.api.letsencrypt.org/directory","account":""}
{"level":"debug","ts":1663056624.3291736,"logger":"tls.issuance.acme.acme_client","msg":"http request","method":"POST","url":"https://acme-v02.api.letsencrypt.org/acme/new-order","headers":{"Content-Type":["application/jose+json"],"User-Agent":["Caddy/2.5.1 CertMagic acmez (linux; amd64)"]},"response_headers":{"Boulder-Requester":["729455287"],"Cache-Control":["public, max-age=0, no-cache"],"Content-Length":["343"],"Content-Type":["application/json"],"Date":["Tue, 13 Sep 2022 08:10:24 GMT"],"Link":["<https://acme-v02.api.letsencrypt.org/directory>;rel=\"index\""],"Location":["https://acme-v02.api.letsencrypt.org/acme/order/729455287/125056839507"],"Replay-Nonce":["0002JA1LTwk8itnMYkFvdGrPquzw7sbnYpzwsJ37I86iq24"],"Server":["nginx"],"Strict-Transport-Security":["max-age=604800"],"X-Frame-Options":["DENY"]},"status_code":201}
{"level":"debug","ts":1663056624.4745715,"logger":"tls.issuance.acme.acme_client","msg":"http request","method":"POST","url":"https://acme-v02.api.letsencrypt.org/acme/authz-v3/152854639307","headers":{"Content-Type":["application/jose+json"],"User-Agent":["Caddy/2.5.1 CertMagic acmez (linux; amd64)"]},"response_headers":{"Boulder-Requester":["729455287"],"Cache-Control":["public, max-age=0, no-cache"],"Content-Length":["802"],"Content-Type":["application/json"],"Date":["Tue, 13 Sep 2022 08:10:24 GMT"],"Link":["<https://acme-v02.api.letsencrypt.org/directory>;rel=\"index\""],"Replay-Nonce":["0002tKVQFD2Gfu180A364Cy9v5BvpHWqvcvqJx_-brEAFXc"],"Server":["nginx"],"Strict-Transport-Security":["max-age=604800"],"X-Frame-Options":["DENY"]},"status_code":200}
{"level":"debug","ts":1663056624.4749,"logger":"tls.issuance.acme.acme_client","msg":"no solver configured","challenge_type":"dns-01"}
{"level":"info","ts":1663056624.4749258,"logger":"tls.issuance.acme.acme_client","msg":"trying to solve challenge","identifier":"6.hachedigital.net","challenge_type":"http-01","ca":"https://acme-v02.api.letsencrypt.org/directory"}
{"level":"debug","ts":1663056624.768603,"logger":"tls.issuance.acme.acme_client","msg":"http request","method":"POST","url":"https://acme-v02.api.letsencrypt.org/acme/chall-v3/152854636887/q60tmQ","headers":{"Content-Type":["application/jose+json"],"User-Agent":["Caddy/2.5.1 CertMagic acmez (linux; amd64)"]},"response_headers":{"Boulder-Requester":["729455237"],"Cache-Control":["public, max-age=0, no-cache"],"Content-Length":["187"],"Content-Type":["application/json"],"Date":["Tue, 13 Sep 2022 08:10:24 GMT"],"Link":["<https://acme-v02.api.letsencrypt.org/directory>;rel=\"index\"","<https://acme-v02.api.letsencrypt.org/acme/authz-v3/152854636887>;rel=\"up\""],"Location":["https://acme-v02.api.letsencrypt.org/acme/chall-v3/152854636887/q60tmQ"],"Replay-Nonce":["0101oCi7YfgoyWiWkQgDUuWUxemDBtYe6AUMN85rZLaBxiA"],"Server":["nginx"],"Strict-Transport-Security":["max-age=604800"],"X-Frame-Options":["DENY"]},"status_code":200}
{"level":"debug","ts":1663056624.7687318,"logger":"tls.issuance.acme.acme_client","msg":"challenge accepted","identifier":"hachedigital.net","challenge_type":"http-01"}
{"level":"info","ts":1663056624.9096093,"logger":"tls.issuance.acme","msg":"served key authentication","identifier":"hachedigital.net","challenge":"http-01","remote":"10.1.2.209:61396","distributed":false}
{"level":"info","ts":1663056625.0354965,"logger":"tls.issuance.acme","msg":"served key authentication","identifier":"hachedigital.net","challenge":"http-01","remote":"10.20.11.1:4547","distributed":false}
{"level":"info","ts":1663056625.0690525,"logger":"tls.issuance.acme","msg":"served key authentication","identifier":"hachedigital.net","challenge":"http-01","remote":"10.1.2.207:4278","distributed":false}
{"level":"debug","ts":1663056625.074502,"logger":"tls.issuance.acme.acme_client","msg":"http request","method":"POST","url":"https://acme-v02.api.letsencrypt.org/acme/chall-v3/152854639307/M9TW8Q","headers":{"Content-Type":["application/jose+json"],"User-Agent":["Caddy/2.5.1 CertMagic acmez (linux; amd64)"]},"response_headers":{"Boulder-Requester":["729455287"],"Cache-Control":["public, max-age=0, no-cache"],"Content-Length":["187"],"Content-Type":["application/json"],"Date":["Tue, 13 Sep 2022 08:10:25 GMT"],"Link":["<https://acme-v02.api.letsencrypt.org/directory>;rel=\"index\"","<https://acme-v02.api.letsencrypt.org/acme/authz-v3/152854639307>;rel=\"up\""],"Location":["https://acme-v02.api.letsencrypt.org/acme/chall-v3/152854639307/M9TW8Q"],"Replay-Nonce":["0001Ouh4lt6p0QuuW2v_h0XaS9y0iJyYF_7sdDeXKliDTYU"],"Server":["nginx"],"Strict-Transport-Security":["max-age=604800"],"X-Frame-Options":["DENY"]},"status_code":200}
{"level":"debug","ts":1663056625.0746148,"logger":"tls.issuance.acme.acme_client","msg":"challenge accepted","identifier":"6.hachedigital.net","challenge_type":"http-01"}
{"level":"debug","ts":1663056625.1664946,"logger":"tls.issuance.acme.acme_client","msg":"http request","method":"POST","url":"https://acme-v02.api.letsencrypt.org/acme/authz-v3/152854636887","headers":{"Content-Type":["application/jose+json"],"User-Agent":["Caddy/2.5.1 CertMagic acmez (linux; amd64)"]},"response_headers":{"Boulder-Requester":["729455237"],"Cache-Control":["public, max-age=0, no-cache"],"Content-Length":["800"],"Content-Type":["application/json"],"Date":["Tue, 13 Sep 2022 08:10:25 GMT"],"Link":["<https://acme-v02.api.letsencrypt.org/directory>;rel=\"index\""],"Replay-Nonce":["0101CVqVOyc5G7RYgQdOF-CYig4lBvV0dD6QnnjDMCa-D9I"],"Server":["nginx"],"Strict-Transport-Security":["max-age=604800"],"X-Frame-Options":["DENY"]},"status_code":200}
{"level":"info","ts":1663056625.3200924,"logger":"tls.issuance.acme","msg":"served key authentication","identifier":"hachedigital.net","challenge":"http-01","remote":"10.1.2.209:42508","distributed":false}
{"level":"info","ts":1663056625.3350475,"logger":"tls.issuance.acme","msg":"served key authentication","identifier":"6.hachedigital.net","challenge":"http-01","remote":"10.20.11.1:14622","distributed":false}
{"level":"debug","ts":1663056625.4677505,"logger":"tls.issuance.acme.acme_client","msg":"http request","method":"POST","url":"https://acme-v02.api.letsencrypt.org/acme/authz-v3/152854639307","headers":{"Content-Type":["application/jose+json"],"User-Agent":["Caddy/2.5.1 CertMagic acmez (linux; amd64)"]},"response_headers":{"Boulder-Requester":["729455287"],"Cache-Control":["public, max-age=0, no-cache"],"Content-Length":["802"],"Content-Type":["application/json"],"Date":["Tue, 13 Sep 2022 08:10:25 GMT"],"Link":["<https://acme-v02.api.letsencrypt.org/directory>;rel=\"index\""],"Replay-Nonce":["00013yw2j1rg_cjuUXIvUdBbR-3CrNvEj10oRdVFM_BNn-o"],"Server":["nginx"],"Strict-Transport-Security":["max-age=604800"],"X-Frame-Options":["DENY"]},"status_code":200}
{"level":"info","ts":1663056625.550031,"logger":"tls.issuance.acme","msg":"served key authentication","identifier":"6.hachedigital.net","challenge":"http-01","remote":"10.1.2.209:10790","distributed":false}
{"level":"info","ts":1663056625.5589337,"logger":"tls.issuance.acme","msg":"served key authentication","identifier":"6.hachedigital.net","challenge":"http-01","remote":"10.1.2.207:32483","distributed":false}
{"level":"debug","ts":1663056625.5641637,"logger":"tls.issuance.acme.acme_client","msg":"http request","method":"POST","url":"https://acme-v02.api.letsencrypt.org/acme/authz-v3/152854636887","headers":{"Content-Type":["application/jose+json"],"User-Agent":["Caddy/2.5.1 CertMagic acmez (linux; amd64)"]},"response_headers":{"Boulder-Requester":["729455237"],"Cache-Control":["public, max-age=0, no-cache"],"Content-Length":["761"],"Content-Type":["application/json"],"Date":["Tue, 13 Sep 2022 08:10:25 GMT"],"Link":["<https://acme-v02.api.letsencrypt.org/directory>;rel=\"index\""],"Replay-Nonce":["0101ffE_0Shshf4Kr5z3Q22yCKRTrzx7Q9uttFwyzAXo48Y"],"Server":["nginx"],"Strict-Transport-Security":["max-age=604800"],"X-Frame-Options":["DENY"]},"status_code":200}
{"level":"info","ts":1663056625.8058574,"logger":"tls.issuance.acme.acme_client","msg":"validations succeeded; finalizing order","order":"https://acme-v02.api.letsencrypt.org/acme/order/729455237/125056837367"}
{"level":"debug","ts":1663056625.862454,"logger":"tls.issuance.acme.acme_client","msg":"http request","method":"POST","url":"https://acme-v02.api.letsencrypt.org/acme/authz-v3/152854639307","headers":{"Content-Type":["application/jose+json"],"User-Agent":["Caddy/2.5.1 CertMagic acmez (linux; amd64)"]},"response_headers":{"Boulder-Requester":["729455287"],"Cache-Control":["public, max-age=0, no-cache"],"Content-Length":["767"],"Content-Type":["application/json"],"Date":["Tue, 13 Sep 2022 08:10:25 GMT"],"Link":["<https://acme-v02.api.letsencrypt.org/directory>;rel=\"index\""],"Replay-Nonce":
{"level":"info","ts":1663056627.2162027,"logger":"tls.issuance.acme.acme_client","msg":"successfully downloaded available certificate chains","count":2,"first_url":"https://acme-v02.api.letsencrypt.org/acme/cert/030b9afcdb666e29529f7c7aa6366f41bb64"}
{"level":"error","ts":1663056627.5372224,"logger":"tls.obtain","msg":"will retry","error":"[6.hachedigital.net] Obtain: saving assets: mkdir /data/caddy/certificates/acme-v02.api.letsencrypt.org-directory/6.hachedigital.net: operation canceled","attempt":1,"retrying_in":60,"elapsed":6.740961155,"max_duration":2592000}
{"level":"error","ts":1663056628.180641,"logger":"tls.obtain","msg":"will retry","error":"[hachedigital.net] Obtain: saving assets: open /data/caddy/certificates/acme-v02.api.letsencrypt.org-directory/hachedigital.net/hachedigital.net.crt: operation canceled","attempt":1,"retrying_in":60,"elapsed":9.139617547,"max_duration":2592000}
{"level":"info","ts":1663056628.5626104,"msg":"[ERROR] Keeping lock file fresh: sync /data/caddy/locks/issue_cert_5.hachedigital.net.lock: operation canceled - terminating lock maintenance (lockfile: /data/caddy/locks/issue_cert_5.hachedigital.net.lock)"}
And this is the content of the Cloud Storage bucket, where you can see only a few certificate created and some of this directory are empty.
Thanks for helping @matt
Regards
