Yeah, on the backend.
Ok, I did, but they have not reappeared after restarting everything.
What’s in your logs?
Frontend:
Feb 18 09:33:38 iot caddy[23373]: {"level":"debug","ts":1645194818.0479314,"logger":"http.handlers.reverse_proxy","msg":"upstream roundtrip","upstream":"cloud.y8s.casa:443","request":{"remote_addr":"173.8.14.69:61151","proto":"HTTP/1.1","method":"GET","host":"cloud.y8s.casa:443","uri":"/","headers":{"X-Forwarded-Proto":["https"],"X-Forwarded-For":["173.8.14.69"],"User-Agent":["curl/7.79.1"],"Accept":["*/*"],"X-Forwarded-Host":["test.eastcapitol.us"]},"tls":{"resumed":false,"version":771,"cipher_suite":49196,"proto":"http/1.1","proto_mutual":true,"server_name":"test.eastcapitol.us"}},"duration":0.019397671,"error":"x509: certificate has expired or is not yet valid: current time 2022-02-18T09:33:38-05:00 is after 2022-02-18T10:25:54Z"}
Feb 18 09:33:38 iot caddy[23373]: {"level":"error","ts":1645194818.0482728,"logger":"http.log.error","msg":"x509: certificate has expired or is not yet valid: current time 2022-02-18T09:33:38-05:00 is after 2022-02-18T10:25:54Z","request":{"remote_addr":"173.8.14.69:61151","proto":"HTTP/1.1","method":"GET","host":"test.eastcapitol.us","uri":"/","headers":{"User-Agent":["curl/7.79.1"],"Accept":["*/*"]},"tls":{"resumed":false,"version":771,"cipher_suite":49196,"proto":"http/1.1","proto_mutual":true,"server_name":"test.eastcapitol.us"}},"duration":0.020228262,"status":502,"err_id":"bazyprk9m","err_trace":"reverseproxy.statusError (reverseproxy.go:861)"}
backend:
Feb 18 09:33:38 cloud caddy[1620]: {"level":"debug","ts":1645194818.0468225,"logger":"tls.handshake","msg":"choosing certificate","identifier":"cloud.y8s.casa","num_choices":1}
Feb 18 09:33:38 cloud caddy[1620]: {"level":"debug","ts":1645194818.0472898,"logger":"tls.handshake","msg":"default certificate selection results","identifier":"cloud.y8s.casa","subjects":["cloud.y8s.casa"],"managed":true,"issuer_key":"caddy.y8s.casa-acme-local-directory","hash":"d99effb6eea719a3624529d0b184b95c0e1dd28d7973f41ab71fd37a324d5ec8"}
Feb 18 09:33:38 cloud caddy[1620]: {"level":"debug","ts":1645194818.0473847,"logger":"tls.handshake","msg":"matched certificate in cache","subjects":["cloud.y8s.casa"],"managed":true,"expiration":1645179954,"hash":"d99effb6eea719a3624529d0b184b95c0e1dd28d7973f41ab71fd37a324d5ec8"}
Feb 18 09:33:38 cloud caddy[1620]: {"level":"debug","ts":1645194818.056496,"logger":"http.stdlib","msg":"http: TLS handshake error from 10.10.10.40:37894: remote error: tls: bad certificate"}
So I don’t have to delete any of the root or intermediate certs again, correct?
What about the /var/lib/caddy/.local/share/caddy/acme on the FRONTend?
Maybe drastic, but is there any harm in just removing both machines /var/lib/caddy/* and reissuing everything?
Yeah you should only need to remove stuff from the backend only. It’s the backend’s TLS certificate that’s expired. That’s strange though. Please read back in your backend’s logs, find the last time it issued a certificate for that domain you tried. When did it do it? Was there any errors in issuance?
Here’s what a restart of the service on the backend looks like:
Feb 18 10:10:32 cloud systemd[1]: Starting Caddy...
Feb 18 10:10:33 cloud caddy[5464]: caddy.HomeDir=/var/lib/caddy
Feb 18 10:10:33 cloud caddy[5464]: caddy.AppDataDir=/var/lib/caddy/.local/share/caddy
Feb 18 10:10:33 cloud caddy[5464]: caddy.AppConfigDir=/var/lib/caddy/.config/caddy
Feb 18 10:10:33 cloud caddy[5464]: caddy.ConfigAutosavePath=/var/lib/caddy/.config/caddy/autosave.json
Feb 18 10:10:33 cloud caddy[5464]: caddy.Version=v2.4.6 h1:HGkGICFGvyrodcqOOclHKfvJC0qTU7vny/7FhYp9hNw=
Feb 18 10:10:33 cloud caddy[5464]: runtime.GOOS=linux
Feb 18 10:10:33 cloud caddy[5464]: runtime.GOARCH=amd64
Feb 18 10:10:33 cloud caddy[5464]: runtime.Compiler=gc
Feb 18 10:10:33 cloud caddy[5464]: runtime.NumCPU=2
Feb 18 10:10:33 cloud caddy[5464]: runtime.GOMAXPROCS=2
Feb 18 10:10:33 cloud caddy[5464]: runtime.Version=go1.17.2
Feb 18 10:10:33 cloud caddy[5464]: os.Getwd=/
Feb 18 10:10:33 cloud caddy[5464]: LANG=en_US.UTF-8
Feb 18 10:10:33 cloud caddy[5464]: PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
Feb 18 10:10:33 cloud caddy[5464]: NOTIFY_SOCKET=/run/systemd/notify
Feb 18 10:10:33 cloud caddy[5464]: HOME=/var/lib/caddy
Feb 18 10:10:33 cloud caddy[5464]: LOGNAME=caddy
Feb 18 10:10:33 cloud caddy[5464]: USER=caddy
Feb 18 10:10:33 cloud caddy[5464]: INVOCATION_ID=b5a1fb39deb449bba6c270fc82d730d0
Feb 18 10:10:33 cloud caddy[5464]: JOURNAL_STREAM=8:35959
Feb 18 10:10:33 cloud caddy[5464]: {"level":"info","ts":1645197033.0483146,"msg":"using provided configuration","config_file":"/etc/caddy/Caddyfile","config_adapter":""}
Feb 18 10:10:33 cloud caddy[5464]: {"level":"warn","ts":1645197033.0636997,"msg":"input is not formatted with 'caddy fmt'","adapter":"caddyfile","file":"/etc/caddy/Caddyfile","line":7}
Feb 18 10:10:33 cloud caddy[5464]: {"level":"warn","ts":1645197033.067639,"logger":"caddy.logging.encoders.single_field","msg":"the 'single_field' encoder is deprecated and will be removed soon!"}
Feb 18 10:10:33 cloud caddy[5464]: {"level":"info","ts":1645197033.0703583,"logger":"admin","msg":"admin endpoint started","address":"tcp/localhost:2019","enforce_origin":false,"origins":["localhost:2019","[::1]:2019","127.0.0.1:2019"]}
Feb 18 10:10:33 cloud caddy[5464]: {"level":"info","ts":1645197033.0711641,"logger":"tls.cache.maintenance","msg":"started background certificate maintenance","cache":"0xc0001a87e0"}
Feb 18 10:10:33 cloud caddy[5464]: {"level":"info","ts":1645197033.0722432,"logger":"http","msg":"server is listening only on the HTTPS port but has no TLS connection policies; adding one to enable TLS","server_name":"srv0","https_port":443}
Feb 18 10:10:33 cloud caddy[5464]: {"level":"info","ts":1645197033.0726316,"logger":"http","msg":"enabling automatic HTTP->HTTPS redirects","server_name":"srv0"}
Feb 18 10:10:33 cloud caddy[5464]: {"level":"info","ts":1645197033.0752678,"logger":"tls","msg":"cleaning storage unit","description":"FileStorage:/var/lib/caddy/.local/share/caddy"}
Feb 18 10:10:33 cloud caddy[5464]: {"level":"info","ts":1645197033.0762537,"logger":"tls","msg":"finished cleaning storage units"}
Feb 18 10:10:33 cloud caddy[5464]: {"level":"debug","ts":1645197033.077,"logger":"http","msg":"starting server loop","address":"[::]:80","http3":false,"tls":false}
Feb 18 10:10:33 cloud caddy[5464]: {"level":"debug","ts":1645197033.0775847,"logger":"http","msg":"starting server loop","address":"[::]:443","http3":false,"tls":true}
Feb 18 10:10:33 cloud caddy[5464]: {"level":"info","ts":1645197033.0779214,"logger":"http","msg":"enabling automatic TLS certificate management","domains":["cloud.y8s.casa"]}
Feb 18 10:10:33 cloud caddy[5464]: {"level":"warn","ts":1645197033.0796323,"logger":"tls","msg":"stapling OCSP","error":"no OCSP stapling for [cloud.y8s.casa]: no OCSP server specified in certificate"}
Feb 18 10:10:33 cloud caddy[5464]: {"level":"debug","ts":1645197033.0800273,"logger":"tls.cache","msg":"added certificate to cache","subjects":["cloud.y8s.casa"],"expiration":1645179954,"managed":true,"issuer_key":"caddy.y8s.casa-acme-local-directory","hash":"d99effb6eea719a3624529d0b184b95c0e1dd28d7973f41ab71fd37a324d5ec8","cache_size":1,"cache_capacity":10000}
Feb 18 10:10:33 cloud caddy[5464]: {"level":"info","ts":1645197033.0811322,"logger":"tls.renew","msg":"acquiring lock","identifier":"cloud.y8s.casa"}
Feb 18 10:10:33 cloud caddy[5464]: {"level":"info","ts":1645197033.1011677,"msg":"autosaved config (load with --resume flag)","file":"/var/lib/caddy/.config/caddy/autosave.json"}
Feb 18 10:10:33 cloud caddy[5464]: {"level":"info","ts":1645197033.1019442,"msg":"serving initial configuration"}
Feb 18 10:10:33 cloud systemd[1]: Started Caddy.
Feb 18 10:10:33 cloud caddy[5464]: {"level":"info","ts":1645197033.1464634,"logger":"tls.renew","msg":"lock acquired","identifier":"cloud.y8s.casa"}
Feb 18 10:10:33 cloud caddy[5464]: {"level":"info","ts":1645197033.147103,"logger":"tls.renew","msg":"renewing certificate","identifier":"cloud.y8s.casa","remaining":-17079.147099447}
Feb 18 10:10:33 cloud caddy[5464]: {"level":"warn","ts":1645197033.1694987,"logger":"tls.issuance.acme.acme_client","msg":"HTTP request failed; retrying","url":"https://caddy.y8s.casa/acme/local/directory","error":"performing request: Get \"https://caddy.y8s.casa/acme/local/directory\": x509: certificate signed by unknown authority (possibly because of \"x509: ECDSA verification failure\" while trying to verify candidate authority certificate \"Caddy Local Authority - 2022 ECC Root\")"}
Feb 18 10:10:33 cloud caddy[5464]: {"level":"warn","ts":1645197033.4378345,"logger":"tls.issuance.acme.acme_client","msg":"HTTP request failed; retrying","url":"https://caddy.y8s.casa/acme/local/directory","error":"performing request: Get \"https://caddy.y8s.casa/acme/local/directory\": x509: certificate signed by unknown authority (possibly because of \"x509: ECDSA verification failure\" while trying to verify candidate authority certificate \"Caddy Local Authority - 2022 ECC Root\")"}
Feb 18 10:10:33 cloud caddy[5464]: {"level":"warn","ts":1645197033.7024932,"logger":"tls.issuance.acme.acme_client","msg":"HTTP request failed; retrying","url":"https://caddy.y8s.casa/acme/local/directory","error":"performing request: Get \"https://caddy.y8s.casa/acme/local/directory\": x509: certificate signed by unknown authority (possibly because of \"x509: ECDSA verification failure\" while trying to verify candidate authority certificate \"Caddy Local Authority - 2022 ECC Root\")"}
Feb 18 10:10:33 cloud caddy[5464]: {"level":"error","ts":1645197033.7027595,"logger":"tls.renew","msg":"could not get certificate from issuer","identifier":"cloud.y8s.casa","issuer":"caddy.y8s.casa-acme-local-directory","error":"registering account [] with server: provisioning client: performing request: Get \"https://caddy.y8s.casa/acme/local/directory\": x509: certificate signed by unknown authority (possibly because of \"x509: ECDSA verification failure\" while trying to verify candidate authority certificate \"Caddy Local Authority - 2022 ECC Root\")"}
Feb 18 10:10:33 cloud caddy[5464]: {"level":"error","ts":1645197033.702909,"logger":"tls.renew","msg":"will retry","error":"[cloud.y8s.casa] Renew: registering account [] with server: provisioning client: performing request: Get \"https://caddy.y8s.casa/acme/local/directory\": x509: certificate signed by unknown authority (possibly because of \"x509: ECDSA verification failure\" while trying to verify candidate authority certificate \"Caddy Local Authority - 2022 ECC Root\")","attempt":1,"retrying_in":60,"elapsed":0.556371264,"max_duration":2592000}
Feb 18 10:11:33 cloud caddy[5464]: {"level":"info","ts":1645197093.7066252,"logger":"tls.renew","msg":"renewing certificate","identifier":"cloud.y8s.casa","remaining":-17139.706064293}
Feb 18 10:11:33 cloud caddy[5464]: {"level":"warn","ts":1645197093.726447,"logger":"tls.issuance.acme.acme_client","msg":"HTTP request failed; retrying","url":"https://caddy.y8s.casa/acme/local/directory","error":"performing request: Get \"https://caddy.y8s.casa/acme/local/directory\": x509: certificate signed by unknown authority (possibly because of \"x509: ECDSA verification failure\" while trying to verify candidate authority certificate \"Caddy Local Authority - 2022 ECC Root\")"}
Feb 18 10:11:33 cloud caddy[5464]: {"level":"warn","ts":1645197093.9925673,"logger":"tls.issuance.acme.acme_client","msg":"HTTP request failed; retrying","url":"https://caddy.y8s.casa/acme/local/directory","error":"performing request: Get \"https://caddy.y8s.casa/acme/local/directory\": x509: certificate signed by unknown authority (possibly because of \"x509: ECDSA verification failure\" while trying to verify candidate authority certificate \"Caddy Local Authority - 2022 ECC Root\")"}
Feb 18 10:11:34 cloud caddy[5464]: {"level":"warn","ts":1645197094.2588966,"logger":"tls.issuance.acme.acme_client","msg":"HTTP request failed; retrying","url":"https://caddy.y8s.casa/acme/local/directory","error":"performing request: Get \"https://caddy.y8s.casa/acme/local/directory\": x509: certificate signed by unknown authority (possibly because of \"x509: ECDSA verification failure\" while trying to verify candidate authority certificate \"Caddy Local Authority - 2022 ECC Root\")"}
Feb 18 10:11:34 cloud caddy[5464]: {"level":"error","ts":1645197094.259805,"logger":"tls.renew","msg":"could not get certificate from issuer","identifier":"cloud.y8s.casa","issuer":"caddy.y8s.casa-acme-local-directory","error":"registering account [] with server: provisioning client: performing request: Get \"https://caddy.y8s.casa/acme/local/directory\": x509: certificate signed by unknown authority (possibly because of \"x509: ECDSA verification failure\" while trying to verify candidate authority certificate \"Caddy Local Authority - 2022 ECC Root\")"}
Feb 18 10:11:34 cloud caddy[5464]: {"level":"error","ts":1645197094.2604587,"logger":"tls.renew","msg":"will retry","error":"[cloud.y8s.casa] Renew: registering account [] with server: provisioning client: performing request: Get \"https://caddy.y8s.casa/acme/local/directory\": x509: certificate signed by unknown authority (possibly because of \"x509: ECDSA verification failure\" while trying to verify candidate authority certificate \"Caddy Local Authority - 2022 ECC Root\")","attempt":2,"retrying_in":120,"elapsed":61.113916498,"max_duration":2592000}
Okay well it looks like the backend doesn’t trust the cert for the caddy.y8s.casa domain of the frontend. Maybe you do need to re-copy the root cert from the frontend to the backend at this point, if at some point you did cause the frontend to regenerate the CA.
the root certs are identical. I even diffed them to make sure.
-rw------- 1 caddy caddy 627 Feb 17 20:54 /etc/ssl/certs/root.crt
permissions/ownership/location are correct? does it matter?
Sorry for chippin’ in late.
Since the creation of the guide I have had this issue where the certificate won’t renew properly. The ONLY way I could make this work was to delete on the front-end ~/.local/share/caddy/certificates/acme-v02.api.letsencrypt.org-directory and on the back-end ~/.local/share/caddy/certificates. After deleting both, reload Front-end and then Back-end and your good until it expires again.
Back when I wrote the guide I used a test environment using caddy start instead of systemd. I guessed that broke the setup and my intention was to restart but you know… time flies. SInce I’m stuck at home now anyway I decided to give it a spin.
First thing I tried was to drop caddy start and instead set it up with systemd. This worked until the certificate expired. So I deleted the folders again but now in /var/lib/caddy.... This didn’t work!
I thought I broke something else but when I use caddy start, it works again using the old data paths.
Then I noticed the same error message in the logs as found by @francislavoie here
I created a new root.crt and imported it to the back-end without a change.
Still looking…
Umm, make sure the caddy user has access to read the root CA cert – that might make the difference.
all the root certs on both machines are owned by caddy:caddy… I will try Robbert’s suggestion.
edit: caddy start didn’t really change things (used on both)
Did you copy the “new” CA root.crt to the the back-end?
Where did you copy it to? Is acme_ca_root pointing to that file?
oh right, for caddy start vs. systemd. here’s the frontend/backend logs after curl -v
2022/02/18 16:50:50.063 DEBUG tls.handshake choosing certificate {"identifier": "test.eastcapitol.us", "num_choices": 1}
2022/02/18 16:50:50.063 DEBUG tls.handshake default certificate selection results {"identifier": "test.eastcapitol.us", "subjects": ["test.eastcapitol.us"], "managed": true, "issuer_key": "acme-v02.api.letsencrypt.org-directory", "hash": "81b95004aa93fcad377af0951c86da3d56e46209bbb7920fd16bd5604467d5ee"}
2022/02/18 16:50:50.063 DEBUG tls.handshake matched certificate in cache {"subjects": ["test.eastcapitol.us"], "managed": true, "expiration": "2022/05/19 15:17:51.000", "hash": "81b95004aa93fcad377af0951c86da3d56e46209bbb7920fd16bd5604467d5ee"}
2022/02/18 16:50:50.152 DEBUG http.handlers.reverse_proxy upstream roundtrip {"upstream": "cloud.y8s.casa:443", "duration": 0.019996936, "request": {"remote_addr": "173.8.14.69:64838", "proto": "HTTP/1.1", "method": "GET", "host": "cloud.y8s.casa:443", "uri": "/", "headers": {"X-Forwarded-Proto": ["https"], "X-Forwarded-Host": ["test.eastcapitol.us"], "X-Forwarded-For": ["173.8.14.69"], "User-Agent": ["curl/7.79.1"], "Accept": ["*/*"]}, "tls": {"resumed": false, "version": 771, "cipher_suite": 49195, "proto": "http/1.1", "proto_mutual": true, "server_name": "test.eastcapitol.us"}}, "headers": {"Server": ["Caddy"], "Strict-Transport-Security": ["max-age=31536000;"], "Content-Length": ["0"], "Date": ["Fri, 18 Feb 2022 16:50:50 GMT"]}, "status": 502}
backend:
2022/02/18 16:50:50.146 DEBUG http.handlers.rewrite rewrote request {"request": {"remote_addr": "10.10.10.40:60654", "proto": "HTTP/2.0", "method": "GET", "host": "cloud.y8s.casa:443", "uri": "/", "headers": {"X-Forwarded-Proto": ["https"], "X-Forwarded-Host": ["test.eastcapitol.us"], "X-Forwarded-For": ["173.8.14.69"], "User-Agent": ["curl/7.79.1"], "Accept": ["*/*"], "Accept-Encoding": ["gzip"]}, "tls": {"resumed": false, "version": 772, "cipher_suite": 4867, "proto": "h2", "proto_mutual": true, "server_name": "cloud.y8s.casa"}}, "method": "GET", "uri": "/index.php"}
2022/02/18 16:50:50.153 DEBUG http.reverse_proxy.transport.fastcgi roundtrip {"request": {"remote_addr": "10.10.10.40:60654", "proto": "HTTP/2.0", "method": "GET", "host": "cloud.y8s.casa:443", "uri": "/index.php", "headers": {"X-Forwarded-Host": ["test.eastcapitol.us"], "X-Forwarded-For": ["173.8.14.69, 10.10.10.40"], "User-Agent": ["curl/7.79.1"], "Accept": ["*/*"], "Accept-Encoding": ["gzip"], "X-Forwarded-Proto": ["https"]}, "tls": {"resumed": false, "version": 772, "cipher_suite": 4867, "proto": "h2", "proto_mutual": true, "server_name": "cloud.y8s.casa"}}, "dial": "127.0.0.1:9000", "env": {"AUTH_TYPE":"","CONTENT_LENGTH":"","CONTENT_TYPE":"","DOCUMENT_ROOT":"/var/www/nextcloud","DOCUMENT_URI":"/index.php","GATEWAY_INTERFACE":"CGI/1.1","HTTPS":"on","HTTP_ACCEPT":"*/*","HTTP_ACCEPT_ENCODING":"gzip","HTTP_HOST":"cloud.y8s.casa:443","HTTP_USER_AGENT":"curl/7.79.1","HTTP_X_FORWARDED_FOR":"173.8.14.69, 10.10.10.40","HTTP_X_FORWARDED_HOST":"test.eastcapitol.us","HTTP_X_FORWARDED_PROTO":"https","PATH":"/bin","PATH_INFO":"","QUERY_STRING":"","REMOTE_ADDR":"10.10.10.40","REMOTE_HOST":"10.10.10.40","REMOTE_IDENT":"","REMOTE_PORT":"60654","REMOTE_USER":"","REQUEST_METHOD":"GET","REQUEST_SCHEME":"https","REQUEST_URI":"/","SCRIPT_FILENAME":"/var/www/nextcloud/index.php","SCRIPT_NAME":"/index.php","SERVER_NAME":"cloud.y8s.casa","SERVER_PORT":"443","SERVER_PROTOCOL":"HTTP/2.0","SERVER_SOFTWARE":"Caddy/v2.4.6","SSL_CIPHER":"TLS_CHACHA20_POLY1305_SHA256","SSL_PROTOCOL":"TLSv1.3"}}
2022/02/18 16:50:50.157 DEBUG http.handlers.reverse_proxy upstream roundtrip {"upstream": "127.0.0.1:9000", "duration": 0.004760661, "request": {"remote_addr": "10.10.10.40:60654", "proto": "HTTP/2.0", "method": "GET", "host": "cloud.y8s.casa:443", "uri": "/index.php", "headers": {"Accept-Encoding": ["gzip"], "X-Forwarded-Proto": ["https"], "X-Forwarded-Host": ["test.eastcapitol.us"], "X-Forwarded-For": ["173.8.14.69, 10.10.10.40"], "User-Agent": ["curl/7.79.1"], "Accept": ["*/*"]}, "tls": {"resumed": false, "version": 772, "cipher_suite": 4867, "proto": "h2", "proto_mutual": true, "server_name": "cloud.y8s.casa"}}, "error": "dialing backend: dial tcp 127.0.0.1:9000: connect: connection refused"}
2022/02/18 16:50:50.158 ERROR http.log.error.log0 dialing backend: dial tcp 127.0.0.1:9000: connect: connection refused {"request": {"remote_addr": "10.10.10.40:60654", "proto": "HTTP/2.0", "method": "GET", "host": "cloud.y8s.casa:443", "uri": "/", "headers": {"X-Forwarded-Proto": ["https"], "X-Forwarded-Host": ["test.eastcapitol.us"], "X-Forwarded-For": ["173.8.14.69"], "User-Agent": ["curl/7.79.1"], "Accept": ["*/*"], "Accept-Encoding": ["gzip"]}, "tls": {"resumed": false, "version": 772, "cipher_suite": 4867, "proto": "h2", "proto_mutual": true, "server_name": "cloud.y8s.casa"}}, "duration": 0.013550281, "status": 502, "err_id": "sf7xgtpdm", "err_trace": "reverseproxy.statusError (reverseproxy.go:886)"}
2022/02/18 16:50:50.158 ERROR http.log.access.log0 handled request {"request": {"remote_addr": "10.10.10.40:60654", "proto": "HTTP/2.0", "method": "GET", "host": "cloud.y8s.casa:443", "uri": "/", "headers": {"X-Forwarded-Host": ["test.eastcapitol.us"], "X-Forwarded-For": ["173.8.14.69"], "User-Agent": ["curl/7.79.1"], "Accept": ["*/*"], "Accept-Encoding": ["gzip"], "X-Forwarded-Proto": ["https"]}, "tls": {"resumed": false, "version": 772, "cipher_suite": 4867, "proto": "h2", "proto_mutual": true, "server_name": "cloud.y8s.casa"}}, "common_log": "10.10.10.40 - - [18/Feb/2022:11:50:50 -0500] \"GET / HTTP/2.0\" 502 0", "user_id": "", "duration": 0.013550281, "size": 0, "status": 502, "resp_headers": {"Server": ["Caddy"], "Strict-Transport-Security": ["max-age=31536000;"]}}
seems like progress with the certs, but now something else is causing issues. Maybe nextcloud config.
Yeah, that reverse_proxy issue seems to suggest the TLS connection between the frontend and backend has now succeeded, but your backend to your upstream app is failing. Make sure you have PHP-FPM running I guess.
I can’t get it to work through systemd 
I started from scratch with 2 brand new deployed Debian based VM’s, installed Caddy and created /etc/caddy/Caddyfile, copied from the front-end /var/lib/caddy/.local/share/caddy/pki/authorities/local/root.crt to the back-end location indicated in the back-end Caddyfile
Feb 18 19:32:57 RJ-Caddy caddy[2203]: {"level":"error","ts":1645209177.2372653,"logger":"http.log.error","msg":"x509: certificate signed by unknown authority (possibly because of \"x509: ECDSA verification failure\" while trying to verify candidate authority certificate \"Caddy Local Authority - 2022 ECC Root\""request":{"remote_addr":"192.168.2.200:52150","proto":"HTTP/2.0","method":"POST","host":"bpass.robbert.com","uri":"/identity/connect/token","headers":{"Accept":["application/json"],"Accept-Language":["en-GB,en;q=0.5"],"Pragma":["no-cache"],"Accept-Encoding":["gzip, deflate, br"],"Device-Type":["3"],"Origin":["moz-ension://fa1d4d8a-3175-4671-9c06-8150bf64cb83"],"Sec-Fetch-Dest":["empty"],"Sec-Fetch-Mode":["cors"],"Cache-Control":["no-cache"],"User-Agent":["Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:96.0) Gecko/20100101 Firefox/96.0"],"Content-Type":["application/x-www-form-urlencoded; charset=utf-8"],"Content-Length":[1"],"Sec-Fetch-Site":["same-origin"],"Te":["trailers"]},"tls":{"resumed":false,"version":772,"cipher_suite":4865,"proto":"h2","proto_mutual":true,"server_name":"bpass.robbert.com"}},"duration":0.007062436,"status":502,"err_id":"bizb4sy73","err_trace":"reverseproxy.statusError (reverseproxy.go:886)"}
Then I copied /etc/caddy/Caddyfile on font-end and back-end, first started front-end to copy the root.crt, then started the back-end… working.
It seems that the root CA is not matching.
Where did you put the root cert, exactly? Does the entire chain of parent directories have executable permission such that the caddy user can see it?
Forgot to mention that.
Originally I have the root CA under /etc/ssl/certs/root.crt I checked that path and user caddy has the correct user rights to get there.
But as a test, I also copied the CA to /var/lib/caddy/root.cert and chmod 777 And of course adjusted /etc/caddy/Caddyfile to make it point to the new location acme_ca_root /var/lib/caddy/root.crt
Same results in both ways.
Okay – are you sure you copied the right root cert from the frontend? If you’re running as a systemd service then you should be grabbing /var/lib/caddy/.local/share/caddy/pki/authorities/local/root.crt and not the one in your own user’s HOME (as would be generated/used if you ran caddy start)
Yep, that’s exactly what I did.
btw, on the back-end in /var/lib/caddy/.local/share/caddy/certificates/acme.orion-acme-local-directory a certificate is created. Also when I delete acme.orion-acme-local-directory and systemctl restart caddy this folder is recreated with the certificate inside.
Would that happen anyway?
this is the log from the back-end when after restarting caddy.
root@RJ-Cloud .../caddy/certificates# systemctl start caddy; journalctl -fu caddy
-- Logs begin at Fri 2022-02-18 17:40:23 CET. --
Feb 19 00:31:04 RJ-Cloud caddy[11573]: {"level":"debug","ts":1645227064.0857167,"logger":"http","msg":"starting server loop","address":"[::]:443","http3":false,"tls":true}
Feb 19 00:31:04 RJ-Cloud caddy[11573]: {"level":"debug","ts":1645227064.0858264,"logger":"http","msg":"starting server loop","address":"[::]:80","http3":false,"tls":false}
Feb 19 00:31:04 RJ-Cloud caddy[11573]: {"level":"info","ts":1645227064.0858436,"logger":"http","msg":"enabling automatic TLS certificate management","domains":["nextcloud.roadrunner","bitwarden.roadrunner"]}
Feb 19 00:31:04 RJ-Cloud caddy[11573]: {"level":"info","ts":1645227064.086441,"msg":"autosaved config (load with --resume flag)","file":"/var/lib/caddy/.config/caddy/autosave.json"}
Feb 19 00:31:04 RJ-Cloud systemd[1]: Started Caddy.
Feb 19 00:31:04 RJ-Cloud caddy[11573]: {"level":"info","ts":1645227064.0881436,"logger":"tls.obtain","msg":"acquiring lock","identifier":"nextcloud.roadrunner"}
Feb 19 00:31:04 RJ-Cloud caddy[11573]: {"level":"info","ts":1645227064.0886598,"msg":"serving initial configuration"}
Feb 19 00:31:04 RJ-Cloud caddy[11573]: {"level":"info","ts":1645227064.088732,"logger":"tls","msg":"cleaning storage unit","description":"FileStorage:/var/lib/caddy/.local/share/caddy"}
Feb 19 00:31:04 RJ-Cloud caddy[11573]: {"level":"info","ts":1645227064.0888329,"logger":"tls","msg":"finished cleaning storage units"}
Feb 19 00:31:04 RJ-Cloud caddy[11573]: {"level":"info","ts":1645227064.089314,"logger":"tls.obtain","msg":"acquiring lock","identifier":"bitwarden.roadrunner"}
Feb 19 00:31:04 RJ-Cloud caddy[11573]: {"level":"info","ts":1645227064.1030514,"logger":"tls.obtain","msg":"lock acquired","identifier":"nextcloud.roadrunner"}
Feb 19 00:31:04 RJ-Cloud caddy[11573]: {"level":"debug","ts":1645227064.1045785,"logger":"tls.obtain","msg":"trying issuer 1/1","issuer":"acme.roadrunner-acme-local-directory"}
Feb 19 00:31:04 RJ-Cloud caddy[11573]: {"level":"info","ts":1645227064.1065798,"logger":"tls.issuance.acme","msg":"waiting on internal rate limiter","identifiers":["nextcloud.roadrunner"],"ca":"https://acme.roadrunner/acme/local/directory","account":""}
Feb 19 00:31:04 RJ-Cloud caddy[11573]: {"level":"info","ts":1645227064.1070085,"logger":"tls.obtain","msg":"lock acquired","identifier":"bitwarden.roadrunner"}
Feb 19 00:31:04 RJ-Cloud caddy[11573]: {"level":"info","ts":1645227064.1070871,"logger":"tls.issuance.acme","msg":"done waiting on internal rate limiter","identifiers":["nextcloud.roadrunner"],"ca":"https://acme.roadrunner/acme/local/directory","account":""}
Feb 19 00:31:04 RJ-Cloud caddy[11573]: {"level":"debug","ts":1645227064.1076863,"logger":"tls.obtain","msg":"trying issuer 1/1","issuer":"acme.roadrunner-acme-local-directory"}
Feb 19 00:31:04 RJ-Cloud caddy[11573]: {"level":"info","ts":1645227064.108724,"logger":"tls.issuance.acme","msg":"waiting on internal rate limiter","identifiers":["bitwarden.roadrunner"],"ca":"https://acme.roadrunner/acme/local/directory","account":""}
Feb 19 00:31:04 RJ-Cloud caddy[11573]: {"level":"info","ts":1645227064.1087651,"logger":"tls.issuance.acme","msg":"done waiting on internal rate limiter","identifiers":["bitwarden.roadrunner"],"ca":"https://acme.roadrunner/acme/local/directory","account":""}
Feb 19 00:31:04 RJ-Cloud caddy[11573]: {"level":"debug","ts":1645227064.1184716,"logger":"tls.handshake","msg":"no matching certificates and no custom selection logic","identifier":"192.168.2.4"}
Feb 19 00:31:04 RJ-Cloud caddy[11573]: {"level":"debug","ts":1645227064.1185207,"logger":"tls.handshake","msg":"no certificate matching TLS ClientHello","server_name":"","remote":"192.168.2.200:53795","identifier":"192.168.2.4","cipher_suites":[49200,49196,49192,49188,49172,49162,165,163,161,159,107,106,105,104,57,56,55,54,49202,49198,49194,49190,49167,49157,157,61,53,136,135,134,133,132,49199,49195,49191,49187,49171,49161,164,162,160,158,103,64,63,62,51,50,49,48,49201,49197,49193,49189,49166,49156,156,60,47,154,153,152,151,69,68,67,66,150,65,7,49169,49159,49164,49154,5,4,49170,49160,22,19,16,13,49165,49155,10,21,18,15,12,9,20,17,8,6,3,255],"cert_cache_fill":0,"load_if_necessary":true,"obtain_if_necessary":true,"on_demand":false}
Feb 19 00:31:04 RJ-Cloud caddy[11573]: {"level":"debug","ts":1645227064.1186545,"logger":"http.stdlib","msg":"http: TLS handshake error from 192.168.2.200:53795: no certificate available for '192.168.2.4'"}
Feb 19 00:31:04 RJ-Cloud caddy[11573]: {"level":"debug","ts":1645227064.1202247,"logger":"tls.issuance.acme.acme_client","msg":"http request","method":"GET","url":"https://acme.roadrunner/acme/local/directory","headers":{"User-Agent":["Caddy/2.4.6 CertMagic acmez (linux; amd64)"]},"response_headers":{"Content-Length":["277"],"Content-Type":["application/json"],"Date":["Fri, 18 Feb 2022 23:31:04 GMT"],"Server":["Caddy"]},"status_code":200}
Feb 19 00:31:04 RJ-Cloud caddy[11573]: {"level":"debug","ts":1645227064.1335292,"logger":"tls.handshake","msg":"no matching certificates and no custom selection logic","identifier":"192.168.2.4"}
Feb 19 00:31:04 RJ-Cloud caddy[11573]: {"level":"debug","ts":1645227064.1339169,"logger":"tls.handshake","msg":"no certificate matching TLS ClientHello","server_name":"","remote":"192.168.2.200:53796","identifier":"192.168.2.4","cipher_suites":[49200,49196,49192,49188,49172,49162,165,163,161,159,107,106,105,104,57,56,55,54,49202,49198,49194,49190,49167,49157,157,61,53,136,135,134,133,132,49199,49195,49191,49187,49171,49161,164,162,160,158,103,64,63,62,51,50,49,48,49201,49197,49193,49189,49166,49156,156,60,47,154,153,152,151,69,68,67,66,150,65,7,49169,49159,49164,49154,5,4,49170,49160,22,19,16,13,49165,49155,10,21,18,15,12,9,20,17,8,6,3,255],"cert_cache_fill":0,"load_if_necessary":true,"obtain_if_necessary":true,"on_demand":false}
Feb 19 00:31:04 RJ-Cloud caddy[11573]: {"level":"debug","ts":1645227064.1343658,"logger":"http.stdlib","msg":"http: TLS handshake error from 192.168.2.200:53796: no certificate available for '192.168.2.4'"}
Feb 19 00:31:04 RJ-Cloud caddy[11573]: {"level":"debug","ts":1645227064.1373436,"logger":"tls.issuance.acme.acme_client","msg":"http request","method":"HEAD","url":"https://acme.roadrunner/acme/local/new-nonce","headers":{"User-Agent":["Caddy/2.4.6 CertMagic acmez (linux; amd64)"]},"response_headers":{"Cache-Control":["no-store"],"Date":["Fri, 18 Feb 2022 23:31:04 GMT"],"Link":["<https://acme.roadrunner/acme/local/directory>;rel=\"index\""],"Replay-Nonce":["c0xPaldWMjl5ckJ1MWd2V09lRUtoMU9hRmlHZ2tLZzE"],"Server":["Caddy"]},"status_code":200}
Feb 19 00:31:04 RJ-Cloud caddy[11573]: {"level":"debug","ts":1645227064.1460996,"logger":"tls.issuance.acme.acme_client","msg":"http request","method":"HEAD","url":"https://acme.roadrunner/acme/local/new-nonce","headers":{"User-Agent":["Caddy/2.4.6 CertMagic acmez (linux; amd64)"]},"response_headers":{"Cache-Control":["no-store"],"Date":["Fri, 18 Feb 2022 23:31:04 GMT"],"Link":["<https://acme.roadrunner/acme/local/directory>;rel=\"index\""],"Replay-Nonce":["MnBNUGNBZTc0TnNkUjBNTmxvMXZvWWgxMmJYNTg5NUo"],"Server":["Caddy"]},"status_code":200}
Feb 19 00:31:04 RJ-Cloud caddy[11573]: {"level":"debug","ts":1645227064.1487856,"logger":"http.stdlib","msg":"http: TLS handshake error from 192.168.2.200:53797: tls: client offered only unsupported versions: [301]"}
Feb 19 00:31:04 RJ-Cloud caddy[11573]: {"level":"debug","ts":1645227064.1633446,"logger":"http.stdlib","msg":"http: TLS handshake error from 192.168.2.200:53798: tls: client offered only unsupported versions: []"}
Feb 19 00:31:04 RJ-Cloud caddy[11573]: {"level":"debug","ts":1645227064.1804004,"logger":"http.stdlib","msg":"http: TLS handshake error from 192.168.2.200:53799: tls: unsupported SSLv2 handshake received"}
Feb 19 00:31:04 RJ-Cloud caddy[11573]: {"level":"debug","ts":1645227064.2733192,"logger":"tls.issuance.acme.acme_client","msg":"http request","method":"POST","url":"https://acme.roadrunner/acme/local/new-order","headers":{"Content-Type":["application/jose+json"],"User-Agent":["Caddy/2.4.6 CertMagic acmez (linux; amd64)"]},"response_headers":{"Cache-Control":["no-store"],"Content-Length":["401"],"Content-Type":["application/json"],"Date":["Fri, 18 Feb 2022 23:31:04 GMT"],"Link":["<https://acme.roadrunner/acme/local/directory>;rel=\"index\""],"Location":["https://acme.roadrunner/acme/local/order/d1v473zZYKMZIVI1oNrqsLcfWCnmtBxm"],"Replay-Nonce":["c3Zpb1ZodlVtSnNhNkMzY25HYTU2SWRmNmdEckUyeWQ"],"Server":["Caddy"]},"status_code":201}
Feb 19 00:31:04 RJ-Cloud caddy[11573]: {"level":"debug","ts":1645227064.290564,"logger":"tls.issuance.acme.acme_client","msg":"http request","method":"POST","url":"https://acme.roadrunner/acme/local/new-order","headers":{"Content-Type":["application/jose+json"],"User-Agent":["Caddy/2.4.6 CertMagic acmez (linux; amd64)"]},"response_headers":{"Cache-Control":["no-store"],"Content-Length":["401"],"Content-Type":["application/json"],"Date":["Fri, 18 Feb 2022 23:31:04 GMT"],"Link":["<https://acme.roadrunner/acme/local/directory>;rel=\"index\""],"Location":["https://acme.roadrunner/acme/local/order/PVxs95JaGzVPDuRCbh7gsXBxKOpwh3AC"],"Replay-Nonce":["bEtSRUNkWmJzelZPY21VcTFKSXdydXJtcW9mamZRUHo"],"Server":["Caddy"]},"status_code":201}
Feb 19 00:31:04 RJ-Cloud caddy[11573]: {"level":"debug","ts":1645227064.3077621,"logger":"tls.issuance.acme.acme_client","msg":"http request","method":"POST","url":"https://acme.roadrunner/acme/local/authz/R1dSppm7VVnTff719S5d5u3foxx23SY2","headers":{"Content-Type":["application/jose+json"],"User-Agent":["Caddy/2.4.6 CertMagic acmez (linux; amd64)"]},"response_headers":{"Cache-Control":["no-store"],"Content-Length":["728"],"Content-Type":["application/json"],"Date":["Fri, 18 Feb 2022 23:31:04 GMT"],"Link":["<https://acme.roadrunner/acme/local/directory>;rel=\"index\""],"Location":["https://acme.roadrunner/acme/local/authz/R1dSppm7VVnTff719S5d5u3foxx23SY2"],"Replay-Nonce":["OFRXc2tUM2tZdVN1MmFZM0FZSXpkbzVQM055R1hkcWo"],"Server":["Caddy"]},"status_code":200}
Feb 19 00:31:04 RJ-Cloud caddy[11573]: {"level":"debug","ts":1645227064.3083925,"logger":"tls.issuance.acme.acme_client","msg":"no solver configured","challenge_type":"dns-01"}
Feb 19 00:31:04 RJ-Cloud caddy[11573]: {"level":"info","ts":1645227064.308452,"logger":"tls.issuance.acme.acme_client","msg":"trying to solve challenge","identifier":"nextcloud.roadrunner","challenge_type":"http-01","ca":"https://acme.roadrunner/acme/local/directory"}
Feb 19 00:31:04 RJ-Cloud caddy[11573]: {"level":"debug","ts":1645227064.3237603,"logger":"tls.issuance.acme.acme_client","msg":"http request","method":"POST","url":"https://acme.roadrunner/acme/local/authz/Lb2LJXu8lD8lSuESSjpdjpLSZfZ3HaKo","headers":{"Content-Type":["application/jose+json"],"User-Agent":["Caddy/2.4.6 CertMagic acmez (linux; amd64)"]},"response_headers":{"Cache-Control":["no-store"],"Content-Length":["728"],"Content-Type":["application/json"],"Date":["Fri, 18 Feb 2022 23:31:04 GMT"],"Link":["<https://acme.roadrunner/acme/local/directory>;rel=\"index\""],"Location":["https://acme.roadrunner/acme/local/authz/Lb2LJXu8lD8lSuESSjpdjpLSZfZ3HaKo"],"Replay-Nonce":["MGVZQXczNEdnYkhWeUlEVzRyODZjWENhMlcyVDZlZ0I"],"Server":["Caddy"]},"status_code":200}
Feb 19 00:31:04 RJ-Cloud caddy[11573]: {"level":"debug","ts":1645227064.3244915,"logger":"tls.issuance.acme.acme_client","msg":"no solver configured","challenge_type":"dns-01"}
Feb 19 00:31:04 RJ-Cloud caddy[11573]: {"level":"info","ts":1645227064.3248203,"logger":"tls.issuance.acme.acme_client","msg":"trying to solve challenge","identifier":"bitwarden.roadrunner","challenge_type":"http-01","ca":"https://acme.roadrunner/acme/local/directory"}
Feb 19 00:31:04 RJ-Cloud caddy[11573]: {"level":"info","ts":1645227064.3447092,"logger":"tls.issuance.acme","msg":"served key authentication","identifier":"nextcloud.roadrunner","challenge":"http-01","remote":"192.168.2.2:47296","distributed":false}
Feb 19 00:31:04 RJ-Cloud caddy[11573]: {"level":"info","ts":1645227064.3616967,"logger":"tls.issuance.acme","msg":"served key authentication","identifier":"bitwarden.roadrunner","challenge":"http-01","remote":"192.168.2.2:47298","distributed":false}
Feb 19 00:31:04 RJ-Cloud caddy[11573]: {"level":"debug","ts":1645227064.3661535,"logger":"tls.issuance.acme.acme_client","msg":"http request","method":"POST","url":"https://acme.roadrunner/acme/local/challenge/R1dSppm7VVnTff719S5d5u3foxx23SY2/RTnNZngOzsRNGaB1baM2BgtnC65uKLvN","headers":{"Content-Type":["application/jose+json"],"User-Agent":["Caddy/2.4.6 CertMagic acmez (linux; amd64)"]},"response_headers":{"Cache-Control":["no-store"],"Content-Length":["228"],"Content-Type":["application/json"],"Date":["Fri, 18 Feb 2022 23:31:04 GMT"],"Link":["<https://acme.roadrunner/acme/local/directory>;rel=\"index\"","<https://acme.roadrunner/acme/local/authz/R1dSppm7VVnTff719S5d5u3foxx23SY2>;rel=\"up\""],"Location":["https://acme.roadrunner/acme/local/challenge/R1dSppm7VVnTff719S5d5u3foxx23SY2/RTnNZngOzsRNGaB1baM2BgtnC65uKLvN"],"Replay-Nonce":["UzZjMmQ4OUtRM0M2cEx6QUs2cVViQUk2SVZDTkNkOHk"],"Server":["Caddy"]},"status_code":200}
Feb 19 00:31:04 RJ-Cloud caddy[11573]: {"level":"debug","ts":1645227064.3669584,"logger":"tls.issuance.acme.acme_client","msg":"challenge accepted","identifier":"nextcloud.roadrunner","challenge_type":"http-01"}
Feb 19 00:31:04 RJ-Cloud caddy[11573]: {"level":"debug","ts":1645227064.3762105,"logger":"tls.issuance.acme.acme_client","msg":"http request","method":"POST","url":"https://acme.roadrunner/acme/local/challenge/Lb2LJXu8lD8lSuESSjpdjpLSZfZ3HaKo/taIYxhEceOrRKMiCWFq0jyekEm1QWBmY","headers":{"Content-Type":["application/jose+json"],"User-Agent":["Caddy/2.4.6 CertMagic acmez (linux; amd64)"]},"response_headers":{"Cache-Control":["no-store"],"Content-Length":["228"],"Content-Type":["application/json"],"Date":["Fri, 18 Feb 2022 23:31:04 GMT"],"Link":["<https://acme.roadrunner/acme/local/directory>;rel=\"index\"","<https://acme.roadrunner/acme/local/authz/Lb2LJXu8lD8lSuESSjpdjpLSZfZ3HaKo>;rel=\"up\""],"Location":["https://acme.roadrunner/acme/local/challenge/Lb2LJXu8lD8lSuESSjpdjpLSZfZ3HaKo/taIYxhEceOrRKMiCWFq0jyekEm1QWBmY"],"Replay-Nonce":["d1lveFlmWFg1TkEyR0NFS0xGT1FzNG5vNWJ1bFJodnQ"],"Server":["Caddy"]},"status_code":200}
Feb 19 00:31:04 RJ-Cloud caddy[11573]: {"level":"debug","ts":1645227064.376402,"logger":"tls.issuance.acme.acme_client","msg":"challenge accepted","identifier":"bitwarden.roadrunner","challenge_type":"http-01"}
Feb 19 00:31:04 RJ-Cloud caddy[11573]: {"level":"debug","ts":1645227064.6462443,"logger":"tls.issuance.acme.acme_client","msg":"http request","method":"POST","url":"https://acme.roadrunner/acme/local/authz/R1dSppm7VVnTff719S5d5u3foxx23SY2","headers":{"Content-Type":["application/jose+json"],"User-Agent":["Caddy/2.4.6 CertMagic acmez (linux; amd64)"]},"response_headers":{"Cache-Control":["no-store"],"Content-Length":["759"],"Content-Type":["application/json"],"Date":["Fri, 18 Feb 2022 23:31:04 GMT"],"Link":["<https://acme.roadrunner/acme/local/directory>;rel=\"index\""],"Location":["https://acme.roadrunner/acme/local/authz/R1dSppm7VVnTff719S5d5u3foxx23SY2"],"Replay-Nonce":["UFRLR2tKNk1rcWlhY25NY3hTNjNFcXhPTzBqeGFWc2Q"],"Server":["Caddy"]},"status_code":200}
Feb 19 00:31:04 RJ-Cloud caddy[11573]: {"level":"info","ts":1645227064.6480546,"logger":"tls.issuance.acme.acme_client","msg":"validations succeeded; finalizing order","order":"https://acme.roadrunner/acme/local/order/d1v473zZYKMZIVI1oNrqsLcfWCnmtBxm"}
Feb 19 00:31:04 RJ-Cloud caddy[11573]: {"level":"debug","ts":1645227064.690357,"logger":"tls.issuance.acme.acme_client","msg":"http request","method":"POST","url":"https://acme.roadrunner/acme/local/authz/Lb2LJXu8lD8lSuESSjpdjpLSZfZ3HaKo","headers":{"Content-Type":["application/jose+json"],"User-Agent":["Caddy/2.4.6 CertMagic acmez (linux; amd64)"]},"response_headers":{"Cache-Control":["no-store"],"Content-Length":["759"],"Content-Type":["application/json"],"Date":["Fri, 18 Feb 2022 23:31:04 GMT"],"Link":["<https://acme.roadrunner/acme/local/directory>;rel=\"index\""],"Location":["https://acme.roadrunner/acme/local/authz/Lb2LJXu8lD8lSuESSjpdjpLSZfZ3HaKo"],"Replay-Nonce":["ckw2SU8ydnNKRTBFZGRLdFd0MUtOZENXeHlpNmxsbXI"],"Server":["Caddy"]},"status_code":200}
Feb 19 00:31:04 RJ-Cloud caddy[11573]: {"level":"info","ts":1645227064.690859,"logger":"tls.issuance.acme.acme_client","msg":"validations succeeded; finalizing order","order":"https://acme.roadrunner/acme/local/order/PVxs95JaGzVPDuRCbh7gsXBxKOpwh3AC"}
Feb 19 00:31:04 RJ-Cloud caddy[11573]: {"level":"debug","ts":1645227064.7605941,"logger":"tls.issuance.acme.acme_client","msg":"http request","method":"POST","url":"https://acme.roadrunner/acme/local/order/d1v473zZYKMZIVI1oNrqsLcfWCnmtBxm/finalize","headers":{"Content-Type":["application/jose+json"],"User-Agent":["Caddy/2.4.6 CertMagic acmez (linux; amd64)"]},"response_headers":{"Cache-Control":["no-store"],"Content-Length":["490"],"Content-Type":["application/json"],"Date":["Fri, 18 Feb 2022 23:31:04 GMT"],"Link":["<https://acme.roadrunner/acme/local/directory>;rel=\"index\""],"Location":["https://acme.roadrunner/acme/local/order/d1v473zZYKMZIVI1oNrqsLcfWCnmtBxm"],"Replay-Nonce":["SUkxaFRHNml0Tm1OT3BCZ0JpbkEwQzRNTFduT0lhQW4"],"Server":["Caddy"]},"status_code":200}
Feb 19 00:31:04 RJ-Cloud caddy[11573]: {"level":"debug","ts":1645227064.8049784,"logger":"tls.issuance.acme.acme_client","msg":"http request","method":"POST","url":"https://acme.roadrunner/acme/local/certificate/52ZfAfyRifgNaqVcfSgp0TkNNzmJlRgU","headers":{"Content-Type":["application/jose+json"],"User-Agent":["Caddy/2.4.6 CertMagic acmez (linux; amd64)"]},"response_headers":{"Cache-Control":["no-store"],"Content-Length":["1393"],"Content-Type":["application/pem-certificate-chain; charset=utf-8"],"Date":["Fri, 18 Feb 2022 23:31:04 GMT"],"Link":["<https://acme.roadrunner/acme/local/directory>;rel=\"index\""],"Replay-Nonce":["VFJIRTVqSzlicGNITFgwa2s1NkJ3SU9hbUg2a0NKT3Q"],"Server":["Caddy"]},"status_code":200}
Feb 19 00:31:04 RJ-Cloud caddy[11573]: {"level":"info","ts":1645227064.8051486,"logger":"tls.issuance.acme.acme_client","msg":"successfully downloaded available certificate chains","count":1,"first_url":"https://acme.roadrunner/acme/local/certificate/52ZfAfyRifgNaqVcfSgp0TkNNzmJlRgU"}
Feb 19 00:31:04 RJ-Cloud caddy[11573]: {"level":"info","ts":1645227064.8059049,"logger":"tls.obtain","msg":"certificate obtained successfully","identifier":"nextcloud.roadrunner"}
Feb 19 00:31:04 RJ-Cloud caddy[11573]: {"level":"info","ts":1645227064.8059373,"logger":"tls.obtain","msg":"releasing lock","identifier":"nextcloud.roadrunner"}
Feb 19 00:31:04 RJ-Cloud caddy[11573]: {"level":"warn","ts":1645227064.8069315,"logger":"tls","msg":"stapling OCSP","error":"no OCSP stapling for [nextcloud.roadrunner]: no OCSP server specified in certificate"}
Feb 19 00:31:04 RJ-Cloud caddy[11573]: {"level":"debug","ts":1645227064.806975,"logger":"tls.cache","msg":"added certificate to cache","subjects":["nextcloud.roadrunner"],"expiration":1645270264,"managed":true,"issuer_key":"acme.roadrunner-acme-local-directory","hash":"5afb0ba072dc727a144f96afb83b1dceeb041e606ba1485018bef826f1dafb94","cache_size":1,"cache_capacity":10000}
Feb 19 00:31:04 RJ-Cloud caddy[11573]: {"level":"debug","ts":1645227064.8126915,"logger":"tls.issuance.acme.acme_client","msg":"http request","method":"POST","url":"https://acme.roadrunner/acme/local/order/PVxs95JaGzVPDuRCbh7gsXBxKOpwh3AC/finalize","headers":{"Content-Type":["application/jose+json"],"User-Agent":["Caddy/2.4.6 CertMagic acmez (linux; amd64)"]},"response_headers":{"Cache-Control":["no-store"],"Content-Length":["490"],"Content-Type":["application/json"],"Date":["Fri, 18 Feb 2022 23:31:04 GMT"],"Link":["<https://acme.roadrunner/acme/local/directory>;rel=\"index\""],"Location":["https://acme.roadrunner/acme/local/order/PVxs95JaGzVPDuRCbh7gsXBxKOpwh3AC"],"Replay-Nonce":["ZlVxdjh4elVWS0JIWVpXb2hIcXA4bFo2bEIyT1owaWs"],"Server":["Caddy"]},"status_code":200}
Feb 19 00:31:04 RJ-Cloud caddy[11573]: {"level":"debug","ts":1645227064.832822,"logger":"tls.issuance.acme.acme_client","msg":"http request","method":"POST","url":"https://acme.roadrunner/acme/local/certificate/imuyQqlzAHzeZJXQGVkEz8DWq10g599W","headers":{"Content-Type":["application/jose+json"],"User-Agent":["Caddy/2.4.6 CertMagic acmez (linux; amd64)"]},"response_headers":{"Cache-Control":["no-store"],"Content-Length":["1393"],"Content-Type":["application/pem-certificate-chain; charset=utf-8"],"Date":["Fri, 18 Feb 2022 23:31:04 GMT"],"Link":["<https://acme.roadrunner/acme/local/directory>;rel=\"index\""],"Replay-Nonce":["MDVvZHNvbVpFaVJTWDAwVEdYTnduMVp1bEQwWFl0eFc"],"Server":["Caddy"]},"status_code":200}
Feb 19 00:31:04 RJ-Cloud caddy[11573]: {"level":"info","ts":1645227064.832958,"logger":"tls.issuance.acme.acme_client","msg":"successfully downloaded available certificate chains","count":1,"first_url":"https://acme.roadrunner/acme/local/certificate/imuyQqlzAHzeZJXQGVkEz8DWq10g599W"}
Feb 19 00:31:04 RJ-Cloud caddy[11573]: {"level":"info","ts":1645227064.8336368,"logger":"tls.obtain","msg":"certificate obtained successfully","identifier":"bitwarden.roadrunner"}
Feb 19 00:31:04 RJ-Cloud caddy[11573]: {"level":"info","ts":1645227064.8336632,"logger":"tls.obtain","msg":"releasing lock","identifier":"bitwarden.roadrunner"}
Feb 19 00:31:04 RJ-Cloud caddy[11573]: {"level":"warn","ts":1645227064.8345842,"logger":"tls","msg":"stapling OCSP","error":"no OCSP stapling for [bitwarden.roadrunner]: no OCSP server specified in certificate"}
Feb 19 00:31:04 RJ-Cloud caddy[11573]: {"level":"debug","ts":1645227064.8346257,"logger":"tls.cache","msg":"added certificate to cache","subjects":["bitwarden.roadrunner"],"expiration":1645270264,"managed":true,"issuer_key":"acme.roadrunner-acme-local-directory","hash":"6b20673a5c056053d396e79c54ea3abee50c19f1447d05f09e2e70ab1171ccb0","cache_size":2,"cache_capacity":10000}
ps all nextcloud entries in the log can be ignored. I have not yet seup that docker container after I created the new VMs…