Did you copy the “new” CA root.crt to the the back-end?
Where did you copy it to? Is acme_ca_root pointing to that file?
oh right, for caddy start vs. systemd. here’s the frontend/backend logs after curl -v
2022/02/18 16:50:50.063 DEBUG tls.handshake choosing certificate {"identifier": "test.eastcapitol.us", "num_choices": 1}
2022/02/18 16:50:50.063 DEBUG tls.handshake default certificate selection results {"identifier": "test.eastcapitol.us", "subjects": ["test.eastcapitol.us"], "managed": true, "issuer_key": "acme-v02.api.letsencrypt.org-directory", "hash": "81b95004aa93fcad377af0951c86da3d56e46209bbb7920fd16bd5604467d5ee"}
2022/02/18 16:50:50.063 DEBUG tls.handshake matched certificate in cache {"subjects": ["test.eastcapitol.us"], "managed": true, "expiration": "2022/05/19 15:17:51.000", "hash": "81b95004aa93fcad377af0951c86da3d56e46209bbb7920fd16bd5604467d5ee"}
2022/02/18 16:50:50.152 DEBUG http.handlers.reverse_proxy upstream roundtrip {"upstream": "cloud.y8s.casa:443", "duration": 0.019996936, "request": {"remote_addr": "173.8.14.69:64838", "proto": "HTTP/1.1", "method": "GET", "host": "cloud.y8s.casa:443", "uri": "/", "headers": {"X-Forwarded-Proto": ["https"], "X-Forwarded-Host": ["test.eastcapitol.us"], "X-Forwarded-For": ["173.8.14.69"], "User-Agent": ["curl/7.79.1"], "Accept": ["*/*"]}, "tls": {"resumed": false, "version": 771, "cipher_suite": 49195, "proto": "http/1.1", "proto_mutual": true, "server_name": "test.eastcapitol.us"}}, "headers": {"Server": ["Caddy"], "Strict-Transport-Security": ["max-age=31536000;"], "Content-Length": ["0"], "Date": ["Fri, 18 Feb 2022 16:50:50 GMT"]}, "status": 502}
backend:
2022/02/18 16:50:50.146 DEBUG http.handlers.rewrite rewrote request {"request": {"remote_addr": "10.10.10.40:60654", "proto": "HTTP/2.0", "method": "GET", "host": "cloud.y8s.casa:443", "uri": "/", "headers": {"X-Forwarded-Proto": ["https"], "X-Forwarded-Host": ["test.eastcapitol.us"], "X-Forwarded-For": ["173.8.14.69"], "User-Agent": ["curl/7.79.1"], "Accept": ["*/*"], "Accept-Encoding": ["gzip"]}, "tls": {"resumed": false, "version": 772, "cipher_suite": 4867, "proto": "h2", "proto_mutual": true, "server_name": "cloud.y8s.casa"}}, "method": "GET", "uri": "/index.php"}
2022/02/18 16:50:50.153 DEBUG http.reverse_proxy.transport.fastcgi roundtrip {"request": {"remote_addr": "10.10.10.40:60654", "proto": "HTTP/2.0", "method": "GET", "host": "cloud.y8s.casa:443", "uri": "/index.php", "headers": {"X-Forwarded-Host": ["test.eastcapitol.us"], "X-Forwarded-For": ["173.8.14.69, 10.10.10.40"], "User-Agent": ["curl/7.79.1"], "Accept": ["*/*"], "Accept-Encoding": ["gzip"], "X-Forwarded-Proto": ["https"]}, "tls": {"resumed": false, "version": 772, "cipher_suite": 4867, "proto": "h2", "proto_mutual": true, "server_name": "cloud.y8s.casa"}}, "dial": "127.0.0.1:9000", "env": {"AUTH_TYPE":"","CONTENT_LENGTH":"","CONTENT_TYPE":"","DOCUMENT_ROOT":"/var/www/nextcloud","DOCUMENT_URI":"/index.php","GATEWAY_INTERFACE":"CGI/1.1","HTTPS":"on","HTTP_ACCEPT":"*/*","HTTP_ACCEPT_ENCODING":"gzip","HTTP_HOST":"cloud.y8s.casa:443","HTTP_USER_AGENT":"curl/7.79.1","HTTP_X_FORWARDED_FOR":"173.8.14.69, 10.10.10.40","HTTP_X_FORWARDED_HOST":"test.eastcapitol.us","HTTP_X_FORWARDED_PROTO":"https","PATH":"/bin","PATH_INFO":"","QUERY_STRING":"","REMOTE_ADDR":"10.10.10.40","REMOTE_HOST":"10.10.10.40","REMOTE_IDENT":"","REMOTE_PORT":"60654","REMOTE_USER":"","REQUEST_METHOD":"GET","REQUEST_SCHEME":"https","REQUEST_URI":"/","SCRIPT_FILENAME":"/var/www/nextcloud/index.php","SCRIPT_NAME":"/index.php","SERVER_NAME":"cloud.y8s.casa","SERVER_PORT":"443","SERVER_PROTOCOL":"HTTP/2.0","SERVER_SOFTWARE":"Caddy/v2.4.6","SSL_CIPHER":"TLS_CHACHA20_POLY1305_SHA256","SSL_PROTOCOL":"TLSv1.3"}}
2022/02/18 16:50:50.157 DEBUG http.handlers.reverse_proxy upstream roundtrip {"upstream": "127.0.0.1:9000", "duration": 0.004760661, "request": {"remote_addr": "10.10.10.40:60654", "proto": "HTTP/2.0", "method": "GET", "host": "cloud.y8s.casa:443", "uri": "/index.php", "headers": {"Accept-Encoding": ["gzip"], "X-Forwarded-Proto": ["https"], "X-Forwarded-Host": ["test.eastcapitol.us"], "X-Forwarded-For": ["173.8.14.69, 10.10.10.40"], "User-Agent": ["curl/7.79.1"], "Accept": ["*/*"]}, "tls": {"resumed": false, "version": 772, "cipher_suite": 4867, "proto": "h2", "proto_mutual": true, "server_name": "cloud.y8s.casa"}}, "error": "dialing backend: dial tcp 127.0.0.1:9000: connect: connection refused"}
2022/02/18 16:50:50.158 ERROR http.log.error.log0 dialing backend: dial tcp 127.0.0.1:9000: connect: connection refused {"request": {"remote_addr": "10.10.10.40:60654", "proto": "HTTP/2.0", "method": "GET", "host": "cloud.y8s.casa:443", "uri": "/", "headers": {"X-Forwarded-Proto": ["https"], "X-Forwarded-Host": ["test.eastcapitol.us"], "X-Forwarded-For": ["173.8.14.69"], "User-Agent": ["curl/7.79.1"], "Accept": ["*/*"], "Accept-Encoding": ["gzip"]}, "tls": {"resumed": false, "version": 772, "cipher_suite": 4867, "proto": "h2", "proto_mutual": true, "server_name": "cloud.y8s.casa"}}, "duration": 0.013550281, "status": 502, "err_id": "sf7xgtpdm", "err_trace": "reverseproxy.statusError (reverseproxy.go:886)"}
2022/02/18 16:50:50.158 ERROR http.log.access.log0 handled request {"request": {"remote_addr": "10.10.10.40:60654", "proto": "HTTP/2.0", "method": "GET", "host": "cloud.y8s.casa:443", "uri": "/", "headers": {"X-Forwarded-Host": ["test.eastcapitol.us"], "X-Forwarded-For": ["173.8.14.69"], "User-Agent": ["curl/7.79.1"], "Accept": ["*/*"], "Accept-Encoding": ["gzip"], "X-Forwarded-Proto": ["https"]}, "tls": {"resumed": false, "version": 772, "cipher_suite": 4867, "proto": "h2", "proto_mutual": true, "server_name": "cloud.y8s.casa"}}, "common_log": "10.10.10.40 - - [18/Feb/2022:11:50:50 -0500] \"GET / HTTP/2.0\" 502 0", "user_id": "", "duration": 0.013550281, "size": 0, "status": 502, "resp_headers": {"Server": ["Caddy"], "Strict-Transport-Security": ["max-age=31536000;"]}}
seems like progress with the certs, but now something else is causing issues. Maybe nextcloud config.
Yeah, that reverse_proxy issue seems to suggest the TLS connection between the frontend and backend has now succeeded, but your backend to your upstream app is failing. Make sure you have PHP-FPM running I guess.
I can’t get it to work through systemd 
I started from scratch with 2 brand new deployed Debian based VM’s, installed Caddy and created /etc/caddy/Caddyfile, copied from the front-end /var/lib/caddy/.local/share/caddy/pki/authorities/local/root.crt to the back-end location indicated in the back-end Caddyfile
Feb 18 19:32:57 RJ-Caddy caddy[2203]: {"level":"error","ts":1645209177.2372653,"logger":"http.log.error","msg":"x509: certificate signed by unknown authority (possibly because of \"x509: ECDSA verification failure\" while trying to verify candidate authority certificate \"Caddy Local Authority - 2022 ECC Root\""request":{"remote_addr":"192.168.2.200:52150","proto":"HTTP/2.0","method":"POST","host":"bpass.robbert.com","uri":"/identity/connect/token","headers":{"Accept":["application/json"],"Accept-Language":["en-GB,en;q=0.5"],"Pragma":["no-cache"],"Accept-Encoding":["gzip, deflate, br"],"Device-Type":["3"],"Origin":["moz-ension://fa1d4d8a-3175-4671-9c06-8150bf64cb83"],"Sec-Fetch-Dest":["empty"],"Sec-Fetch-Mode":["cors"],"Cache-Control":["no-cache"],"User-Agent":["Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:96.0) Gecko/20100101 Firefox/96.0"],"Content-Type":["application/x-www-form-urlencoded; charset=utf-8"],"Content-Length":[1"],"Sec-Fetch-Site":["same-origin"],"Te":["trailers"]},"tls":{"resumed":false,"version":772,"cipher_suite":4865,"proto":"h2","proto_mutual":true,"server_name":"bpass.robbert.com"}},"duration":0.007062436,"status":502,"err_id":"bizb4sy73","err_trace":"reverseproxy.statusError (reverseproxy.go:886)"}
Then I copied /etc/caddy/Caddyfile on font-end and back-end, first started front-end to copy the root.crt, then started the back-end… working.
It seems that the root CA is not matching.
Where did you put the root cert, exactly? Does the entire chain of parent directories have executable permission such that the caddy user can see it?
Forgot to mention that.
Originally I have the root CA under /etc/ssl/certs/root.crt I checked that path and user caddy has the correct user rights to get there.
But as a test, I also copied the CA to /var/lib/caddy/root.cert and chmod 777 And of course adjusted /etc/caddy/Caddyfile to make it point to the new location acme_ca_root /var/lib/caddy/root.crt
Same results in both ways.
Okay – are you sure you copied the right root cert from the frontend? If you’re running as a systemd service then you should be grabbing /var/lib/caddy/.local/share/caddy/pki/authorities/local/root.crt and not the one in your own user’s HOME (as would be generated/used if you ran caddy start)
Yep, that’s exactly what I did.
btw, on the back-end in /var/lib/caddy/.local/share/caddy/certificates/acme.orion-acme-local-directory a certificate is created. Also when I delete acme.orion-acme-local-directory and systemctl restart caddy this folder is recreated with the certificate inside.
Would that happen anyway?
this is the log from the back-end when after restarting caddy.
root@RJ-Cloud .../caddy/certificates# systemctl start caddy; journalctl -fu caddy
-- Logs begin at Fri 2022-02-18 17:40:23 CET. --
Feb 19 00:31:04 RJ-Cloud caddy[11573]: {"level":"debug","ts":1645227064.0857167,"logger":"http","msg":"starting server loop","address":"[::]:443","http3":false,"tls":true}
Feb 19 00:31:04 RJ-Cloud caddy[11573]: {"level":"debug","ts":1645227064.0858264,"logger":"http","msg":"starting server loop","address":"[::]:80","http3":false,"tls":false}
Feb 19 00:31:04 RJ-Cloud caddy[11573]: {"level":"info","ts":1645227064.0858436,"logger":"http","msg":"enabling automatic TLS certificate management","domains":["nextcloud.roadrunner","bitwarden.roadrunner"]}
Feb 19 00:31:04 RJ-Cloud caddy[11573]: {"level":"info","ts":1645227064.086441,"msg":"autosaved config (load with --resume flag)","file":"/var/lib/caddy/.config/caddy/autosave.json"}
Feb 19 00:31:04 RJ-Cloud systemd[1]: Started Caddy.
Feb 19 00:31:04 RJ-Cloud caddy[11573]: {"level":"info","ts":1645227064.0881436,"logger":"tls.obtain","msg":"acquiring lock","identifier":"nextcloud.roadrunner"}
Feb 19 00:31:04 RJ-Cloud caddy[11573]: {"level":"info","ts":1645227064.0886598,"msg":"serving initial configuration"}
Feb 19 00:31:04 RJ-Cloud caddy[11573]: {"level":"info","ts":1645227064.088732,"logger":"tls","msg":"cleaning storage unit","description":"FileStorage:/var/lib/caddy/.local/share/caddy"}
Feb 19 00:31:04 RJ-Cloud caddy[11573]: {"level":"info","ts":1645227064.0888329,"logger":"tls","msg":"finished cleaning storage units"}
Feb 19 00:31:04 RJ-Cloud caddy[11573]: {"level":"info","ts":1645227064.089314,"logger":"tls.obtain","msg":"acquiring lock","identifier":"bitwarden.roadrunner"}
Feb 19 00:31:04 RJ-Cloud caddy[11573]: {"level":"info","ts":1645227064.1030514,"logger":"tls.obtain","msg":"lock acquired","identifier":"nextcloud.roadrunner"}
Feb 19 00:31:04 RJ-Cloud caddy[11573]: {"level":"debug","ts":1645227064.1045785,"logger":"tls.obtain","msg":"trying issuer 1/1","issuer":"acme.roadrunner-acme-local-directory"}
Feb 19 00:31:04 RJ-Cloud caddy[11573]: {"level":"info","ts":1645227064.1065798,"logger":"tls.issuance.acme","msg":"waiting on internal rate limiter","identifiers":["nextcloud.roadrunner"],"ca":"https://acme.roadrunner/acme/local/directory","account":""}
Feb 19 00:31:04 RJ-Cloud caddy[11573]: {"level":"info","ts":1645227064.1070085,"logger":"tls.obtain","msg":"lock acquired","identifier":"bitwarden.roadrunner"}
Feb 19 00:31:04 RJ-Cloud caddy[11573]: {"level":"info","ts":1645227064.1070871,"logger":"tls.issuance.acme","msg":"done waiting on internal rate limiter","identifiers":["nextcloud.roadrunner"],"ca":"https://acme.roadrunner/acme/local/directory","account":""}
Feb 19 00:31:04 RJ-Cloud caddy[11573]: {"level":"debug","ts":1645227064.1076863,"logger":"tls.obtain","msg":"trying issuer 1/1","issuer":"acme.roadrunner-acme-local-directory"}
Feb 19 00:31:04 RJ-Cloud caddy[11573]: {"level":"info","ts":1645227064.108724,"logger":"tls.issuance.acme","msg":"waiting on internal rate limiter","identifiers":["bitwarden.roadrunner"],"ca":"https://acme.roadrunner/acme/local/directory","account":""}
Feb 19 00:31:04 RJ-Cloud caddy[11573]: {"level":"info","ts":1645227064.1087651,"logger":"tls.issuance.acme","msg":"done waiting on internal rate limiter","identifiers":["bitwarden.roadrunner"],"ca":"https://acme.roadrunner/acme/local/directory","account":""}
Feb 19 00:31:04 RJ-Cloud caddy[11573]: {"level":"debug","ts":1645227064.1184716,"logger":"tls.handshake","msg":"no matching certificates and no custom selection logic","identifier":"192.168.2.4"}
Feb 19 00:31:04 RJ-Cloud caddy[11573]: {"level":"debug","ts":1645227064.1185207,"logger":"tls.handshake","msg":"no certificate matching TLS ClientHello","server_name":"","remote":"192.168.2.200:53795","identifier":"192.168.2.4","cipher_suites":[49200,49196,49192,49188,49172,49162,165,163,161,159,107,106,105,104,57,56,55,54,49202,49198,49194,49190,49167,49157,157,61,53,136,135,134,133,132,49199,49195,49191,49187,49171,49161,164,162,160,158,103,64,63,62,51,50,49,48,49201,49197,49193,49189,49166,49156,156,60,47,154,153,152,151,69,68,67,66,150,65,7,49169,49159,49164,49154,5,4,49170,49160,22,19,16,13,49165,49155,10,21,18,15,12,9,20,17,8,6,3,255],"cert_cache_fill":0,"load_if_necessary":true,"obtain_if_necessary":true,"on_demand":false}
Feb 19 00:31:04 RJ-Cloud caddy[11573]: {"level":"debug","ts":1645227064.1186545,"logger":"http.stdlib","msg":"http: TLS handshake error from 192.168.2.200:53795: no certificate available for '192.168.2.4'"}
Feb 19 00:31:04 RJ-Cloud caddy[11573]: {"level":"debug","ts":1645227064.1202247,"logger":"tls.issuance.acme.acme_client","msg":"http request","method":"GET","url":"https://acme.roadrunner/acme/local/directory","headers":{"User-Agent":["Caddy/2.4.6 CertMagic acmez (linux; amd64)"]},"response_headers":{"Content-Length":["277"],"Content-Type":["application/json"],"Date":["Fri, 18 Feb 2022 23:31:04 GMT"],"Server":["Caddy"]},"status_code":200}
Feb 19 00:31:04 RJ-Cloud caddy[11573]: {"level":"debug","ts":1645227064.1335292,"logger":"tls.handshake","msg":"no matching certificates and no custom selection logic","identifier":"192.168.2.4"}
Feb 19 00:31:04 RJ-Cloud caddy[11573]: {"level":"debug","ts":1645227064.1339169,"logger":"tls.handshake","msg":"no certificate matching TLS ClientHello","server_name":"","remote":"192.168.2.200:53796","identifier":"192.168.2.4","cipher_suites":[49200,49196,49192,49188,49172,49162,165,163,161,159,107,106,105,104,57,56,55,54,49202,49198,49194,49190,49167,49157,157,61,53,136,135,134,133,132,49199,49195,49191,49187,49171,49161,164,162,160,158,103,64,63,62,51,50,49,48,49201,49197,49193,49189,49166,49156,156,60,47,154,153,152,151,69,68,67,66,150,65,7,49169,49159,49164,49154,5,4,49170,49160,22,19,16,13,49165,49155,10,21,18,15,12,9,20,17,8,6,3,255],"cert_cache_fill":0,"load_if_necessary":true,"obtain_if_necessary":true,"on_demand":false}
Feb 19 00:31:04 RJ-Cloud caddy[11573]: {"level":"debug","ts":1645227064.1343658,"logger":"http.stdlib","msg":"http: TLS handshake error from 192.168.2.200:53796: no certificate available for '192.168.2.4'"}
Feb 19 00:31:04 RJ-Cloud caddy[11573]: {"level":"debug","ts":1645227064.1373436,"logger":"tls.issuance.acme.acme_client","msg":"http request","method":"HEAD","url":"https://acme.roadrunner/acme/local/new-nonce","headers":{"User-Agent":["Caddy/2.4.6 CertMagic acmez (linux; amd64)"]},"response_headers":{"Cache-Control":["no-store"],"Date":["Fri, 18 Feb 2022 23:31:04 GMT"],"Link":["<https://acme.roadrunner/acme/local/directory>;rel=\"index\""],"Replay-Nonce":["c0xPaldWMjl5ckJ1MWd2V09lRUtoMU9hRmlHZ2tLZzE"],"Server":["Caddy"]},"status_code":200}
Feb 19 00:31:04 RJ-Cloud caddy[11573]: {"level":"debug","ts":1645227064.1460996,"logger":"tls.issuance.acme.acme_client","msg":"http request","method":"HEAD","url":"https://acme.roadrunner/acme/local/new-nonce","headers":{"User-Agent":["Caddy/2.4.6 CertMagic acmez (linux; amd64)"]},"response_headers":{"Cache-Control":["no-store"],"Date":["Fri, 18 Feb 2022 23:31:04 GMT"],"Link":["<https://acme.roadrunner/acme/local/directory>;rel=\"index\""],"Replay-Nonce":["MnBNUGNBZTc0TnNkUjBNTmxvMXZvWWgxMmJYNTg5NUo"],"Server":["Caddy"]},"status_code":200}
Feb 19 00:31:04 RJ-Cloud caddy[11573]: {"level":"debug","ts":1645227064.1487856,"logger":"http.stdlib","msg":"http: TLS handshake error from 192.168.2.200:53797: tls: client offered only unsupported versions: [301]"}
Feb 19 00:31:04 RJ-Cloud caddy[11573]: {"level":"debug","ts":1645227064.1633446,"logger":"http.stdlib","msg":"http: TLS handshake error from 192.168.2.200:53798: tls: client offered only unsupported versions: []"}
Feb 19 00:31:04 RJ-Cloud caddy[11573]: {"level":"debug","ts":1645227064.1804004,"logger":"http.stdlib","msg":"http: TLS handshake error from 192.168.2.200:53799: tls: unsupported SSLv2 handshake received"}
Feb 19 00:31:04 RJ-Cloud caddy[11573]: {"level":"debug","ts":1645227064.2733192,"logger":"tls.issuance.acme.acme_client","msg":"http request","method":"POST","url":"https://acme.roadrunner/acme/local/new-order","headers":{"Content-Type":["application/jose+json"],"User-Agent":["Caddy/2.4.6 CertMagic acmez (linux; amd64)"]},"response_headers":{"Cache-Control":["no-store"],"Content-Length":["401"],"Content-Type":["application/json"],"Date":["Fri, 18 Feb 2022 23:31:04 GMT"],"Link":["<https://acme.roadrunner/acme/local/directory>;rel=\"index\""],"Location":["https://acme.roadrunner/acme/local/order/d1v473zZYKMZIVI1oNrqsLcfWCnmtBxm"],"Replay-Nonce":["c3Zpb1ZodlVtSnNhNkMzY25HYTU2SWRmNmdEckUyeWQ"],"Server":["Caddy"]},"status_code":201}
Feb 19 00:31:04 RJ-Cloud caddy[11573]: {"level":"debug","ts":1645227064.290564,"logger":"tls.issuance.acme.acme_client","msg":"http request","method":"POST","url":"https://acme.roadrunner/acme/local/new-order","headers":{"Content-Type":["application/jose+json"],"User-Agent":["Caddy/2.4.6 CertMagic acmez (linux; amd64)"]},"response_headers":{"Cache-Control":["no-store"],"Content-Length":["401"],"Content-Type":["application/json"],"Date":["Fri, 18 Feb 2022 23:31:04 GMT"],"Link":["<https://acme.roadrunner/acme/local/directory>;rel=\"index\""],"Location":["https://acme.roadrunner/acme/local/order/PVxs95JaGzVPDuRCbh7gsXBxKOpwh3AC"],"Replay-Nonce":["bEtSRUNkWmJzelZPY21VcTFKSXdydXJtcW9mamZRUHo"],"Server":["Caddy"]},"status_code":201}
Feb 19 00:31:04 RJ-Cloud caddy[11573]: {"level":"debug","ts":1645227064.3077621,"logger":"tls.issuance.acme.acme_client","msg":"http request","method":"POST","url":"https://acme.roadrunner/acme/local/authz/R1dSppm7VVnTff719S5d5u3foxx23SY2","headers":{"Content-Type":["application/jose+json"],"User-Agent":["Caddy/2.4.6 CertMagic acmez (linux; amd64)"]},"response_headers":{"Cache-Control":["no-store"],"Content-Length":["728"],"Content-Type":["application/json"],"Date":["Fri, 18 Feb 2022 23:31:04 GMT"],"Link":["<https://acme.roadrunner/acme/local/directory>;rel=\"index\""],"Location":["https://acme.roadrunner/acme/local/authz/R1dSppm7VVnTff719S5d5u3foxx23SY2"],"Replay-Nonce":["OFRXc2tUM2tZdVN1MmFZM0FZSXpkbzVQM055R1hkcWo"],"Server":["Caddy"]},"status_code":200}
Feb 19 00:31:04 RJ-Cloud caddy[11573]: {"level":"debug","ts":1645227064.3083925,"logger":"tls.issuance.acme.acme_client","msg":"no solver configured","challenge_type":"dns-01"}
Feb 19 00:31:04 RJ-Cloud caddy[11573]: {"level":"info","ts":1645227064.308452,"logger":"tls.issuance.acme.acme_client","msg":"trying to solve challenge","identifier":"nextcloud.roadrunner","challenge_type":"http-01","ca":"https://acme.roadrunner/acme/local/directory"}
Feb 19 00:31:04 RJ-Cloud caddy[11573]: {"level":"debug","ts":1645227064.3237603,"logger":"tls.issuance.acme.acme_client","msg":"http request","method":"POST","url":"https://acme.roadrunner/acme/local/authz/Lb2LJXu8lD8lSuESSjpdjpLSZfZ3HaKo","headers":{"Content-Type":["application/jose+json"],"User-Agent":["Caddy/2.4.6 CertMagic acmez (linux; amd64)"]},"response_headers":{"Cache-Control":["no-store"],"Content-Length":["728"],"Content-Type":["application/json"],"Date":["Fri, 18 Feb 2022 23:31:04 GMT"],"Link":["<https://acme.roadrunner/acme/local/directory>;rel=\"index\""],"Location":["https://acme.roadrunner/acme/local/authz/Lb2LJXu8lD8lSuESSjpdjpLSZfZ3HaKo"],"Replay-Nonce":["MGVZQXczNEdnYkhWeUlEVzRyODZjWENhMlcyVDZlZ0I"],"Server":["Caddy"]},"status_code":200}
Feb 19 00:31:04 RJ-Cloud caddy[11573]: {"level":"debug","ts":1645227064.3244915,"logger":"tls.issuance.acme.acme_client","msg":"no solver configured","challenge_type":"dns-01"}
Feb 19 00:31:04 RJ-Cloud caddy[11573]: {"level":"info","ts":1645227064.3248203,"logger":"tls.issuance.acme.acme_client","msg":"trying to solve challenge","identifier":"bitwarden.roadrunner","challenge_type":"http-01","ca":"https://acme.roadrunner/acme/local/directory"}
Feb 19 00:31:04 RJ-Cloud caddy[11573]: {"level":"info","ts":1645227064.3447092,"logger":"tls.issuance.acme","msg":"served key authentication","identifier":"nextcloud.roadrunner","challenge":"http-01","remote":"192.168.2.2:47296","distributed":false}
Feb 19 00:31:04 RJ-Cloud caddy[11573]: {"level":"info","ts":1645227064.3616967,"logger":"tls.issuance.acme","msg":"served key authentication","identifier":"bitwarden.roadrunner","challenge":"http-01","remote":"192.168.2.2:47298","distributed":false}
Feb 19 00:31:04 RJ-Cloud caddy[11573]: {"level":"debug","ts":1645227064.3661535,"logger":"tls.issuance.acme.acme_client","msg":"http request","method":"POST","url":"https://acme.roadrunner/acme/local/challenge/R1dSppm7VVnTff719S5d5u3foxx23SY2/RTnNZngOzsRNGaB1baM2BgtnC65uKLvN","headers":{"Content-Type":["application/jose+json"],"User-Agent":["Caddy/2.4.6 CertMagic acmez (linux; amd64)"]},"response_headers":{"Cache-Control":["no-store"],"Content-Length":["228"],"Content-Type":["application/json"],"Date":["Fri, 18 Feb 2022 23:31:04 GMT"],"Link":["<https://acme.roadrunner/acme/local/directory>;rel=\"index\"","<https://acme.roadrunner/acme/local/authz/R1dSppm7VVnTff719S5d5u3foxx23SY2>;rel=\"up\""],"Location":["https://acme.roadrunner/acme/local/challenge/R1dSppm7VVnTff719S5d5u3foxx23SY2/RTnNZngOzsRNGaB1baM2BgtnC65uKLvN"],"Replay-Nonce":["UzZjMmQ4OUtRM0M2cEx6QUs2cVViQUk2SVZDTkNkOHk"],"Server":["Caddy"]},"status_code":200}
Feb 19 00:31:04 RJ-Cloud caddy[11573]: {"level":"debug","ts":1645227064.3669584,"logger":"tls.issuance.acme.acme_client","msg":"challenge accepted","identifier":"nextcloud.roadrunner","challenge_type":"http-01"}
Feb 19 00:31:04 RJ-Cloud caddy[11573]: {"level":"debug","ts":1645227064.3762105,"logger":"tls.issuance.acme.acme_client","msg":"http request","method":"POST","url":"https://acme.roadrunner/acme/local/challenge/Lb2LJXu8lD8lSuESSjpdjpLSZfZ3HaKo/taIYxhEceOrRKMiCWFq0jyekEm1QWBmY","headers":{"Content-Type":["application/jose+json"],"User-Agent":["Caddy/2.4.6 CertMagic acmez (linux; amd64)"]},"response_headers":{"Cache-Control":["no-store"],"Content-Length":["228"],"Content-Type":["application/json"],"Date":["Fri, 18 Feb 2022 23:31:04 GMT"],"Link":["<https://acme.roadrunner/acme/local/directory>;rel=\"index\"","<https://acme.roadrunner/acme/local/authz/Lb2LJXu8lD8lSuESSjpdjpLSZfZ3HaKo>;rel=\"up\""],"Location":["https://acme.roadrunner/acme/local/challenge/Lb2LJXu8lD8lSuESSjpdjpLSZfZ3HaKo/taIYxhEceOrRKMiCWFq0jyekEm1QWBmY"],"Replay-Nonce":["d1lveFlmWFg1TkEyR0NFS0xGT1FzNG5vNWJ1bFJodnQ"],"Server":["Caddy"]},"status_code":200}
Feb 19 00:31:04 RJ-Cloud caddy[11573]: {"level":"debug","ts":1645227064.376402,"logger":"tls.issuance.acme.acme_client","msg":"challenge accepted","identifier":"bitwarden.roadrunner","challenge_type":"http-01"}
Feb 19 00:31:04 RJ-Cloud caddy[11573]: {"level":"debug","ts":1645227064.6462443,"logger":"tls.issuance.acme.acme_client","msg":"http request","method":"POST","url":"https://acme.roadrunner/acme/local/authz/R1dSppm7VVnTff719S5d5u3foxx23SY2","headers":{"Content-Type":["application/jose+json"],"User-Agent":["Caddy/2.4.6 CertMagic acmez (linux; amd64)"]},"response_headers":{"Cache-Control":["no-store"],"Content-Length":["759"],"Content-Type":["application/json"],"Date":["Fri, 18 Feb 2022 23:31:04 GMT"],"Link":["<https://acme.roadrunner/acme/local/directory>;rel=\"index\""],"Location":["https://acme.roadrunner/acme/local/authz/R1dSppm7VVnTff719S5d5u3foxx23SY2"],"Replay-Nonce":["UFRLR2tKNk1rcWlhY25NY3hTNjNFcXhPTzBqeGFWc2Q"],"Server":["Caddy"]},"status_code":200}
Feb 19 00:31:04 RJ-Cloud caddy[11573]: {"level":"info","ts":1645227064.6480546,"logger":"tls.issuance.acme.acme_client","msg":"validations succeeded; finalizing order","order":"https://acme.roadrunner/acme/local/order/d1v473zZYKMZIVI1oNrqsLcfWCnmtBxm"}
Feb 19 00:31:04 RJ-Cloud caddy[11573]: {"level":"debug","ts":1645227064.690357,"logger":"tls.issuance.acme.acme_client","msg":"http request","method":"POST","url":"https://acme.roadrunner/acme/local/authz/Lb2LJXu8lD8lSuESSjpdjpLSZfZ3HaKo","headers":{"Content-Type":["application/jose+json"],"User-Agent":["Caddy/2.4.6 CertMagic acmez (linux; amd64)"]},"response_headers":{"Cache-Control":["no-store"],"Content-Length":["759"],"Content-Type":["application/json"],"Date":["Fri, 18 Feb 2022 23:31:04 GMT"],"Link":["<https://acme.roadrunner/acme/local/directory>;rel=\"index\""],"Location":["https://acme.roadrunner/acme/local/authz/Lb2LJXu8lD8lSuESSjpdjpLSZfZ3HaKo"],"Replay-Nonce":["ckw2SU8ydnNKRTBFZGRLdFd0MUtOZENXeHlpNmxsbXI"],"Server":["Caddy"]},"status_code":200}
Feb 19 00:31:04 RJ-Cloud caddy[11573]: {"level":"info","ts":1645227064.690859,"logger":"tls.issuance.acme.acme_client","msg":"validations succeeded; finalizing order","order":"https://acme.roadrunner/acme/local/order/PVxs95JaGzVPDuRCbh7gsXBxKOpwh3AC"}
Feb 19 00:31:04 RJ-Cloud caddy[11573]: {"level":"debug","ts":1645227064.7605941,"logger":"tls.issuance.acme.acme_client","msg":"http request","method":"POST","url":"https://acme.roadrunner/acme/local/order/d1v473zZYKMZIVI1oNrqsLcfWCnmtBxm/finalize","headers":{"Content-Type":["application/jose+json"],"User-Agent":["Caddy/2.4.6 CertMagic acmez (linux; amd64)"]},"response_headers":{"Cache-Control":["no-store"],"Content-Length":["490"],"Content-Type":["application/json"],"Date":["Fri, 18 Feb 2022 23:31:04 GMT"],"Link":["<https://acme.roadrunner/acme/local/directory>;rel=\"index\""],"Location":["https://acme.roadrunner/acme/local/order/d1v473zZYKMZIVI1oNrqsLcfWCnmtBxm"],"Replay-Nonce":["SUkxaFRHNml0Tm1OT3BCZ0JpbkEwQzRNTFduT0lhQW4"],"Server":["Caddy"]},"status_code":200}
Feb 19 00:31:04 RJ-Cloud caddy[11573]: {"level":"debug","ts":1645227064.8049784,"logger":"tls.issuance.acme.acme_client","msg":"http request","method":"POST","url":"https://acme.roadrunner/acme/local/certificate/52ZfAfyRifgNaqVcfSgp0TkNNzmJlRgU","headers":{"Content-Type":["application/jose+json"],"User-Agent":["Caddy/2.4.6 CertMagic acmez (linux; amd64)"]},"response_headers":{"Cache-Control":["no-store"],"Content-Length":["1393"],"Content-Type":["application/pem-certificate-chain; charset=utf-8"],"Date":["Fri, 18 Feb 2022 23:31:04 GMT"],"Link":["<https://acme.roadrunner/acme/local/directory>;rel=\"index\""],"Replay-Nonce":["VFJIRTVqSzlicGNITFgwa2s1NkJ3SU9hbUg2a0NKT3Q"],"Server":["Caddy"]},"status_code":200}
Feb 19 00:31:04 RJ-Cloud caddy[11573]: {"level":"info","ts":1645227064.8051486,"logger":"tls.issuance.acme.acme_client","msg":"successfully downloaded available certificate chains","count":1,"first_url":"https://acme.roadrunner/acme/local/certificate/52ZfAfyRifgNaqVcfSgp0TkNNzmJlRgU"}
Feb 19 00:31:04 RJ-Cloud caddy[11573]: {"level":"info","ts":1645227064.8059049,"logger":"tls.obtain","msg":"certificate obtained successfully","identifier":"nextcloud.roadrunner"}
Feb 19 00:31:04 RJ-Cloud caddy[11573]: {"level":"info","ts":1645227064.8059373,"logger":"tls.obtain","msg":"releasing lock","identifier":"nextcloud.roadrunner"}
Feb 19 00:31:04 RJ-Cloud caddy[11573]: {"level":"warn","ts":1645227064.8069315,"logger":"tls","msg":"stapling OCSP","error":"no OCSP stapling for [nextcloud.roadrunner]: no OCSP server specified in certificate"}
Feb 19 00:31:04 RJ-Cloud caddy[11573]: {"level":"debug","ts":1645227064.806975,"logger":"tls.cache","msg":"added certificate to cache","subjects":["nextcloud.roadrunner"],"expiration":1645270264,"managed":true,"issuer_key":"acme.roadrunner-acme-local-directory","hash":"5afb0ba072dc727a144f96afb83b1dceeb041e606ba1485018bef826f1dafb94","cache_size":1,"cache_capacity":10000}
Feb 19 00:31:04 RJ-Cloud caddy[11573]: {"level":"debug","ts":1645227064.8126915,"logger":"tls.issuance.acme.acme_client","msg":"http request","method":"POST","url":"https://acme.roadrunner/acme/local/order/PVxs95JaGzVPDuRCbh7gsXBxKOpwh3AC/finalize","headers":{"Content-Type":["application/jose+json"],"User-Agent":["Caddy/2.4.6 CertMagic acmez (linux; amd64)"]},"response_headers":{"Cache-Control":["no-store"],"Content-Length":["490"],"Content-Type":["application/json"],"Date":["Fri, 18 Feb 2022 23:31:04 GMT"],"Link":["<https://acme.roadrunner/acme/local/directory>;rel=\"index\""],"Location":["https://acme.roadrunner/acme/local/order/PVxs95JaGzVPDuRCbh7gsXBxKOpwh3AC"],"Replay-Nonce":["ZlVxdjh4elVWS0JIWVpXb2hIcXA4bFo2bEIyT1owaWs"],"Server":["Caddy"]},"status_code":200}
Feb 19 00:31:04 RJ-Cloud caddy[11573]: {"level":"debug","ts":1645227064.832822,"logger":"tls.issuance.acme.acme_client","msg":"http request","method":"POST","url":"https://acme.roadrunner/acme/local/certificate/imuyQqlzAHzeZJXQGVkEz8DWq10g599W","headers":{"Content-Type":["application/jose+json"],"User-Agent":["Caddy/2.4.6 CertMagic acmez (linux; amd64)"]},"response_headers":{"Cache-Control":["no-store"],"Content-Length":["1393"],"Content-Type":["application/pem-certificate-chain; charset=utf-8"],"Date":["Fri, 18 Feb 2022 23:31:04 GMT"],"Link":["<https://acme.roadrunner/acme/local/directory>;rel=\"index\""],"Replay-Nonce":["MDVvZHNvbVpFaVJTWDAwVEdYTnduMVp1bEQwWFl0eFc"],"Server":["Caddy"]},"status_code":200}
Feb 19 00:31:04 RJ-Cloud caddy[11573]: {"level":"info","ts":1645227064.832958,"logger":"tls.issuance.acme.acme_client","msg":"successfully downloaded available certificate chains","count":1,"first_url":"https://acme.roadrunner/acme/local/certificate/imuyQqlzAHzeZJXQGVkEz8DWq10g599W"}
Feb 19 00:31:04 RJ-Cloud caddy[11573]: {"level":"info","ts":1645227064.8336368,"logger":"tls.obtain","msg":"certificate obtained successfully","identifier":"bitwarden.roadrunner"}
Feb 19 00:31:04 RJ-Cloud caddy[11573]: {"level":"info","ts":1645227064.8336632,"logger":"tls.obtain","msg":"releasing lock","identifier":"bitwarden.roadrunner"}
Feb 19 00:31:04 RJ-Cloud caddy[11573]: {"level":"warn","ts":1645227064.8345842,"logger":"tls","msg":"stapling OCSP","error":"no OCSP stapling for [bitwarden.roadrunner]: no OCSP server specified in certificate"}
Feb 19 00:31:04 RJ-Cloud caddy[11573]: {"level":"debug","ts":1645227064.8346257,"logger":"tls.cache","msg":"added certificate to cache","subjects":["bitwarden.roadrunner"],"expiration":1645270264,"managed":true,"issuer_key":"acme.roadrunner-acme-local-directory","hash":"6b20673a5c056053d396e79c54ea3abee50c19f1447d05f09e2e70ab1171ccb0","cache_size":2,"cache_capacity":10000}
ps all nextcloud entries in the log can be ignored. I have not yet seup that docker container after I created the new VMs…
Oh, I think I know what’s going on now… The reverse_proxy on frontend isn’t trusting the certificate from the backend, because the frontend server hasn’t trusted the root cert that was generated.
This bit is complicated for now unfortunately but will be fixed in v2.5.0 probably with pki: Implement `GET /pki/certificates/<id>` API, rework `caddy trust` by francislavoie · Pull Request #4443 · caddyserver/caddy · GitHub but for now you’ll need to run this command (just once) to make sure the right root cert is added to the system’s trust store:
sudo HOME=~caddy caddy trust
You can follow the discussion on the issue that PR references for a deeper explanation.
1 Like
That was it!
Just for extra confirmation: After that fix I also successfully restored the original path of the CA root back to /etc/ssl/cert with user rights: -rw-r--r-- 1 root root
Just one question, while running the suggested command I got a warning:
root@RJ-Cloud .../caddy/certificates# HOME=~caddy caddy trust
2022/02/19 07:46:49.568 WARN ca.local installing root certificate (you might be prompted for password) {"path": "storage:pki/authorities/local/root.crt"}
2022/02/19 08:46:49 Warning: "certutil" is not available, install "certutil" with "apt install libnss3-tools" or "yum install nss-tools" and try again
2022/02/19 08:46:49 define JAVA_HOME environment variable to use the Java trust
2022/02/19 08:46:50 certificate installed properly in linux trusts
I also noticed that the same warning was in the caddy logs. Do I need this util?
Back to the OPs issue, now that I seem to have Caddy working with systemd, I can continue to look into the certificate renewal. The suggested cron schedule sounds like a solid workaround.
@francislavoie Thanks alot again for your great devotion and time spend on our troubles
1 Like
NSS is the library that underlies Firefox’s TLS implementation. You would only need that installed if you’re running Firefox on the same machine as you’re running Caddy and using it to browse to your site where you need Firefox to trust Caddy’s CA.
Same with Java, only needed if you’re running Java apps that need to trust Caddy’s CA, on that machine.
so I have a mess of cert subfolders on both machines now. if I wanted to completely blow away all the caddy storage and copied certs (caddy start and systemd) and start over with just the binary and caddyfile, then run the fix, will I run into a problem?
Yeah that’s probably fine to do. Just remember to do the main steps of copying the root cert over from the frontend to the backend, and running the trust command I wrote above on the frontend to make sure the system trusts it.
Yes of course. Thanks for the help!
1 Like
So I removed the caddy user and /var/lib/caddy
recopied the root.crt
ran sudo HOME=~caddy caddy trust on the frontend
force reloaded both
restarted caddy on both
now my already working domains (hosted on the frontend) are broken too.
I paired down my caddyfile to test and now I’m rate limited.
Don’t remove the caddy user! That’s necessary for the systemd service to run properly.
Sorry, that wasn’t clear. I mistakenly deleted the caddy dir and had to recreate it properly. I deleted the caddy user and then recreated it using the instructions here: Keep Caddy Running — Caddy Documentation
So now that I am rate limited, can I restore a backup with certs from a few days ago and will that work?
Does Let’s Encrypt care, or will it say newer ones have been created and try to reobtain them?
Phew the answer is YES my old certs work with the stuff hosted on the front end. Now just have to un-break the backend nextcloud install.
Probably. But Caddy will fallback to ZeroSSL if you’re rate limited by Let’s Encrypt (as long as you don’t explicitly configure Caddy to use only Let’s Encrypt), so you should be fine anyways.
Let’s Encrypt can’t care, there’s no communication back to them after certs are issued.