Thanks for pointing out some possible areas to check.
First of all, after trying again the first time again since I wrote this post (11d ago), I noticed some different error message on the frontend caddy today:
{
"level": "error",
"ts": 1632770320.2249467,
"logger": "http.log.error",
"msg": "x509: certificate has expired or is not yet valid: current time 2021-09-27T19:18:40Z is after 2021-09-23T13:34:24Z",
"request": {
"remote_addr": "192.168.160.1:36340",
"proto": "HTTP/2.0",
"method": "GET",
"host": "test.mydomain.tld",
"uri": "/",
"headers": {
"Accept": [
"text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8"
],
"Accept-Language": [
"en-US,en;q=0.5"
],
"Accept-Encoding": [
"gzip, deflate, br"
],
"Sec-Fetch-Dest": [
"document"
],
"Te": [
"trailers"
],
"User-Agent": [
"Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:92.0) Gecko/20100101 Firefox/92.0"
],
"Cookie": [
"nc_sameSiteCookielax=true; nc_sameSiteCookiestrict=true; oc_sessionPassphrase=e1RNAlEmNFjO8Fork%2FEL9jXN3XpZ3DB82ah%2BhRZt37gIJBJIpywG1Db0l0WfCwDH2cLmVVXQhJaKDYTSXEFHQmAcQ6NkIVCdkHFj5xg%2Bymyo1uur2jbQ0Qq%2BqW5hv4q0; oc7zsgh7x0fv=143a056701a303c033f2679bf07c6777"
],
"Upgrade-Insecure-Requests": [
"1"
],
"Sec-Fetch-Mode": [
"navigate"
],
"Sec-Fetch-Site": [
"cross-site"
]
},
"tls": {
"resumed": false,
"version": 772,
"cipher_suite": 4867,
"proto": "h2",
"proto_mutual": true,
"server_name": "test.mydomain.tld"
}
},
"duration": 0.018968927,
"status": 502,
"err_id": "2kciiib0w",
"err_trace": "reverseproxy.statusError (reverseproxy.go:858)"
}
After searching a bit, I found this topic, which seems to be very much related:
https://caddy.community/t/mtls-tls-internal-error/12807/23
Unfortunately, I cannot post there, since the topic is auto-closed already.
However, from what I saw, there were some problems with updating the mTLS. You also pointed out:
Edit: Sorry, 7 days is the lifetime of the intermediate CA cert.
In my case, it is also 7 days after I started the Frontend Caddy (with ACME server) the last time: 16 September (shortly before I started this topic) till 23 September (from the error message above).
So apparently Caddy did not manage to update the intermediate CA cert properly. Not sure if my observations are helpful in some ways (e.g. to further analyze that other issue)
After I restarted the Frontend Caddy container, the intermediate CA cert was updated and now I’m back to the original problem.
What I’ve checked today:
- When bind mounting the root CA cert manually into the Frontend Caddy container (via docker-compose.yml)
then it also works right away. In the logs I can see that during Caddy start the Root CA cert is recognized as already trusted:volumes: - ./etc-caddy:/etc/caddy:ro - caddydata:/data - ./proxy_ca_root.crt:/etc/ssl/certs/proxy_ca_root.crt:ro{"level":"info","ts":1632780209.0057275,"logger":"pki.ca.local","msg":"root certificate is already trusted by system","path":"storage:pki/authorities/local/root.crt"} - That gave me the idea to reset/restart/reload Caddy without stopping the whole container regularly:
Which resulted in:caddy stop
And this time it also worked. I don’t get the 502 anymore.{"level":"warn","ts":1632780207.6874425,"logger":"admin.api","msg":"exiting; byeee!! 👋"} {"level":"debug","ts":1632780207.6947668,"logger":"http.handlers.acme_server","msg":"unloading unused CA database","db_key":"local"} {"level":"info","ts":1632780207.6948857,"logger":"tls.cache.maintenance","msg":"stopped background certificate maintenance","cache":"0x4000483960"} {"level":"info","ts":1632780207.6968102,"logger":"admin","msg":"stopped previous server","address":"tcp/localhost:2019"} {"level":"info","ts":1632780207.6969922,"logger":"admin.api","msg":"shutdown complete","exit_code":0} {"level":"info","ts":1632780208.8935914,"msg":"using provided configuration","config_file":"/etc/caddy/Caddyfile","config_adapter":"caddyfile"} {"level":"info","ts":1632780208.9062464,"logger":"admin","msg":"admin endpoint started","address":"tcp/localhost:2019","enforce_origin":false,"origins":["localhost:2019","[::1]:2019","127.0.0.1:2019"]} {"level":"info","ts":1632780208.90713,"logger":"tls.cache.maintenance","msg":"started background certificate maintenance","cache":"0x4000486770"} {"level":"info","ts":1632780208.9084504,"logger":"http","msg":"server is listening only on the HTTPS port but has no TLS connection policies; adding one to enable TLS","server_name":"srv0","https_port":443} {"level":"info","ts":1632780208.908786,"logger":"http","msg":"server is listening only on the HTTP port, so no automatic HTTPS will be applied to this server","server_name":"srv1","http_port":80} {"level":"info","ts":1632780209.0057275,"logger":"pki.ca.local","msg":"root certificate is already trusted by system","path":"storage:pki/authorities/local/root.crt"}
So, as conclusion, when starting the Docker container regularly, we see this in the logs:
{"level":"warn","ts":1631830791.917869,"logger":"pki.ca.local","msg":"installing root certificate (you might be prompted for password)","path":"storage:pki/authorities/local/root.crt"}
2021/09/16 22:19:51 Warning: "certutil" is not available, install "certutil" with "apt install libnss3-tools" or "yum install nss-tools" and try again
2021/09/16 22:19:51 define JAVA_HOME environment variable to use the Java trust
2021/09/16 22:19:52 certificate installed properly in linux trusts
At this point the Root CA certificate is copied to /etc/ssl/certs (and appended to /etc/ssl/cert.pem -> /etc/ssl/certs/ca-certificates.crt) but apparently it is not recognized by the Caddy runtime (?) yet.
Only next time the Caddy application is started (and the cert file is already in /etc/ssl/certs), it is recognized properly and considered as trusted.
Not sure if this is expected or acceptable behavior.