Caddy as ACME server - not renewing intermediate certificate

Help
1

1. Caddy version (caddy version):

v2.5.2 in a docker container

2. How I run Caddy:

docker compose up -d

a. System environment:

Docker on Debian Bullseye with all updates.

b. Command:

Paste command here.

c. Service/unit/compose file:

version: '3.5'

services:
  caddy:
    image: caddy:latest
    restart: unless-stopped
    container_name: caddy
    hostname: ca.intern.meinedomain.xyz
    ports:
      - 80:80
      - 443:443
    volumes:
      - /etc/caddy/Caddyfile:/etc/caddy/Caddyfile
      - /etc/ssl/certs/00_intern.meinedomain.xyz_root-ca.pem:/etc/ssl/certs/root.crt:ro
      - /etc/ssl/private/root_ca_nopw.key:/etc/ssl/private/root.key:ro
      - "caddy-config:/config"
      - "caddy-data:/data"
    environment:
      - LOG_FILE=/data/access.log
    networks:
      - baikal-network

volumes:
    caddy-config:
    caddy-data:
networks:
    baikal-network:
      external: true
      name: baikal_baikal-network

d. My complete Caddyfile or JSON config:

{
        # General Options
        #       debug

        #PKI options
        pki {
                ca {
                        root_cn ca.intern.meinedomain.xyz
                        root {
                                format pem_file
                                cert /etc/ssl/certs/root.crt
                                key /etc/ssl/private/root.key
                        }
                }
        }
}

#ACME server
ca.intern.meinedomain.xyz {
        acme_server
        tls internal
}
###########################################################################################
###### Baikal ### Baikal ### Baikal ### Baikal ### Calendarserver ####### Baikal ##########
###########################################################################################
# Baikal Calendarserver
calendarserver.intern.meinedomain.xyz:443 {
        # This setting may have compatibility issues with some browsers
        # (e.g., attachment downloading on Firefox). Try disabling this
        # if you encounter issues.
        encode gzip

        # Proxy everything to calendarserver
        handle_path /.well-known/carddav {
                rewrite * /dav.php{uri}
        }
        handle_path /.well-known/caldav {
                rewrite * /dav.php{uri}
        }

        reverse_proxy baikal_nginx:80

        #TLS certificate
        #        tls /etc/caddy/cert.pem /etc/caddy/cert.key.pem
        tls {
                ca https://ca.intern.meinedomain.xyz/acme/local/directory # point to ACME server
                ca_root /etc/ssl/certs/root.crt # define root certificate
        }
}

3. The problem I’m having:

This is a lab environment with one host and all services are running on it in their own environement (container/network/etc.).
I have one Caddy instance running in a docker container with 2 tasks.
1.) acme server
2.) reverse-proxy for the services
The root key and certificate are allready existing and mapped into the container so caddy can use it to generate an intermediate ca and all needed certificates.

On the first start caddy is generating the needed intermediate key and cert via acme and everything works like expected for 5 days. After 5 days renewing of the certs stopps - because the intermediate certificate expired and is not renewed.

4. Error messages and/or full log output:

working:

{"level":"info","ts":1658301410.085073,"logger":"tls.renew","msg":"certificate renewed successfully","identifier":"kochbuch.intern.meinedomain.xyz"}
{"level":"info","ts":1658301410.085133,"logger":"tls.renew","msg":"releasing lock","identifier":"kochbuch.intern.meinedomain.xyz"}
{"level":"info","ts":1658301410.0855236,"logger":"tls","msg":"reloading managed certificate","identifiers":["kochbuch.intern.meinedomain.xyz"]}
{"level":"warn","ts":1658301410.0866222,"logger":"tls","msg":"stapling OCSP","error":"no OCSP stapling for [kochbuch.intern.meinedomain.xyz]: no OCSP server specified in certificate","identifiers":["kochbuch.intern.meinedomain.xyz"]}
{"level":"info","ts":1658301410.086671,"logger":"tls.cache","msg":"replaced certificate in cache","subjects":["kochbuch.intern.meinedomain.xyz"],"new_expiration":1658344608}
{"level":"info","ts":1658303807.9442873,"logger":"tls.cache.maintenance","msg":"certificate expires soon; queuing for renewal","identifiers":["ca.intern.meinedomain.xyz"],"remaining":14399.055717542}
{"level":"info","ts":1658303807.9448261,"logger":"tls.cache.maintenance","msg":"attempting certificate renewal","identifiers":["ca.intern.meinedomain.xyz"],"remaining":14399.055178352}
{"level":"info","ts":1658303807.9453542,"logger":"tls.renew","msg":"acquiring lock","identifier":"ca.intern.meinedomain.xyz"}
{"level":"info","ts":1658303807.9616904,"logger":"tls.renew","msg":"lock acquired","identifier":"ca.intern.meinedomain.xyz"}
{"level":"info","ts":1658303807.9624262,"logger":"tls.renew","msg":"renewing certificate","identifier":"ca.intern.meinedomain.xyz","remaining":14399.037577131}
{"level":"info","ts":1658303807.9654067,"logger":"tls.renew","msg":"certificate renewed successfully","identifier":"ca.intern.meinedomain.xyz"}
{"level":"info","ts":1658303807.9654405,"logger":"tls.renew","msg":"releasing lock","identifier":"ca.intern.meinedomain.xyz"}
{"level":"info","ts":1658303807.965591,"logger":"tls","msg":"reloading managed certificate","identifiers":["ca.intern.meinedomain.xyz"]}
{"level":"warn","ts":1658303807.966644,"logger":"tls","msg":"stapling OCSP","error":"no OCSP stapling for [ca.intern.meinedomain.xyz]: no OCSP server specified in certificate","identifiers":["ca.intern.meinedomain.xyz"]}
{"level":"info","ts":1658303807.9666934,"logger":"tls.cache","msg":"replaced certificate in cache","subjects":["ca.intern.meinedomain.xyz"],"new_expiration":1658347007}
{"level":"info","ts":1658330208.04375,"logger":"tls.cache.maintenance","msg":"certificate expires soon; queuing for renewal","identifiers":["seafile.intern.meinedomain.xyz"],"remaining":14399.956254088}
{"level":"info","ts":1658330208.0438552,"logger":"tls.cache.maintenance","msg":"certificate expires soon; queuing for renewal","identifiers":["calendarserver.intern.meinedomain.xyz"],"remaining":14399.956146663}
{"level":"info","ts":1658330208.0438793,"logger":"tls.cache.maintenance","msg":"certificate expires soon; queuing for renewal","identifiers":["kochbuch.intern.meinedomain.xyz"],"remaining":14399.956121918}
{"level":"info","ts":1658330208.0439186,"logger":"tls.cache.maintenance","msg":"attempting certificate renewal","identifiers":["seafile.intern.meinedomain.xyz"],"remaining":14399.956094024}
{"level":"info","ts":1658330208.0445092,"logger":"tls.renew","msg":"acquiring lock","identifier":"seafile.intern.meinedomain.xyz"}
{"level":"info","ts":1658330208.0448155,"logger":"tls.cache.maintenance","msg":"attempting certificate renewal","identifiers":["calendarserver.intern.meinedomain.xyz"],"remaining":14399.955187609}
{"level":"info","ts":1658330208.0455167,"logger":"tls.renew","msg":"acquiring lock","identifier":"calendarserver.intern.meinedomain.xyz"}
{"level":"info","ts":1658330208.0549724,"logger":"tls.cache.maintenance","msg":"attempting certificate renewal","identifiers":["kochbuch.intern.meinedomain.xyz"],"remaining":14399.945034959}
{"level":"info","ts":1658330208.055483,"logger":"tls.renew","msg":"acquiring lock","identifier":"kochbuch.intern.meinedomain.xyz"}
{"level":"info","ts":1658330208.0664124,"logger":"tls.renew","msg":"lock acquired","identifier":"seafile.intern.meinedomain.xyz"}
{"level":"info","ts":1658330208.067199,"logger":"tls.renew","msg":"renewing certificate","identifier":"seafile.intern.meinedomain.xyz","remaining":14399.932805727}
{"level":"info","ts":1658330208.0685902,"logger":"tls.issuance.acme","msg":"waiting on internal rate limiter","identifiers":["seafile.intern.meinedomain.xyz"],"ca":"https://ca.intern.meinedomain.xyz/acme/local/directory","account":""}
{"level":"info","ts":1658330208.0686283,"logger":"tls.issuance.acme","msg":"done waiting on internal rate limiter","identifiers":["seafile.intern.meinedomain.xyz"],"ca":"https://ca.intern.meinedomain.xyz/acme/local/directory","account":""}
{"level":"info","ts":1658330208.0895457,"logger":"tls.renew","msg":"lock acquired","identifier":"calendarserver.intern.meinedomain.xyz"}
{"level":"info","ts":1658330208.0903957,"logger":"tls.renew","msg":"renewing certificate","identifier":"calendarserver.intern.meinedomain.xyz","remaining":14399.909608918}
{"level":"info","ts":1658330208.0917702,"logger":"tls.issuance.acme","msg":"waiting on internal rate limiter","identifiers":["calendarserver.intern.meinedomain.xyz"],"ca":"https://ca.intern.meinedomain.xyz/acme/local/directory","account":""}
{"level":"info","ts":1658330208.0918412,"logger":"tls.issuance.acme","msg":"done waiting on internal rate limiter","identifiers":["calendarserver.intern.meinedomain.xyz"],"ca":"https://ca.intern.meinedomain.xyz/acme/local/directory","account":""}
{"level":"info","ts":1658330208.1004567,"logger":"tls.renew","msg":"lock acquired","identifier":"kochbuch.intern.meinedomain.xyz"}
{"level":"info","ts":1658330208.1013303,"logger":"tls.renew","msg":"renewing certificate","identifier":"kochbuch.intern.meinedomain.xyz","remaining":14399.898674639}
{"level":"info","ts":1658330208.1028984,"logger":"tls.issuance.acme","msg":"waiting on internal rate limiter","identifiers":["kochbuch.intern.meinedomain.xyz"],"ca":"https://ca.intern.meinedomain.xyz/acme/local/directory","account":""}
{"level":"info","ts":1658330208.1030445,"logger":"tls.issuance.acme","msg":"done waiting on internal rate limiter","identifiers":["kochbuch.intern.meinedomain.xyz"],"ca":"https://ca.intern.meinedomain.xyz/acme/local/directory","account":""}
{"level":"info","ts":1658330208.7105727,"msg":"{\"id\":\"jna1w77nKs1Z0YG8aoEDW0YViB82CKCD\",\"status\":\"pending\",\"expires\":\"2022-07-21T15:16:48Z\",\"identifiers\":[{\"type\":\"dns\",\"value\":\"seafile.intern.meinedomain.xyz\"}],\"notBefore\":\"2022-07-20T15:15:48Z\",\"notAfter\":\"2022-07-21T03:16:48Z\",\"authorizations\":[\"https://ca.intern.meinedomain.xyz/acme/local/authz/sgrq5oa0hdPEimmctoMH5bDOzd8BTZMY\"],\"finalize\":\"https://ca.intern.meinedomain.xyz/acme/local/order/jna1w77nKs1Z0YG8aoEDW0YViB82CKCD/finalize\"}"}
{"level":"info","ts":1658330208.75726,"msg":"{\"id\":\"Az6X91EmziDtLwnSORXyfSISBnXjIqKF\",\"status\":\"pending\",\"expires\":\"2022-07-21T15:16:48Z\",\"identifiers\":[{\"type\":\"dns\",\"value\":\"calendarserver.intern.meinedomain.xyz\"}],\"notBefore\":\"2022-07-20T15:15:48Z\",\"notAfter\":\"2022-07-21T03:16:48Z\",\"authorizations\":[\"https://ca.intern.meinedomain.xyz/acme/local/authz/ogN7gDXtiiZkhK13kHH5ze2psZdz9bKx\"],\"finalize\":\"https://ca.intern.meinedomain.xyz/acme/local/order/Az6X91EmziDtLwnSORXyfSISBnXjIqKF/finalize\"}"}
{"level":"info","ts":1658330208.8025637,"msg":"{\"id\":\"2JT0ef9TF9npSD5S3xz5XLC58Puf9i2E\",\"status\":\"pending\",\"expires\":\"2022-07-21T15:16:48Z\",\"identifiers\":[{\"type\":\"dns\",\"value\":\"kochbuch.intern.meinedomain.xyz\"}],\"notBefore\":\"2022-07-20T15:15:48Z\",\"notAfter\":\"2022-07-21T03:16:48Z\",\"authorizations\":[\"https://ca.intern.meinedomain.xyz/acme/local/authz/Jespk0uSWk1gdjtxMt22LQ76H4srePLC\"],\"finalize\":\"https://ca.intern.meinedomain.xyz/acme/local/order/2JT0ef9TF9npSD5S3xz5XLC58Puf9i2E/finalize\"}"}
{"level":"info","ts":1658330208.8488204,"msg":"{\"identifier\":{\"type\":\"dns\",\"value\":\"seafile.intern.meinedomain.xyz\"},\"status\":\"pending\",\"challenges\":[{\"type\":\"dns-01\",\"status\":\"pending\",\"token\":\"Xl1Qm8nGW8arhzszNTKGStKsq7UYY34K\",\"url\":\"https://ca.intern.meinedomain.xyz/acme/local/challenge/sgrq5oa0hdPEimmctoMH5bDOzd8BTZMY/6tLhRJzq8JWKq9c3FWN1iKWITZhXV1Wv\"},{\"type\":\"http-01\",\"status\":\"pending\",\"token\":\"Xl1Qm8nGW8arhzszNTKGStKsq7UYY34K\",\"url\":\"https://ca.intern.meinedomain.xyz/acme/local/challenge/sgrq5oa0hdPEimmctoMH5bDOzd8BTZMY/5zOsOwNQiYQ35wW5TEtgxsx6SOen4z8h\"},{\"type\":\"tls-alpn-01\",\"status\":\"pending\",\"token\":\"Xl1Qm8nGW8arhzszNTKGStKsq7UYY34K\",\"url\":\"https://ca.intern.meinedomain.xyz/acme/local/challenge/sgrq5oa0hdPEimmctoMH5bDOzd8BTZMY/uoI507LIfB1R1KiAKh2f4NHMsRclJHdG\"}],\"wildcard\":false,\"expires\":\"2022-07-21T15:16:48Z\"}"}
{"level":"info","ts":1658330208.849938,"logger":"tls.issuance.acme.acme_client","msg":"trying to solve challenge","identifier":"seafile.intern.meinedomain.xyz","challenge_type":"http-01","ca":"https://ca.intern.meinedomain.xyz/acme/local/directory"}
{"level":"info","ts":1658330208.894568,"msg":"{\"identifier\":{\"type\":\"dns\",\"value\":\"calendarserver.intern.meinedomain.xyz\"},\"status\":\"pending\",\"challenges\":[{\"type\":\"dns-01\",\"status\":\"pending\",\"token\":\"2kDVNcBAsPBiwdC5mXk4hsdGZR8H988C\",\"url\":\"https://ca.intern.meinedomain.xyz/acme/local/challenge/ogN7gDXtiiZkhK13kHH5ze2psZdz9bKx/f9Zo42TlVBOv9fmQGN1BEIqM6bbXmmwW\"},{\"type\":\"http-01\",\"status\":\"pending\",\"token\":\"2kDVNcBAsPBiwdC5mXk4hsdGZR8H988C\",\"url\":\"https://ca.intern.meinedomain.xyz/acme/local/challenge/ogN7gDXtiiZkhK13kHH5ze2psZdz9bKx/1WxwTSjSG06o1qhiqJ0fHsIxnRuAuEAa\"},{\"type\":\"tls-alpn-01\",\"status\":\"pending\",\"token\":\"2kDVNcBAsPBiwdC5mXk4hsdGZR8H988C\",\"url\":\"https://ca.intern.meinedomain.xyz/acme/local/challenge/ogN7gDXtiiZkhK13kHH5ze2psZdz9bKx/GVV7ZrgjHlXmpw4n98i70i24MO26Nfyj\"}],\"wildcard\":false,\"expires\":\"2022-07-21T15:16:48Z\"}"}
{"level":"info","ts":1658330208.8955808,"logger":"tls.issuance.acme.acme_client","msg":"trying to solve challenge","identifier":"calendarserver.intern.meinedomain.xyz","challenge_type":"http-01","ca":"https://ca.intern.meinedomain.xyz/acme/local/directory"}
{"level":"info","ts":1658330208.9399152,"msg":"{\"identifier\":{\"type\":\"dns\",\"value\":\"kochbuch.intern.meinedomain.xyz\"},\"status\":\"pending\",\"challenges\":[{\"type\":\"dns-01\",\"status\":\"pending\",\"token\":\"omvWc76GoknjZ39cNNgQiUBDGdndDeBP\",\"url\":\"https://ca.intern.meinedomain.xyz/acme/local/challenge/Jespk0uSWk1gdjtxMt22LQ76H4srePLC/ChB72lbEW1quvL8jYKbTAHiFACWruhyl\"},{\"type\":\"http-01\",\"status\":\"pending\",\"token\":\"omvWc76GoknjZ39cNNgQiUBDGdndDeBP\",\"url\":\"https://ca.intern.meinedomain.xyz/acme/local/challenge/Jespk0uSWk1gdjtxMt22LQ76H4srePLC/TCAL7sqhMcq8sW82AXKBnKLhJ4vcnR6Q\"},{\"type\":\"tls-alpn-01\",\"status\":\"pending\",\"token\":\"omvWc76GoknjZ39cNNgQiUBDGdndDeBP\",\"url\":\"https://ca.intern.meinedomain.xyz/acme/local/challenge/Jespk0uSWk1gdjtxMt22LQ76H4srePLC/XhXZVhYIxObSlSEglwwmsE83lvYsO9pI\"}],\"wildcard\":false,\"expires\":\"2022-07-21T15:16:48Z\"}"}
{"level":"info","ts":1658330208.9406652,"logger":"tls.issuance.acme.acme_client","msg":"trying to solve challenge","identifier":"kochbuch.intern.meinedomain.xyz","challenge_type":"http-01","ca":"https://ca.intern.meinedomain.xyz/acme/local/directory"}
{"level":"info","ts":1658330209.0506094,"logger":"tls.issuance.acme","msg":"served key authentication","identifier":"seafile.intern.meinedomain.xyz","challenge":"http-01","remote":"172.19.0.1:47794","distributed":false}
{"level":"info","ts":1658330209.0755281,"msg":"{\"type\":\"http-01\",\"status\":\"valid\",\"token\":\"Xl1Qm8nGW8arhzszNTKGStKsq7UYY34K\",\"validated\":\"2022-07-20T15:16:49Z\",\"url\":\"https://ca.intern.meinedomain.xyz/acme/local/challenge/sgrq5oa0hdPEimmctoMH5bDOzd8BTZMY/5zOsOwNQiYQ35wW5TEtgxsx6SOen4z8h\"}"}
{"level":"info","ts":1658330209.0895958,"logger":"tls.issuance.acme","msg":"served key authentication","identifier":"kochbuch.intern.meinedomain.xyz","challenge":"http-01","remote":"172.19.0.1:47796","distributed":false}
{"level":"info","ts":1658330209.109323,"msg":"{\"type\":\"http-01\",\"status\":\"valid\",\"token\":\"omvWc76GoknjZ39cNNgQiUBDGdndDeBP\",\"validated\":\"2022-07-20T15:16:49Z\",\"url\":\"https://ca.intern.meinedomain.xyz/acme/local/challenge/Jespk0uSWk1gdjtxMt22LQ76H4srePLC/TCAL7sqhMcq8sW82AXKBnKLhJ4vcnR6Q\"}"}
{"level":"info","ts":1658330209.1128495,"logger":"tls.issuance.acme","msg":"served key authentication","identifier":"calendarserver.intern.meinedomain.xyz","challenge":"http-01","remote":"172.19.0.1:47798","distributed":false}
{"level":"info","ts":1658330209.1321495,"msg":"{\"type\":\"http-01\",\"status\":\"valid\",\"token\":\"2kDVNcBAsPBiwdC5mXk4hsdGZR8H988C\",\"validated\":\"2022-07-20T15:16:49Z\",\"url\":\"https://ca.intern.meinedomain.xyz/acme/local/challenge/ogN7gDXtiiZkhK13kHH5ze2psZdz9bKx/1WxwTSjSG06o1qhiqJ0fHsIxnRuAuEAa\"}"}
{"level":"info","ts":1658330209.412276,"msg":"{\"identifier\":{\"type\":\"dns\",\"value\":\"seafile.intern.meinedomain.xyz\"},\"status\":\"valid\",\"challenges\":[{\"type\":\"dns-01\",\"status\":\"pending\",\"token\":\"Xl1Qm8nGW8arhzszNTKGStKsq7UYY34K\",\"url\":\"https://ca.intern.meinedomain.xyz/acme/local/challenge/sgrq5oa0hdPEimmctoMH5bDOzd8BTZMY/6tLhRJzq8JWKq9c3FWN1iKWITZhXV1Wv\"},{\"type\":\"http-01\",\"status\":\"valid\",\"token\":\"Xl1Qm8nGW8arhzszNTKGStKsq7UYY34K\",\"validated\":\"2022-07-20T15:16:49Z\",\"url\":\"https://ca.intern.meinedomain.xyz/acme/local/challenge/sgrq5oa0hdPEimmctoMH5bDOzd8BTZMY/5zOsOwNQiYQ35wW5TEtgxsx6SOen4z8h\"},{\"type\":\"tls-alpn-01\",\"status\":\"pending\",\"token\":\"Xl1Qm8nGW8arhzszNTKGStKsq7UYY34K\",\"url\":\"https://ca.intern.meinedomain.xyz/acme/local/challenge/sgrq5oa0hdPEimmctoMH5bDOzd8BTZMY/uoI507LIfB1R1KiAKh2f4NHMsRclJHdG\"}],\"wildcard\":false,\"expires\":\"2022-07-21T15:16:48Z\"}"}
{"level":"info","ts":1658330209.4137304,"logger":"tls.issuance.acme.acme_client","msg":"validations succeeded; finalizing order","order":"https://ca.intern.meinedomain.xyz/acme/local/order/jna1w77nKs1Z0YG8aoEDW0YViB82CKCD"}
{"level":"info","ts":1658330209.5275977,"msg":"{\"identifier\":{\"type\":\"dns\",\"value\":\"kochbuch.intern.meinedomain.xyz\"},\"status\":\"valid\",\"challenges\":[{\"type\":\"dns-01\",\"status\":\"pending\",\"token\":\"omvWc76GoknjZ39cNNgQiUBDGdndDeBP\",\"url\":\"https://ca.intern.meinedomain.xyz/acme/local/challenge/Jespk0uSWk1gdjtxMt22LQ76H4srePLC/ChB72lbEW1quvL8jYKbTAHiFACWruhyl\"},{\"type\":\"http-01\",\"status\":\"valid\",\"token\":\"omvWc76GoknjZ39cNNgQiUBDGdndDeBP\",\"validated\":\"2022-07-20T15:16:49Z\",\"url\":\"https://ca.intern.meinedomain.xyz/acme/local/challenge/Jespk0uSWk1gdjtxMt22LQ76H4srePLC/TCAL7sqhMcq8sW82AXKBnKLhJ4vcnR6Q\"},{\"type\":\"tls-alpn-01\",\"status\":\"pending\",\"token\":\"omvWc76GoknjZ39cNNgQiUBDGdndDeBP\",\"url\":\"https://ca.intern.meinedomain.xyz/acme/local/challenge/Jespk0uSWk1gdjtxMt22LQ76H4srePLC/XhXZVhYIxObSlSEglwwmsE83lvYsO9pI\"}],\"wildcard\":false,\"expires\":\"2022-07-21T15:16:48Z\"}"}
{"level":"info","ts":1658330209.5289657,"logger":"tls.issuance.acme.acme_client","msg":"validations succeeded; finalizing order","order":"https://ca.intern.meinedomain.xyz/acme/local/order/2JT0ef9TF9npSD5S3xz5XLC58Puf9i2E"}
{"level":"info","ts":1658330209.573187,"msg":"{\"identifier\":{\"type\":\"dns\",\"value\":\"calendarserver.intern.meinedomain.xyz\"},\"status\":\"valid\",\"challenges\":[{\"type\":\"dns-01\",\"status\":\"pending\",\"token\":\"2kDVNcBAsPBiwdC5mXk4hsdGZR8H988C\",\"url\":\"https://ca.intern.meinedomain.xyz/acme/local/challenge/ogN7gDXtiiZkhK13kHH5ze2psZdz9bKx/f9Zo42TlVBOv9fmQGN1BEIqM6bbXmmwW\"},{\"type\":\"http-01\",\"status\":\"valid\",\"token\":\"2kDVNcBAsPBiwdC5mXk4hsdGZR8H988C\",\"validated\":\"2022-07-20T15:16:49Z\",\"url\":\"https://ca.intern.meinedomain.xyz/acme/local/challenge/ogN7gDXtiiZkhK13kHH5ze2psZdz9bKx/1WxwTSjSG06o1qhiqJ0fHsIxnRuAuEAa\"},{\"type\":\"tls-alpn-01\",\"status\":\"pending\",\"token\":\"2kDVNcBAsPBiwdC5mXk4hsdGZR8H988C\",\"url\":\"https://ca.intern.meinedomain.xyz/acme/local/challenge/ogN7gDXtiiZkhK13kHH5ze2psZdz9bKx/GVV7ZrgjHlXmpw4n98i70i24MO26Nfyj\"}],\"wildcard\":false,\"expires\":\"2022-07-21T15:16:48Z\"}"}

not working:

{"level":"info","ts":1658361408.0425854,"logger":"tls.cache.maintenance","msg":"certificate expires soon; queuing for renewal","identifiers":["ca.intern.meinedomain.xyz"],"remaining":14399.957419457}
{"level":"info","ts":1658361408.042717,"logger":"tls.cache.maintenance","msg":"attempting certificate renewal","identifiers":["ca.intern.meinedomain.xyz"],"remaining":14399.957286186}
{"level":"info","ts":1658361408.043284,"logger":"tls.renew","msg":"acquiring lock","identifier":"ca.intern.meinedomain.xyz"}
{"level":"info","ts":1658361408.067695,"logger":"tls.renew","msg":"lock acquired","identifier":"ca.intern.meinedomain.xyz"}
{"level":"info","ts":1658361408.0685546,"logger":"tls.renew","msg":"renewing certificate","identifier":"ca.intern.meinedomain.xyz","remaining":14399.931450816}
{"level":"info","ts":1658361408.0724697,"logger":"tls.renew","msg":"certificate renewed successfully","identifier":"ca.intern.meinedomain.xyz"}
{"level":"info","ts":1658361408.072546,"logger":"tls.renew","msg":"releasing lock","identifier":"ca.intern.meinedomain.xyz"}
{"level":"info","ts":1658361408.072766,"logger":"tls","msg":"reloading managed certificate","identifiers":["ca.intern.meinedomain.xyz"]}
{"level":"warn","ts":1658361408.0731246,"logger":"tls","msg":"stapling OCSP","error":"no OCSP stapling for [ca.intern.meinedomain.xyz]: no OCSP server specified in certificate","identifiers":["ca.intern.meinedomain.xyz"]}
{"level":"info","ts":1658361408.0731466,"logger":"tls.cache","msg":"replaced certificate in cache","subjects":["ca.intern.meinedomain.xyz"],"new_expiration":1658404608}
{"level":"info","ts":1658387807.9444728,"logger":"tls.cache.maintenance","msg":"certificate expires soon; queuing for renewal","identifiers":["seafile.intern.meinedomain.xyz"],"remaining":14400.055531346}
{"level":"info","ts":1658387807.944581,"logger":"tls.cache.maintenance","msg":"certificate expires soon; queuing for renewal","identifiers":["calendarserver.intern.meinedomain.xyz"],"remaining":14400.055420761}
{"level":"info","ts":1658387807.9446054,"logger":"tls.cache.maintenance","msg":"certificate expires soon; queuing for renewal","identifiers":["kochbuch.intern.meinedomain.xyz"],"remaining":14400.055395627}
{"level":"info","ts":1658387807.9450932,"logger":"tls.cache.maintenance","msg":"attempting certificate renewal","identifiers":["seafile.intern.meinedomain.xyz"],"remaining":14400.054919135}
{"level":"info","ts":1658387807.9453645,"logger":"tls.cache.maintenance","msg":"attempting certificate renewal","identifiers":["calendarserver.intern.meinedomain.xyz"],"remaining":14400.054638841}
{"level":"info","ts":1658387807.9457,"logger":"tls.renew","msg":"acquiring lock","identifier":"seafile.intern.meinedomain.xyz"}
{"level":"info","ts":1658387807.945888,"logger":"tls.renew","msg":"acquiring lock","identifier":"calendarserver.intern.meinedomain.xyz"}
{"level":"info","ts":1658387807.9609087,"logger":"tls.renew","msg":"lock acquired","identifier":"calendarserver.intern.meinedomain.xyz"}
{"level":"info","ts":1658387807.961789,"logger":"tls.renew","msg":"renewing certificate","identifier":"calendarserver.intern.meinedomain.xyz","remaining":14400.038216494}
{"level":"info","ts":1658387807.9628115,"logger":"tls.cache.maintenance","msg":"attempting certificate renewal","identifiers":["kochbuch.intern.meinedomain.xyz"],"remaining":14400.037192817}
{"level":"info","ts":1658387807.9633515,"logger":"tls.renew","msg":"acquiring lock","identifier":"kochbuch.intern.meinedomain.xyz"}
{"level":"info","ts":1658387807.964639,"logger":"tls.issuance.acme","msg":"waiting on internal rate limiter","identifiers":["calendarserver.intern.meinedomain.xyz"],"ca":"https://ca.intern.meinedomain.xyz/acme/local/directory","account":""}
{"level":"info","ts":1658387807.9646966,"logger":"tls.issuance.acme","msg":"done waiting on internal rate limiter","identifiers":["calendarserver.intern.meinedomain.xyz"],"ca":"https://ca.intern.meinedomain.xyz/acme/local/directory","account":""}
{"level":"warn","ts":1658387807.9673185,"logger":"tls.issuance.acme.acme_client","msg":"HTTP request failed; retrying","url":"https://ca.intern.meinedomain.xyz/acme/local/new-nonce","error":"performing request: Head \"https://ca.intern.meinedomain.xyz/acme/local/new-nonce\": x509: certificate signed by unknown authority"}
{"level":"info","ts":1658387807.9718478,"logger":"tls.renew","msg":"lock acquired","identifier":"seafile.intern.meinedomain.xyz"}
{"level":"info","ts":1658387807.9725118,"logger":"tls.renew","msg":"renewing certificate","identifier":"seafile.intern.meinedomain.xyz","remaining":14400.027491526}
{"level":"info","ts":1658387807.9736643,"logger":"tls.issuance.acme","msg":"waiting on internal rate limiter","identifiers":["seafile.intern.meinedomain.xyz"],"ca":"https://ca.intern.meinedomain.xyz/acme/local/directory","account":""}
{"level":"info","ts":1658387807.9737191,"logger":"tls.issuance.acme","msg":"done waiting on internal rate limiter","identifiers":["seafile.intern.meinedomain.xyz"],"ca":"https://ca.intern.meinedomain.xyz/acme/local/directory","account":""}
{"level":"warn","ts":1658387807.9763732,"logger":"tls.issuance.acme.acme_client","msg":"HTTP request failed; retrying","url":"https://ca.intern.meinedomain.xyz/acme/local/new-nonce","error":"performing request: Head \"https://ca.intern.meinedomain.xyz/acme/local/new-nonce\": x509: certificate signed by unknown authority"}
{"level":"info","ts":1658387807.9946344,"logger":"tls.renew","msg":"lock acquired","identifier":"kochbuch.intern.meinedomain.xyz"}
{"level":"info","ts":1658387807.9957125,"logger":"tls.renew","msg":"renewing certificate","identifier":"kochbuch.intern.meinedomain.xyz","remaining":14400.00429297}
{"level":"info","ts":1658387807.997501,"logger":"tls.issuance.acme","msg":"waiting on internal rate limiter","identifiers":["kochbuch.intern.meinedomain.xyz"],"ca":"https://ca.intern.meinedomain.xyz/acme/local/directory","account":""}
{"level":"info","ts":1658387807.9975646,"logger":"tls.issuance.acme","msg":"done waiting on internal rate limiter","identifiers":["kochbuch.intern.meinedomain.xyz"],"ca":"https://ca.intern.meinedomain.xyz/acme/local/directory","account":""}
{"level":"warn","ts":1658387808.0041416,"logger":"tls.issuance.acme.acme_client","msg":"HTTP request failed; retrying","url":"https://ca.intern.meinedomain.xyz/acme/local/new-nonce","error":"performing request: Head \"https://ca.intern.meinedomain.xyz/acme/local/new-nonce\": x509: certificate signed by unknown authority"}

5. What I already tried:

Restarting the container does not help.
Deleting all volumes (including data & config) restarts the counter (5 days) but this is not a satisfying solution.

6. Links to relevant resources:

2

I believe that error means the certificate isn’t signed by a known authority; i.e. the signer’s certificate is not installed into the trust store or is not configured as trusted.

Is there anything in the logs that says “intermediate expires soon”? What do the logs show if you turn on debug mode? (Uncomment the “debug” in your global options.) Look for anything in the logs that says “intermediate” (but the full logs would help – although this is a good start).

3

I could not upload the whole log file because it was too large (>65k characters and I could upload only 32k characters).
This error only pops up after the intermediate expired.
The log has only two lines about the intermediate:

{"level":"info","ts":1658353608.5594645,"logger":"pki","msg":"intermediate expires soon; renewing","ca":"local","time_remaining":120456.440537902}
{"level":"info","ts":1658353608.5631812,"logger":"pki","msg":"renewed intermediate","ca":"local","new_expiration":1658958408}

Maybe it renewed the intermediate but the new cert is never used (even after a complete reboot of the host the old intermediate is used).
I can activate the debug mode , but how to upload the log file?

1 Like
4

Are you sure this is the version you’re using?

Please double-check by running caddy version inside the container.

This was fixed a few versions ago, which is why I want to make absolutely sure.

1 Like
5

Yeah I’m having a slight panic attack over here :mildpanic:

Strange that a reboot doesn’t pick up the new cert. That seems like there is something else going on.

@migibtsnet Can you please 100% verify that the old certificate is used? A dump from curl -v and/or openssl s_client would be ideal. I want to see NotAfter dates and also what’s on disk in the pki subfolder of Caddy’s data folder: Conventions — Caddy Documentation

6

Here are the results:

> docker exec -it caddy /bin/sh
> /srv # caddy version
v2.5.2 h1:eCJdLyEyAGzuQTa5Mh3gETnYWDClo1LjtQm2q9RNZrs=

> openssl s_client -connect calendarserver.intern.meinedomain.xyz:443
CONNECTED(00000003)
depth=1 CN = Caddy Local Authority - ECC Intermediate
verify error:num=20:unable to get local issuer certificate
verify return:1
depth=0
verify error:num=10:certificate has expired
notAfter=Jul 21 11:16:48 2022 GMT
verify return:1
depth=0
notAfter=Jul 21 11:16:48 2022 GMT
verify return:1
---
Certificate chain
 0 s:
   i:CN = Caddy Local Authority - ECC Intermediate
 1 s:CN = Caddy Local Authority - ECC Intermediate
   i:CN = ca.intern.meinedomain.xyz
---
Server certificate
-----BEGIN CERTIFICATE-----
MIIB+TCCAaCgAwIBAgIQRzXuZNZe89FXZlMmtILu7jAKBggqhkjOPQQDAjAzMTEw
LwYDVQQDEyhDYWRkeSBMb2NhbCBBdXRob3JpdHkgLSBFQ0MgSW50ZXJtZWRpYXRl
MB4XDTIyMDcyMDIzMTU0OFoXDTIyMDcyMTExMTY0OFowADBZMBMGByqGSM49AgEG
CCqGSM49AwEHA0IABAWfLEOI8njTCnBS0wab0LJsNe1cMixMbZv9Syq6nRY3mEKm
5MGqgh24kL9wDrwvSCDQj5fj6KUrITN+h5WrvB2jgcgwgcUwDgYDVR0PAQH/BAQD
AgeAMB0GA1UdJQQWMBQGCCsGAQUFBwMBBggrBgEFBQcDAjAdBgNVHQ4EFgQUpTw2
uH6nuJQE8qXF53gzf4iL8DswHwYDVR0jBBgwFoAUtfR9M2rmwF+qEAwQQEkFzWq1
kGAwNAYDVR0RAQH/BCowKIImY2FsZW5kYXJzZXJ2ZXIuaW50ZXJuLmx1dHRlbmJl
cmdlci54eXowHgYMKwYBBAGCpGTGKEABBA4wDAIBBgQFbG9jYWwEADAKBggqhkjO
PQQDAgNHADBEAiALsm0bQvBXwIbHueL+5wQLgdZZJbadybk7wH9Ysihq9gIgOzIU
ebiuVxQuzl8v2PaGn9ASU+BlkBFEWJebaz2Mb3k=
-----END CERTIFICATE-----
subject=

issuer=CN = Caddy Local Authority - ECC Intermediate

---
No client certificate CA names sent
Peer signing digest: SHA256
Peer signature type: ECDSA
Server Temp Key: X25519, 253 bits
---
SSL handshake has read 1318 bytes and written 394 bytes
Verification error: certificate has expired
---
New, TLSv1.3, Cipher is TLS_AES_128_GCM_SHA256
Server public key is 256 bit
Secure Renegotiation IS NOT supported
Compression: NONE
Expansion: NONE
No ALPN negotiated
Early data was not sent
Verify return code: 10 (certificate has expired)
---
---
Post-Handshake New Session Ticket arrived:
SSL-Session:
    Protocol  : TLSv1.3
    Cipher    : TLS_AES_128_GCM_SHA256
    Session-ID: A5A25929D1DB6C792BD08B3EDE1E124EE74793E89259C475A747AC8AA07A1267
    Session-ID-ctx:
    Resumption PSK: 0A6AFFB8EC4C29654A0E5C568B803B235E6A92672F939376495B22C732AD9C32
    PSK identity: None
    PSK identity hint: None
    SRP username: None
    TLS session ticket lifetime hint: 604800 (seconds)
    TLS session ticket:
    0000 - 56 31 0e 4c 70 4a ba 71-02 1b 9f f2 6f 52 89 de   V1.LpJ.q....oR..
    0010 - 39 f6 40 cd c5 5f bc 6a-44 e2 b1 8c 4c 75 13 3b   9.@.._.jD...Lu.;
    0020 - 68 20 da f5 4b 3c bb e7-95 83 f1 c5 ea 30 7a ff   h ..K<.......0z.
    0030 - b5 89 6b d7 19 a6 3a fd-ca aa 16 63 27 b0 48 17   ..k...:....c'.H.
    0040 - 08 c1 00 15 ac 9a 22 f4-29 f8 d8 98 b9 0f 0b d1   ......".).......
    0050 - e7 7f 58 86 23 3b 06 98-45 bb 31 50 14 f7 9d 34   ..X.#;..E.1P...4
    0060 - a6 90 02 74 a1 9b a5 24-d7 2d e0 bc cc ec 86 69   ...t...$.-.....i
    0070 - 11                                                .

    Start Time: 1658784613
    Timeout   : 7200 (sec)
    Verify return code: 10 (certificate has expired)
    Extended master secret: no
    Max Early Data: 0
---
read R BLOCK

After this I restarted docker again (not a reboot of the host but of the docker-service with all containers). But again - the old intermediate is offered.
Very interesting is the content of the pki folder:

> /data/caddy/pki/authorities/local # ls -la
total 16
drwx------    2 root     root            86 Jul 20 21:46 .
drwx------    3 root     root            19 Jul 15 07:14 ..
-rw-------    1 root     root           664 Jul 20 21:46 intermediate.crt
-rw-------    1 root     root           227 Jul 20 21:46 intermediate.key
-rw-------    1 root     root           599 Jul 20 21:46 root.crt
-rw-------    1 root     root           227 Jul 20 21:46 root.key

> openssl x509 -noout -text -in intermediate.crt
Certificate:
    Data:
        Version: 3 (0x2)
        Serial Number:
            49:e4:84:89:17:93:d0:d4:df:42:b3:b2:db:69:fe:79
        Signature Algorithm: ecdsa-with-SHA256
        Issuer: CN = ca.intern.meinedomain.xyz
        Validity
            Not Before: Jul 20 21:46:48 2022 GMT
            Not After : Jul 27 21:46:48 2022 GMT
        Subject: CN = Caddy Local Authority - ECC Intermediate
        Subject Public Key Info:
            Public Key Algorithm: id-ecPublicKey
                Public-Key: (256 bit)
                pub:
                    04:83:b4:60:bb:d5:69:4f:f8:58:6a:72:f0:57:d4:
                    81:9e:ac:e8:37:5c:f3:c0:a8:7d:5b:c4:59:87:15:
                    56:31:09:cf:d8:70:a9:30:6c:a9:be:f9:01:eb:dc:
                    45:39:d2:a3:b4:c7:d1:d8:3c:8b:52:0e:32:1b:12:
                    18:e1:3c:9c:67
                ASN1 OID: prime256v1
                NIST CURVE: P-256
        X509v3 extensions:
            X509v3 Key Usage: critical
                Certificate Sign, CRL Sign
            X509v3 Basic Constraints: critical
                CA:TRUE, pathlen:0
            X509v3 Subject Key Identifier:
                B5:F4:7D:33:6A:E6:C0:5F:AA:10:0C:10:40:49:05:CD:6A:B5:90:60
            X509v3 Authority Key Identifier:
                keyid:2E:AF:1F:B5:01:90:51:7F:25:CC:6B:C6:5A:25:5D:F4:AD:23:BC:3A

    Signature Algorithm: ecdsa-with-SHA256
         30:45:02:20:51:c9:19:e2:61:88:42:28:61:4b:ff:4c:a1:ce:
         3a:79:d7:4f:0e:56:01:75:ba:5a:76:c7:e9:76:70:42:b0:f5:
         02:21:00:90:a1:c5:35:82:f0:27:eb:da:28:42:cb:26:f9:84:
         b2:ba:b5:8c:24:fd:8f:f7:b7:30:52:14:68:da:8b:73:04

So a new intermediate certificate was generated but it seems that it is not loaded for some reason

1 Like
7

I don’t understand from where it could possibly be taking that expired certificate.

Something’s fishy here with the storage…

8

Thanks for the added information. That’s very interesting!

I have to admit I am stumped how the file on disk is renewed but Caddy continues to somehow load the old certificate after a reboot even though that certificate doesn’t exist anymore. :thinking:

Are you 100% sure the volumes and underlying storage are properly configured?

As a last resort we could probably provide a commit with additional debug logs, you could build from source and we could see what the output is to try to learn more.

9

Thanks for the fast answer. I am pretty sure the storage/volume is configured the right way (I used the docker manual as reference)
But I don’t understand, why this should be a storage error? There are multiple services with and without container on this host running using this volume. Only one service has a problem with one file which is not loaded on the expected time.
To me it does not sound like a file system error but like some sort of app-caching problem. Do you cache the certificates for faster renewal of client certificates or is there some sort of cert-store or a combined cert-file with all certs in it? Maybe a missing/not working script?
Is there anything I can do for the moment? What will happen, if I delete the intermediate crt/key in the pki folder?

10

Only in memory. It’s not stored anywhere else.

Which is why we’re so confused, because from where the heck is it loading the expired one???

The intermediate cert and key are definitely only stored at /data/caddy/pki/authorities/local/intermediate.{crt,key}

It should regenerate it. Just make sure to restart Caddy to trigger it.

1 Like
11

I move the container to a new volume on a new disk.
I will also create a second container to compare the old and a new one. Maybe I find something to work with.
This will last 1 or 2 days so please don’t close the ticket.

1 Like
12

Ok, I think I found the problem.
The problem is the way, the client certificate ist saved. You save the cert including the intermediate certificate - so you have a nice cert-path but with the old intermediate certificate.
Take a look at this (all certs are real so only the first and the last line is used):
First the old, still not working Caddy-instance:
The actual intermediate looks like this:

-----BEGIN CERTIFICATE-----
MIIBvTCCAWKgAwIBAgIQRoX3pm8uJdf4ix8xZ4hA1jAKBggqhkjOPQQDAjAlMSMw
[…]
zLXFygmVVQma3z8d9UqUjeQ=
-----END CERTIFICATE-----

and the old one:

-----BEGIN CERTIFICATE-----
MIIB+TCCAaCgAwIBAgIQRzXuZNZe89FXZlMmtILu7jAKBggqhkjOPQQDAjAzMTEw
[…]
ebiuVxQuzl8v2PaGn9ASU+BlkBFEWJebaz2Mb3k=
-----END CERTIFICATE-----

Now take a look at the cert-file of the calendarserver:
> ls /data/caddy/certificates/ca.intern.meinedomain.xyz-acme-local-directory/calendarserver.intern.meinedomain.xyz/
calendarserver.intern.meinedomain.xyz.crt calendarserver.intern.meinedomain.xyz.json calendarserver.intern.meinedomain.xyz.key
> cat /data/caddy/certificates/ca.intern.meinedomain.xyz-acme-local-directory/calendarserver.intern.meinedomain.xyz/calendarserver.intern.meinedomain.xyz.crt

-----BEGIN CERTIFICATE-----
MIIB+TCCAaCgAwIBAgIQRzXuZNZe89FXZlMmtILu7jAKBggqhkjOPQQDAjAzMTEw
[…]
ebiuVxQuzl8v2PaGn9ASU+BlkBFEWJebaz2Mb3k=
-----END CERTIFICATE-----
-----BEGIN CERTIFICATE-----
MIIBvDCCAWKgAwIBAgIQSeSEiReT0NTfQrOy22n+eTAKBggqhkjOPQQDAjAlMSMw
[…]
urWMJP2P97cwUhRo2otzBA==
-----END CERTIFICATE-----

Quick check of the new instance (the one, I started today):
A look at the intermediate certificate:
> cat /data/caddy/pki/authorities/local/intermediate.crt

-----BEGIN CERTIFICATE-----
MIID8jCCAdqgAwIBAgIQKM3ve47yaGKXHmlxcCr8bDANBgkqhkiG9w0BAQsFADCB
[…]
eLn3axmC
-----END CERTIFICATE-----

Now the certificate of the calendarserver:
> cat _data/caddy/certificates/ca.intern.meinedomain.xyz-acme-local-directory/calendarserver.intern.meinedomain.xyz/calendarserver.intern.meinedomain.xyz.crt

-----BEGIN CERTIFICATE-----
MIIB+TCCAaCgAwIBAgIQOjc+t/5yxYC7SdPShvZxHTAKBggqhkjOPQQDAjAzMTEw
[…]
GxowIIKNtLfX4DHLBAQQtb2+4+X4FxDTFQIdZ9g=
-----END CERTIFICATE-----
-----BEGIN CERTIFICATE-----
MIID8jCCAdqgAwIBAgIQKM3ve47yaGKXHmlxcCr8bDANBgkqhkiG9w0BAQsFADCB
[…]
eLn3axmC
-----END CERTIFICATE-----

Now it is clear, why I still get the old intermediate certificate - because it is hard coded in the client certificate. :slight_smile:

1 Like
13

Ah yeah, that makes sense! Thanks for digging a little deeper.

So why is the leaf certificate not being renewed… Hmm. Intermediates should be renewed well before it is used to issue any certs that go beyond the intermediate’s lifetime. I think. What are the NotBefore and NotAfter dates of the certs?

14

To be honest, it is a little bit confusing to me what is happening with the cert-files but this are the actual dates from the not working and the new Caddy-instance (both started a few minutes ago)

Old Caddy:
Actual intermediate.crt (but not used):

Not Before: Jul 26 14:02:14 2022 GMT
Not After : Aug 2 14:02:14 2022 GMT

Old intermediate.crt (the one in the certificates):

Not Before: Jul 20 21:46:48 2022 GMT
Not After : Jul 27 21:46:48 2022 GMT

calendarserver.crt (not renewed anymore):

Not Before: Jul 20 23:15:48 2022 GMT
Not After : Jul 21 11:16:48 2022 GMT

New Caddy:
Actual intermediate.crt (also in the calendarserver-cert-file):

Not Before: Jul 28 06:26:18 2022 GMT
Not After : Aug 4 06:26:18 2022 GMT

calendarserver.crt:

Not Before: Jul 28 22:25:26 2022 GMT
Not After : Jul 29 10:26:26 2022 GMT

1 Like
15

Hmm. I think you could just do tls internal here instead, since they’re on the same Caddy instance.

That would be the same thing, but actually using the internal issuer instead of the acme issuer which will do HTTP requests.

1 Like
16

My first attempt was with tls internal but that didn’t work as expected. All container used https but my existing root-certificate was never used (maybe a configuration error, but I don’t have the config-files anymore). This wasn’t what I needed so I switched to the acme-server.
The other thing is - in my test lab all containers are running on one host but the acme server is a really cool feature which I want to use in one or more production environments. So finding and fixing the error would be nice :slight_smile:

17

This topic was automatically closed after 30 days. New replies are no longer allowed.