Caddy in Docker container does not trust its own Root CA certificate automatically

1. Caddy version (caddy version):

v2.4.5

2. How I run Caddy:

a. System environment:

I’m trying to implement this here at my home: Caddy reverse proxy + Nextcloud + Collabora + Bitwarden_rs with local HTTPS

  • Docker on Raspi 4 (hostname: rowena) behind private DSL (port 80 and 443 forwarded to Pi, DynDNS etc. all working fine)
  • Caddy container to be used as reverse proxy
  • Nextcloud as backend service
  • Caddy and Nextcloud service connected via Docker network

Goal: TLS between Frontend (reverse proxy) and backend (Nextcloud)

b. Command:

docker-compose up

c. Service/unit/compose file:

version: "3.8"

volumes:
  caddydata:
    name: proxy-caddydata
    external: true

networks:
  proxy:
    name: proxy-network

services:
  caddy:
    container_name: proxy
    image: caddy:latest
    restart: unless-stopped
    security_opt:
      - no-new-privileges:true
    networks:
      - proxy
    ports:
      - 80:80
      - 443:443
    volumes:
      - ./etc-caddy:/etc/caddy:ro
      - caddydata:/data

d. My complete Caddyfile or JSON config:

Caddyfile of reverse proxy container (FRONTEND):

{
	debug
	email hostmaster@mydomain.tld
	auto_https disable_redirects
}

localhost, rowena, rowena.local, proxy {
	acme_server
	tls internal
}

https://localhost, https://rowena, https://rowena.local {
	tls internal

        # just some test forwarding to my laptop
	reverse_proxy /8000 http://ronbook:8000 {
		header_up Host {http.reverse_proxy.upstream.hostport}
		header_up X-Forwarded-Host {host}
	}
        # just some test forwarding to another Raspi
	reverse_proxy http://altair {
		header_up Host {http.reverse_proxy.upstream.hostport}
		header_up X-Forwarded-Host {host}
	}
}

http://localhost, http://rowena, http://rowena.local {
	respond "Hello World!"
}

# proxying to another Raspi with old Nextcloud setup (incl. TLS)
# not relevant for the bug
cloud.mydomain.tld {
	reverse_proxy https://altair {
		#header_up Host {http.reverse_proxy.upstream.hostport}
		header_up Host cloud.mydomain.tld:443
		header_up X-Forwarded-Host {host}
		transport http {
			tls_server_name cloud.mydomain.tld
		}
	}
}

# this is what I want to get working
test.mydomain.tld {
	reverse_proxy https://ncweb {
		header_up Host {http.reverse_proxy.upstream.hostport}
		header_up X-Forwarded-Host {host}
	}
}

Caddyfile of Nextcloud service (BACKEND):

{
	debug

	# TLS Options
	acme_ca https://proxy/acme/local/directory
	acme_ca_root /etc/ssl/certs/proxy_ca_root.crt
}

https://ncweb {	
    # omitted here to keep it shorter
}

So I’m using the ACME server on the frontend caddy to issue a certificate to the backend caddy.
In Backend Caddyfile I point to the ACME server. I also provisioned the root certificate of the Frontend ACME to the Backend container and I point to it in the Backend Caddyfile.

3. The problem I’m having:

First time I started both containers (frontend and backend), certificate for the backend service was issued properly by the ACME server on the frontend.

If I now try to open the webpage by calling test.mydomain.tld, I get a 502 in the browser and error messages in the logs about an untrusted certificate.

4. Error messages and/or full log output:

On Backend Caddy (Nextcloud):

ncweb_1    | {"level":"debug","ts":1631832875.4458847,"logger":"http.stdlib","msg":"http: TLS handshake error from 192.168.128.2:42892: remote error: tls: bad certificate"}

On Frontend Caddy:

{
    "level": "debug",
    "ts": 1631829601.2283535,
    "logger": "http.handlers.reverse_proxy",
    "msg": "upstream roundtrip",
    "upstream": "ncweb:443",
    "request": {
        "remote_addr": "172.29.0.1:45316",
        "proto": "HTTP/2.0",
        "method": "GET",
        "host": "ncweb:443",
        "uri": "/",
        "headers": {
            "Te": [
                "trailers"
            ],
            "X-Forwarded-Proto": [
                "https"
            ],
            "Cookie": [
                # ....
            ],
            "X-Forwarded-Host": [
                "test.mydomain.tld"
            ],
            "Cache-Control": [
                "max-age=0"
            ],
            "User-Agent": [
                "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:91.0) Gecko/20100101 Firefox/91.0"
            ],
            "Accept": [
                "text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8"
            ],
            "Sec-Fetch-Site": [
                "none"
            ],
            "Upgrade-Insecure-Requests": [
                "1"
            ],
            "X-Forwarded-For": [
                "172.29.0.1"
            ],
            "Sec-Fetch-User": [
                "?1"
            ],
            "Accept-Encoding": [
                "gzip, deflate, br"
            ],
            "Sec-Fetch-Mode": [
                "navigate"
            ],
            "Accept-Language": [
                "en-US,en;q=0.5"
            ],
            "Sec-Fetch-Dest": [
                "document"
            ]
        },
        "tls": {
            "resumed": false,
            "version": 772,
            "cipher_suite": 4867,
            "proto": "h2",
            "proto_mutual": true,
            "server_name": "test.mydomain.tld"
        }
    },
    "duration": 0.017470574,
    "error": "x509: certificate signed by unknown authority"
}


{
    "level": "error",
    "ts": 1631829601.2297077,
    "logger": "http.log.error",
    "msg": "x509: certificate signed by unknown authority",
    "request": {
        "remote_addr": "172.29.0.1:45316",
        "proto": "HTTP/2.0",
        "method": "GET",
        "host": "test.mydomain.tld",
        "uri": "/",
        "headers": {
            "Sec-Fetch-Mode": [
                "navigate"
            ],
            "Cache-Control": [
                "max-age=0"
            ],
            "Accept-Language": [
                "en-US,en;q=0.5"
            ],
            "Accept-Encoding": [
                "gzip, deflate, br"
            ],
            "Sec-Fetch-Dest": [
                "document"
            ],
            "Upgrade-Insecure-Requests": [
                "1"
            ],
            "Sec-Fetch-Site": [
                "none"
            ],
            "Sec-Fetch-User": [
                "?1"
            ],
            "Te": [
                "trailers"
            ],
            "User-Agent": [
                "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:91.0) Gecko/20100101 Firefox/91.0"
            ],
            "Accept": [
                "text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8"
            ],
            "Cookie": [
                "nc_sameSiteCookielax=true; nc_sameSiteCookiestrict=true; oc_sessionPassphrase=e1RNAlEmNFjO8Fork%2FEL9jXN3XpZ3DB82ah%2BhRZt37gIJBJIpywG1Db0l0WfCwDH2cLmVVXQhJaKDYTSXEFHQmAcQ6NkIVCdkHFj5xg%2Bymyo1uur2jbQ0Qq%2BqW5hv4q0; oc7zsgh7x0fv=c5a4cc056e1d173c83ac82f997ce7bb5"
            ]
        },
        "tls": {
            "resumed": false,
            "version": 772,
            "cipher_suite": 4867,
            "proto": "h2",
            "proto_mutual": true,
            "server_name": "test.mydomain.tld"
        }
    },
    "duration": 0.019264899,
    "status": 502,
    "err_id": "13in481tx",
    "err_trace": "reverseproxy.statusError (reverseproxy.go:858)"
}

Frontend Caddy logs show this during startup:

{"level":"warn","ts":1631830791.917869,"logger":"pki.ca.local","msg":"installing root certificate (you might be prompted for password)","path":"storage:pki/authorities/local/root.crt"}
2021/09/16 22:19:51 Warning: "certutil" is not available, install "certutil" with "apt install libnss3-tools" or "yum install nss-tools" and try again
2021/09/16 22:19:51 define JAVA_HOME environment variable to use the Java trust
2021/09/16 22:19:52 certificate installed properly in linux trusts

5. What I already tried:

/srv # caddy trust

2021/09/16 23:42:22.650	INFO	ca.local	root certificate is already trusted by system	{"path": "storage:pki/authorities/local/root.crt"}

Caddy Root CA cert is already in trusted cert list:

/srv # ls -lrt /etc/ssl/certs/

    ...

-rw-r--r--    1 root     root        214685 Sep 16 22:53 ca-certificates.crt
lrwxrwxrwx    1 root     root           114 Sep 16 22:53 ca-cert-Caddy_Local_Authority_-_2021_ECC_Root_265478200015055389009959270412194820536.pem -> /usr/local/share/ca-certificates/Caddy_Local_Authority_-_2021_ECC_Root_265478200015055389009959270412194820536.crt
lrwxrwxrwx    1 root     root            89 Sep 16 22:53 1e027192.0 -> ca-cert-Caddy_Local_Authority_-_2021_ECC_Root_265478200015055389009959270412194820536.pem

We definitely have the same certificates everywhere:

/srv # sha1sum /usr/local/share/ca-certificates/Caddy_Local_Authority_-_2021_ECC_Root_2654782000150553890099592704121948
20536.crt /data/caddy/pki/authorities/local/root.crt

f69b4b1d3850f7dca6e43a691e293567352c5e3d  /usr/local/share/ca-certificates/Caddy_Local_Authority_-_2021_ECC_Root_265478200015055389009959270412194820536.crt
f69b4b1d3850f7dca6e43a691e293567352c5e3d  /data/caddy/pki/authorities/local/root.crt

I even installed curl into the Frontend Caddy Container and called the backend manually, with success (no complaints about the certificate presented by the backend):

/srv # apk add curl
OK: 8 MiB in 20 packages

/srv # curl https://ncweb

<!DOCTYPE html>
<html class="ng-csp" data-placeholder-focus="false" lang="en" data-locale="en" >
	<head
...

BUT If I modify the Frontend Caddyfile and point to its own Root CA certificate explicitly, then it works:

test.mydomain.tld {
        reverse_proxy https://ncweb {
                header_up Host {http.reverse_proxy.upstream.hostport}
                header_up X-Forwarded-Host {host}
                transport http {
                       tls_trusted_ca_certs /data/caddy/pki/authorities/local/root.crt
                }
        }
}

So what is going wrong here? Why do I have to specify it explicitly, even though it is already in the trusted certificates on this container.

1 Like

Apparently Alpine uses a file /etc/ssl/cert.pem which bundles all the certificates. See the code in Go which specifies the list of locations:

The code that Caddy uses for trusting the cert is here:

Hopefully that points you in the right direction. It might be a bug in Go, or it might be a bug in smallstep/truststore. I don’t know.

1 Like

Thanks for pointing out some possible areas to check.

First of all, after trying again the first time again since I wrote this post (11d ago), I noticed some different error message on the frontend caddy today:

{
    "level": "error",
    "ts": 1632770320.2249467,
    "logger": "http.log.error",
    "msg": "x509: certificate has expired or is not yet valid: current time 2021-09-27T19:18:40Z is after 2021-09-23T13:34:24Z",
    "request": {
        "remote_addr": "192.168.160.1:36340",
        "proto": "HTTP/2.0",
        "method": "GET",
        "host": "test.mydomain.tld",
        "uri": "/",
        "headers": {
            "Accept": [
                "text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8"
            ],
            "Accept-Language": [
                "en-US,en;q=0.5"
            ],
            "Accept-Encoding": [
                "gzip, deflate, br"
            ],
            "Sec-Fetch-Dest": [
                "document"
            ],
            "Te": [
                "trailers"
            ],
            "User-Agent": [
                "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:92.0) Gecko/20100101 Firefox/92.0"
            ],
            "Cookie": [
                "nc_sameSiteCookielax=true; nc_sameSiteCookiestrict=true; oc_sessionPassphrase=e1RNAlEmNFjO8Fork%2FEL9jXN3XpZ3DB82ah%2BhRZt37gIJBJIpywG1Db0l0WfCwDH2cLmVVXQhJaKDYTSXEFHQmAcQ6NkIVCdkHFj5xg%2Bymyo1uur2jbQ0Qq%2BqW5hv4q0; oc7zsgh7x0fv=143a056701a303c033f2679bf07c6777"
            ],
            "Upgrade-Insecure-Requests": [
                "1"
            ],
            "Sec-Fetch-Mode": [
                "navigate"
            ],
            "Sec-Fetch-Site": [
                "cross-site"
            ]
        },
        "tls": {
            "resumed": false,
            "version": 772,
            "cipher_suite": 4867,
            "proto": "h2",
            "proto_mutual": true,
            "server_name": "test.mydomain.tld"
        }
    },
    "duration": 0.018968927,
    "status": 502,
    "err_id": "2kciiib0w",
    "err_trace": "reverseproxy.statusError (reverseproxy.go:858)"
}

After searching a bit, I found this topic, which seems to be very much related:
https://caddy.community/t/mtls-tls-internal-error/12807/23
Unfortunately, I cannot post there, since the topic is auto-closed already.

However, from what I saw, there were some problems with updating the mTLS. You also pointed out:

Edit: Sorry, 7 days is the lifetime of the intermediate CA cert.

In my case, it is also 7 days after I started the Frontend Caddy (with ACME server) the last time: 16 September (shortly before I started this topic) till 23 September (from the error message above).

So apparently Caddy did not manage to update the intermediate CA cert properly. Not sure if my observations are helpful in some ways (e.g. to further analyze that other issue)

After I restarted the Frontend Caddy container, the intermediate CA cert was updated and now I’m back to the original problem.


What I’ve checked today:

  1. When bind mounting the root CA cert manually into the Frontend Caddy container (via docker-compose.yml)
    volumes:
      - ./etc-caddy:/etc/caddy:ro
      - caddydata:/data
      - ./proxy_ca_root.crt:/etc/ssl/certs/proxy_ca_root.crt:ro
    
    then it also works right away. In the logs I can see that during Caddy start the Root CA cert is recognized as already trusted:
    {"level":"info","ts":1632780209.0057275,"logger":"pki.ca.local","msg":"root certificate is already trusted by system","path":"storage:pki/authorities/local/root.crt"}
    
  2. That gave me the idea to reset/restart/reload Caddy without stopping the whole container regularly:
    caddy stop
    
    Which resulted in:
    {"level":"warn","ts":1632780207.6874425,"logger":"admin.api","msg":"exiting; byeee!! 👋"}
    {"level":"debug","ts":1632780207.6947668,"logger":"http.handlers.acme_server","msg":"unloading unused CA database","db_key":"local"}
    {"level":"info","ts":1632780207.6948857,"logger":"tls.cache.maintenance","msg":"stopped background certificate maintenance","cache":"0x4000483960"}
    {"level":"info","ts":1632780207.6968102,"logger":"admin","msg":"stopped previous server","address":"tcp/localhost:2019"}
    {"level":"info","ts":1632780207.6969922,"logger":"admin.api","msg":"shutdown complete","exit_code":0}
    {"level":"info","ts":1632780208.8935914,"msg":"using provided configuration","config_file":"/etc/caddy/Caddyfile","config_adapter":"caddyfile"}
    {"level":"info","ts":1632780208.9062464,"logger":"admin","msg":"admin endpoint started","address":"tcp/localhost:2019","enforce_origin":false,"origins":["localhost:2019","[::1]:2019","127.0.0.1:2019"]}
    {"level":"info","ts":1632780208.90713,"logger":"tls.cache.maintenance","msg":"started background certificate maintenance","cache":"0x4000486770"}
    {"level":"info","ts":1632780208.9084504,"logger":"http","msg":"server is listening only on the HTTPS port but has no TLS connection policies; adding one to enable TLS","server_name":"srv0","https_port":443}
    {"level":"info","ts":1632780208.908786,"logger":"http","msg":"server is listening only on the HTTP port, so no automatic HTTPS will be applied to this server","server_name":"srv1","http_port":80}
    {"level":"info","ts":1632780209.0057275,"logger":"pki.ca.local","msg":"root certificate is already trusted by system","path":"storage:pki/authorities/local/root.crt"}
    
    And this time it also worked. I don’t get the 502 anymore.

So, as conclusion, when starting the Docker container regularly, we see this in the logs:

{"level":"warn","ts":1631830791.917869,"logger":"pki.ca.local","msg":"installing root certificate (you might be prompted for password)","path":"storage:pki/authorities/local/root.crt"}
2021/09/16 22:19:51 Warning: "certutil" is not available, install "certutil" with "apt install libnss3-tools" or "yum install nss-tools" and try again
2021/09/16 22:19:51 define JAVA_HOME environment variable to use the Java trust
2021/09/16 22:19:52 certificate installed properly in linux trusts

At this point the Root CA certificate is copied to /etc/ssl/certs (and appended to /etc/ssl/cert.pem -> /etc/ssl/certs/ca-certificates.crt) but apparently it is not recognized by the Caddy runtime (?) yet.
Only next time the Caddy application is started (and the cert file is already in /etc/ssl/certs), it is recognized properly and considered as trusted.

Not sure if this is expected or acceptable behavior.

This topic was automatically closed after 30 days. New replies are no longer allowed.