Help with ERR_CERT_AUTHORITY_INVALID error in caddy v2.7.6

Help
1

1. The problem I’m having:

After updating to caddy version 2.7.6 I am now getting “NET::ERR_CERT_AUTHORITY_INVALID” errors when i try to navigate to my proxied pages. I run my own step-ca as a root crt issuer. I don’t think the step-ca is the problem as caddy is still able to auto renew certs and my update logs show that a little while after I updated caddy, my services started not working correctly due to crt errors. The chain of trust I had working was Root CA → Caddy crt → server crt (that caddy generated). I would then trust the Root CA and the chain is fully trusted.

2. Error messages and/or full log output:

ERROR MESSAGE:

NET::ERR_CERT_AUTHORITY_INVALID
Subject:

Issuer: Caddy Local Authority - ECC Intermediate

Expires on: Dec 11, 2023

Current date: Dec 10, 2023

PEM encoded chain:
-----BEGIN CERTIFICATE-----
MIIBxDCCAWugAwIBAgIQGR1eTWBq+/YuYZoe/QYKkDAKBggqhkjOPQQDAjAzMTEw
LwYDVQQDEyhDYWRkeSBMb2NhbCBBdXRob3JpdHkgLSBFQ0MgSW50ZXJtZWRpYXRl
MB4XDTIzMTIxMDIwMjcwMloXDTIzMTIxMTA4MjcwMlowADBZMBMGByqGSM49AgEG
CCqGSM49AwEHA0IABIh4KyB9WjEGnvgS4b3Tx99i4MMnwOmNkSsl7jSTigt7sumu
mIjyA/5kqUWt41VZz89jywvd26Cg+b/B1E1k6TCjgZMwgZAwDgYDVR0PAQH/BAQD
AgeAMB0GA1UdJQQWMBQGCCsGAQUFBwMBBggrBgEFBQcDAjAdBgNVHQ4EFgQUVVrZ
9yUxdTc68bRAJjZzrgvV2MMwHwYDVR0jBBgwFoAUiems4q+huMGeib2d6jGo8a48
DmYwHwYDVR0RAQH/BBUwE4IRcHJveG1veC5ob21lLmFycGEwCgYIKoZIzj0EAwID
RwAwRAIgZeBig70GjVz10Won9HwNGuHyIqoiCTFdJ6cbKYs6TMgCIHHqWJwFc9iD
zsmTTtDNQgrJhYWTjbov/AWwuOi65QTa
-----END CERTIFICATE-----
-----BEGIN CERTIFICATE-----
MIIBxzCCAW2gAwIBAgIQOMsb3oyuDGTAOrP5SmwCyDAKBggqhkjOPQQDAjAwMS4w
LAYDVQQDEyVDYWRkeSBMb2NhbCBBdXRob3JpdHkgLSAyMDIzIEVDQyBSb290MB4X
DTIzMTIwOTA0MDcwNFoXDTIzMTIxNjA0MDcwNFowMzExMC8GA1UEAxMoQ2FkZHkg
TG9jYWwgQXV0aG9yaXR5IC0gRUNDIEludGVybWVkaWF0ZTBZMBMGByqGSM49AgEG
CCqGSM49AwEHA0IABG2erOzhl70CGHyC+SL/499eybblVarJZypm1ygzYomvaRT7
D1nqxfXA6fwib5sb8TzD21R94O4jx1pKhCx9EP+jZjBkMA4GA1UdDwEB/wQEAwIB
BjASBgNVHRMBAf8ECDAGAQH/AgEAMB0GA1UdDgQWBBSJ6azir6G4wZ6JvZ3qMajx
rjwOZjAfBgNVHSMEGDAWgBRV63izljPjklTlMTxaDe9suLIz6jAKBggqhkjOPQQD
AgNIADBFAiBByTxxc/zfIQA6u7DGyHi1HQrKDVRuK+z+ZLyejkBAwQIhANAxQ4uL
t//ACsoccd1Bs3DIcia7deU4UrYFvoOxyNhS
-----END CERTIFICATE-----

using curl to hit one of my pages:

curl: (60) SSL certificate problem: unable to get local issuer certificate
More details here: https://curl.se/docs/sslcerts.html

curl failed to verify the legitimacy of the server and therefore could not
establish a secure connection to it. To learn more about this situation and
how to fix it, please visit the web page mentioned above.

Cert being hosted by the root CA:

Common Name (CN)	Step Online CA
Organization (O)	<Not Part Of Certificate>
Organizational Unit (OU)	<Not Part Of Certificate>
Common Name (CN)	Bastion Yubikey
Organization (O)	adminH
Organizational Unit (OU)	<Not Part Of Certificate>

3. Caddy version:

v2.7.6

4. How I installed and ran Caddy:

a. System environment:

debian 12, x86, in docker container

b. Command:

N/A, runninnig in docker

c. Service/unit/compose file:

version: "3.9"
services:
  caddy:
    image: caddy:latest
    restart: unless-stopped
    container_name: caddy
    ports:
      - 80:80
      - 443:443
      - "443:443/udp"
    networks:
      - caddy-network
    volumes:
      - caddy_data:/data
      - caddy_config:/config
      - /data/caddyfile/Caddyfile:/etc/caddy/Caddyfile:ro
      - /usr/local/share/ca-certificates/bastion.crt:/etc/ssl/certs/bastion.crt:ro
    logging:
            driver: "json-file"
            options:
                max-size: "50m"

networks:
  caddy-network:
    external: true



volumes:
  caddy_data:
  caddy_config:

d. My complete Caddy config:

#Global options
{
        #Our local ACME server
        acme_ca https://tinyca.home.arpa/acme/acme/directory
}

#A single server which will get a TLS certificate automatically
truenas.home.arpa {
	reverse_proxy 192.168.9.6
}
proxmox.home.arpa {
	reverse_proxy 192.168.9.5:8006 {
		transport http {
			tls
			tls_insecure_skip_verify
		}
	}		
}
unifi.home.arpa {
	reverse_proxy 192.168.9.2:8443 {
		transport http {
			tls
			tls_insecure_skip_verify
		}
	}		
}
gitea.home.arpa {
	reverse_proxy gitea:3000
}
speedtest.home.arpa {
	reverse_proxy speedtest
}
documentserver.home.arpa {
	reverse_proxy documentserver
}
nextcloud.home.arpa {
    redir /.well-known/carddav /remote.php/dav 301
    redir /.well-known/caldav /remote.php/dav 301
    header Strict-Transport-Security max-age=31536000;

    reverse_proxy nextcloud:443 {
        transport http {
                tls_insecure_skip_verify
        }
    }
    header {
        Strict-Transport-Security max-age=31536000;
    }
}
backup.home.arpa {
	reverse_proxy urbackup:55414		
}
backup-service.home.arpa
{
	reverse_proxy urbackup:55415		
}

grafana.home.arpa {
	reverse_proxy grafana:3000	
}
prometheus.home.arpa {
	basicauth {
		box password_here
	}
	reverse_proxy prometheus:9090	
}
ansible.home.arpa {
	reverse_proxy ansiblesemaphore:3000	
}
kuma.home.arpa {
	reverse_proxy uptimekuma:3001
}
opnsense.home.arpa {
	reverse_proxy https://192.168.9.1 {
		transport http {
			tls
			tls_insecure_skip_verify
		}
	}		
}
pi-hole.home.arpa {
	encode zstd gzip
	redir / /admin{uri}
	reverse_proxy 192.168.9.2
}
homeassistant.home.arpa {
	reverse_proxy 192.168.9.97:8123 
}
wazuh.home.arpa {
	reverse_proxy wazuh.dashboard:5601 {
		transport http {
			tls
			tls_insecure_skip_verify
		}
	}		

}
dashy.home.arpa {
	reverse_proxy dashy:80 
}
portainer.home.arpa {
	reverse_proxy portainer:9443 {
		transport http {
			tls
			tls_insecure_skip_verify
		}
	}		

}
pbs.home.arpa {
	reverse_proxy 192.168.9.8:8007 {
		transport http {
			tls
			tls_insecure_skip_verify
		}
	}		
}
ntfy.home.arpa {
    reverse_proxy ntfy:80
}

5. Links to relevant resources:

2

Did you actually add the root cert to your system’s trust store?

Are you sure the root cert is still the same (i.e. the CA’s storage didn’t get lost/reset somehow?)

3

Yes I did. I even readded it to my windows machine that is trying to access the pages to be sure and still gettting the error. The root CA is still there and is not lost. The root ca is in a seperarte computer away from caddy. I just rolled back to caddy 2.7.5 and the error is gone and I can reach my sites just fine. So somthing is going in on in the new vesion, 2.7.6, that is breaking for me. Either a bug or a config change that is breaking with my configs.
This is report from the cert on the webpages in 2.7.5

Common Name (CN)	<Not Part Of Certificate>
Organization (O)	<Not Part Of Certificate>
Organizational Unit (OU)	<Not Part Of Certificate>
Common Name (CN)	Bastion Yubikey
Organization (O)	adminH
Organizational Unit (OU)	<Not Part Of Certificate>

Comparing it to the error message I got above, it looks like caddy is issuing its own ca’s as root and not as an intermediatory under my root ca in 2.7.6. Is there a setting that changed that I need to add?

4

Oh, yeah :grimacing: I see it.

We added .home.arpa to the list of TLDs that we automatically configure tls internal for. We did that because it slightly simplifies config for most users using Caddy.

We didn’t consider that it could be a breaking change for anyone using an CA external to Caddy itself for that TLD! (I’m kinda surprised anyone even does, to be frank).

I think what you’d need to do is explicitly configure tls in each site block instead of using the global acme_ca option unfortunately, because you need to explicitly override the implicit tls internal default.

You can use snippets though which slightly reduces the amount of config. Might look something like this:

(tinyca) {
	tls {
		ca https://tinyca.home.arpa/acme/acme/directory
	}
}

truenas.home.arpa {
	import tinyca
	reverse_proxy 192.168.9.6
}

...

Sorry for the trouble :sweat:

1 Like
5

No worries. I wanted my root CA private key to be on a hardware key and both due to technical complexity and security, I run the ste-ca on a completely seperate server than where caddy is running. Reason I am using .home.arpa is that I wanted a local only domain and that is the only reserved local domain, so in this case we had similar ideas.

2 Likes
6

Justed implemented it and its working, thanks for the help.

1 Like
7

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.