Reverse proxy x509: certificate signed by unknown authority

Hi ! :slight_smile:

1. The problem I’m having:

Caddy can’t manage to get SSL certificate. All I get is X509 error. It was working a few months ago when I started this project. Today, I updated the website and Caddy didn’t want to restart…
The ca-certificate package is installed on the alpine, I’ve checked that.
I tried to build the caddy image locally and pushed it to the server, but I got the same error.

2. Error messages and/or full log output:

{"level":"info","ts":1680390122.2776883,"logger":"tls.obtain","msg":"acquiring lock","identifier":"admin.monsite.fr"}
{"level":"info","ts":1680390122.2800052,"logger":"tls.obtain","msg":"lock acquired","identifier":"admin.monsite.fr"}
{"level":"info","ts":1680390122.2803776,"logger":"tls.obtain","msg":"obtaining certificate","identifier":"admin.monsite.fr"}
{"level":"info","ts":1680390122.283984,"msg":"autosaved config (load with --resume flag)","file":"/config/caddy/autosave.json"}
{"level":"info","ts":1680390122.2843254,"msg":"serving initial configuration"}
{"level":"info","ts":1680390122.2868705,"logger":"tls.obtain","msg":"acquiring lock","identifier":"monsite.fr"}
{"level":"info","ts":1680390122.2905958,"logger":"tls.obtain","msg":"lock acquired","identifier":"monsite.fr"}
{"level":"info","ts":1680390122.2912974,"logger":"tls.obtain","msg":"obtaining certificate","identifier":"monsite.fr"}
{"level":"warn","ts":1680390122.606337,"logger":"http.acme_client","msg":"HTTP request failed; retrying","url":"https://acme-staging-v02.api.letsencrypt.org/directory","error":"performing request: Get \"https://acme-staging-v02.api.letsencrypt.org/directory\": x509: certificate signed by unknown authority"}
{"level":"warn","ts":1680390124.1852741,"logger":"http.acme_client","msg":"HTTP request failed; retrying","url":"https://acme-staging-v02.api.letsencrypt.org/directory","error":"performing request: Get \"https://acme-staging-v02.api.letsencrypt.org/directory\": x509: certificate signed by unknown authority"}
{"level":"warn","ts":1680390124.7808793,"logger":"http.acme_client","msg":"HTTP request failed; retrying","url":"https://acme-staging-v02.api.letsencrypt.org/directory","error":"performing request: Get \"https://acme-staging-v02.api.letsencrypt.org/directory\": x509: certificate signed by unknown authority"}
{"level":"error","ts":1680390124.7809825,"logger":"tls.obtain","msg":"could not get certificate from issuer","identifier":"admin.monsite.fr","issuer":"acme-staging-v02.api.letsencrypt.org-directory","error":"registering account [mailto:quentin.esnault@lilo.org] with server: provisioning client: performing request: Get \"https://acme-staging-v02.api.letsencrypt.org/directory\": x509: certificate signed by unknown authority"}
{"level":"error","ts":1680390124.9620564,"logger":"tls.obtain","msg":"could not get certificate from issuer","identifier":"admin.monsite.fr","issuer":"acme-staging-v02.api.letsencrypt.org-directory","error":"account pre-registration callback: performing EAB credentials request: Post \"https://api.zerossl.com/acme/eab-credentials-email\": x509: certificate signed by unknown authority"}
{"level":"error","ts":1680390124.962114,"logger":"tls.obtain","msg":"will retry","error":"[admin.monsite.fr] Obtain: account pre-registration callback: performing EAB credentials request: Post \"https://api.zerossl.com/acme/eab-credentials-email\": x509: certificate signed by unknown authority","attempt":1,"retrying_in":60,"elapsed":2.681900097,"max_duration":2592000}
{"level":"warn","ts":1680390125.1294262,"logger":"http.acme_client","msg":"HTTP request failed; retrying","url":"https://acme-staging-v02.api.letsencrypt.org/directory","error":"performing request: Get \"https://acme-staging-v02.api.letsencrypt.org/directory\": x509: certificate signed by unknown authority"}
{"level":"warn","ts":1680390125.6982532,"logger":"http.acme_client","msg":"HTTP request failed; retrying","url":"https://acme-staging-v02.api.letsencrypt.org/directory","error":"performing request: Get \"https://acme-staging-v02.api.letsencrypt.org/directory\": x509: certificate signed by unknown authority"}
{"level":"warn","ts":1680390126.2816727,"logger":"http.acme_client","msg":"HTTP request failed; retrying","url":"https://acme-staging-v02.api.letsencrypt.org/directory","error":"performing request: Get \"https://acme-staging-v02.api.letsencrypt.org/directory\": x509: certificate signed by unknown authority"}
{"level":"error","ts":1680390126.281713,"logger":"tls.obtain","msg":"could not get certificate from issuer","identifier":"monsite.fr","issuer":"acme-staging-v02.api.letsencrypt.org-directory","error":"registering account [mailto:quentin.esnault@lilo.org] with server: provisioning client: performing request: Get \"https://acme-staging-v02.api.letsencrypt.org/directory\": x509: certificate signed by unknown authority"}
{"level":"error","ts":1680390126.4910443,"logger":"tls.obtain","msg":"could not get certificate from issuer","identifier":"monsite.fr","issuer":"acme-staging-v02.api.letsencrypt.org-directory","error":"account pre-registration callback: performing EAB credentials request: Post \"https://api.zerossl.com/acme/eab-credentials-email\": x509: certificate signed by unknown authority"}
{"level":"error","ts":1680390126.4911246,"logger":"tls.obtain","msg":"will retry","error":"[monsite.fr] Obtain: account pre-registration callback: performing EAB credentials request: Post \"https://api.zerossl.com/acme/eab-credentials-email\": x509: certificate signed by unknown authority","attempt":1,"retrying_in":60,"elapsed":4.200117065,"max_duration":2592000}

3. Caddy version:

2.6.4

4. How I installed and ran Caddy:

I’m running caddy inside a docker container

a. System environment:

Docker is running on debian.
Here is version of all docker package :

docker-ce-cli/bullseye,now 5:23.0.2-1~debian.11~bullseye amd64 [installed]
docker-ce-rootless-extras/bullseye,now 5:23.0.2-1~debian.11~bullseye amd64 [installed,automatic]
docker-ce/bullseye,now 5:23.0.2-1~debian.11~bullseye amd64 [installed]
docker-clean/stable 2.0.4-3 all
docker-compose-plugin/bullseye,now 2.17.2-1~debian.11~bullseye amd64 [installed]
docker-compose/stable 1.25.0-1 all
docker-doc/stable 20.10.5+dfsg1-1+deb11u2 all
docker-registry/stable 2.7.1+ds2-7+b6 amd64
docker-scan-plugin/bullseye,now 0.23.0~debian-bullseye amd64 [installed,automatic]
docker.io/stable 20.10.5+dfsg1-1+deb11u2 amd64
docker2aci/stable 0.17.2+dfsg-2.1+b5 amd64
docker/stable 1.5-2 all
elpa-dockerfile-mode/stable 1.2-2 all
golang-docker-credential-helpers/stable 0.6.3-1+b6 amd64
golang-github-appc-docker2aci-dev/stable 0.17.2+dfsg-2.1 all
golang-github-docker-distribution-dev/stable 2.7.1+ds2-7 all
golang-github-docker-docker-credential-helpers-dev/stable 0.6.3-1 all
golang-github-docker-docker-dev/stable 20.10.5+dfsg1-1+deb11u2 all
golang-github-docker-go-connections-dev/stable 0.4.0-3 all
golang-github-docker-go-dev/stable 0.0~git20160303.0.d30aec9-3 all
golang-github-docker-go-events-dev/stable 0.0~git20190806.e31b211-1 all
golang-github-docker-go-metrics-dev/stable 0.0.1-1 all
golang-github-docker-go-units-dev/stable 0.4.0-3 all
golang-github-docker-leadership-dev/stable 0.1.0-1.1 all
golang-github-docker-libkv-dev/stable 0.2.1-2 all
golang-github-docker-libtrust-dev/stable 0.0~git20150526.0.9cbd2a1-3.1 all
golang-github-docker-notary-dev/stable 0.6.1~ds2-6 all
golang-github-docker-spdystream-dev/stable 0.2.0-1 all
golang-github-fsouza-go-dockerclient-dev/stable 1.6.6-1 all
golang-github-samalba-dockerclient-dev/stable 0.0~git20160531.0.a303626-2 all
kdocker/stable 5.3-1 amd64
libnss-docker/stable 0.02-1+b1 amd64
ovn-docker/bullseye-backports 21.06.0+ds1-2~bpo11+1 amd64
python3-docker/stable 4.1.0-1.2 all
python3-dockerpty/stable 0.4.1-2 all
python3-dockerpycreds/stable 0.3.0-1.1 all
ruby-docker-api/stable 1.22.2-1.1 all
wmdocker/stable 1.5-2 amd64

b. Command:

docker compose up -d 

c. Service/unit/compose file:

version: '3.4'
services:
  demofront:
    build:  demofront
    container_name: demofront
    working_dir: /root/monsiteFront
    command: npm run dev
    volumes:
      - type: bind
        source: ./demofront/scripts/
        target: /root/scripts/
      - type: bind
        source: ./demofront/config/nuxt.config.ts
        target: /root/KomemliaFront/nuxt.config.ts

          
  demoback:
    build:  demoback
    container_name: demoback
    working_dir: /root/monsiteBack
    command: npm run develop
    volumes: 
      - type: bind
        source: ./demoback/data_back
        target: /root/monsiteBack/.tmp
      - type: bind
        source: ./demoback/scripts/
        target: /root/scripts/
      - type: bind
        source: ./demoback/uploads_back
        target: /root/monsiteBack/public/uploads

  caddyserver:
    image: caddy:latest
    container_name: caddy
    ports:
      - "80:80"
      - "443:443"
      - "443:443/udp"
      - "24678:24678"
      
    volumes:
      - type: bind
        source: ./caddy/
        target: /etc/caddy/
      - type : bind
        source: ./caddy_data
        target: /data
      - type: bind
        source : ./caddy_config
        target : /config  
      - type: bind
        source: ./caddyetcssl
        target: /etc/ssl 
networks:
  default:
  internal:
    internal:  true

d. My complete Caddy config:

admin.monsite.fr {
	tls truc.muche@lilo.org 
	reverse_proxy demoback:1338
}
monsite.fr {

 	tls truc.muche@lilo.org 
   
	handle /_nuxt/hmr/ {
		reverse_proxy http://demofront:24678 
	}

	handle {
		reverse_proxy http://demofront:3001
	}

}

5. Links to relevant resources:

I’ve also tried to build my own caddy image on a debian image. But I’m still getting the same issue.

FROM debian:latest
RUN apt -y update
RUN apt -y upgrade
RUN apt-get -y install curl vim htop ca-certificates wget mailcap
RUN apt install -y debian-keyring debian-archive-keyring apt-transport-https
RUN curl -1sLf 'https://dl.cloudsmith.io/public/caddy/stable/gpg.key' |  gpg --dearmor -o /usr/share/keyrings/caddy-stable-archive-keyring.gpg
RUN curl -1sLf 'https://dl.cloudsmith.io/public/caddy/stable/debian.deb.txt' |  tee /etc/apt/sources.list.d/caddy-stable.list
RUN apt update
RUN apt install caddy
RUN mkdir -p \
		/config/caddy \
		/data/caddy \
		/etc/caddy \
		/usr/share/caddy \
	; \
	wget -O /etc/caddy/Caddyfile "https://github.com/caddyserver/dist/raw/305fe484cc8a9ac72900e8cc172d652102a87240/config/Caddyfile"; \
	wget -O /usr/share/caddy/index.html "https://github.com/caddyserver/dist/raw/305fe484cc8a9ac72900e8cc172d652102a87240/welcome/index.html"
CMD ["caddy", "run", "--config", "/etc/caddy/Caddyfile", "--adapter", "caddyfile"]

I’ve installed caddy without docker container on a debian server, with the same Caddyfile a above. And then caddy managed to get a response from zerossl without x509 error.
So, it seems to be docker related.

I believe this stanza

overrides the existing CA certs in /etc/ssl within the base image, which makes the OS inside the container not able to verify any CA because the dir is empty inside the container. I find this handy:

1 Like

Thanks for your help. I’ll check that this evening.

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.