Secure Connection Failed when using self-signed certificates, reverse_proxy, and docker-compose

Not sure if this belongs in the forums as a usage error or as an issue. I posted on both just in case and for search-ability if someone runs into similar issues. (Secure Connection Failed when using self-signed certificates, reverse_proxy, and docker-compose · Issue #3777 · caddyserver/caddy · GitHub).

1. Caddy version (caddy version):

v2.2.0 h1:sMUFqTbVIRlmA8NkFnNt9l7s0e+0gw+7GPIrhty905A=

2. How I run Caddy:

Everything is run through docker-compose. Caddy binds port :80 and :443 to localhost on the host machine and reverse proxies traffic to various services within the docker network named global. Specifically, I need to route traffic to a simple golang web server identified as golang:4200 on the docker network. I also need to route http://localhost:3100/loki/api/v1/push on the host machine to an instance of Loki for log aggregation.

This implementation where traffic is proxied into the docker network through caddy is intended to get around the fact that docker networking implementations slightly different on every operating system and often cause issues by users of different platforms.

a. System environment:

  • Ubuntu 18.04.5 LTS (although this solution is meant to be cross-platform via docker-compose)
  • docker-compose version 1.25.5, build unknown
  • docker network create global

b. Command:

docker network create global 
docker plugin install grafana/loki-docker-driver:latest --alias loki --grant-all-permissions
// restart docker to ensure the plugin was installed
docker-compose up

c. Service/unit/compose file:

version: '3.8'
x-logging: 
  &default-logging 
  driver: loki
  options:
    loki-url: 'http://localhost:3100/loki/api/v1/push'

services:
  grafana:
    logging: *default-logging 
    container_name: grafana
    image: 'grafana/grafana:latest'
    networks:
      - global
    ports:
      - '3000:3000'
    user: '472'
    volumes:
      - 'grafana-storage:/var/lib/grafana'

  loki:
    logging: *default-logging 
    container_name: loki
    image: 'grafana/loki:1.5.0'
    networks:
      - global
    command: '-config.file=/etc/loki/local-config.yaml'
    volumes:
      - 'loki-storage:/loki'

  golang:
    logging: *default-logging 
    tty: true
    container_name: golang
    image: ...image that served a web application on port 4200
    networks:
      - global

  caddy:
    logging: *default-logging 
    container_name: caddy
    image: 'caddy:latest'
    networks:
      - global
    depends_on:
      - golang
      - loki
    ports:
      - '80:80'
      - '443:443'
      - '3100:3100'
    volumes:
      - 'caddy-data:/data'
      - 'caddy-config:/config'
      - './Caddyfile:/etc/caddy/Caddyfile'

volumes:
  loki-storage: 
  caddy-data: 
  caddy-config: 
  grafana-storage: 

networks:
  global:
    external: true

d. My complete Caddyfile or JSON config:

{
  debug 
}

localhost {
  metrics /metrics

  reverse_proxy /loki {
    to loki:3100
  }

  reverse_proxy / {
    to golang:4200
  }
}

3. The problem I’m having:

Attempting to access localhost or localhost:3100 via firefox on the host machine shows an error page with message:

Secure Connection Failed

An error occurred during a connection to localhost. Peer’s certificate has an invalid signature.

Error code: SEC_ERROR_BAD_SIGNATURE

4. Error messages and/or full log output:

{"level":"info","ts":1601753875.2739213,"msg":"using provided configuration","config_file":"/etc/caddy/Caddyfile","config_adapter":"caddyfile"}
{"level":"info","ts":1601753875.2749262,"logger":"admin","msg":"admin endpoint started","address":"tcp/localhost:2019","enforce_origin":false,"origins":["localhost:2019","[::1]:2019","127.0.0.1:2019"]}
{"level":"info","ts":1601753875.2749944,"logger":"http","msg":"server is listening only on the HTTPS port but has no TLS connection policies; adding one to enable TLS","server_name":"srv0","https_port":443}
{"level":"info","ts":1601753875.2750032,"logger":"http","msg":"enabling automatic HTTP->HTTPS redirects","server_name":"srv0"}
{"level":"info","ts":1601753875.2751086,"logger":"tls.cache.maintenance","msg":"started background certificate maintenance","cache":"0xc0003d8e00"}
{"level":"info","ts":1601753875.2820623,"logger":"tls","msg":"setting internal issuer for automation policy that has only internal subjects but no issuer configured","subjects":["localhost"]}
{"level":"info","ts":1601753875.282647,"logger":"tls","msg":"cleaned up storage units"}
{"level":"warn","ts":1601753875.3019483,"logger":"pki.ca.local","msg":"installing root certificate (you might be prompted for password)","path":"storage:pki/authorities/local/root.crt"}
2020/10/03 19:37:55 Warning: "certutil" is not available, install "certutil" with "apt install libnss3-tools" or "yum install nss-tools" and try again
2020/10/03 19:37:55 define JAVA_HOME environment variable to use the Java trust
2020/10/03 19:37:55 certificate installed properly in linux trusts
{"level":"debug","ts":1601753875.3226576,"logger":"http","msg":"starting server loop","address":"[::]:443","http3":false,"tls":true}
{"level":"debug","ts":1601753875.3226876,"logger":"http","msg":"starting server loop","address":"[::]:80","http3":false,"tls":false}
{"level":"info","ts":1601753875.3226933,"logger":"http","msg":"enabling automatic TLS certificate management","domains":["localhost"]}
{"level":"warn","ts":1601753875.323083,"logger":"tls","msg":"stapling OCSP","error":"no OCSP stapling for [localhost]: no OCSP server specified in certificate"}
{"level":"info","ts":1601753875.323183,"msg":"autosaved config","file":"/config/caddy/autosave.json"}
{"level":"info","ts":1601753875.3231888,"msg":"serving initial configuration"}
{"level":"debug","ts":1601753889.3679104,"logger":"http.stdlib","msg":"http: TLS handshake error from 172.20.0.1:58788: remote error: tls: bad certificate"}

5. What I already tried:

I’ve already tried installing libnss3-tools on the host machine.

I swear this setup was working yesterday with the omission of the second reverse_proxy directive, although now neither caddyfile implementations work.

I wonder if this has to do with self-signed certificates being trusted on the host machine? I attempted to delete the certificates in firefox as well as deleting the caddy related volumes such that new ones would be created, but neither work.

Any help for a working solution of reverse proxying localhost → golang:4200 & localhost:3100 → loki:3100 with a self-signed certificate working properly would be greatly appreciated.

6. Links to relevant resources:

I’m not exactly sure why you’re getting that TLS error, but there are a couple issues with your Caddyfile.

Path matching is exact-match in Caddy v2, so /loki will only match exactly /loki and not /loki/foo for example. And / will only match /. Also, you said you wanted loclahost:3100 to proxy to loki, but you never tell Caddy to serve a site on port 3100.

I think this is closer to what you want:

{
	debug
}

localhost {
	metrics /metrics

	reverse_proxy golang:4200
}

localhost:3100 {
	reverse_proxy loki:3100
}

When Caddy is trying to set up the self signed CA, it’s trying to install it inside of the container. Caddy isn’t aware of the host machine at all, so it’s not able to install it on the host. You’d need to install it yourself by pulling the CA cert out of the container. It should be stored in /data/pki/authorities/local/root.crt

Those certutil warnings are harmless, it just means Caddy wasn’t able to install the cert to web browser trust stores inside of the container, because the container doesn’t have web browsers nor the appropriate libs installed. You can ignore those.

1 Like

Thanks for the quick response! I’ll go through your suggestions.

This topic was automatically closed after 30 days. New replies are no longer allowed.