1. Caddy version (caddy version):
v2.4.5
2. How I run Caddy:
a. System environment:
I’m trying to implement this here at my home: Caddy reverse proxy + Nextcloud + Collabora + Bitwarden_rs with local HTTPS
- Docker on Raspi 4 (hostname: rowena) behind private DSL (port 80 and 443 forwarded to Pi, DynDNS etc. all working fine)
- Caddy container to be used as reverse proxy
- Nextcloud as backend service
- Caddy and Nextcloud service connected via Docker network
Goal: TLS between Frontend (reverse proxy) and backend (Nextcloud)
b. Command:
docker-compose up
c. Service/unit/compose file:
version: "3.8"
volumes:
caddydata:
name: proxy-caddydata
external: true
networks:
proxy:
name: proxy-network
services:
caddy:
container_name: proxy
image: caddy:latest
restart: unless-stopped
security_opt:
- no-new-privileges:true
networks:
- proxy
ports:
- 80:80
- 443:443
volumes:
- ./etc-caddy:/etc/caddy:ro
- caddydata:/data
d. My complete Caddyfile or JSON config:
Caddyfile of reverse proxy container (FRONTEND):
{
debug
email hostmaster@mydomain.tld
auto_https disable_redirects
}
localhost, rowena, rowena.local, proxy {
acme_server
tls internal
}
https://localhost, https://rowena, https://rowena.local {
tls internal
# just some test forwarding to my laptop
reverse_proxy /8000 http://ronbook:8000 {
header_up Host {http.reverse_proxy.upstream.hostport}
header_up X-Forwarded-Host {host}
}
# just some test forwarding to another Raspi
reverse_proxy http://altair {
header_up Host {http.reverse_proxy.upstream.hostport}
header_up X-Forwarded-Host {host}
}
}
http://localhost, http://rowena, http://rowena.local {
respond "Hello World!"
}
# proxying to another Raspi with old Nextcloud setup (incl. TLS)
# not relevant for the bug
cloud.mydomain.tld {
reverse_proxy https://altair {
#header_up Host {http.reverse_proxy.upstream.hostport}
header_up Host cloud.mydomain.tld:443
header_up X-Forwarded-Host {host}
transport http {
tls_server_name cloud.mydomain.tld
}
}
}
# this is what I want to get working
test.mydomain.tld {
reverse_proxy https://ncweb {
header_up Host {http.reverse_proxy.upstream.hostport}
header_up X-Forwarded-Host {host}
}
}
Caddyfile of Nextcloud service (BACKEND):
{
debug
# TLS Options
acme_ca https://proxy/acme/local/directory
acme_ca_root /etc/ssl/certs/proxy_ca_root.crt
}
https://ncweb {
# omitted here to keep it shorter
}
So I’m using the ACME server on the frontend caddy to issue a certificate to the backend caddy.
In Backend Caddyfile I point to the ACME server. I also provisioned the root certificate of the Frontend ACME to the Backend container and I point to it in the Backend Caddyfile.
3. The problem I’m having:
First time I started both containers (frontend and backend), certificate for the backend service was issued properly by the ACME server on the frontend.
If I now try to open the webpage by calling test.mydomain.tld, I get a 502 in the browser and error messages in the logs about an untrusted certificate.
4. Error messages and/or full log output:
On Backend Caddy (Nextcloud):
ncweb_1 | {"level":"debug","ts":1631832875.4458847,"logger":"http.stdlib","msg":"http: TLS handshake error from 192.168.128.2:42892: remote error: tls: bad certificate"}
On Frontend Caddy:
{
"level": "debug",
"ts": 1631829601.2283535,
"logger": "http.handlers.reverse_proxy",
"msg": "upstream roundtrip",
"upstream": "ncweb:443",
"request": {
"remote_addr": "172.29.0.1:45316",
"proto": "HTTP/2.0",
"method": "GET",
"host": "ncweb:443",
"uri": "/",
"headers": {
"Te": [
"trailers"
],
"X-Forwarded-Proto": [
"https"
],
"Cookie": [
# ....
],
"X-Forwarded-Host": [
"test.mydomain.tld"
],
"Cache-Control": [
"max-age=0"
],
"User-Agent": [
"Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:91.0) Gecko/20100101 Firefox/91.0"
],
"Accept": [
"text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8"
],
"Sec-Fetch-Site": [
"none"
],
"Upgrade-Insecure-Requests": [
"1"
],
"X-Forwarded-For": [
"172.29.0.1"
],
"Sec-Fetch-User": [
"?1"
],
"Accept-Encoding": [
"gzip, deflate, br"
],
"Sec-Fetch-Mode": [
"navigate"
],
"Accept-Language": [
"en-US,en;q=0.5"
],
"Sec-Fetch-Dest": [
"document"
]
},
"tls": {
"resumed": false,
"version": 772,
"cipher_suite": 4867,
"proto": "h2",
"proto_mutual": true,
"server_name": "test.mydomain.tld"
}
},
"duration": 0.017470574,
"error": "x509: certificate signed by unknown authority"
}
{
"level": "error",
"ts": 1631829601.2297077,
"logger": "http.log.error",
"msg": "x509: certificate signed by unknown authority",
"request": {
"remote_addr": "172.29.0.1:45316",
"proto": "HTTP/2.0",
"method": "GET",
"host": "test.mydomain.tld",
"uri": "/",
"headers": {
"Sec-Fetch-Mode": [
"navigate"
],
"Cache-Control": [
"max-age=0"
],
"Accept-Language": [
"en-US,en;q=0.5"
],
"Accept-Encoding": [
"gzip, deflate, br"
],
"Sec-Fetch-Dest": [
"document"
],
"Upgrade-Insecure-Requests": [
"1"
],
"Sec-Fetch-Site": [
"none"
],
"Sec-Fetch-User": [
"?1"
],
"Te": [
"trailers"
],
"User-Agent": [
"Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:91.0) Gecko/20100101 Firefox/91.0"
],
"Accept": [
"text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8"
],
"Cookie": [
"nc_sameSiteCookielax=true; nc_sameSiteCookiestrict=true; oc_sessionPassphrase=e1RNAlEmNFjO8Fork%2FEL9jXN3XpZ3DB82ah%2BhRZt37gIJBJIpywG1Db0l0WfCwDH2cLmVVXQhJaKDYTSXEFHQmAcQ6NkIVCdkHFj5xg%2Bymyo1uur2jbQ0Qq%2BqW5hv4q0; oc7zsgh7x0fv=c5a4cc056e1d173c83ac82f997ce7bb5"
]
},
"tls": {
"resumed": false,
"version": 772,
"cipher_suite": 4867,
"proto": "h2",
"proto_mutual": true,
"server_name": "test.mydomain.tld"
}
},
"duration": 0.019264899,
"status": 502,
"err_id": "13in481tx",
"err_trace": "reverseproxy.statusError (reverseproxy.go:858)"
}
Frontend Caddy logs show this during startup:
{"level":"warn","ts":1631830791.917869,"logger":"pki.ca.local","msg":"installing root certificate (you might be prompted for password)","path":"storage:pki/authorities/local/root.crt"}
2021/09/16 22:19:51 Warning: "certutil" is not available, install "certutil" with "apt install libnss3-tools" or "yum install nss-tools" and try again
2021/09/16 22:19:51 define JAVA_HOME environment variable to use the Java trust
2021/09/16 22:19:52 certificate installed properly in linux trusts
5. What I already tried:
/srv # caddy trust
2021/09/16 23:42:22.650 INFO ca.local root certificate is already trusted by system {"path": "storage:pki/authorities/local/root.crt"}
Caddy Root CA cert is already in trusted cert list:
/srv # ls -lrt /etc/ssl/certs/
...
-rw-r--r-- 1 root root 214685 Sep 16 22:53 ca-certificates.crt
lrwxrwxrwx 1 root root 114 Sep 16 22:53 ca-cert-Caddy_Local_Authority_-_2021_ECC_Root_265478200015055389009959270412194820536.pem -> /usr/local/share/ca-certificates/Caddy_Local_Authority_-_2021_ECC_Root_265478200015055389009959270412194820536.crt
lrwxrwxrwx 1 root root 89 Sep 16 22:53 1e027192.0 -> ca-cert-Caddy_Local_Authority_-_2021_ECC_Root_265478200015055389009959270412194820536.pem
We definitely have the same certificates everywhere:
/srv # sha1sum /usr/local/share/ca-certificates/Caddy_Local_Authority_-_2021_ECC_Root_2654782000150553890099592704121948
20536.crt /data/caddy/pki/authorities/local/root.crt
f69b4b1d3850f7dca6e43a691e293567352c5e3d /usr/local/share/ca-certificates/Caddy_Local_Authority_-_2021_ECC_Root_265478200015055389009959270412194820536.crt
f69b4b1d3850f7dca6e43a691e293567352c5e3d /data/caddy/pki/authorities/local/root.crt
I even installed curl into the Frontend Caddy Container and called the backend manually, with success (no complaints about the certificate presented by the backend):
/srv # apk add curl
OK: 8 MiB in 20 packages
/srv # curl https://ncweb
<!DOCTYPE html>
<html class="ng-csp" data-placeholder-focus="false" lang="en" data-locale="en" >
<head
...
BUT If I modify the Frontend Caddyfile and point to its own Root CA certificate explicitly, then it works:
test.mydomain.tld {
reverse_proxy https://ncweb {
header_up Host {http.reverse_proxy.upstream.hostport}
header_up X-Forwarded-Host {host}
transport http {
tls_trusted_ca_certs /data/caddy/pki/authorities/local/root.crt
}
}
}
So what is going wrong here? Why do I have to specify it explicitly, even though it is already in the trusted certificates on this container.