Ok, thanks I’ll run through it again – apologies for posting in this way, was just starting to get frustrated, on top of the already frustrating problem! Haha.
Looks like it’s because you added caddy fmt next to the opening ``` for your Caddyfile. Remove that bit.
At a glance, it looks like your tls file paths are not the same between your compose file and Caddyfile.
2. How I run Caddy:
Internet → Cloudflare DNS+Proxy → Firewall → Traefik [VM1; docker] → Caddy (Searx, Morty, Filtron) [VM2; docker]
I am trying to spin up a public instance of Searx, which has Caddy baked in. I have isolated it inside its own Debian10 VM for security and want to use my existing Traefik instance to point to Caddy.
Cloudflare is set to strict-full using Origin Certificates. mTLS is also in place and working (Cloudflare call it Authenticated Origin Pulls).
I am somewhat tied in, in the situation because I have many other services running on my main Docker VM with Traefik as the back bone to serving all of it, so I cannot move away from it.
a. System environment:
Debian10 - minimal install
Docker
b. Command:
Caddy is launched from docker-compose.yaml using a start script ‘start.sh’ in the searx/searx-docker distribution. start.sh just uses launches docker-compose.yaml with a basic:
$DOCKERCOMPOSE -f $DOCKERCOMPOSEFILE down
$DOCKERCOMPOSE -f $DOCKERCOMPOSEFILE rm -fv
$DOCKERCOMPOSE -f $DOCKERCOMPOSEFILE up
c. Service/unit/compose file:
version: '3.7'
services:
caddy:
container_name: caddy
image: caddy:2-alpine
restart: on-failure
network_mode: host
volumes:
- ./Caddyfile:/etc/caddy/Caddyfile:ro
- caddy-data:/data:rw
- caddy-config:/config:rw
- /certs/tomlawson.io.pem:/certs/tomlawson.io.pem:ro
- /certs/tomlawson.io.key:/certs/tomlawson.io.key:ro
environment:
- SEARX_HOSTNAME=${SEARX_HOSTNAME:-localhost}
- SEARX_TLS=${LETSENCRYPT_EMAIL:-internal}
cap_drop:
- ALL
cap_add:
- NET_BIND_SERVICE
- DAC_OVERRIDE
filtron:
container_name: filtron
image: dalf/filtron
restart: always
ports:
- "127.0.0.1:4040:4040"
- "127.0.0.1:4041:4041"
networks:
- searx
command: -listen 0.0.0.0:4040 -api 0.0.0.0:4041 -target searx:8080
volumes:
- ./rules.json:/etc/filtron/rules.json:rw
read_only: true
cap_drop:
- ALL
searx:
container_name: searx
image: searx/searx:latest
restart: always
networks:
- searx
command: ${SEARX_COMMAND:-}
volumes:
- ./searx:/etc/searx:rw
environment:
- BIND_ADDRESS=0.0.0.0:8080
- BASE_URL=https://${SEARX_HOSTNAME:-localhost}/
- MORTY_URL=https://${SEARX_HOSTNAME:-localhost}/morty/
- MORTY_KEY=${MORTY_KEY}
cap_drop:
- ALL
cap_add:
- CHOWN
- SETGID
- SETUID
- DAC_OVERRIDE
morty:
container_name: morty
image: dalf/morty
restart: always
ports:
- "127.0.0.1:3000:3000"
networks:
- searx
command: -timeout 6 -ipv6
environment:
- MORTY_KEY=${MORTY_KEY}
- MORTY_ADDRESS=0.0.0.0:3000
logging:
driver: none
read_only: true
cap_drop:
- ALL
networks:
searx:
ipam:
driver: default
volumes:
caddy-data:
caddy-config:
Environment file:
# By default listen on https : / / localhost
# To change this:
# * uncomment SEARX_HOSTNAME, and replace <host> by the searx hostname
# * uncomment LETSENCRYPT_EMAIL, and replace <email> by your email (require to create a Let's Encrypt certificate)
SEARX_HOSTNAME=[REMOVED DUE TO NEW USER LIMIT ON POSTING LINKS]
# LETSENCRYPT_EMAIL=<email>
# automatically update settings to the new version
# comment this line if you made / will make some modifications to the settings
#SEARX_COMMAND=-f
# use openssl rand -base64 33
MORTY_KEY=base64key_goes_here
3. The problem I’m having:
Navigating to the subdomain presents a blank page when using auto-generated certs
Navigating to the subdomain does not load anything at all with Cloudflare cert+key pair (auto-generate is off, no root CA provided).
Please note this is specifically an issue isolated to TLS – the instance functions locally (direct IP) 100% correctly if I auto-issue a cert via caddy. It does not work when proxied though.
4. Error messages and/or full log output:
docker logs caddy
{"level":"info","ts":1618110663.9686768,"msg":"using provided configuration","config_file":"/etc/caddy/Caddyfile","config_adapter":"caddyfile"}
{"level":"warn","ts":1618110663.978929,"logger":"admin","msg":"admin endpoint disabled"}
{"level":"info","ts":1618110663.9799156,"logger":"tls.cache.maintenance","msg":"started background certificate maintenance","cache":"0xc000353030"}
{"level":"warn","ts":1618110663.9819667,"logger":"tls","msg":"stapling OCSP","error":"no OCSP stapling for [cloudflare origin certificate *.tomlawson.io tomlawson.io]: no URL to issuing certificate"}
{"level":"info","ts":1618110663.986797,"msg":"autosaved config","file":"/config/caddy/autosave.json"}
{"level":"info","ts":1618110663.9868426,"msg":"serving initial configuration"}
{"level":"info","ts":1618110663.9907296,"logger":"tls","msg":"cleaned up storage units"}
5. What I already tried:
I have tried the following:
- Turning off auto certs & mounting the cert files directly into Caddy as mounted volumes (like I usually do)
- Turning on auto-certs & generating a cert, which works locally, but does not work when forwarded by Traefik
- Generating my own TLS cert/key pair instead of Origin Certs
Thanks, yes – you’re absolutely right. Sorry that’s because I forgot to change the file back before posting. I read somewhere to try it with the domain as the filename.
docker-compose.yaml
version: '3.7'
services:
caddy:
container_name: caddy
image: caddy:2-alpine
restart: on-failure
network_mode: host
volumes:
- ./Caddyfile:/etc/caddy/Caddyfile:ro
- caddy-data:/data:rw
- caddy-config:/config:rw
- /certs/tomlawson.io-cf-origin.pem:/certs/tomlawson.io-cf-origin.pem:ro
- /certs/tomlawson.io-cf-origin.key:/certs/tomlawson.io-cf-origin.key:ro
environment:
- SEARX_HOSTNAME=${SEARX_HOSTNAME:-localhost}
- SEARX_TLS=${LETSENCRYPT_EMAIL:-internal}
cap_drop:
- ALL
cap_add:
- NET_BIND_SERVICE
- DAC_OVERRIDE
Caddyfile
{
admin off
auto_https off
}
{$SEARX_HOSTNAME} {
log {
output discard
}
# tls {$SEARX_TLS}
tls /certs/tomlawson.io-cf-origin.pem /certs/tomlawson.io-cf-origin.key
# tls {
# client_auth {
# mode require_and_verify
# trusted_ca_cert_file /certs/rootCA.cer
# }
# }
@api {
path /config
path /status
}
@static {
path /static/*
}
@notstatic {
not path /static/*
}
@morty {
path /morty/*
}
@notmorty {
not path /morty/*
}
header {
# Enable HTTP Strict Transport Security (HSTS) to force clients to always connect via HTTPS
Strict-Transport-Security "max-age=31536000; includeSubDomains; preload"
# Enable cross-site filter (XSS) and tell browser to block detected attacks
X-XSS-Protection "1; mode=block"
# Prevent some browsers from MIME-sniffing a response away from the declared Content-Type
X-Content-Type-Options "nosniff"
# Disallow the site to be rendered within a frame (clickjacking protection)
X-Frame-Options "SAMEORIGIN"
# Disable some features
Feature-Policy "accelerometer 'none';ambient-light-sensor 'none'; autoplay 'none';camera 'none';encrypted-media 'none';focus-without-user-activation 'none'; geolocation 'none';gyroscope 'none';magnetometer 'none';microphone 'none';midi 'none';payment 'none';picture-in-picture 'none'; speaker 'none';sync-xhr 'none';usb 'none';v$
# Referer
Referrer-Policy "no-referrer"
# X-Robots-Tag
X-Robots-Tag "noindex, noarchive, nofollow"
# Remove Server header
-Server
}
header @api {
Access-Control-Allow-Methods "GET, OPTIONS"
Access-Control-Allow-Origin "*"
}
# Cache
header @static {
# Cache
Cache-Control "public, max-age=31536000"
defer
}
header @notstatic {
# No Cache
Cache-Control "no-cache, no-store"
Pragma "no-cache"
}
# CSP (see http://content-security-policy.com/ )
header @morty {
Content-Security-Policy "default-src 'none'; style-src 'self' 'unsafe-inline'; form-action 'self'; frame-ancestors 'self'; base-uri 'self'; img-src 'self' data:; font-src 'self'; frame-src 'self'"
}
header @notmorty {
Content-Security-Policy "upgrade-insecure-requests; default-src 'none'; script-src 'self'; style-src 'self' 'unsafe-inline'; form-action 'self'; font-src 'self'; frame-ancestors 'self'; base-uri 'self'; connect-src 'self' https://overpass-api.de; img-src 'self' data: https://*.tile.openstreetmap.org; frame-src https://www.yout$
}
# Morty
handle @morty {
reverse_proxy localhost:3000
}
# Filtron
handle {
encode zstd gzip
reverse_proxy localhost:4040 {
header_up X-Forwarded-Port {http.request.port}
header_up X-Forwarded-Proto {http.request.scheme}
header_up X-Forwarded-TlsProto {tls_protocol}
header_up X-Forwarded-TlsCipher {tls_cipher}
header_up X-Forwarded-HttpsProto {proto}
}
}
}
docker logs caddy
{"level":"info","ts":1618146649.0440636,"msg":"using provided configuration","config_file":"/etc/caddy/Caddyfile","config_adapter":"caddyfile"}
{"level":"warn","ts":1618146649.0515978,"logger":"admin","msg":"admin endpoint disabled"}
{"level":"info","ts":1618146649.052488,"logger":"tls.cache.maintenance","msg":"started background certificate maintenance","cache":"0xc0003d2a80"}
{"level":"warn","ts":1618146649.0857294,"logger":"tls","msg":"stapling OCSP","error":"no OCSP stapling for [cloudflare origin certificate *.tomlawson.io tomlawson.io]: no URL to issuing certificate"}
{"level":"warn","ts":1618146649.239062,"logger":"pki.ca.local","msg":"installing root certificate (you might be prompted for password)","path":"storage:pki/authorities/local/root.crt"}
2021/04/11 13:10:49 Warning: "certutil" is not available, install "certutil" with "apt install libnss3-tools" or "yum install nss-tools" and try again
2021/04/11 13:10:49 define JAVA_HOME environment variable to use the Java trust
2021/04/11 13:10:49 certificate installed properly in linux trusts
{"level":"info","ts":1618146649.372279,"msg":"autosaved config","file":"/config/caddy/autosave.json"}
{"level":"info","ts":1618146649.3723478,"msg":"serving initial configuration"}
{"level":"info","ts":1618146649.3734577,"logger":"tls","msg":"cleaned up storage units"}
If I allow auto-https, the service can be accessed @ https://server-ip/
docker logs caddy
{"level":"info","ts":1618146996.6874382,"msg":"using provided configuration","config_file":"/etc/caddy/Caddyfile","config_adapter":"caddyfile"}
{"level":"warn","ts":1618146996.69644,"logger":"admin","msg":"admin endpoint disabled"}
{"level":"info","ts":1618146996.6977289,"logger":"tls.cache.maintenance","msg":"started background certificate maintenance","cache":"0xc00036e620"}
{"level":"warn","ts":1618146996.7001317,"logger":"tls","msg":"stapling OCSP","error":"no OCSP stapling for [cloudflare origin certificate *.tomlawson.io tomlawson.io]: no URL to issuing certificate"}
{"level":"info","ts":1618146996.7006288,"logger":"http","msg":"enabling automatic HTTP->HTTPS redirects","server_name":"srv0"}
{"level":"info","ts":1618146996.754106,"logger":"tls","msg":"cleaned up storage units"}
{"level":"warn","ts":1618146996.8651402,"logger":"pki.ca.local","msg":"installing root certificate (you might be prompted for password)","path":"storage:pki/authorities/local/root.crt"}
2021/04/11 13:16:36 Warning: "certutil" is not available, install "certutil" with "apt install libnss3-tools" or "yum install nss-tools" and try again
2021/04/11 13:16:36 define JAVA_HOME environment variable to use the Java trust
2021/04/11 13:16:36 certificate installed properly in linux trusts
{"level":"info","ts":1618146996.9894936,"logger":"http","msg":"enabling automatic TLS certificate management","domains":["192.168.11.23"]}
{"level":"warn","ts":1618146996.991861,"logger":"tls","msg":"stapling OCSP","error":"no OCSP stapling for [192.168.11.23]: no OCSP server specified in certificate"}
{"level":"info","ts":1618146996.9978938,"msg":"autosaved config","file":"/config/caddy/autosave.json"}
{"level":"info","ts":1618146996.997971,"msg":"serving initial configuration"}
{"level":"info","ts":1618146996.9988823,"logger":"tls.renew","msg":"acquiring lock","identifier":"192.168.11.23"}
{"level":"info","ts":1618146996.99958,"logger":"tls.renew","msg":"lock acquired","identifier":"192.168.11.23"}
{"level":"info","ts":1618146997.0008771,"logger":"tls.renew","msg":"renewing certificate","identifier":"192.168.11.23","remaining":3478.999129224}
{"level":"info","ts":1618146997.0054393,"logger":"tls.renew","msg":"certificate renewed successfully","identifier":"192.168.11.23"}
{"level":"info","ts":1618146997.0054946,"logger":"tls.renew","msg":"releasing lock","identifier":"192.168.11.23"}
{"level":"info","ts":1618146997.0058832,"logger":"tls","msg":"reloading managed certificate","identifiers":["192.168.11.23"]}
{"level":"warn","ts":1618146997.0073195,"logger":"tls","msg":"stapling OCSP","error":"no OCSP stapling for [192.168.11.23]: no OCSP server specified in certificate"}
{"level":"info","ts":1618146997.0073934,"logger":"tls.cache","msg":"replaced certificate in cache","identifiers":["192.168.11.23"],"new_expiration":1618190197}
However, when I try to forward to Caddy from Traefik, I get a blank screen?
Okay, so… progress:
If I allow auto-gen I can now access the site via my subdomain. The fix in my case was to alter the Caddyfile such that both the local IP and sub-domain were listed in the Caddyfile.
However, this does not resolve my issue, as it relies on the auto-generated self-signed certificate and does not use the Origin Certs, which I cannot seem to get to be used.
Why are you using Traefik? Caddy and Traefik can serve the same role. Why not just use one or the other?
When you try to proxy HTTPS traffic, something has to terminate the TLS connection. As far as I know, Traefik can’t proxy raw TLS bytes without terminating TLS.
This means you have essentially two hops of TLS in your setup, i.e. Cloudflare to Traefik, then Traefik to Caddy (then possibly Caddy to your app if you proxy over https:// which you don’t seem to be here, which is fine).
If I were you, I’d just cut out Traefik altogether, it’ll simplify things significantly.
As mentioned above, Traefik serves my whole network. Caddy is baked into a single service I’m running in an isolated VM for security. I’m aware it creates an additional TLS hop but there’s no real way around it because Caddy is heavily integrated into Searx.
Re-working the entire network/server to use Caddy isn’t an option and would still result in two hops, because then I’d have Caddy1 to Caddy2 anyway.
You could use mTLS with Caddy to secure the connection between your front Caddy instance (which acts like an ACME CA) and your backend one.
Example setup here:
Or just proxy over HTTP (plaintext) between Traefik and Caddy if you can’t manage to establish trust between them.
Thanks, I’ve book marked that guide as a next step 
The part that is confusing to me, is that if I allow Caddy to auto-generate a certificate then the setup works and I can forward to the service perfectly, but it serves the Caddy generated certificate.
What I’m trying to do instead, is to use Caddy to serve my existing cert.pem and key.pem but it doesn’t seem to serve them.
I think the problem stems from this warning:
{"level":"warn","ts":1618151666.779669,"logger":"tls","msg":"stapling OCSP","error":"no OCSP stapling for [cloudflare origin certificate *.tomlawson.io tomlawson.io]: no URL to issuing certificate"}
When Caddy creates a certificate from its own root CA, I’m assuming it does the ‘OCSP stapling’ that it wants to see, and therefore it works with the self signed cert and that this means that the CF origin certs won’t work unless they include the OCSP stapling.
Thinking about it, I’ll try to setup the mTLS now as it may resolve my issue
Ps,
Caddy does actually look really good. I’d never seen it before but will definitely be playing with it to see where else I can use it. In this particular scenario it would just be hours and hours of rework as I’d literally be reworking every service I host to make one new one work.
1 Like
No that’s just a warning, it’s irrelevant here. OCSP is not required but it’s generally recommended.
I haven’t seen your Traefik setup, but I honestly doubt that. Caddy is generally very simple to use (this one app you’re serving is overly complicated with its header config). Much simpler config than Traefik’s generally.
I don’t think the problem is Caddy here, but instead with Traefik and what it’s accepting from Caddy. You haven’t really shown any evidence of that piece of the puzzle so I can’t really offer any help. And I don’t really want to give support for Traefik anyways.
Ultimately the underlying thing about TLS is trust. If Traefik doesn’t trust the certificate Caddy presents, then it won’t complete the round-trip.
You can play around with making the request directly to Caddy with curl -v (and make sure you set up SNI in the curl command so Caddy understands what hostname the request is for; google it) and see what it looks like. If that looks good, then you know the problem is Traefik.
Fully appreciate you wouldn’t want to support a competing product, I’m not sure where the problem is so it’s difficult to pick the right place to ask questions.
I also understand that (naturally) you’re advocating for Caddy here as I’m on a Caddy community forum. I’m neither saying, nor implying Traefik is better than Caddy. It’s just what I’ve been using to date and as a hobbyist with a limited skill set, the prospect of a full switch is too much work for the free time I have outside of family and work. My docker-compose.yml on my main docker VM makes extensive use of Traefik’s labels for about 30 containers and then there’s reverse proxying to other things outside of docker.
I also find it hard to understand how it can (and does) work just fine with an auto-generated cert from Caddy but won’t work with a different cert. I think it’s a reasonable piece of logic/rationale to think that if it works with one cert as expected, I should be able to use a cert with different details.
I’ll delve further into it in the Traefik community forums, Google etc.
Thank you for all your help, I do appreciate it.
You should probably enable whatever logging facilities Traefik provides (debug level etc) and see what it says when proxying the request to Caddy.
Or like I said, play with curl -v pointing directly to Caddy (bypassing Traefik) to see how it behaves.
1 Like
Thanks, the SNI hint was very helpful and you’re absolutely right, it would appear the problem is with Traefik.
For anyone else, the curl command I used was:
curl -vik --resolve example.com:443:<local IP> https://example.com/
I can see the page content and the correct TLS certificate is being served (the one I want it to). I really appreciate the help in identifying that and have actually bookmarked the resources I found when searching for info on curl & SNI as it’s clearly a very useful troubleshooting tool. Doubly helpful.
Resources for others in future:
https://hacksbrain.com/2018/08/27/testing-sni-enabled-servers-with-curl/
https://curl.se/docs/manpage.html
2 Likes
This topic was automatically closed after 30 days. New replies are no longer allowed.