I think the intermittent nature might related to timing and when an HTTP connection between cloudflared and caddy is reused, versus when a new one is opened.
What’s in your Caddyfile now? Do you have one certificate or multiple?
First of all, thanks for sticking with me in trying to resolve my issue in my application of Caddy.
Answering your question @hmoffatt, this is how my Caddyfile is.
{
debug
log {
output file /var/log/access.log
}
auto_https prefer_wildcard
}
(localonly) {
@denied not remote_ip private_ranges
abort @denied
}
(mtls) {
tls {
client_auth {
mode require_and_verify
trust_pool file <file_directory>/ca.pem
}
}
}
*.mydomain.net {
tls {
dns cloudflare {env.CLOUDFLARE_API_KEY}
}
}
serviceX.mydomain.net {
reverse_proxy <serviceX_ip>:<serviceX_port>
}
serviceY.mydomain.net {
reverse_proxy <serviceY_ip>:<serviceY_port>
}
serviceZ.mydomain.net { #this service is not accessible through the Cloudflare tunnel, e.g.: not exposed to the internet
import localonly
import mtls
reverse_proxy <serviceZ_ip>:<serviceZ_port>
}
Have you read this?
Yes @victor, I have read all wiki and posts (here, reddit, github, and whatever google could throw at me) possible regarding Caddy + Cloudflare tunnel.
They don’t really work with my configuration, setup and I want to do.
The general idea of what I want to do works with NGINX Proxy Manager (NPM), but as I said, NPM is buggy and I appreciate how fast , easy to change and manage, and reliable Caddyserver is.
I have Caddyserver reverse proxy meeting all the criteria I want for locally accessed services, but when I throw the exposed services into the mix, it doesn’t really work. The goal is to make Caddy work with the same requirements as NPM currently does.
These are the criteria I want:
- Access to services with reverse proxy, either locally or externally exposed
- Let’s encrypt certificate with DNS challenge
- When exposed, done with Cloudflare tunnel
- Not all services are externally accessible, but all of the services are to use the same Let’s encrypt certificate
- I want to be able to chose which services will only be accessible with mTLS (which is self-signed), when accessed locally.
For criteria 5, it is easy with NPM, because all I need to do is the following configuration for a given endpoint. And it still works with Cloudflare tunnel and the DNS challenge certificate.
So far, with the help of @TheRettom and @hmoffatt I’ve been able to partially meet the criteria. I feel we are almost there, but I just to figure out @hmoffatt last response in terms of configuration.
1 Like
So, I was able to make it work with all the criteria I wanted above, but I needed to add strict_sni_host insecure_off which I’m not sure if this is the recommended configuration.
{
debug
auto_https prefer_wildcard
servers {
strict_sni_host insecure_off
}
}
(localonly) {
@denied not remote_ip private_ranges
abort @denied
}
(mtls) {
tls {
client_auth {
mode require_and_verify
trust_pool file <file_directory>/ca.pem
}
}
}
*.mydomain.net {
tls {
dns cloudflare {env.CLOUDFLARE_API_KEY}
}
}
serviceX.mydomain.net {
reverse_proxy <serviceX_ip>:<serviceX_port>
}
serviceY.mydomain.net {
reverse_proxy <serviceY_ip>:<serviceY_port>
}
serviceZ.mydomain.net { #this service is not accessible through the Cloudflare tunnel, e.g.: not exposed to the internet
import localonly
import mtls
reverse_proxy <serviceZ_ip>:<serviceZ_port>
}
What certificates do you see Caddy generating? I think (but I’m not sure) that you are telling it to generate four different certs in your example, which is contrary to your requirement of having a single cert, and also probably confusing things with cloudflared.
Caddy is only issuing two certificates: *.mydomain.net and mydomain.net.
But the issue that’s holding me back is this:
TLS ServerName (*.mydomain.net) and HTTP Host (serviceX.mydomain.net) values differ
This happens when I don’t have the global setting strict_sni_host insecure_off, which it sounds to me is not recommended.
If there’s no way around it, I will call it as the a solution, but make it clear that is not recommended.
I’m not really sure if it makes sense to have *.mydomain.net then service1.mydomain.net etc as separate blocks. We need an expert here to comment.
The documentation for wildcards gives an example with one block and matching within. Common Caddyfile Patterns — Caddy Documentation
I think another solution to your issue would be to use different ports for different servers, as this will force cloudflared not to mix requests for different hosts on a single HTTPS connection. But I think the problem is still your configuration.
Yeah, I have tried the pattern available in the docs, the problem is that it does not fulfill all of my criteria list.
Still, you could try it as part of debugging.
Which criteria aren’t met? I think you said it’s the mTLS part, which you’re not using through cloudflared anyway so that could still be its own section.
Could you give an example on how to separate that?
I’m not sure exactly what you mean, but what I’m suggesting is
*.mydomain.net {
tls {
dns cloudflare {env.CLOUDFLARE_API_KEY}
}
@serviceX host serviceX.mydomain.net
handle @serviceX {
reverse_proxy <serviceX_ip>:<serviceX_port>
}
@serviceY host serviceY.mydomain.net
handle @serviceY {
reverse_proxy <serviceY_ip>:<serviceY_port>
}
}
Ignore serviceZ for now as you’re not using it with the tunnel.
I’ve tried that already, and it works, but it doesn’t fulfill all of my criteria. Maybe Caddy is not the way to go with what I want to do…
I apprecite the support from y’all anyway.
This seems fishy. It’ll set the Host header to '*.mydomain.net, which is invalid. I’m not versed in cloudflare tunnnel config, so I don’t know how to make it variable based on the incoming request.
Unless you can set the httpHostHeader with the accurate Host value which Caddy should expect, I’d recommend making multiple entries of this per each host name. That probably should resolve your issue.
I’m not even convinced that any httpHostHeader configuration is required here. I think cloudflared will use the original if this is not configured.
The last resort would be to put the different hosts on different ports.
1 Like
I decided to take the debug logs while accessing from different ways, including your suggestion @Mohammed90, so we can compare.
I still can’t figure out what is going on, but with your suggestion, the screen is white, so nothing loads and no error is mentioned. Looking at the last block below, the handshake is matched, but the request doesn’t move to the reserve_proxy bit, as it does with the local access.
## Local access - via my AdgaurdHome DNS instance that also points to Caddy via a wildcard hostname (*.mydomain.net -> <caddy_lxc_ip>) - this works fine
2025/02/24 17:16:09.076 DEBUG events event {"name": "tls_get_certificate", "id": "<id>", "origin": "tls", "data": {"client_hello":{"CipherSuites":[4865,4866,4867],"ServerName":"serviceX.mydomain.net","SupportedCurves":[4588,29,23,24],"SupportedPoints":null,"SignatureSchemes":[1027,2052,1025,1283,2053,1281,2054,1537,513],"SupportedProtos":["h3"],"SupportedVersions":[772],"RemoteAddr":{"IP":"<cloudflared_lxc_ip>5","Port":62464,"Zone":""},"LocalAddr":{"IP":"<caddy_lxc_ip>","Port":443,"Zone":""}}}}
2025/02/24 17:16:09.076 DEBUG tls.handshake no matching certificates and no custom selection logic {"identifier": "serviceX.mydomain.net"}
2025/02/24 17:16:09.076 DEBUG tls.handshake choosing certificate {"identifier": "*.mydomain.net", "num_choices": 1}
2025/02/24 17:16:09.076 DEBUG tls.handshake default certificate selection results {"identifier": "*.mydomain.net", "subjects": ["*.mydomain.net"], "managed": true, "issuer_key": "acme-v02.api.letsencrypt.org-directory", "hash": "<hash>"}
2025/02/24 17:16:09.076 DEBUG tls.handshake matched certificate in cache {"remote_ip": "<cloudflared_lxc_ip>5", "remote_port": "62464", "subjects": ["*.mydomain.net"], "managed": true, "expiration": "2025/05/02 05:45:30.000", "hash": "<hash>"}
2025/02/24 17:16:09.085 DEBUG http.handlers.reverse_proxy selected upstream {"dial": "<serivceX_ip>", "total_upstreams": 1}
2025/02/24 17:16:09.091 DEBUG http.handlers.reverse_proxy upstream roundtrip {"upstream": "<serivceX_ip>", "duration": 0.005240565, "request": {"remote_ip": "<cloudflared_lxc_ip>5", "remote_port": "62464", "client_ip": "<cloudflared_lxc_ip>5", "proto": "HTTP/3.0", "method": "GET", "host": "serviceX.mydomain.net", "uri": "/api/auth-enabled", "headers": {"Sec-Ch-Ua": ["\"Not(A:Brand\";v=\"99\", \"Brave\";v=\"133\", \"Chromium\";v=\"133\""], "Accept": ["application/json"], "Referer": ["https://serviceX.mydomain.net/"], "X-Forwarded-Host": ["serviceX.mydomain.net"], "Sec-Gpc": ["1"], "Sec-Fetch-Site": ["same-origin"], "Priority": ["u=1, i"], "Sec-Ch-Ua-Platform": ["\"Windows\""], "X-Forwarded-Proto": ["https"], "User-Agent": ["Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/133.0.0.0 Safari/537.36"], "Sec-Ch-Ua-Mobile": ["?0"], "Accept-Encoding": ["gzip, deflate, br, zstd"], "Accept-Language": ["en-US,en;q=0.6"], "Sec-Fetch-Mode": ["cors"], "Sec-Fetch-Dest": ["empty"], "If-None-Match": ["W/\"1c-Vt/6TvvrO9SE5YmGKzP/ETOJLzw\""], "X-Forwarded-For": ["<cloudflared_lxc_ip>5"]}, "tls": {"resumed": false, "version": 772, "cipher_suite": 4865, "proto": "h3", "server_name": "serviceX.mydomain.net"}}, "headers": {"Vary": ["Origin"], "Access-Control-Allow-Credentials": ["true"], "X-Ratelimit-Remaining": ["493"], "Keep-Alive": ["timeout=5"], "X-Powered-By": ["Express"], "X-Ratelimit-Limit": ["500"], "Date": ["Mon, 24 Feb 2025 17:16:09 GMT"], "X-Ratelimit-Reset": ["1740419528"], "Etag": ["W/\"1c-Vt/6TvvrO9SE5YmGKzP/ETOJLzw\""], "Connection": ["keep-alive"]}, "status": 304}
2025/02/24 17:16:09.139 DEBUG http.handlers.reverse_proxy selected upstream {"dial": "<serivceX_ip>", "total_upstreams": 1}
2025/02/24 17:16:09.140 DEBUG http.handlers.reverse_proxy selected upstream {"dial": "<serivceX_ip>", "total_upstreams": 1}
2025/02/24 17:16:09.142 DEBUG http.handlers.reverse_proxy upstream roundtrip {"upstream": "<serivceX_ip>", "duration": 0.003571468, "request": {"remote_ip": "<cloudflared_lxc_ip>5", "remote_port": "62464", "client_ip": "<cloudflared_lxc_ip>5", "proto": "HTTP/3.0", "method": "GET", "host": "serviceX.mydomain.net", "uri": "/logo.svg", "headers": {"Sec-Ch-Ua-Mobile": ["?0"], "Accept-Encoding": ["gzip, deflate, br, zstd"], "Accept": ["image/avif,image/webp,image/apng,image/svg+xml,image/*,*/*;q=0.8"], "If-Modified-Since": ["Fri, 20 Dec 2024 09:50:26 GMT"], "Priority": ["i"], "Sec-Ch-Ua-Platform": ["\"Windows\""], "X-Forwarded-For": ["<cloudflared_lxc_ip>5"], "X-Forwarded-Proto": ["https"], "Sec-Gpc": ["1"], "Sec-Fetch-Site": ["same-origin"], "Sec-Fetch-Dest": ["image"], "Sec-Ch-Ua": ["\"Not(A:Brand\";v=\"99\", \"Brave\";v=\"133\", \"Chromium\";v=\"133\""], "Sec-Fetch-Mode": ["no-cors"], "If-None-Match": ["W/\"767-193e379bad0\""], "User-Agent": ["Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/133.0.0.0 Safari/537.36"], "Accept-Language": ["en-US,en;q=0.6"], "Referer": ["https://serviceX.mydomain.net/"], "X-Forwarded-Host": ["serviceX.mydomain.net"]}, "tls": {"resumed": false, "version": 772, "cipher_suite": 4865, "proto": "h3", "server_name": "serviceX.mydomain.net"}}, "headers": {"Last-Modified": ["Fri, 20 Dec 2024 09:50:26 GMT"], "Etag": ["W/\"767-193e379bad0\""], "Date": ["Mon, 24 Feb 2025 17:16:09 GMT"], "Connection": ["keep-alive"], "Keep-Alive": ["timeout=5"], "X-Powered-By": ["Express"], "Accept-Ranges": ["bytes"], "Cache-Control": ["public, max-age=0"]}, "status": 304}
2025/02/24 17:16:09.143 DEBUG http.handlers.reverse_proxy upstream roundtrip {"upstream": "<serivceX_ip>", "duration": 0.002275191, "request": {"remote_ip": "<cloudflared_lxc_ip>5", "remote_port": "62464", "client_ip": "<cloudflared_lxc_ip>5", "proto": "HTTP/3.0", "method": "GET", "host": "serviceX.mydomain.net", "uri": "/assets/MaterialIcons-Regular-DOtZ65Va.woff2", "headers": {"Sec-Ch-Ua": ["\"Not(A:Brand\";v=\"99\", \"Brave\";v=\"133\", \"Chromium\";v=\"133\""], "Referer": ["https://serviceX.mydomain.net/assets/index-BsI52nnX.css"], "If-Modified-Since": ["Fri, 20 Dec 2024 09:50:27 GMT"], "X-Forwarded-For": ["<cloudflared_lxc_ip>5"], "X-Forwarded-Proto": ["https"], "Origin": ["https://serviceX.mydomain.net"], "Accept-Encoding": ["gzip, deflate, br, zstd"], "Sec-Ch-Ua-Mobile": ["?0"], "Accept-Language": ["en-US,en;q=0.6"], "Sec-Fetch-Mode": ["cors"], "User-Agent": ["Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/133.0.0.0 Safari/537.36"], "Sec-Fetch-Dest": ["font"], "X-Forwarded-Host": ["serviceX.mydomain.net"], "Sec-Gpc": ["1"], "Sec-Ch-Ua-Platform": ["\"Windows\""], "Accept": ["*/*"], "If-None-Match": ["W/\"1e8bc-193e379beb8\""], "Sec-Fetch-Site": ["same-origin"], "Priority": ["u=0"]}, "tls": {"resumed": false, "version": 772, "cipher_suite": 4865, "proto": "h3", "server_name": "serviceX.mydomain.net"}}, "headers": {"Date": ["Mon, 24 Feb 2025 17:16:09 GMT"], "Connection": ["keep-alive"], "Keep-Alive": ["timeout=5"], "X-Powered-By": ["Express"], "Accept-Ranges": ["bytes"], "Cache-Control": ["public, max-age=0"], "Last-Modified": ["Fri, 20 Dec 2024 09:50:27 GMT"], "Etag": ["W/\"1e8bc-193e379beb8\""]}, "status": 304}
2025/02/24 17:16:10.720 DEBUG http.handlers.reverse_proxy selected upstream {"dial": "<serivceX_ip>", "total_upstreams": 1}
2025/02/24 17:16:10.722 DEBUG http.handlers.reverse_proxy upstream roundtrip {"upstream": "<serivceX_ip>", "duration": 0.002012722, "request": {"remote_ip": "<cloudflared_lxc_ip>5", "remote_port": "62464", "client_ip": "<cloudflared_lxc_ip>5", "proto": "HTTP/3.0", "method": "GET", "host": "serviceX.mydomain.net", "uri": "/sw.js", "headers": {"Service-Worker": ["script"], "Accept-Language": ["en-US,en;q=0.6"], "Sec-Fetch-Site": ["same-origin"], "Sec-Fetch-Dest": ["serviceworker"], "Sec-Fetch-Mode": ["same-origin"], "Accept": ["*/*"], "Sec-Gpc": ["1"], "X-Forwarded-Host": ["serviceX.mydomain.net"], "User-Agent": ["Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/133.0.0.0 Safari/537.36"], "X-Forwarded-Proto": ["https"], "Cache-Control": ["max-age=0"], "X-Forwarded-For": ["<cloudflared_lxc_ip>5"], "Priority": ["u=4, i"], "If-None-Match": ["W/\"5239-193e379c2a0\""], "Referer": ["https://serviceX.mydomain.net/sw.js"], "Accept-Encoding": ["gzip, deflate, br, zstd"], "If-Modified-Since": ["Fri, 20 Dec 2024 09:50:28 GMT"]}, "tls": {"resumed": false, "version": 772, "cipher_suite": 4865, "proto": "h3", "server_name": "serviceX.mydomain.net"}}, "headers": {"Connection": ["keep-alive"], "Keep-Alive": ["timeout=5"], "X-Powered-By": ["Express"], "Accept-Ranges": ["bytes"], "Cache-Control": ["public, max-age=0"], "Last-Modified": ["Fri, 20 Dec 2024 09:50:28 GMT"], "Etag": ["W/\"5239-193e379c2a0\""], "Date": ["Mon, 24 Feb 2025 17:16:10 GMT"]}, "status": 304}
## Cloudflare tunnel config with wildcard hostname (*.mydomain.net) and not originRequest options
2025/02/24 17:17:43.689 DEBUG events event {"name": "tls_get_certificate", "id": "<id>", "origin": "tls", "data": {"client_hello":{"CipherSuites":[49195,49199,49196,49200,52393,52392,49161,49171,49162,49172,49170,4865,4866,4867],"ServerName":"","SupportedCurves":[29,23,24,25],"SupportedPoints":"AA==","SignatureSchemes":[2052,1027,2055,2053,2054,1025,1281,1537,1283,1539,513,515],"SupportedProtos":null,"SupportedVersions":[772,771],"RemoteAddr":{"IP":"<cloudflared_lxc_ip>","Port":46962,"Zone":""},"LocalAddr":{"IP":"<caddy_lxc_ip>","Port":443,"Zone":""}}}}
2025/02/24 17:17:43.690 DEBUG tls.handshake no matching certificates and no custom selection logic {"identifier": "<caddy_lxc_ip>"}
2025/02/24 17:17:43.690 DEBUG tls.handshake no certificate matching TLS ClientHello {"remote_ip": "<cloudflared_lxc_ip>", "remote_port": "46962", "server_name": "", "remote": "<cloudflared_lxc_ip>:46962", "identifier": "<caddy_lxc_ip>", "cipher_suites": [49195, 49199, 49196, 49200, 52393, 52392, 49161, 49171, 49162, 49172, 49170, 4865, 4866, 4867], "cert_cache_fill": 0.0001, "load_or_obtain_if_necessary": true, "on_demand": false}
2025/02/24 17:17:43.690 DEBUG http.stdlib http: TLS handshake error from <cloudflared_lxc_ip>:46962: no certificate available for '<caddy_lxc_ip>'
## Cloudflare tunnel config with wildcard hostname (*.mydomain.net) or direct hostname (serviceX.mydomain.net) and with the originRequest options - the screen is white, nothing loads
2025/02/24 17:20:47.836 DEBUG tls.handshake choosing certificate {"identifier": "*.mydomain.net", "num_choices": 1}
2025/02/24 17:20:47.836 DEBUG tls.handshake default certificate selection results {"identifier": "*.mydomain.net", "subjects": ["*.mydomain.net"], "managed": true, "issuer_key": "acme-v02.api.letsencrypt.org-directory", "hash": "<hash>"}
2025/02/24 17:20:47.836 DEBUG tls.handshake matched certificate in cache {"remote_ip": "<cloudflared_lxc_ip>", "remote_port": "35280", "subjects": ["*.mydomain.net"], "managed": true, "expiration": "2025/05/02 05:45:30.000", "hash": "<hash>"}
You will have to share what the config files look like now
{
debug
auto_https prefer_wildcard
}
*.mydomain.net {
tls {
dns cloudflare {env.CLOUDFLARE_API_KEY}
}
}
serviceX.mydomain.net {
reverse_proxy <serviceX_ip>:<serviceX_port>
}
serviceY.mydomain.net {
reverse_proxy <serviceY_ip>:<serviceY_port>
}
What about cloudflared config? Have you changed anything? What does it look like now?