Frontend Websocket connection errors with mTLS Minio Backend

1. Caddy version (caddy version):

activated and installed caddy via copr repo @caddy/caddy

v2.4.5 with Hetzner DNS plugin

2. How I run Caddy:

systemd (copr repo caddy install provided)

a. System environment:

Rocky Linux, LXC unpriviliged Container

b. Command:

systemctl start caddy.

c. Service/unit/compose file:

[Unit]
Description=Caddy
Documentation=https://caddyserver.com/docs/
After=network.target network-online.target
Requires=network-online.target

[Service]
Type=notify
User=caddy
Group=caddy
ExecStart=/usr/bin/caddy run --environ --config /etc/caddy/Caddyfile
ExecReload=/usr/bin/caddy reload --config /etc/caddy/Caddyfile
TimeoutStopSec=5s
LimitNOFILE=1048576
LimitNPROC=512
PrivateTmp=true
ProtectSystem=full
AmbientCapabilities=CAP_NET_BIND_SERVICE

[Install]
WantedBy=multi-user.target

d. My complete Caddyfile or JSON config:

frontend

##############
### Snippets ###
##############
(log_file) {
        log {
                format logfmt
                output file /var/log/caddy/{host}.access.log {
                        roll_keep 7
                }
        }
}

(custom_header) {
	header_up X-Forwarded-Host {host}
	header_up Host {http.reverse_proxy.upstream.hostport}
}

(privacy_header) {
	header {
		# disable FLoC tracking
		Permissions-Policy interest-cohort=()
		# enable HSTS
		Strict-Transport-Security max-age=31536000;
		# disable clients from sniffing the media type
		X-Content-Type-Options nosniff
		# clickjacking protection
		X-Frame-Options DENY
		# keep referrer data off of HTTP connections
		Referrer-Policy no-referrer-when-downgrade
	}
}

(hetzner_dns01) {
	tls admin@mydomain.tld {
		dns hetzner APIKEYREDACTED
		ca https://acme-v02.api.letsencrypt.org/directory
	}
}

##############
### Global ###
##############
{
	debug
}
##############
### config ###
##############
proxin.dmz.mydomain.tld {
	acme_server
	tls internal
}

s3.mydomain.tld {
	import hetzner_dns01
	import privacy_header
	reverse_proxy https://s3.subdomain.mydomain.tld {
		import custom_header
	}
}

console.s3.mydomain.tld {
	#basicauth * {
	#	user PASSWORDREDACTED
	#}
	import hetzner_dns01
	import privacy_header
	@websockets {
		header Connection *Upgrade*
		header Upgrade    websocket
	}
	reverse_proxy @websockets console.s3.subdomain.mydomain.tld/ws/* {
		#import custom_header
	}
	reverse_proxy https://console.s3.subdomain.mydomain.tld {
		import custom_header
	}
}

backend

########### snippets ##################################
(custom_headers) {
	#header_up X-Forwarded-Ssl on
	header_up Host {host}
	header_up X-Real-IP {remote}
	#header_up X-Forwarded-For {remote}
	#header_up X-Forwarded-Port {server_port}
	#header_up X-Url-Scheme {scheme}
	#header_up X-Forwarded-Host {host}
	#header_up X-Forwarded-Proto {scheme}
}

(proxin_acme) {
	tls {
		ca https://proxin.dmz.mydomain.tld/acme/local/directory
		ca_root /etc/ssl/certs/root.crt
	}
}
######################################################
{
	debug
}

s3.subdomain.mydomain.tld {
	import proxin_acme
	reverse_proxy s3.subdomain.mydomain.tld:9000 {
		import custom_headers
	}
}

console.s3.subdomain.mydomain.tld {
	import proxin_acme
	reverse_proxy console.s3.subdomain.mydomain.tld:9001 {
		import custom_headers
	}
}

3. The problem I’m having:

Download of Diagnose Data via minio webconsole is failing for me, when trying to access the webconsole through frontend.
browser devtools showing that the websocket connection cant be established with the following error
connection is closed by by server with code:1006

However, access and Download Works fine if i access the minio webconsole directly via console.s3.subdomain.mydomain.tld

So my guess was that something is missing to pass trough the websocket connections at the frontend and ended up with the websockets handler config in the frontend with no luck.
Also trying to play with the header-upoptions due to no success

I am using mTLS for frontend/backend communication and everything just works fine with no errors after a few tries, thanks for making the setup a piece of cake! donations incoming :wink:

i was looking for minio use cases here in the forum i stumbled upon a minio config from user m90 and would like to know more about this part, especially the header_down options.
thanks in advance

mysite.com {
  log
  encode gzip
  reverse_proxy {
    to minio:9000
    header_down -Server
    header_down -Content-Security-Policy
    header_down -X-Amz-Request-Id
  }

  header {
    Permissions-Policy interest-cohort=()
  }

  @noslash {
    not expression {path}.endsWith("/")
    path_regexp ^[a-z\-/]*$
  }

  @with_query {
    not expression {query}.size() == 0
  }

  handle @noslash {
    redir @with_query {path}/?{query} permanent
    redir {path}/ permanent
  }

  handle {
    @document expression {path}.endsWith("/")
    route @document {
      rewrite {path}index.html
    }
    rewrite * /bucket{path}
  }
...

I see you removed some sections of the help topic template. Next time, please fill out all sections, it’s important. For example, I’m not seeing Caddy’s logs, which would help show what Caddy is actually doing with your requests.

Small thing, as of v2.4.4, you can shorten that line to this:

header_up Host {upstream_hostport}

You have /ws/* appended to the end of your upstream here. What exactly are you trying to do with that?

Caddy’s reverse_proxy module doesn’t support performing rewrites at the same time, because that’s not one of its concerns. If you need to do a rewrite before sending the websocket upstream, use the rewrite directive for that.

If you don’t need a rewrite to happen, and requests already have /ws/ in the path, then you can probably just remove that chunk, because your other reverse_proxy immediately following will proxy websocket requests upstream all the same.

Those header_down lines are just removing some headers that that user decided they didn’t want to send back to the client. I don’t think they’re unlikely to be useful for you, unless you specifically need them.

1 Like

thanks for you help and clearing things up for me

error catched in frontend while trying to access diagnosis:

3.subdomain.mydomain.tld:443","uri":"/ws/health-info?deadline=1h","headers":{"X-Real-Ip":["10.111.0.100:49341"],"Cookie":["token=TOKENIDREDACTED"],"User-Agent":["Mozilla/5.0 (X11; Fedora; Linux x86_64; rv:93.0) Gecko/20100101 Firefox/93.0"],"Sec-Websocket-Key":["WEBSOCKETKEYREDACTED"],"Sec-Fetch-Mode":["websocket"],"Upgrade":["websocket"],"Accept-Encoding":["gzip, deflate, br"],"Accept":["*/*"],"Sec-Fetch-Dest":["websocket"],"Sec-Fetch-Site":["same-origin"],"Pragma":["no-cache"],"X-Forwarded-For":["10.111.0.100"],"Accept-Language":["en-US,en;q=0.5"],"X-Forwarded-Host":["console.s3.mydomain.tld"],"Origin":["https://console.s3.mydomain.tld"],"Connection":["Upgrade"],"Cache-Control":["no-cache"],"Sec-Websocket-Extensions":["permessage-deflate"],"Sec-Websocket-Version":["13"],"X-Forwarded-Proto":["https"]},"tls":{"resumed":true,"version":772,"cipher_suite":4865,"proto":"http/1.1","proto_mutual":true,"server_name":"console.s3.mydomain.tld"}},"headers":{"Content-Length":["96"],"Content-Type":["text/plain; charset=utf-8"],"Sec-Websocket-Version":["13"],"Server":["Caddy","MinIO Console"],"X-Content-Type-Options":["nosniff"],"Date":["Mon, 18 Oct 2021 01:33:13 GMT"],"X-Frame-Options":["DENY"],"X-Xss-Protection":["1; mode=block"]},"status":403}

corresponding backend error

Oct 18 01:33:13 s3 caddy[903]: {"level":"debug","ts":1634520793.0490696,"logger":"http.handlers.reverse_proxy","msg":"upstream roundtrip","upstream":"console.s3.subdomain.mydomain.tld:9001","request":{"remote_addr":"10.9.8.76:33198","proto":"HTTP/1.1","method":"GET","host":"console.s3.subdomain.mydomain.tld","uri":"/ws/health-info?deadline=1h","headers":{"Origin":["https://console.s3.mydomain.tld"],"X-Real-Ip":["10.9.8.76:33198"],"Cache-Control":["no-cache"],"X-Forwarded-Host":["console.s3.mydomain.tld"],"Accept-Encoding":["gzip, deflate, br"],"X-Forwarded-Proto":["https"],"User-Agent":["Mozilla/5.0 (X11; Fedora; Linux x86_64; rv:93.0) Gecko/20100101 Firefox/93.0"],"Accept-Language":["en-US,en;q=0.5"],"Sec-Fetch-Dest":["websocket"],"Pragma":["no-cache"],"Sec-Fetch-Site":["same-origin"],"Cookie":["token=TOKENIDREDACTED"],"Sec-Websocket-Key":["WEBSOCKETKEYREDACTED"],"X-Forwarded-For":["10.111.0.100, 10.9.8.76"],"Upgrade":["websocket"],"Sec-Fetch-Mode":["websocket"],"Sec-Websocket-Extensions":["permessage-deflate"],"Connection":["Upgrade"],"Accept":["*/*"],"Sec-Websocket-Version":["13"]},"tls":{"resumed":false,"version":772,"cipher_suite":4865,"proto":"","proto_mutual":true,"server_name":"console.s3.subdomain.mydomain.tld"}},"headers":{"Content-Type":["text/plain; charset=utf-8"],"X-Xss-Protection":["1; mode=block"],"Sec-Websocket-Version":["13"],"Server":["MinIO Console"],"X-Content-Type-Options":["nosniff"],"X-Frame-Options":["DENY"],"Date":["Mon, 18 Oct 2021 01:33:13 GMT"],"Content-Length":["96"]},"status":403}

Well, this tells me that the actual app is what responded with a 403 here. I couldn’t say why, that’ll be something specific to the app. Caddy seems to have properly proxied the request as far as I can tell. Check your Minio logs to see what might be going on, I guess.

ok thanks i will have another look into it, but i am not sure about this.
Why is it working then when accessing it via backend directly?

I don’t know. It could be any number of things. I don’t use Minio so I don’t have an understanding of when or why they would respond with a 403.

i have minio (and the console) running behind caddy (but in a docker environment).
in my Caddyfile section for the console i don’t have a section for the websockets (just the reverese_proxy for the console itself) and i could get a diagnostic file from the console.

aswell with local acme server backends? are you care to share your frontend/backend/minio config ?

no, without local acme server backends.
i have a single host (vps) with a caddy in docker, a minio (with console) in docker and several services using minio,
so my Caddyfile is pretty simple:

minio.example.com {
        tls email@example.com

        reverse_proxy minio:9000 {
                header_down Strict-Transport-Security "max-age=15768000; includeSubDomains; preload;"
                header_up X-Real-IP {remote_host}
        }

        header Strict-Transport-Security Strict-Transport-Security "max-age=15768000; includeSubDomains; preload;"
}

console.minio.example.com {
        tls email@example.com

        reverse_proxy minio:9001 {
                header_down Strict-Transport-Security "max-age=15768000; includeSubDomains; preload;"
                header_up X-Real-IP {remote_host}
        }
}

In the minio container i have an environment variable defined " MINIO_BROWSER_REDIRECT_URL", which i set to the url of the console. (see GitHub - minio/minio: High Performance, Kubernetes Native Object Storage ). I don’t know if this will work with your frontend/backend configuration (you can set this variable to either the frontend url or the backend url, not both)

OK, this works fine for me and the error is gone when i disable TLS backends!

My guess is that Headers need to be slightly different for a config with Acme Server Backend but i couldnt it get to work for now…

This topic was automatically closed after 30 days. New replies are no longer allowed.