Hi @xnaas, welcome to the Caddy community.
There’s no functionality available in the browse directive to supply a list of files/directories that shouldn’t be listed.
That said, Caddy uses Go’s built-in text/template package to render the index pages for browse.
The template included by default can be seen here: https://github.com/mholt/caddy/blob/8369a1211544224b2967dd3ac43372a2ef432291/caddyhttp/browse/setup.go#L115-L515
You could take that, alter it to manually check that the file/folder doesn’t match what you don’t want to list, and supply the template back in the Caddyfile. It’d be a bit complicated if you’re not familiar with text/template already, though.
You could also open a ticket at https://github.com/mholt/caddy/issues and request this as a feature. One of our contributors might be able to implement what you’re after.