Perhaps I’m mistaking it, but it seems to me like the nginx configuration you posted is set to listen on your IP address (server_name my.host-ip.whatever) and reverse proxy to your public website address over HTTPS (proxy_pass https://my-public-website-address), which seems odd to me.
Anyway, from the documentation for proxy:
Because nginx’s location is /, that’s what you want for a base path here. Add in the headers, and it should look something like this:
Now, the problem with your proxy directive at the moment is that you’ve got your IP address as a base path. That means, effectively, that Caddy won’t proxy unless you request https://my-public-address/my-host-ip. However, the browse directive means that if it’s not proxying, it should be showing you what’s in the site root (which you haven’t set, so the site root is whichever directory Caddy is running in - it’s best to set root /path/to/www/html whenever you’re using browse).
The fact you’re getting “connection refused” makes me think that possibly you are trying to access your site over HTTP while you’ve configured Caddy only to listen for HTTPS connections to this site. If that’s the case, and you want Caddy to upgrade HTTP->S automatically, just remove the scheme from my-public-address and Caddy will take care of both.