No, that’s a fundamental requirement of internet protocols.
Your Caddy is connecting to the upstream server in order to retrieve a response on behalf of its client, and for the upstream server to return any kind of response, it needs to know where to address it - specifically, your Caddy server’s IP address.