To clarify, I think that simple Caddyfile should work just fine. For context, another user confirmed recently that it worked for them:
The only thing you might want to add is a rule to block .php files from the /uploads directory:
respond /uploads/*.php 404
I feel that the HTTP 500 error is unrelated to the way Caddy is passing the request to your server, but instead some other issue. Best to find the error logs and see what it says.