I have used the json format to configure caddy and there is it quite easy to configure a handler when the remote ip does not match.
You can see a example in this question.
You can configure caddy via api with the json file or add the -adapter jsonc to caddy and use the json as -config caddy.json
Here is the command for xcaddy to add the adapter
xcaddy build --with github.com/caddyserver/jsonc-adapter
# run caddy with json config
caddy run -adapter jsonc -config caddy.json