I think I’ve encountered a similar thing. Docker essentially runs NAT with port forwarding (as defined in the “ports” section of your compose file). So from caddy’s perspective all connections are indeed coming from the bridge gateway.
You’ll want to put this container in host network mode, so it directly attaches.