Check if there’s something like ufw running on that machine. Triple check that your IP address is indeed correct.
I’m not sure what else to tell you, this isn’t a problem with Caddy in particular, it’s a problem in networking somewhere between the outside world and Caddy.