Caddy permits multiplexing multiple listeners onto the same bound socket using its Listen() methods – which make graceful reloads possible. When multiple servers bind to the same socket, it’s nondeterministic which one “gets” the connection. The “random” swapping might be related to connection lifetimes / timeouts.
Generally, it is user error to bind separate services on the same socket, but as I said, Caddy technically allows it so a new config can gracefully take the place of an old one. This is why graceful reloads work on Windows, not just POSIX-compliant systems; a relatively unique feature.