Not really. Maybe like 5ms at most with Cloudflare Tunnels. Negligible.
You don’t have to use acmedns. You can use the DNS provider module for your DNS provider:
And building Caddy with plugins is very easy. You can just download a build directly from Download Caddy with the plugins you need, or follow these instructions Build from source — Caddy Documentation which is as simple as downloading Go, downloading xcaddy, then running xcaddy build --with <plugins>.