Pretty sure yours is the same issue as: v2: add one server route config will effect all server route · Issue #2938 · caddyserver/caddy · GitHub
Your HTTP server reverse-proxies to the admin endpoint. The admin endpoint is commanded to load a new configuration, so it shuts down the HTTP reverse proxy (gracefully). The reverse proxy is waiting for the upstream request to complete, which is in turn waiting for itself to shut down, so… yeah… it’s a circular dependency: the admin endpoint and the reverse proxy are both waiting for each other to stop first.