Are you perfectly certain it is Caddy ?
Have you compared performance with other applications ?
Is the container or host doing any domain/hostname resolve ?
If you are using Docker for Mac, I know docker-compose is slow in executing commands but that should not affect a running container.
You will need to provide more details to enable us narrow down the issue.