No, you port forward 80 and 443 to your server running Caddy, and the rest of the ports stay internal to your local network. This is explained in this article:
If you want to run a service inside a Local Area Network (LAN) such as your home or office – and especially if you want to be able to access it from outside that network – Caddy can help you accomplish this quite easily. This guide will show you how. It assumes you’ve never done this before, but that you have some technical proficiency and are somewhat knowledgable about your own network.