So this was it the entire time!!! Thank you so much!!! It is working now. So now I just need to get back to my docker container so Im not using precious RAM in my ESXi server.
So on my docker image I am using Portainer as a crutch. Basically I am running it with a few mapped volumes and ports but not much else.
My ports are
My volumes are:
HOST: /opt/caddy/Caddyfile to Caddy: /etc/Caddyfile
HOST: /opt/caddy/.caddy to Caddy: /root/.caddy
HOST: /opt/caddy/srv to Caddy: /srv