If you don’t want Caddy to use port 443 (which is the default for production sites), you need to specify http:// or :80 in the address in the Caddyfile or use tls off in your site description.
Neither apache nor nginx automatically use ACME to obtain a certificate and if you configure them to serve HTTPS without a certificate (the equivalent of what you are doing here), they will hard fail.
I am definitely with matt here, if I configure my webserver just so and it cannot do what I have asked it to, I would prefer it to fail than silently decide to do something else. Draconian is exactly what I want.
But, as a possible solution to your woes, you could maybe use an environment variable so your config is static but the tls option is configured externally.