To be specific, the automatic redirection behaviour can be disabled (by adding https:// or :443 to your site label). This will not stop Caddy from attempting to bind to :80 in order to complete ACME challenges for the purpose of managing your certificates, though.
From memory, if you specify a HTTPS-only label with DNS validation enabled, Caddy won’t bind to :80.