It’s because you have these lines:
This sets different configuration for your each site, and those will inherit the default order because it’s not specified there.
If you run caddy adapt --pretty on your config and look at the tls section, you’ll probably see two automation policies (you omitted the rest of your sites so I’m making an assumption here). The first one will have a subjects “matcher” which makes the first apply – this one will have it in the order of acme then zerossl. But there will be a second automation policy without a subjects matcher which has the order of zerossl then acme.
The solution in your case would be to remove the cert_issuer globals and move it to your tls snippet with the issuer subdirective. This will make sure that the order properly applies to all your sites.
That’s because you’re still using the same account (email) so Let’s Encrypt “remembers” you, and says “hey, you already have an issued cert for that account, so here you go” and has you download it again, without issuing a new one.
You should probably switch to the Let’s Encrypt staging endpoint if you want to test issuance.
Yeah, with the tls_trusted_ca_certs transport option. You may also need to configure tls_server_name so that the SNI matches what domain is actually in the certificate even though you’re connecting with an IP address. If the self-signed cert doesn’t have a domain in it, you probably should make sure it has one (or that IP address).
Alternatively, you could run Caddy on that server too and do mTLS between the two Caddy instances - this means the publicly accessible Caddy instance would act as an ACME server (with the acme_server directive) and your private one would point to the public one with the acme_ca global option.
Here’s a nice wiki explaining how that works: