I couldn’t configure the HTTPS record yet.
However, I can confirm that once I make OpenWrt no longer include the HTTPS record in the DNS response, I no longer have such problem.
To make that happen, I needed to not only set my DNS record in OpenWrt > Network > DHCP and DNS > DNS Records > Hostnames, but also add the /lan/<domain>/ record in OpenWrt > Network > DHCP and DNS > General > Addresses.
This isn’t ideal yet because without the HTTPS record, HTTP/2 and HTTP/3 between my browser and Caddy cannot be established as optimally as otherwise.