UPDATE:
I happened to see [WARNING] Stapling OCSP: no OCSP stapling for [my.site.com]: making OCSP request: Post "http://ocsp.int-x3.letsencrypt.org": dial tcp 66.220.149.18:80: i/o timeout in the caddy log today, and realized that the OCSP stapling sometimes just fails on my server.
This may well be a firewall issue.