TLS throttling on Raspberry Pi 4B

1. The problem I’m having:

I believe I’m getting throttled by Caddy TLS proxying on my Raspberry Pi 4B.
On LAN via a 1Gbit cable connection between my PC and Pi (router in-between) I get around 112M/s download speed from Pi server when using either direct connection, or Caddy without TLS (configuration entry starting with http://). That’s about 90% of a Gigabit link and I guess acceptable.
However, when using TLS, the speed drops to around 75M/s. I don’t think this is CPU throttling issue, because at the time of transfer Caddy is only at 30% cpu as reported by btop (but 30% on each of 4 cores, so maybe it is CPU throttled?). It’s also not going with through WAN, as my ISP limits that to around 35M/s.
The file for download speed testing is also large (3G), so this is not a connection initiation overhead issue.
Any way I can mitigate this? Thanks.

3. Caddy version:

v2.10.2 h1:g/gTYjGMD0dec+UgMw8SnfmJ3I9+M2TdvoRL/Ovu6U8=

4. How I installed and ran Caddy:

First via dpkg, then via caddy upgrade

a. System environment:

Linux pi 6.12.47+rpt-rpi-v8 #1 SMP PREEMPT Debian 1:6.12.47-1+rpt1 (2025-09-16) aarch64 GNU/Linux
Raspberry Pi 4B 8gb

d. My complete Caddy config:

http://b.domain.com {
        reverse_proxy localhost:8004 # this gives full speed
}
a.domain.com {
        reverse_proxy localhost:8004 # this throttles to around 75M/s or 65% of 112M/s
}

5. Links to relevant resources:

the sped is then tested from PC by downloading a file from Pi server with LAN connection
I’m also running a custom DNS server that maps *.domain.com to 192.168.1.227 (Pi IP)

curl http://192.168.1.227:8004/file > /dev/null # this is fast
curl http://a.domain.com/file > /dev/null # this is also fast
curl https://b.domain.com/file > /dev/null # this is throttled

I tested some more by writing a simple TLS server in Go, and it seems that those 75M/s are the best speed that RPi hardware can do with Go ChaCha20 TLS implementation. So I guess it can’t really be fixed in Caddy.

could this be what you’re running into?