Yeah, the builder image just ships with the latest version of xcaddy and Go. You need to use a builder image with at least Go 1.20, and the the 2.7.2 image does.
Yes, those are automated rebuilds from the Docker Official Images program. Those rebuilds are normal, depending on upstream dependencies triggering updates.
I had the same issue, since I needed to build caddy to add a couple of modules.
This dockerfile worked for me, after trial and error, since github.com/caddy-dns/cloudflare does not accept any version lower than 2.7.3:
FROM caddy:2.7.2-builder AS builder
RUN xcaddy build v2.7.3 \
--with github.com/caddy-dns/cloudflare \
--with github.com/hslatman/caddy-crowdsec-bouncer/http
FROM caddy:2.7.2
COPY --from=builder /usr/bin/caddy /usr/bin/caddy