@andrewhamon suggested something really cool: ONBUILD.
ONBUILD would allow us to create a base Caddy image that would include everything needed to build Caddy+plugins. Then a consumer would make a new Dockerfile to use it like this:
FROM caddy/caddy:base
ENV PLUGINS git,hugo
// Entrypoint & CMD
That way users get to easily choose their plugins without having to wait for compilation whenever they run the image.