I want to know the creators perspective on this as well.
Personally i will use caddy as a base only if i want to add functionality on top of a webserver / proxy server.
Caddy has wonderful docs on how you can extend Caddy to add your custom logic - Extending Caddy — Caddy Documentation
For everything else, i would stick to the standard http library (Go by Example: HTTP Servers) or using something like gin (GitHub - gin-gonic/gin: Gin is a HTTP web framework written in Go (Golang). It features a Martini-like API with much better performance -- up to 40 times faster. If you need smashing performance, get yourself some Gin.)
But again, this is my personal opinion. Would like to know the maintainer’s views on this.