This is a (hopefully) quick question about caddy’s internals — I can create a demo repo but I figured it might be reasonable just to ask, to start.
I’ve encountered what I think is a bug in an extension — the deferred headers aren’t being applied to the response. It calls io.Copy (it’s caddy-s3-proxy, so it’s copying from s3). If I call the WriteHeader method of ResponseWriter before calling io.Copy then I see the headers; if I call afterwards then I don’t. (The extension doesn’t call it at all.)
I see from http.ResponseWriter that calling the Write method automatically calls WriteHeader. And, of course, calling io.Copy does not — I see https://cs.opensource.google/go/go/+/refs/tags/go1.21.4:src/io/io.go;l=376-457 and I’d guess it’s using the WriterTo interface (i.e. for the ResponseWriter destination; how else would it set the Content-Length automatically?) but I haven’t been able to find the WriteTo implementation. I’m not familiar with go, which doesn’t help.
Does that sound like the right behaviour to expect?
2. Error messages and/or full log output:
There isn’t really log output — if necessary I can put together the example repo, and then demonstrate here.
3. Caddy version:
2.7.5
4. How I installed and ran Caddy:
Again, not that relevant, but using gomod2nix to build the version with the extension, and running as a systemd service.
I’m not sure I follow, but yes it’s correct that headers are flushed before response body is written, and it’s not possible to add more headers after the body has started being written out (because that’s how HTTP works, headers are first, then the body — except for “trailers” Trailer - HTTP | MDN which Go stdlib does support). Plugins implementing ServeHTTP and writing a response body should call WriteHeader() with a status code before starting to write the response body to flush the headers and write the HTTP status code.
“Deferred” header aren’t actually written out after the response (as you said on Github), they’re just queued to be written as soon as the status code is written. That’s done by wrapping http.ResponseWriter with a struct that waits for WriteHeader or Write (whichever comes first) and writes the headers right then just before actually writing the response status code.
Your PR to add WriteHeader looks correct. When a handler wants to write a response body, it’s usually “terminal” and should not call next.ServeHTTP() because subsequent handlers can’t reasonably write headers or responses afterwards anyway. Simply return-ing terminates the handler chain and then allows any handlers invoked earlier in the chain to run some code after next.ServeHTTP to do things on the way out (e.g. templates or encode which transform/compress the response body stream).
Thanks, Francis. I hadn’t understood the subtleties around when to call next.ServeHTTP(). I did get that there can be code after the call to next.ServeHTTP() that gets called on the way back up the call stack, but I hadn’t realized there was no abstraction for reordering, between writing to the ResponseWriter and the resulting HTTP response. It makes sense to me, now.
Thanks, too, for your work on this tool and community!
I’m not sure sure what you mean by this, but there is ordering, i.e. the order of the handlers in the config. In the Caddyfile, directives get sorted according to Caddyfile Directives — Caddy Documentation. You can run caddy fmt -p to see your config as JSON to see the actual order after sorting. A request goes top-down through the middleware chain, then back up (depending on how handlers call next.ServeHTTP())
Oh, that I understand. I meant, specifically, I didn’t understand that calling ResponseWriter.Write (or io.Copy) writes directly to a buffer that becomes the http message. When I spoke of reordering, I meant at this level: that subsequent calls to WriteHeader could not add headers (only trailers) as the buffer had already been written.
As I’ve said, I’m not that familiar with go. In languages/tooling I’ve used before, the http message is serialized when the full message is complete, rather than this piecemeal approach, meaning the response parameters can be edited right until it is sent. I presume the approach here is more performant (less intermediate copying?) although caddy is doing tls termination so I guess the response still has to be encrypted and re-packaged before being written to the TCP transmit buffer.
And thanks for pointing out caddy fmt -p — it’s the same ascaddy adapt, without --validate, right?
Yeah, no buffering of the full response body. As soon as it’s being written out, it streams out. Unless you wrap the body struct with something to buffer it, but that can be costly. And for it to be streamed out, the headers need to have been written out first otherwise it’s invalid HTTP (again with asterisks, there’s modes of HTTP that allow chunking etc).
Yeah but encryption is pretty cheap these days with CPU instructions optimizing it depending on the cipher suite.
No, caddy fmt is a separate tool (and different Caddyfile parser… which is a problem I want to solve eventually) which performs some syntax cleanup on the Caddyfile, fixing indentation according to { } nesting.