How to log the HTTP body of the request and the response?

Help
1

I needed today to log the HTTP body of the request and the response and as I found out, Caddy does not support this because they are handled as data streams. It is possible to refer to them like this:

log_append request_body {http.request.body} # won't work

The configuration will be loaded successfully, but the body will not be logged.

I did not find a way with a vanilla Caddy, BUT there is a great tool that can help: mitmproxy. In my case, Caddy is running as a reverse proxy for a server listening on port 8080. So I changed the server configuration to listen on port 8081 and I started mitmproxy this way:

mitmproxy -p 8080 --mode reverse:http://localhost:8081/

It is more effort to set up than logging, and it causes an interruption of the service, but the upside is that there is a nice user interface to navigate between the requests and all the details.

image
image1103Ă—370 22.8 KB

https://www.chrissearle.org/_ipx/_/images/posts/2020/12/mitm-curl-response.png (sorry, I do not have the right to embed more than one image as a new user).

You can even replay the requests, with or without modification…

There are even more features. Here is the documentation:

2

Yes, Caddy does allow you to do that. What version are you using? We even have an example here log_append (Caddyfile directive) — Caddy Documentation

For debugging purposes only (not for use in production), the handler has specialized handling when the value is one of these placeholders: {http.request.body}, {http.request.body_base64}, {http.response.body}, or {http.response.body_base64}. If a request body placeholder is used, then “early” mode is implicitly enabled, and the request body will be buffered. If a response body placeholder is used, response buffering is enabled to capture the response body and the field is added to the log “late”, as the response is being written.

For debugging purposes, add the request and response bodies to the logs (not for use in production, as this harms performance and makes the logs very noisy). If you expect the bodies to be binary data with non-printable characters, you can use the base64 variants of the placeholders instead (e.g. {http.request.body_base64} and {http.response.body_base64}), which will be easier to copy and inspect:

example.com {
	log
	log_append req_body {http.request.body}
	log_append resp_body {http.response.body}

	reverse_proxy localhost:9000
}
2 Likes