Log request and response bodies

[Alex_Wells (Alex)]

1. The problem I’m having:

Hey. Currently we’re using Nginx and, through Lua plugin, we’re logging (JSON) request and response bodies of incoming HTTP requests. We’re seeing quite a lot of traffic and care for PII, so:

• PII data is obfuscated with Regex
• only first 1k characters of responses are stored
• only error (>=400 status) responses are stored

As far as I know, requests and responses are buffered in that case. This data is very useful for debugging purposes: whenever frontend sends a malformed request (missing required keys, invalid data types, missing client side validation etc), backend responds with an error (4xx) which we do not track otherwise. If a user reports a problem, we can then trace back user’s steps and see what exactly the user did and what went wrong.

The thing is that Caddy does not allow that; and I’m not seeing a 3rd party solution to use either. As mentioned in a similar topic, a placeholder containing request’s body is available, but cannot be used to log messages; there also would be no option to obfuscate it properly it seems. And response body is just not available as a placeholder at all.

One alternative I’ve considered is logging requests/responses on the application side, but this scatters the access log to different places and makes for a worse developer experience searching through both logs. It also makes it application specific, requiring us to implement it for every service we have, which is not ideal.

The most performant solution would be to implement this on the web server side, but I understand this comes with a performance penalty. Either way, that performance hit under conditions I’ve specified is still many times smaller than doing the same thing on the application level.

There were two posts like this in the past; but authors did not communicate why they needed this feature, which is why I created the third thread

So the two questions are:

• is it possible at all to implement it somewhat efficiently, given my conditions above?
• are there any other solutions to this problem that I’m unaware of?

Thanks

2. Error messages and/or full log output:

PASTE OVER THIS, BETWEEN THE ``` LINES.
Please use the preview pane to ensure it looks nice.

3. Caddy version:

v2.8.4 h1:q3pe0wpBj1OcHFZ3n/1nl4V4bxBrYoSoab7rL9BMYNk=

4. How I installed and ran Caddy:

a. System environment:

Docker image

d. My complete Caddy config:

5. Links to relevant resources:

How do I log both the request length and response length?

1. Caddy version (caddy version): v2.4.5 2. How I run Caddy: I run caddy using the JSON config. I write a Caddyfile then Adapt it and load the config. It’s not containerised. a. System environment: Ubuntu 20.04.3 LTS (GNU/Linux 5.4.0-88-generic x86_64) b. Command: caddy start c. Service/unit/compose file: Not being used. d. My complete Caddyfile or JSON config: { on_demand_tls { ask http://my_server/allowed interval 2m burst …

How can i log the request and response body of each request

1. The problem I’m having: I’m trying to implement logging for my caddy config with Grafana and so far i have been successful. But the logs do not contain request body or response body. 2. Error messages and/or full log output: there is no error, I’m trying to implement something new 3. Caddy version: 2.8.4 4. How I installed and ran Caddy: I use the latest docker image version a. System environment: Docker on Ubuntu b. My complete Caddy config: { admin :2019 log { output std…

[francislavoie]

You could implement it as a plugin, yes. You know your constraints, so it’s best implemented as a plugin rather than being built into Caddy itself.

You can add the content to access logs, see how the log_append handler is implemented, you can copy that logic.

You’d need to implement request/response body wrappers which buffer the amount of data you want then stop buffering and just stream once you reach your size limit, etc. There’s examples of wrappers like responseRecorder in Caddy’s source that you can refer to.