Include fields in log instead of delete

[tas (Michal Maciejewski)]

hi,

Is it possible to format logs by specifying fields to be included, rather then marking fields for deletion?

I just want to include a few fields.
Is it because of Go’s regex implementation?

This Caddyfile is getting out of hand:

	log {
		format filter {
			wrap json 
			fields {
				request>method delete
				request>proto delete
				request>headers>Sec-Ch-Ua delete
				request>headers>Upgrade-Insecure-Requests delete
				request>headers>Accept-Encoding delete
				request>headers>Cache-Control delete
				request>headers>If-None-Match delete
				request>headers>If-Modified-Since delete
				request>headers>Sec-Ch-Ua-Platform delete
				request>headers>Accept delete
				request>headers>Accept-Language delete
				request>headers>Referer delete
				request>headers>Cookie delete
				request>headers>Sec-Ch-Ua-Mobile delete
				request>headers>Sec-Fetch-Site delete
				request>headers>Sec-Fetch-User delete
				request>headers>Authorization delete
				request>headers>Sec-Fetch-Mode delete
				request>headers>Sec-Fetch-Dest delete
				request>tls delete
				user_id delete
				duration delete
				size delete
				resp_headers>Cache-Status delete
				resp_headers>Set-Cookie delete
				resp_headers>Djdt-Store-Id delete
				resp_headers>Referrer-Policy delete
				resp_headers>Server-Timing delete
				resp_headers>X-Content-Type-Options delete
				resp_headers>Vary delete
				resp_headers>Content-Length delete
				resp_headers>Content-Type delete
				resp_headers>Last-Modified delete
				resp_headers>X-Frame-Options delete
				resp_headers>Content-Language delete
				resp_headers>Cross-Origin-Opener-Policy delete
				resp_headers>Etag delete
				resp_headers>Access-Control-Allow-Origin delete
				resp_headers>Accept-Ranges delete
				resp_headers>Server delete
				resp_headers>Alt-Svc delete
				resp_headers>X-Country-Code delete
				resp_headers>Cache-Control delete
			}
		}
	}

thanks!

[francislavoie]

We’re not using regexp for log filters. We use GitHub - uber-go/zap: Blazing fast, structured, leveled logging in Go. for logging; the logs are nested Go structures, and each log field is a zapcore.Field which describes that specific field in the log item.

The delete filter just marks the field to be skipped from being written out.

The problem is that filters can’t apply to “all except some” fields because of how filters are set up. Filters need to apply to a specific field because we have to lookup a configured filter when encoding a specific field and this should run in O(1) time to avoid performance issues related to logging.

This feature was asked for previously here, but we couldn’t find a reasonable solution:

[FEATURE][v2]: Provide a new "allow_list" to the log filter

You can probably use the transform encoder plugin as mentioned in a comment in there.

Or you could just log everything as JSON normally, then immediately process the JSON with some external tool to only keep the parts you want (with jq for example).