Some log lines don't reach my encoder plugin

I’m working on a logging encoder plugin, that makes it easier to use Caddy if Elastic Stack Filebeat is used to collect and ship logs.

https://www.elastic.co/guide/en/ecs/current/index.html

{"level":"info","ts":1645912920.7716858,"msg":"using provided configuration","config_file":"/etc/caddy/Caddyfile","config_adapter":""}

I can’t get my encoder to “encode” this particular initial message, but any other log lines gets “touched” by my plugin.

The problem is that my filebeat collects and ships this message, and wrecks all sorts of havoc in my elastic stack setup.
I could create an elasticsearch ingestion pipeline supporting caddys json structure, but I would much rather like Caddy to print ECS compatible json lines.

Here is the plugin
https://github.com/firecow/caddy-elastic-encoder

Here is how the logs looks like, except the one in question.

{"log.level":"info","@timestamp":"2022-02-27T13:15:51.055Z","log.logger":"admin","message":"admin endpoint started","address":"tcp/localhost:2019","enforce_origin":false,"origins":["localhost:2019","[::1]:2019","127.0.0.1:2019"]}
{"log.level":"info","@timestamp":"2022-02-27T13:15:51.055Z","message":"autosaved config (load with --resume flag)","file":"/tmp/caddy/autosave.json"}
{"log.level":"info","@timestamp":"2022-02-27T13:15:51.055Z","message":"serving initial configuration"}

Anyone got any ideas on how to suppress this initial log message, that doesn’t get encoded by plugins, or how i get my plugin to catch em?

The reason that message isn’t encoded is pretty simple: the log message “using provided configuration” is emitted before the config file is loaded and applied, thus any custom log encoder hasn’t been configured yet.

Not all log messages will be structured (due to dependencies, API limitations, etc.), even if that one particular message is skipped or encoded as desired; we recommend that log consumers be graceful in how they handle input they don’t recognize (most I think will just store such messages as an opaque string – less useful, but still recorded).

Roger, that, I’ll get by.

Perhaps we can make it possible to configure log format and log level from CLI/Environment variables in the future.

caddy run --config Caddyfile --log-format elastic --log-level warn

or something like that.

Thanks for your swift response :smiley:

1 Like

Alternatively, you could configure a different log writer, like write the logs to a specific file, and read those logs only. Any logs that happen to stdout/stderr before the config is loaded will never get written there (because the config to change where it emits logs wasn’t loaded yet)

1 Like

This topic was automatically closed after 30 days. New replies are no longer allowed.