I am trying to format/transform the logging output of caddy to resemble a specific format.
EDIT: I think I will use a custom parser in syslog-ng to extract the level values and convert them to the PRI values. My question here is probably out of scope. /EDIT
I’m sending these logs to a syslog-ng server which I have limited control over.
My desired format of the log message is:
{ts} caddy {level} {msg}
{ts} - formatted in rfc3339
caddy - hardcoded to resemble the service name
{level} - a numerical level of the severity
{msg} - the whole log message
This setup already works, except for the {level} key and a bit of trouble with the {msg} key.
I’m having trouble with the {level} key, as I don’t know if it’s possible to transform it into a numerical value of the severity. Can these values be turned into numerical severity? (e.g. 3 - error, 4 - warn, …)
I’m using the “format transform” functionality in order to turn the json format into a new output format. Since I don’t know every possible key that can appear in a log message, is there something like a wildcard? Or an option to append the whole json string in the log message?
Converting to a number seems out of scope, the numbers we’d use may not match those you want to have, nor would it match what other users might expect, etc.
Wouldn’t it be easier to make your net writer a script that takes the logs as JSON then transforms them to your liking?
Or you could fork the transform module to emit things exactly the way you want.
Thank you, you make a lot of valid points. I didn’t think about forking and adjusting the transform module. That sounds like an interesting direction to get caddy log messages transformed into RFC5424 syslog format. I gonna try that sometime and see how it goes.
EDIT:
Since I don’t have any experience with golang, I have opted for a different approach. A parser in syslog-ng. Here is a configuration that translates the levels of the caddy log output into used for RFC5424.
# Parser for applying the correct rewrite rule based on the log message
parser p_caddy_levels {
channel {
filter {
message(".*debug.*") or
message(".*info.*") or
message(".*warn.*") or
message(".*error.*") or
message(".*panic.*") or
message(".*fatal.*");
};
rewrite {
# Map Caddy log levels to syslog severity levels
set-severity("7" condition(message(".*debug.*"))); # DEBUG -> Debug
set-severity("6" condition(message(".*info.*"))); # INFO -> Informational
set-severity("4" condition(message(".*warn.*"))); # WARN -> Warning
set-severity("3" condition(message(".*error.*"))); # ERROR -> Error
set-severity("2" condition(message(".*panic.*"))); # PANIC -> Critical
set-severity("1" condition(message(".*fatal.*"))); # FATAL -> Alert
};
};
};
# Filter for Caddy logs
filter f_local_caddy {
program("caddy");
};
# Destination for Caddy logs
destination d_local_caddy {
file(
"/var/log/caddy/caddy_${YEAR}${MONTH}${DAY}.log"
create-dirs(yes)
flags(syslog-protocol)
);
};
# Log path for processing Caddy logs
log {
source(s_all);
filter(f_local_caddy);
parser(p_caddy_levels);
destination(d_local_caddy);
};
Example output:
<14>1 2024-01-09T09:40:13+00:00 hostname caddy - - [meta sequenceId="28"] info done waiting on internal rate limiter
<11>1 2024-01-09T09:40:15+00:00 hostname caddy - - [meta sequenceId="29"] error could not get certificate from issuer
It can already parse json. I guess this whole thing was partly due to inexperience from me. I only used the transform module because I thought I needed caddy to output its own program name in its log messages. But I already overcame this by using an own unix socket (e.g. /var/caddy/var/run/log) and sending all caddy logs to that.
This way, the source is tagged automatically and I can use a syslog-ng rewrite rule to set the program name in that application. Now I don’t need to modify the output of caddy at all (apart from rfc3339 for time but thats supported.)
Sometimes just writing about things or talking with somebody is already a big help. Thanks for helping me here and on github too.
