Logging transformation/formatting help

1. The problem I’m having:

I am trying to format/transform the logging output of caddy to resemble a specific format.

EDIT: I think I will use a custom parser in syslog-ng to extract the level values and convert them to the PRI values. My question here is probably out of scope. /EDIT

I’m sending these logs to a syslog-ng server which I have limited control over.

My desired format of the log message is:

{ts} caddy {level} {msg}

  • {ts} - formatted in rfc3339
  • caddy - hardcoded to resemble the service name
  • {level} - a numerical level of the severity
  • {msg} - the whole log message

This setup already works, except for the {level} key and a bit of trouble with the {msg} key.

  1. I’m having trouble with the {level} key, as I don’t know if it’s possible to transform it into a numerical value of the severity. Can these values be turned into numerical severity? (e.g. 3 - error, 4 - warn, …)

  2. I’m using the “format transform” functionality in order to turn the json format into a new output format. Since I don’t know every possible key that can appear in a log message, is there something like a wildcard? Or an option to append the whole json string in the log message?

Thank you a lot for any hints. :slight_smile:

2. Error messages and/or full log output:

I don’t have any errors.

3. Caddy version:

v2.7.6 h1:w0NymbG2m9PcvKWsrXO6EEkY9Ru4FJK8uQbYcev1p3A=

4. How I installed and ran Caddy:

a. System environment:


b. Command:

service caddy onestart

c. Service/unit/compose file:


# PROVIDE: caddy
# KEYWORD: shutdown

. /etc/rc.subr


load_rc_config $name

: ${caddy_enable:=NO}
: ${caddy_config_file:="/usr/local/etc/caddy/Caddyfile"}

command_args="-P ${pidfile} -r /usr/local/bin/caddy run --config ${caddy_config_file} >> /var/log/caddy/caddy.log 2>&1"

export XDG_CONFIG_HOME="/usr/local/etc/caddy/.config"

run_rc_command "$1"

d. My complete Caddy config:

There’s only the global block and no domains since its the only thing I’m testing this with right now.

        storage file_system {
                root /usr/local/etc/caddy
        log {
                output net unixgram//var/run/log {
                format transform {ts} caddy {level} {msg} {
                        time_format rfc3339

5. Links to relevant resources:

Caddy Encoder Transform
Caddy Logs Level
RFC5424 Severity

1 Like

We only support these level formats currently:

  • level_format The format for levels. May be one of:
    • lower Lowercase; this is the default.
    • upper Uppercase.
    • color Uppercase, with console colors.

See log (Caddyfile directive) — Caddy Documentation

Converting to a number seems out of scope, the numbers we’d use may not match those you want to have, nor would it match what other users might expect, etc.

Wouldn’t it be easier to make your net writer a script that takes the logs as JSON then transforms them to your liking?

Or you could fork the transform module to emit things exactly the way you want.

1 Like

Thank you, you make a lot of valid points. I didn’t think about forking and adjusting the transform module. That sounds like an interesting direction to get caddy log messages transformed into RFC5424 syslog format. I gonna try that sometime and see how it goes. :slight_smile:

Since I don’t have any experience with golang, I have opted for a different approach. A parser in syslog-ng. Here is a configuration that translates the levels of the caddy log output into used for RFC5424.

# Parser for applying the correct rewrite rule based on the log message
parser p_caddy_levels {
    channel {
        filter {
            message(".*debug.*") or
            message(".*info.*") or
            message(".*warn.*") or
            message(".*error.*") or
            message(".*panic.*") or
        rewrite {
            # Map Caddy log levels to syslog severity levels
            set-severity("7" condition(message(".*debug.*")));  # DEBUG -> Debug
            set-severity("6" condition(message(".*info.*")));   # INFO -> Informational
            set-severity("4" condition(message(".*warn.*")));   # WARN -> Warning
            set-severity("3" condition(message(".*error.*")));  # ERROR -> Error
            set-severity("2" condition(message(".*panic.*")));  # PANIC -> Critical
            set-severity("1" condition(message(".*fatal.*")));  # FATAL -> Alert

# Filter for Caddy logs
filter f_local_caddy {

# Destination for Caddy logs
destination d_local_caddy {

# Log path for processing Caddy logs
log {

Example output:

<14>1 2024-01-09T09:40:13+00:00 hostname caddy - - [meta sequenceId="28"] info done waiting on internal rate limiter
<11>1 2024-01-09T09:40:15+00:00 hostname caddy - - [meta sequenceId="29"] error could not get certificate from issuer
1 Like

Pretty cool solution :+1:

Maybe you should request that syslog-ng adds structured JSON log support?

See How Logging Works — Caddy Documentation for an argument for why JSON logs are better.

1 Like

It can already parse json. I guess this whole thing was partly due to inexperience from me. I only used the transform module because I thought I needed caddy to output its own program name in its log messages. But I already overcame this by using an own unix socket (e.g. /var/caddy/var/run/log) and sending all caddy logs to that.

This way, the source is tagged automatically and I can use a syslog-ng rewrite rule to set the program name in that application. Now I don’t need to modify the output of caddy at all (apart from rfc3339 for time but thats supported.)

Sometimes just writing about things or talking with somebody is already a big help. Thanks for helping me here and on github too. :slight_smile:

1 Like

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.