Is it possible to modify the JSON access log format?

[MrAlex (Alex)]

1. My Caddy version (caddy version):

2.0rc3

2. How I run Caddy:

a. System environment:

Ubuntu 20.04 LTS, installed via apt, enabled by default with systemd.

b. My complete Caddyfile or JSON config:

FQDN {

        # Set this path to your site's directory.
        root * /usr/share/caddy

        # Enable the static file server.
        file_server

        log {
                format json
                output net unix//var/cache/caddy.socket
        }
}

3. The problem I’m having:

I’d like to be able to feed the output from the access log in a custom JSON format via syslog to the matomo live log importer, with the expected json output of:

 '{"ip": "$remote_addr",'
 '"host": "$host",'
 '"path": "$request_uri",'
 '"status": "$status",'
 '"referrer": "$http_referer",'
 '"user_agent": "$http_user_agent",'
 '"length": $bytes_sent,'
 '"generation_time_milli": $request_time,'
 '"date": "$time_iso8601"}';

4. What I already tried:

Output is set to JSON and sent via net unix socket to rsyslog; /etc/rsyslog.d/10-matomo.conf:

$AddUnixListenSocket /var/cache/caddy.socket

$template matomo,"%msg%\n"

if $syslogfacility-text == 'local0' then ^/usr/local/matomo/matomo.sh;matomo

/usr/local/matomo/matomo.sh:

#!/bin/sh

echo "${@}" | /usr/local/matomo/import_logs.py \
 --url=https://localhost/matomo/ --token-auth=<SECRET> \
 --enable-http-errors --enable-http-redirects --enable-static --enable-bots \
 --idsite=1 --recorders=4 --log-format-name=nginx_json -

Of course, there is no data imported as the JSON format is not as expected. Does anyone know if this is possible? I tried getting my head around mmjsonparse to parse the incoming JSON and output the JSON as desired, but couldn’t figure out how to.

Cheers!

[francislavoie]

I think you can use the single_field format and set the field to your custom json string using request placeholders (or their equivalent Caddyfile shorthands)

[MrAlex (Alex)]

I’ve tried format single_field {host}{hostport}{method}{path} just to see what it outputs, but it doesn’t seem to be any different to format json

{
    "level": "info",
    "ts": 1588615390.2087238,
    "logger": "http.log.access.log0",
    "msg": "handled request",
    "request": {
        "method": "GET",
        "uri": "/",
        "proto": "HTTP/2.0",
        "remote_addr": "x.x.x.x:51780",
        "host": "x.x.x",
        "headers": {
            "Accept-Encoding": ["gzip, deflate, br"],
            "If-None-Match": ["\"q9dj779fm\""],
            "Te": ["trailers"],
            "User-Agent": ["Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:68.0) Gecko/20100101 Firefox/68.0"],
            "Accept": ["text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8"],
            "Accept-Language": ["en-US,en-US;q=0.7,en;q=0.3"],
            "Dnt": ["1"],
            "Upgrade-Insecure-Requests": ["1"],
            "If-Modified-Since": ["Sun, 26 Apr 2020 02:35:31 GMT"],
            "Cache-Control": ["max-age=0"]
        },
        "tls": {
            "resumed": false,
            "version": 772,
            "ciphersuite": 4865,
            "proto": "h2",
            "proto_mutual": true,
            "server_name": "x.x.x"
        }
    },
    "common_log": "x.x.x.x - - [04/May/2020:20:03:10 +0200] \"GET / HTTP/2.0\" 304 0",
    "latency": 0.004685767,
    "size": 0,
    "status": 304,
    "resp_headers": {
        "Server": ["Caddy"],
        "Etag": ["\"q9dj779fm\""]
    }
}

[matt (Matt Holt)]

The single_field encoder just uses an existing field from the log entry as the value for the log entry.

Caddy’s log encoders are modular – if one doesn’t suit your needs, it can be swapped out for another one.

You might find this helpful: Making Caddy logs more readable - #5 by JeanLucLacroix

[MrAlex (Alex)]

Thanks, that was a great help. Also congratulations on the release!