Making Caddy logs more readable

Hello

I open this thread to discuss strategies to use the JSON logs created by Caddy 2.

Since I’m mostly using the command tail to consume the logs directly in my terminal window, the default presentation is not great at all :

I found a really simple solution though. I installed jq and now I can output a more readable log just by adding jq after the tail, like this :

tail -f /var/log/caddy/voiretmanger.fr.access.log | jq

This is what I get (I’m using ZSH and oh-my-zsh, which explains the colors used) :

Another benefit of jq is that you can filter the output to display only one information. For instance, if you only want to see the requests uri, you can do :

tail -f /var/log/caddy/voiretmanger.fr.access.log | jq .request.uri

And this is what you will get :

You could use that filter to display the common log, still available in the default log outputted by Caddy 2 :

tail -f /var/log/caddy/voiretmanger.fr.access.log | jq .common_log

You can even display multiple informations at once, using a comma between each element, like so :

tail -f /var/log/caddy/voiretmanger.fr.access.log | jq '.request.uri, .request.remote_addr'

The jq documentation contains more example of filtering available : https://stedolan.github.io/jq/manual/#Basicfilters

It’s pretty basic, but it’s usually enough for my limited needs. If you have a different system, I would be interested to see what you do !

6 Likes

This is a fantastic writeup, @nicolinux!

Would it be alright with you if I made this a wiki so that others can find it easier? I think there’s some really good info here and over time the rest of the community can contribute back to it.

No problem at all, I’m glad it can be useful. :slight_smile:

Alright, this is a really neat little tool and super useful! Thanks for pointing it out, makes it really easy to inspect logs without setting up more advanced log parsing or mangling Caddy’s default comprehensive structured logging.

Thank to this thread I could reformat the caddy json output to something that I am familiar with and that I can easily further process with awk piped into iptables (ipset) for banning purposes.

jq -j '.ts |= strftime("%Y-%m-%d %H:%M:%S") | .request.remote_addr |= .[:-6]  | .ts, "|", .request.remote_addr,"|", .request.uri,"|", .request.method,"|", .request.proto,"|", .status,"|", .request.headers."User-Agent"[]+"\n"' /var/log/caddy/access.log

This jq snippet outputs this

2020-04-22 14:23:05|192.168.0.92|/info.php|GET|HTTP/1.1|200|curl/7.64.0
2020-04-22 14:23:51|192.168.0.92|/info.php|GET|HTTP/1.1|200|Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/80.0.3987.162 Safari/537.36
2020-04-22 14:23:51|192.168.0.92|/favicon.ico|GET|HTTP/1.1|404|Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/80.0.3987.162 Safari/537.36
2020-04-22 14:28:35|192.168.0.92|/info.php|GET|HTTP/1.1|200|Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/80.0.3987.162 Safari/537.36
2020-04-22 14:28:50|192.168.0.92|/|GET|HTTP/1.1|200|Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/80.0.3987.162 Safari/537.36
2020-04-22 14:28:50|192.168.0.92|/favicon.ico|GET|HTTP/1.1|404|Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/80.0.3987.162 Safari/537.36
2020-04-22 14:29:30|192.168.0.92|/info.php|GET|HTTP/1.1|200|Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/80.0.3987.162 Safari/537.36
2020-04-22 14:32:38|192.168.0.92|/info.php|GET|HTTP/1.1|200|Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/80.0.3987.162 Safari/537.36

I am impressed by the power of jq although its syntax is a bit cumbersome.

6 Likes

@JeanLucLacroix are you using fail2ban with your reformatted caddy logs?

No, I use my own awk script that scans the logs and bans ip’s that keep on insisting to connect more that a preset number of times per minutes. Infringing ip’s are added to an ipset hooked to iptables. ipset is handy as it doesn’t require iptables rules to be reloaded when changed.

Sorry for the late answer.

Thanks for getting back to me.

This topic was automatically closed after 180 days. New replies are no longer allowed.