Yes, but doing so may loose some data, unless you output both. If we can simply read the full log and simply extract the values interesting to us at the moment, seem it could be valuable. I used the common log format as example, but it’s possible to use any value from the json.
I decided to expand the idea of logparse into something a little more capable. Instead of just having a hardcoded option, you can now customise how you like.
If you wanted a tab separated CSV, and with the unix timestamp as datetime with ms, simply call logparse with those selectors:
> logparse -s "datetime_ms tab client_ip tab proto tab status tab uri" caddy.log
2024-05-26 13:42:52.601 CET 10.10.10.117 HTTP/2.0 200 /gentoo/gentoo-distfiles/distfiles/9b/tl-hrlatex.source-2021.tar.xz
2024-05-26 13:43:50.508 CET 10.10.10.134 HTTP/2.0 200 /gentoo/gentoo-portage/app-emulation/metadata.xml
2024-05-26 13:44:00.654 CET 10.10.10.132 HTTP/2.0 200 /gentoo/gentoo-distfiles/distfiles/b6/libvirt-python-9.9.0.tar.gz
2024-05-26 02:22:43.414 CET 240e:1:1::1234 HTTP/3.0 200 /res/browse.css
I also added a conversion from caddys TLS and cipher suite decimal form to their common names such as TLS 1.3 and TLS_RSA_WITH_AES_128_CBC_SHA256
It’s possible to change the naming scheme and jq selectors/filters in the config file. I suppose you could create a different config for different sources, not just Caddy’s.
Usage: logparse [-c | -C | -s sel] [-F <file>] [-l] [-t <num> [-f]] filename
Options:
-c, --common Apache Common Log Format (default).
-C, --combined Apache Combined Log Format.
-s, --selector Selector list, comma-separated: sel1,sel2,sel3
multiple -s is allowed; lists are concatenated.
-F, --config-file Use a configuration file.
-t, --tail <num> Output the last NUM lines from log file (default: 15).
-f, --follow Continuously monitor log file for new entries.
-l, --list-selectors List all available selectors and exit.
-h, --help Show this help message and exit.
More examples, details and source code can be found on my wiki logparse: A JSON logparser for Caddy webserver logs.
Source code is available at Forza/logparse: Logparse is a shell script that parses Caddy's JSON log files and outputs them as formatted text. - logparse - Gitea: Git with a cup of tea
I hope it will be useful to someone 
Update: 2025-11-01 - logparse 0.2.0
I’ve updated logparse with more fields from the Caddy logs; for exampl Content-Encoding, logger, various orher request and reaponse headers. req_headers and rh_headers return a concatenated list of all request and response headers.
There is also a –list-selectors option that shows all known fields that can be filtered on. You can also use comma separated selectors with -s.
I’d be very happy to add more fields and options or other feature requests.
3 Likes
I’ve found this is useful for when you’re only logging to standard output:
docker compose logs web | tail -1 | cut -d"|" -f2- | jq
3 Likes
If you’re using journald as a log store; you can prefix most of the jq examples here as follows: journalctl -f -u caddy -o cat | jq <etc>
I used jq, yq before. But now moved to miller. It is more easier to use with a regular programming language like syntax..
Examples
❯ sudo cat /var/log/caddy/abort.log | \
∙ mlr --l2p \
∙ put '$ts = sec2localtime($ts);$ua = truncate($request["headers"]["User-Agent"][1],50);' \
∙ then filter '$ua =~ "[bB]ot"' \
∙ then flatten then cut -of ts,ua,request.uri then head
ts ua request.uri
2024-11-28 12:51:40 Mozilla/5.0 (compatible; Googlebot/2.1; +http://ww /
2024-11-28 13:34:52 Mozilla/5.0 (compatible; DotBot/1.2; +https://open /robots.txt
2024-11-28 13:41:00 Snap URL Preview Service; bot; snapchat; https://d /
2024-11-28 14:23:42 Mozilla/5.0 (compatible; DotBot/1.2; +https://open /robots.txt
2024-11-28 15:13:23 Mozilla/5.0 (compatible; DotBot/1.2; +https://open /robots.txt
2024-11-28 16:03:06 Mozilla/5.0 (compatible; DotBot/1.2; +https://open /robots.txt
2024-11-28 16:13:56 Mozilla/5.0 (compatible; Googlebot/2.1; +http://ww /
2024-11-28 16:52:25 Mozilla/5.0 (compatible; DotBot/1.2; +https://open /robots.txt
2024-11-28 17:01:19 Mozilla/5.0 (compatible; Googlebot/2.1; +http://ww /robots.txt
2024-11-28 17:44:09 Mozilla/5.0 (compatible; DotBot/1.2; +https://open /robots.txt
To see the longer URIs
❯ sudo cat /var/log/caddy/abort.log | \
∙ mlr --l2p \
∙ put '$ts = sec2localtime($ts); $urilen = strlen($request["uri"]); $uriprefix = truncate($request["uri"],50);' \
∙ then sort -nr urilen then cut -of ts,urilen,uriprefix then head
ts urilen uriprefix
2024-12-16 08:43:46 515 /?s=/index/%5Cthink%5Capp/invokefunction&function=
2024-12-16 08:43:46 515 /?s=/index/%5Cthink%5Capp/invokefunction&function=
2025-01-31 16:38:22 468 /cgi-bin/luci/;stok=/locale?form=country&operation
2024-12-21 01:32:24 396 /$((a=${IFS};b=/bin/;c="sudo${a}${b}";${c}cp${a}${
2024-12-21 01:36:06 396 /$((a=${IFS};b=/bin/;c="sudo${a}${b}";${c}cp${a}${
2024-12-21 01:44:48 396 /$((a=${IFS};b=/bin/;c="sudo${a}${b}";${c}cp${a}${
2024-12-21 03:30:11 396 /$((a=${IFS};b=/bin/;c="sudo${a}${b}";${c}cp${a}${
2024-12-21 06:22:20 396 /$((a=${IFS};b=/bin/;c="sudo${a}${b}";${c}cp${a}${
2024-12-21 00:15:10 388 /$((a=${IFS};b=/bin/;c="sudo${a}${b}";${c}cp${a}${
2024-12-21 00:18:20 388 /$((a=${IFS};b=/bin/;c="sudo${a}${b}";${c}cp${a}${
2 Likes