Another benefit of jq is that you can filter the output to display only one information. For instance, if you only want to see the requests uri, you can do :
Would it be alright with you if I made this a wiki so that others can find it easier? I think there’s some really good info here and over time the rest of the community can contribute back to it.
Alright, this is a really neat little tool and super useful! Thanks for pointing it out, makes it really easy to inspect logs without setting up more advanced log parsing or mangling Caddy’s default comprehensive structured logging.
Thank to this thread I could reformat the caddy json output to something that I am familiar with and that I can easily further process with awk piped into iptables (ipset) for banning purposes.
No, I use my own awk script that scans the logs and bans ip’s that keep on insisting to connect more that a preset number of times per minutes. Infringing ip’s are added to an ipset hooked to iptables. ipset is handy as it doesn’t require iptables rules to be reloaded when changed.
Thank you for writing this post very useful if you want to navigate caddy logs via cli.
I wanted to present an alternative method involving grafana, loki (data source for logs) and promtail (loki collector). After installing those in your system, you need to update the loki config to scape journald by adding this to the promtail config (/opt/promtail/promtail-local-config.yaml in my system):
After that, you can go to grafana and explore the loki source to create graphs for the metrics you are interested on.
If anyone knows of a grafana caddy dashboard that we can import please let us know.
I found this one by user mgsh in the grafana dashboards website. It works just fine right after importing and it provides useful info. Definitely a good starting point. Be aware it uses a prometheus data source not loki.
Thanks for making this article, it helped me as well to figure out how to make the logs more readable.
I ended up making some aliases to fetch the logs and update things on a caddy VM I use with systemd and thought, why not share it as well if someone else might find it useful too?
# caddy.service
#
# For using Caddy with a config file.
#
# Make sure the ExecStart and ExecReload commands are correct
# for your installation.
#
# See https://caddyserver.com/docs/install for instructions.
#
# WARNING: This service does not use the --resume flag, so if you
# use the API to make changes, they will be overwritten by the
# Caddyfile next time the service is restarted. If you intend to
# use Caddy's API to configure it, add the --resume flag to the
# `caddy run` command or use the caddy-api.service file instead.
[Unit]
Description=Caddy
Documentation=https://caddyserver.com/docs/
After=network.target network-online.target
Requires=network-online.target
[Service]
Type=notify
User=caddy
Group=caddy
ExecStart=/usr/bin/caddy run --environ --config /etc/caddy/Caddyfile
ExecReload=/usr/bin/caddy reload --config /etc/caddy/Caddyfile
TimeoutStopSec=5s
LimitNOFILE=1048576
LimitNPROC=512
PrivateTmp=true
ProtectSystem=full
AmbientCapabilities=CAP_NET_BIND_SERVICE
[Install]
WantedBy=multi-user.target
We are also using join(" ") to make jq to output one log entry per one line of stdout.
Output looks like
"200 (hidden) DE 79007d973dd4c2be-VIE /"
"200 (hidden) DE 79007d972d8ac27a-VIE /"
"200 (hidden) DE 79007d973c3dc287-VIE /"
"200 (hidden) DE 79007d9729b2c240-VIE /"
"200 (hidden) DE 79007d9788f83244-VIE /state"
While the json output is comprehensive, it is rather combersome to not have a formatted output. Are there any good (free/open source) log file parser and monitor tools?
I do agree with Caddy authors to use the full json format, as it contains all relevant data. However, it is also not super easy to parse with human eyes .
I looked at jq, and it’s a little alien to me, but here is an attempt to make a json parser that output in Common Log format or in Combined Log format.
Perhaps we can improve on it together? I don’t particularly like the timestamp after the client_ip or user_id, but it should easy be able to add additional custom log formats.
logparse.sh
#!/bin/bash
###
### Caddy webserver JSON log parser
###
show_help() {
cat <<END
Usage: $0 [-c|--common] [-C|--combined] [-f|--files] [-h|--help] filename
Options:
-c, --common Apache Common Log Format (default)
-C, --combined Apache Combined Log Format
-f, --files Custom Log Format, focused on file transfers
-h, --help Show this help message and exit
END
}
# Function to output logs in common log format
output_common_log_format() {
jq -r '
. as $log |
$log.request.client_ip as $h |
"-" as $l |
($log.user_id // "-") as $u |
($log.ts | tostring | split(".") | .[1][:3]) as $ms |
($log.ts | floor | todateiso8601 | sub("T"; " ") | sub("Z"; "." + $ms + " +0000")) as $t |
"\($log.request.method) \($log.request.uri) \($log.request.proto)" as $r |
$log.status as $s |
($log.size // "-") as $b |
"\($h) \($l) \($u)[\($t)] \"\($r)\" \($s) \($b)"
' "${1}"
}
# Function to output logs in combined log format
output_combined_log_format() {
jq -r '
. as $log |
$log.request.client_ip as $h |
"-" as $l |
($log.user_id // "-") as $u |
($log.ts | tostring | split(".") | .[1][:3]) as $ms |
($log.ts | floor | todateiso8601 | sub("T"; " ") | sub("Z"; "." + $ms + " +0000")) as $t |
"\($log.request.method) \($log.request.uri) \($log.request.proto)" as $r |
$log.status as $s |
($log.size // "-") as $b |
($log.request.headers.Referer[0] // "-") as $referrer |
($log.request.headers["User-Agent"][0] // "-") as $user_agent |
"\($h) \($l) \($u)[\($t)] \"\($r)\" \($s) \($b) \"\($referrer)\" \"\($user_agent)\""
' "${1}"
}
output_file_log_format() {
jq -r '
. as $log |
$log.request.client_ip as $h |
"-" as $l |
($log.user_id // "-") as $u |
($log.ts | tostring | split(".") | .[1][:3]) as $ms |
($log.ts | floor | todateiso8601 | sub("T"; " ") | sub("Z"; "." + $ms + " +0000")) as $t |
"\($log.request.method) \($log.request.uri) \($log.request.proto)" as $r |
$log.status as $s |
($log.size // "-") as $b |
($log.request.headers.Referer[0] // "-") as $referrer |
($log.request.headers["User-Agent"][0] // "-") as $user_agent |
"\($h) \($l) \($u)[\($t)] \"\($r)\" \($s) \($b) \"\($referrer)\" \"\($user_agent)\""
' "${1}"
}
# Function to list served files
list_served_files() {
jq -r '
.request.client_ip as $client_ip |
(.ts | floor | todateiso8601 | sub("T"; " ") | sub("Z"; " +0000")) as $datetime |
.status as $status_code |
(.size // "-") as $size |
.request.uri as $filename |
"\($client_ip) [\($datetime)] \($status_code) \($size) \"\($filename)\""
' "${1}"
}
# Default to common log format
format="common"
# Parse command line options
while [[ "$#" -gt 0 ]]; do
case $1 in
-c|--common)
format="common"
shift
;;
-C|--combined)
format="combined"
shift
;;
-f|--files)
format="files"
shift
;;
-h|--help)
show_help
exit 0
;;
*)
input="${1}"
shift
;;
esac
done
# Check if a filename is provided
if [ -z "${input}" ]; then
echo -e "Error: No input file provided. \n"
show_help
exit 1
fi
# Output the logs in the chosen format
if [ "$format" = "common" ]; then
output_common_log_format "${input}"
elif [ "$format" = "files" ]; then
list_served_files "${input}"
else
output_combined_log_format "${input}"
fi
# Apache Commong Log format:
# "%h %l %u %t \"%r\" %>s %b"
# Apache Combined LogFormat:
# "%h %l %u %t \"%r\" %>s %b \"%{Referer}i\" \"%{User-agent}i\""
# %h is the remote host (client IP)
# %l is the remote logname (not used, so we'll use -)
# %u is the authenticated user
# %t is the time the request was received
# %r is the request line from the client ("method uri proto")
# %>s is the status code
# %b is the size of the object returned to the client
.