Remote logging and systemd

Aye! I’d like to share some shortcuts to remote logging. That is, in this context, sending of log lines to a different machine (»server«) than Caddy (the »daemon« or »service«) runs on. If you’ve made good experiences with other service providers: Do share!

The advantage is, even if your server does become unresponsive you will be able to read at least parts of the logs. And in case of a breach (»crack«, though youngsters call it »hack/hacking«) it’ll be harder for the adversary to go unnoticed.

So, I found these the easiest to setup and operate:

  1. systemd’s own remote logging.
  2. Papertrail — systemd - Papertrail
  3. Amazon’s CloudWatch. Surprisingly cheap (first month will cost a bit due to setup, though), but searching through logs is a pain.

Utilizing (3) is quite easy with Say Media’s Cloudwatch-Logs. Go through the configuration at AWS as usual (tutorials are too long for a forum post, but plenty to be found), and then use something like this (change/set usernames, paths, etc.):

# /etc/systemd/system/journald-cloudwatch-logs.service
Description=remote logging to AWS CloudWatch

; …


ExecStart=/opt/sbin/journald-cloudwatch-logs /etc/journald-cloudwatch-logs.conf

# /etc/tmpfiles.d/journald-cloudwatch-logs.conf
d /var/lib/journald-cloudwatch-logs 0750 0 0
# /etc/journald-cloudwatch-logs.conf
aws_region = "eu-central-1"
log_group = "staging"
state_file = "/var/lib/journald-cloudwatch-logs/state"

Isn’t there a security risk here in that /etc/systemd/system/journald-cloudwatch-logs.service is likely public readable and contains your AWS_SECRET_ACCESS_KEY?

I’m unsure if systemd .service files are required to be public readable - all of them are doing a cursory check on a local Ubuntu install.