Remote logging and systemd


(Mark) #1

Aye! I’d like to share some shortcuts to remote logging. That is, in this context, sending of log lines to a different machine (»server«) than Caddy (the »daemon« or »service«) runs on. If you’ve made good experiences with other service providers: Do share!

The advantage is, even if your server does become unresponsive you will be able to read at least parts of the logs. And in case of a breach (»crack«, though youngsters call it »hack/hacking«) it’ll be harder for the adversary to go unnoticed.

So, I found these the easiest to setup and operate:

  1. systemd’s own remote logging.
  2. Papertrail — http://help.papertrailapp.com/kb/configuration/configuring-centralized-logging-from-systemd/
  3. Amazon’s CloudWatch. Surprisingly cheap (first month will cost a bit due to setup, though), but searching through logs is a pain.

Utilizing (3) is quite easy with Say Media’s Cloudwatch-Logs. Go through the configuration at AWS as usual (tutorials are too long for a forum post, but plenty to be found), and then use something like this (change/set usernames, paths, etc.):

# /etc/systemd/system/journald-cloudwatch-logs.service
[Unit]
Description=remote logging to AWS CloudWatch
Wants=basic.target network-online.target
After=basic.target network-online.target
AssertPathExists=/var/lib/journald-cloudwatch-logs
ConditionFileNotEmpty=/etc/journald-cloudwatch-logs.conf

[Service]
KillMode=process
; …

Environment=GOMAXPROCS=2
Environment=AWS_ACCESS_KEY_ID=XXXXXXXXXXXXX
Environment=AWS_SECRET_ACCESS_KEY=eeeeeeeeeeeeeeeee

ExecStart=/opt/sbin/journald-cloudwatch-logs /etc/journald-cloudwatch-logs.conf

[Install]
WantedBy=default.target
# /etc/tmpfiles.d/journald-cloudwatch-logs.conf
d /var/lib/journald-cloudwatch-logs 0750 0 0
# /etc/journald-cloudwatch-logs.conf
aws_region = "eu-central-1"
log_group = "staging"
state_file = "/var/lib/journald-cloudwatch-logs/state"

(Dean Taylor) #2

Isn’t there a security risk here in that /etc/systemd/system/journald-cloudwatch-logs.service is likely public readable and contains your AWS_SECRET_ACCESS_KEY?

I’m unsure if systemd .service files are required to be public readable - all of them are doing a cursory check on a local Ubuntu install.