Log backup to s3

Help
1

1. The problem I’m having:

I am trying to export rotated logs to another server (s3 compatible minio server). I cant find any setting on Output module log file module to do so. What is the recommended approach to backing up logs?

2. Caddy version:

v2.7.6

3. How I installed and ran Caddy:

  • Debian/ubuntu installation with cloudflare module
sudo apt install -y debian-keyring debian-archive-keyring apt-transport-https curl
curl -1sLf 'https://dl.cloudsmith.io/public/caddy/stable/gpg.key' | sudo gpg --dearmor -o /usr/share/keyrings/caddy-stable-archive-keyring.gpg
curl -1sLf 'https://dl.cloudsmith.io/public/caddy/stable/debian.deb.txt' | sudo tee /etc/apt/sources.list.d/caddy-stable.list
sudo apt update
sudo apt install caddy

xcaddy build --with github.com/caddy-dns/cloudflare

a. System environment:

  • Ubuntu 22.04.3 LTS
  • arm64

b. Command:

sudo systemctl start caddy

with /etc/caddy/Caddyfile configured.

d. My complete Caddy config:

/etc/caddy/Caddyfile

import /etc/caddy/non_production/*.caddy

/etc/caddy/non_production/log-test.caddy

log.preview.mywebsite.com {
        reverse_proxy :32987
        log access-log {
                output file /var/log/caddy/log-test/access.log {
                        #roll_disabled
                        roll_size 1MiB
                        #roll_uncompressed
                        #roll_local_time
                        roll_keep 5
                        roll_keep_for 720h
                }
                format json
        }
}

docker ps

CONTAINER ID   IMAGE                                                                   COMMAND                  CREATED        STATUS                          PORTS                                                                                                                             NAMES
62f7e851effd   registry.gitlab.com/org/reactjs/myapp/main:latest   "docker-entrypoint.s…"   18 hours ago   Up 18 hours                     0.0.0.0:32987->3000/tcp, :::32987->3000/tcp                                                                                       log-test

5. Rotated logs output.

root@statics:/var/log/caddy/log-test-2# ls -lah
total 136K
drwxr-xr-x 2 caddy caddy 4.0K Jan 31 09:03 .
drwxr-xr-x 4 caddy caddy 4.0K Jan 31 09:00 ..
-rw------- 1 caddy caddy  40K Jan 31 09:03 access-2024-01-31T09-03-42.764.log.gz
-rw------- 1 caddy caddy  86K Jan 31 09:03 access.log

Here, I would like to backup access-2024-01-31T09-03-42.764.log.gz during creation.

2

Our file logging library doesn’t support anything like that.

You should write a cron script that does that. It’s out of scope of Caddy, really.

3

since log rotation is handled by caddy,
Instead of just deleting after 5 logs (setup via roll_keep 5) in above config, It felt like there should also be option to back that up somewhere else…

If this is handled via cron job, I think we should have large number configured in roll_keep and cron job should just back that up to elsewhere and remove the backed up file in current server right?

4

Backup is a lot more complex than simple log rotation. There’s an infinite amount of ways to do it, and every user will need it to work a different way. So it doesn’t make sense for Caddy to dictate how that’s done. We don’t have the bandwidth to support that in addition to everything else.

Sure, how you do that is up to you.

5

Ok, Thank you for your help!

6

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.