Log Configuration in Docker / "docker log" behavior?

Help
1

1. Caddy version (caddy version):

v2.4.3 h1:Y1FaV2N4WO3rBqxSYA8UZsZTQdN+PwcoOcAiZTM8C0I=

2. How I run Caddy:

Via slothcroissant/caddy-cloudflaredns Docker: Docker Hub

a. System environment:

Ubuntu 20.04 Server running Docker in ESXi 6.7

b. Command:

See Docker Compose file below.

c. Service/unit/compose file:

version: "2"

services:
  app:
    image: slothcroissant/caddy-cloudflaredns:dev
    container_name: caddy
    environment:
      - CLOUDFLARE_API_TOKEN=[redacted]
      - ACME_AGREE=true
      - CLOUDFLARE_EMAIL=[redacted]
    volumes:
      - /mnt/zfs/docker/caddy/config:/config
      - /mnt/zfs/docker/caddy/data:/data
      - /mnt/zfs/docker/caddy/Caddyfile:/etc/caddy/Caddyfile
    restart: always
    ports:
      - 80:80
      - 443:443
    networks:
      - network-bridge

networks:
  network-bridge:
    external:
      name: network-bridge

d. My complete Caddyfile or JSON config:

(snippet) {
	header {
		Strict-Transport-Security "max-age=31536000; includeSubdomains"
		X-XSS-Protection "1; mode=block"
		X-Frame-Options "DENY"
		-server
	}
	tls ryan@ryanb.tv {
		dns cloudflare {env.CLOUDFLARE_API_TOKEN}
	}
	log {
		format single_field common_log
		output file /data/logs/caddy.log
	}
}

something.url.com {
	reverse_proxy dockerHost.lan:12345
	import snippet
}

3. The problem I’m having:

My issue stems from (I think) docker logs filling up my container. After a couple weeks of decently heavy logging from Caddy, my docker container was sitting at 12GB+. By deleting it and recreating it, that 12GB+ went away and started fresh at zero.

I’m looking to get info about how to best do logging when running on docker. I’d like to use the log directive to pump logs to a mounted volume, and rotate the logs as needed to prevent the runaway space usage, but I don’t think that the log directive impacts what happens in Docker. For example, above I have specified format single_field common_log, yet my docker logs output shows regular logging in json format.

Can we turn off the HTTP console logging to docker logs? It’d be useful to keep that for application-level logging (renewing LetsEncrypt certs, etc) but I’d like to offload my HTTP logging from there.

4. Error messages and/or full log output:

5. What I already tried:

See config.

6. Links to relevant resources:

Looks like this (or something similar) has come up before: Logs configuration doesn't work in Docker + Caddy2

2

Access logs are off by default.

The log directive turns them on, and if you don’t specify output then it goes to stdout (which goes into what you see in docker logs).

If you do specify output then it should not appear in stdout.

So, I don’t understand what your issue is. What kinds of logs are you actually seeing that take up a lot of room?

You don’t need this anymore btw, this was only a requirement for Caddy v1. It has no effect for Caddy v2.

I recommend making a separate volume for logs, such as /logs.

Caddy can roll/rotate the logs itself:

1 Like
3

I think it’s the logs that would normally show in the docker log output. These are taking up space over days/weeks, etc. Yesterday I got an alert that my docker host was nearing capacity. When I looked, /var/lib/docker/containers/[container_id]/[container_id]-json.log (the same container ID as my caddy container) was sitting at 12GB of usage.

I have my regular logs working, using the existing output specification in the config. This works great, and I have my expected logs in my caddy.log file in my volume (mapped to my host as usual). My issue is that json.log in /var/lib/docker/containers gets wildly huge fairly quickly with all the json logging.

EDIT: I confirmed this is what I think it is, as by using tail or cat it shows all the Caddy JSON logging populating as seen in docker logs command.

Good to know, thanks. Will remove it from my config.

4

That’s what I don’t understand though. It shouldn’t be that big so quickly. What kinds of logs are repeated so often that it’s an issue?

You can configure logging for everything other than access logs using the log global option:

1 Like
5

I had { debug } in my global config, maybe that was doing something. I removed it, and this is my resulting config, and it works as expected:

(snippet) {
	header {
		Strict-Transport-Security "max-age=31536000; includeSubdomains"
		X-XSS-Protection "1; mode=block"
		X-Frame-Options "DENY"
		-server
	}
	tls ryan@ryanb.tv {
		dns cloudflare {env.CLOUDFLARE_API_TOKEN}
	}
	log {
		format single_field common_log
		output file /data/logs/caddy.log
	}
}

something.url.com {
	reverse_proxy dockerHost.lan:12345
	import snippet
}

docker log now only shows application-related logs, and all access logs are now instead output to the log file specified.

Next up, will go change from the old-school clf to the new fancy json logging.

Thanks for your help @francislavoie.

6

Yeah, that would do it. Debug will add a lot of logging for each request. Not necessary in production. Most helpful when playing with the Caddyfile or testing some edgecase.

3 Likes
7

This topic was automatically closed after 30 days. New replies are no longer allowed.