Health Checks are filling up my Caddy logs

1. The problem I’m having:

I have Caddy configured as a reverse proxy for my applications and it’s working great. The problem is that the AWS ELB health checks are filling up my logs and making it difficult to see the real requests.

When I was using Nginx I configured it to drop all 200 status health checks and only log the non-200 status health checks. That way if something went wrong with the health check I could still get some detail on the error. With Caddy I’m pretty sure that I want to use skip_log but I don’t know how to block just the 200 requests.

2. Error messages and/or full log output:

N/A

3. Caddy version:

$ caddy --version
v2.7.6 h1:w0NymbG2m9PcvKWsrXO6EEkY9Ru4FJK8uQbYcev1p3A=

4. How I installed and ran Caddy:

a. System environment:

Linux, Alpine 3.18.6, Docker (caddy:2.7)

b. Command:

caddy run --config /etc/caddy/Caddyfile

c. Service/unit/compose file:

Default Alpine / caddy:2.7 config

d. My complete Caddy config:

# Global config options
# https://caddyserver.com/docs/caddyfile/options
{
        # turn off admin API endpoint
        admin off

        servers {
                trusted_proxies static private_ranges
        }
}

# Internal endpoint for serving healthchecks etc.
:80 {
        respond /up "UP" 200
}

:3000 {
        # Block API calls from the WWW
        @blocked {
                header x-forwarded-proto https
                path /internal_api
        }
        respond @blocked 401

        reverse_proxy cobalt2.{$LH_DATASET}:3000 {
                fail_duration 30s
        }

        log {
                output stdout
                format json
        }
}

:3001 {
        # Block API calls from the WWW
        @blocked {
                header x-forwarded-proto https
                path /internal_api
        }
        respond @blocked 401

        reverse_proxy durin.{$LH_DATASET}:3001 {
                fail_duration 30s
        }

        log {
                output stdout
                format json
        }
}

5. Links to relevant resources:

Please show an example of the logs that you’re trying to quiet.

I’m not clear on what your health requests look like, or what server it hits.

{
    "level": "info",
    "ts": 1711090053.7564049,
    "logger": "http.log.access.log0",
    "msg": "handled request",
    "request":
    {
        "remote_ip": "10.204.49.13",
        "remote_port": "62302",
        "client_ip": "10.204.49.13",
        "proto": "HTTP/1.1",
        "method": "GET",
        "host": "10.204.36.115:3000",
        "uri": "/up",
        "headers":
        {
            "Connection":
            [
                "close"
            ],
            "User-Agent":
            [
                "ELB-HealthChecker/2.0"
            ],
            "Accept-Encoding":
            [
                "gzip, compressed"
            ]
        }
    },
    "bytes_read": 0,
    "user_id": "",
    "duration": 0.010653966,
    "size": 58,
    "status": 200,
    "resp_headers":
    {
        "Cache-Control":
        [
            "max-age=0, private, must-revalidate"
        ],
        "X-Envoy-Upstream-Service-Time":
        [
            "9"
        ],
        "Date":
        [
            "Fri, 22 Mar 2024 06:47:33 GMT"
        ],
        "X-Download-Options":
        [
            "noopen"
        ],
        "X-Permitted-Cross-Domain-Policies":
        [
            "none"
        ],
        "Content-Type":
        [
            "text/html; charset=utf-8"
        ],
        "Content-Length":
        [
            "58"
        ],
        "X-Xss-Protection":
        [
            "1; mode=block"
        ],
        "Vary":
        [
            "Origin"
        ],
        "Etag":
        [
            "W/\"830fe1288600b623f54d204c55fe2f83\""
        ],
        "Referrer-Policy":
        [
            "strict-origin-when-cross-origin"
        ],
        "X-Content-Type-Options":
        [
            "nosniff"
        ],
        "X-Runtime":
        [
            "0.006066"
        ],
        "X-Frame-Options":
        [
            "SAMEORIGIN"
        ],
        "Server":
        [
            "Caddy",
            "envoy"
        ],
        "X-Request-Id":
        [
            "7b305d74-f8a0-4be3-8e92-bcdfd28ed6cd"
        ]
    }
}

Okay, so I think you need to use reverse_proxy’s handle_response subdirective to intercept the 200 response.

reverse_proxy <upstream> {
    @200 status 200
    handle_response @200 {
        skip_log /up
        copy_response
    }
}

Just a heads up, in v2.8.0 we’re renaming skip_log to log_skip (just a consistency change)