Google Cloud Structured Logging Format

1. Caddy version (caddy version):

v2.2.1

2. How I run Caddy:

a. System environment:

ARG CADDY_VERSION=2.2.1
FROM caddy:${CADDY_VERSION}-builder-alpine AS builder

ARG ROUTE53_VERSION=1.0.2
RUN xcaddy build \
    --with github.com/caddy-dns/route53@v${ROUTE53_VERSION}

FROM caddy:${CADDY_VERSION}-alpine
COPY --from=builder /usr/bin/caddy /usr/bin/caddy

RUN apk --no-cache add curl

ENV XDG_DATA_HOME=/data \
    XDG_CONFIG_HOME=/config

COPY static/root /

CMD ["scaddy"]

b. Command:

This is the scaddy bash script referenced in the Dockerfile.

APP_BASIC_AUTH_PASSWORD_HASHED=$(caddy hash-password --plaintext $APP_BASIC_AUTH_PASSWORD) KUEUI_BASIC_AUTH_PASSWORD_HASHED=$(caddy hash-password --plaintext $KUEUI_BASIC_AUTH_PASSWORD) FARCRY_BASIC_AUTH_PASSWORD_HASHED=$(caddy hash-password --plaintext $FARCRY_BASIC_AUTH_PASSWORD) caddy run --config /etc/caddy/Caddyfile --adapter caddyfile

c. Service/unit/compose file:

d. My complete Caddyfile or JSON config:

{$APP_DOMAIN} {
    encode gzip

    log {
        output stdout
        format filter {
            wrap json {
                message_key "message"
                level_key "severity"
                time_key "timestampSeconds"
            }
            fields {
                common_log delete
            }
        }
        level DEBUG
    }

    root * /www/admin
    file_server

    @exceptProxy not path /css/* /fonts/* /images/* /js/* /favicon.ico /robots.txt /version.json
    reverse_proxy @exceptProxy http://app:{$APP_SERVICE_PORT_HTTP}
}

3. The problem I’m having:

I’m trying to structure the JSON logs to match the specified JSON at Structured logging  |  Cloud Logging  |  Google Cloud. This helps with automatic Log ingestion when running on GKE.

4. Error messages and/or full log output:

The closest I can get to is:

{
  "level": "info",
  "timestampSeconds": 1622762635.692175,
  "logger": "http.log.access.log173",
  "message": "handled request",
  "request": {
    "remote_addr": "172.17.0.1:19853",
    "proto": "HTTP/2.0",
    "method": "GET",
    "host": "www.typicalmotors.local.repcoservice.net",
    "uri": "/",
    "headers": {
      "Accept-Encoding": ["gzip, deflate, br"],
      "Cookie": [
        "_ga=GA1.2.262521322.1622506095; _gid=GA1.2.1926935137.1622598217"
      ],
      "Cp-Extension-Installed": ["Yes"],
      "Accept": [
        "text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.9"
      ],
      "Sec-Fetch-Site": ["none"],
      "Sec-Fetch-Dest": ["document"],
      "Sec-Fetch-Mode": ["navigate"],
      "Accept-Language": ["en-AU,en;q=0.9,en-US;q=0.8"],
      "Sec-Ch-Ua": [
        "\" Not A;Brand\";v=\"99\", \"Chromium\";v=\"90\", \"Google Chrome\";v=\"90\""
      ],
      "Sec-Ch-Ua-Mobile": ["?0"],
      "Upgrade-Insecure-Requests": ["1"],
      "Cache-Control": ["max-age=0"],
      "Authorization": ["Basic Y2hhc3Npczp4QkJCYWlZWFhGYW5QRHhoN1lLZEZmdFU="],
      "User-Agent": [
        "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/90.0.4430.212 Safari/537.36"
      ],
      "Sec-Fetch-User": ["?1"]
    },
    "tls": {
      "resumed": false,
      "version": 772,
      "cipher_suite": 4865,
      "proto": "h2",
      "proto_mutual": true,
      "server_name": "www.typicalmotors.local.repcoservice.net"
    }
  },
  "duration": 5.391735864,
  "size": 14852,
  "status": 200,
  "resp_headers": {
    "Server": ["Caddy"],
    "Link": [
      "<https://www.typicalmotors.local.repcoservice.net/>; rel=\"canonical\""
    ],
    "Content-Type": ["text/html; charset=utf-8"],
    "Etag": ["\"eee8-BffRLbLiGMWHIMVYKUw4WkXAIWI\""],
    "Content-Encoding": ["gzip"],
    "Date": ["Thu, 03 Jun 2021 23:23:55 GMT"],
    "X-Powered-By": ["Next.js"],
    "Cache-Control": [
      "private, no-cache, no-store, max-age=0, must-revalidate"
    ],
    "Vary": ["Accept-Encoding"]
  }
}

Notice that level_key didn’t get rewritten?

There is a bunch of other things I want to do like rename request to httpRequest, and then request>method to httpRequest>requestMethod, etc.

Is this possible?

I’ve read through the logs a few times but I can’t find anything.

Any advice would be appreciated. Thanks.

You’d be better off piping the logs through jq or something instead, to transform them.

Please upgrade to v2.4.1, you’re using a pretty old version!

Hey Francis,

JQ hasn’t had a release since 2018 Release jq 1.6 · stedolan/jq · GitHub. Not sure I want to add something so old into my stack which doesn’t seem to be getting any maintenance on it.

So, are you saying it’s not possible with Caddy? It doesn’t look like it is.

Maybe writing a plugin would be better?

cheers,
Scott.

There’s really no better tool. It manipulates JSON. Doesn’t really need that much maintenance.

Also, it is still getting commits, but just no tagged release since then.

Also see:

Caddy is a log emitter. Capabilities for formatting/encoding are limited because it’s really meant to be delegated to other tools that are better suited for it.

Possibly.

1 Like

Also you might consider using GitHub - itchyny/gojq: Pure Go implementation of jq which I just found (hadn’t done and research on re-implementations in a few months, so I’m glad to see this; but haven’t tried it yet to see if it’s properly syntax compatible)

1 Like

Thanks, that looks better!

I looked into this, and yep, that one was configurable, but not used at all :man_facepalming:

1 Like

Alright this is probably not the most efficient way to do it, but it’s what I could come up with in an hour or so of scouring the interwebs:

echo '<log>' | gojq '
    with_entries(if .key == "request" then .key = "httpRequest" else . end)
    | with_entries(if .key == "httpRequest" then .value = (
        .value | with_entries(if .key == "method" then .key = "requestMethod" else . end)
    ) else . end)'

Kinda wacky.

Basically the with_entries function converts each key-value pair into a {key: .key, value: .value} object, which you can then manipulate, and it then converts it back into the key value pair afterwards.

First we do request -> httpRequest key rename (if the key is request then we change the key, else return the key-value pair as-is).

Then we pipe that into another one which finds httpRequest, then manipulates that value to run with_entries on it (cause it’s nested) to do method -> requestMethod key rename.

:sweat:

1 Like

:rofl:

Okay this is way easier:

echo '<log>' | gojq '
    . + {"httpRequest": (
        .request + {"requestMethod": .request.method} | del(.method)
    )}
    | del(.request)'

Basically, clones request to httpRequest, and while doing so, clones method to requestMethod, then deletes method, and then deletes request.

2 Likes

Thanks @francislavoie. This looks like a much better option. I’ll integrate it and post here the entire string.

1 Like