Caddy v2 logfile format

1. Caddy version (caddy version):

v2.3.0 h1:fnrqJLa3G5vfxcxmOH/+kJOcunPLhSBnjgIvjXV/QTA=

2. How I run Caddy:

/home/tmh/caddy/caddy start &

a. System environment:

4.19.0-14-686-pae #1 SMP Debian 4.19.171-2 (2021-01-30) i686 GNU/Linux

d. My complete Caddyfile or JSON config:

https://v4.tld, https://v6.ru {
log {
output file access.log
}
root * /home/tmh/caddy/web

    @blocked1 not maxmind_geolocation {
            db_path "/home/tmh/caddy/geolite/GeoLite2-Country.mmdb"
            allow_countries RU IL FR
    }
    rewrite @blocked1 def.html
    @blocked2 {
            path /site/pvt/folder*
            not maxmind_geolocation {
            db_path "/home/tmh/caddy/geolite/GeoLite2-Country.mmdb"
            allow_countries RU IL
            }
    }
    rewrite @blocked2 def.html
    file_server

}

3. The problem I’m having:

Once again, I report not a problem but a question - how to specify logfile format in Caddy v2 as needed one, not that raw output I have since I started to use Caddy V2 ?

Matt guru has been pointed me to that url - JSON Config Structure - Caddy Documentation - but I do not see no examples there, or more sad fact - no explains for all the fields listed in that page…

Once again, I really want to have the log output like this:
2021.03.04 14:25:32 192.168.0.15 “/favicon.ico” 200
Explained:
2021.03.04 14:25:32 – event timestamp,
192.168.0.15 – visitor IP OR resolved host name of v4 OR v6 addresses,
“/favicon.ico” – the visited path/file,
200 – result code.

In plus - some day later I’d be glad to use one of mentioned fields to specify source browser info, for example…

MANY THANKS in advance - I’ll be VERY happy if someone will explain how to setup such format of logging.

-t

Using & here is redundant, the caddy start command starts a Caddy instance in the background.

How did you install Caddy? I recommend you install with our apt repo, and use systemd to run Caddy:

See the log directive docs, you should set the format subdirective, specifically single_field common_log:

I compiled it by myself since no i386 binaries are available on version2…

Let see this - time_format The format for timestamps. – Sorry, but where are examples of this and other fields ? What kind of variables I can use these, what format for those variables ? Any ideas ?
PS can you point me EXACTLY on what fields related to these:
2021.03.04 14:25:32 – event timestamp,
192.168.0.15 – visitor IP OR resolved host name of v4 OR v6 addresses,
“/favicon.ico” – the visited path/file,
200 – result code.
Please ?

Ah, I see. I still recommend running as a systemd service though. Follow the instructions here:

There’s an example at the bottom of that page:

Use common log format (deprecated, but can be useful for older setups):

log {
    format single_field common_log
}

The template for common_log is this:

{http.request.remote.host} - {http.auth.user.id} [{time.now.common_log}] "{http.request.orig_method} {http.request.orig_uri} {http.request.proto}" {http.response.status} {http.response.size}

If you need more control, then you’ll need to use the format-encoder plugin:

Many thanks for help !! Just tell me - this is the way as it has to be done? :
log {
format single_field common_log
{http.request.remote.host} - {http.auth.user.id} [{time.now.common_log}] “{http.request.orig_method} {http.request.orig_uri} {http.request.proto}” {http.response.status} {http.response.size}
output file access.log
}

– or not… ?
And - where I can read about those variables http.request.remote.host, http.response.status and others ? thanks in advance !

No. Please read this page in the docs to understand:

The common_log field is a single field that exists in the JSON logs Caddy emits. You cannot configure what goes into it.

Like I said, if you need more control, then you’ll need to use the format-encoder plugin.

1 Like

Sorry… Information in those urls you provided are easy to be understanded for programmers, those who knowing JSON and go language, and I am not a coder at all, so that info is a total gibberish for me - without USEFUL examples is the reason, at least…

Could you - PLEASE - specify working format like Apache config file has that is very simple to understand yet to use:
LogFormat “%h %l %u [%{%Y-%m-%d %H:%M:%S}t] “%r” %>s %b “%{Referer}i” “%{User-Agent}i”” combined
– the same for Caddy v2 ? Without ‘format-encoder’ for a while ?

Thanks in advance !

You’re making it overly complex to do the simple thing you need, which is to have logs in Apache Combined Log Format. You didn’t need to look at the time_format option in the docs, although it’s available. The doc page which Francis linked gives you all the available knobs to configure logging, one of which is to define how your logs are formatted when printed (e.g. JSON) via an encoder. One of the format encoders options is single_field, which allows you to pick a single field from the native JSON log line to be printed instead of the entire JSON. The JSON log lines happen to contain a field called common_log which contains the log line that would have been printed for the request in the Apache Common Log format. Therefore you can combine the 2 concepts together, i.e. using single_field format encoder and the common_log field of the JSON, to define a logger which prints the log exactly how Apache Common Log format would print it. In fact, this is one of the prominent examples printed on the doc page:

log {
	format single_field common_log
}

@francislavoie has already given this line.

Back to your need, which is to print the logs in the Apache Combined Log format. Per Apache docs, the Combined format is the same as the Common Log Format with the “Referer” and “User-Agent” headers appended. To construct a custom template of the log line, you will need to use one of Caddy’s non-standard modules called format-encoder (names are hard), as suggested by @francislavoie, with which you define the format/template of how you want your log line to look like and it’ll follow suit.

Just like the single_field logging encoder, this module traverses the native JSON log for the values, so you will need to inspect a sample JSON log line to find the values you need. I’ll beat you to the question of why don’t we provide the structure of the JSON to answer it with because the content of the JSON log is malleable and will have fields added/removed based on various factors. The README of the module shows you how to extract the User-Agent header, which can be easily modified to extract the Referer header as well.

I’ll cut it short and tell you the template/format to use with the format-encoder is probably (unverified):

{common_log} {request>headers>Referer>[0]} {request>headers>User-Agent>[0]}

Again, this is from the top of my head. You’ll have to validate it, play with it, and figure out exactly how you want to change it.

3 Likes

Greetings Mohammed90 – I’ll try this one now:

    log {
            format single_field common_log
            {common_log} {request>headers>Referer>[0]} {request>headers>User-Agent>[0]}
            output file access.log
    }

– is this one correct one ?

No. Again, you will have to use the linked format-encoder. Read the instruction in the docs and the README of the linked module, and don’t just pick up the quoted config lines without understanding how they work together.

3 Likes

OK, guys, this sunny morning gives me ability to write this scheme in Caddyfile:


       log {
        format single_field common_log
        output file access.log
        }

now I see almost-what-I-needed in output
IP.IP.IP.IP - - [05/Mar/2021:07:40:14 +0300] “GET / HTTP/2.0” 304 0

As said earlier this format is a bit outdated (I don’t know why), so - I have to RECOMPILE my version of caddy to include format-encoder…

Once again ppl I want to tell you - I am NOT a programmer at all, I do not understand why VARIABLES are renamed to PLACEHOLDERS - for example - and I do not see NO useful information about those placeholders, for example look here:
{
“format”: “logfmt”,
“message_key”: “”,
“level_key”: “”,
“time_key”: “”,
“name_key”: “”,
“caller_key”: “”,
“stacktrace_key”: “”,
“line_ending”: “”,
“time_format”: “”,
“duration_format”: “”,
“level_format”: “”
}
where I can see full descriptions on these “placeholders” - what those can accept - for example - format “YYYY-mm-DD HH:MM:SS” in “time_format” placeholder ?

Thanks in advance, ppl, I’ll report after I’ll get recompiled my version of caddy v2…
-t

Nowhere does the doc says variables and placeholders are interchangeable, logfmt has been deprecated and removed for awhile and it’s nowhere in the current docs, and you still don’t need to look at those level_key and whatnot so don’t worry about those. The time_format follows Go’s convention outlined here: time package - time - pkg.go.dev

1 Like

Look here: Conventions — Caddy Documentation
(quote from there: “Placeholders are a similar idea to variables in other software. For example, nginx has variables like $uri and $document_root.”)

Yes, as soon I’ll get recompiled binary I’ll report it here with a question - what to do next :wink:
Thanks in advance !

PS. Or - instead of performing brains melting procedure of learning Go language - I will try to modify server.go file at string 583 - commonLogFormat - to set exact log format I need - and then recompile the caddy… “When the Magomed won’t come to mountain then mountain itself comes to Magomed” - very old and truly wise proverb.

I can tell you right now this will not be as simple as one line change. The proverb isn’t serving you well here.

2 Likes

Might be, but I won’t turn away from this :wink:

I’ll dig up the variable in common_log - [{time.now.common_log}] to set desired date+time format.
If I’ll stuck while searching needed info - I hope I can count on You, or someone else - to help me with this case. THANKS in advance ! :slight_smile:

OK
I found how to fix the stuff – in short way:
caddyserver/caddy/v2@v2.3.0>diff replacer.go.orig replacer.go
289c289,290

// return nowFunc().Format(“02/Jan/2006:15:04:05 -0700”), true
return nowFunc().Format(“2006-01-02 15:04:05”), true

Now I’ll try to examine that format-encoder plugin…

Before format-encoder stuff, I want to ask go language gurus about some go related question.

As you ppl know that ‘return nowFunk().Format’ variable is being used in replacer.go file of caddy v2 server.
Please tell me - is it possible to have a variable in config file that will re-define this variable ? I mean, I know that altering source .go files is NOT the right way/not good at all, so - as far I want to have my own date suffix - once again - BEFORE I’ll move onto format-encoder, I want to know about this ability - redefine variable with my on suffix in config file - like this:
log {
myDateTimeFormat “2006-01-02 15:04:05”
}

I think it will be somewhat useful for those who do not want to install addition plugins like format-encoder and take the journey ‘how to configure that thing!’ :slight_smile:

Thanks in advance !
-t

If you’re questioning whether it’s possible, in Golang, to write code to expose that string for the server or web administrator to configure, the answer is yes, it is possible.

In the context of Caddy specifically, a better question might be whether or not that’s feasible.

1 Like

Hello and thanks for the answer!
So - is this ‘twist’ easy to code, then ? Can you show example how this variable re-define can be done in config file ?:slight_smile:

I can’t, sorry.

Just to be unambiguous here, I have my doubts that it would be feasible.

In general, though, the best place to discuss proposals to have Caddy’s code changed isn’t here on the Caddy Community. A better forum for that would be the GitHub repository, perhaps as a feature request, where it can be acknowledged and the merit of any possible actual implementation could be considered and discussed.

If you’re just looking to hardcode the change for your own benefit, you can do that directly in the source file before compiling your binary.

1 Like