Analyzing Caddy's log files real time with GoAccess?

jotteerr · September 4, 2019, 10:45pm

I’m trying to setup GoAccess for real time web log analyzing. Unfortunately, the documentation isn’t very helpful (at least for me, that is).

I’ve got GoAccess up and running and can look at real life data in a terminal, but setting up a HTML page is beyond my understanding. I’m running GoAccess with goaccess /var/log/caddy/access.log --real-time-html -o /var/www/report/report.html -p /etc/goaccess/goaccess.conf. In my Caddyfile I’ve set up www.mydomain.com/report as a site, and the first look at report.html with a browser populates the page with a nice visualization of the data from my Caddy logs.

So far, so good, but the difficulty is that I can’t find out how to make this static page to a real life one. GoAccess sets up a websocket server on port 7890 to deliver the data stream from the logs, and here is the point where I don’t know how to proceed. I tried various variations with the proxy directive, but nothing worked so far.

Has someone got GoAccess working with Caddy? Here is an example with a nginx configuration, but I can’t translate this into a Caddyfile.

hectorm · September 15, 2019, 12:59pm

Hi, here’s an example of Caddy and GoAccess that I’ve been using for months and it works well.

giopas · October 18, 2019, 12:02pm

Hi,

I am trying to set GoAccess as well.

My Caddy.log looks like the following:

2019/09/25 12:35:55 [INFO] ...

Can you please help me to configure it? I am really bad at regex and similar stuff!

Currently my goaccess.conf is similar to this one:

port 7890
log-format COMMON
log-file /share/Public/caddy/caddy.log
date-format %Y/%m/%d
time-format %H:%M:%S
output index.html
real-time-html true

Thanks!

Whitestrake · October 18, 2019, 12:42pm

That kinda looks like the process log. (i.e. -log /path/to/caddy.log)

Is your Caddyfile configured to output access logs to their own file, and is the access log file the one you’re pointing goaccess at?

giopas · October 18, 2019, 12:52pm

Indeed, this is what I am trying to do (ingest the caddy.log in GoAccess to have a monitoring tool of the reverse proxy*).

By the way I need to create a login access to such page (but this is another story).

Whitestrake · October 18, 2019, 12:55pm

Just to clarify and be as unambiguous as possible:

GoAccess can not ingest your Caddy process log.

The Caddy process log is specified by the -log flag when you run Caddy.

You need to ensure you are not outputting the process log to the same file you are ingesting with GoAccess.

You need to ensure that the file you ingest with GoAccess contains only access logs, which are specified with the log directive in your Caddyfile.

giopas · October 18, 2019, 1:04pm

In the Caddy.sh launch script I have

caddy (...) -log /share/Public/caddy/caddy.log

and in the goaccess.conf I have

log-file /share/Public/caddy/caddy.log

As far as I read here[1], it seems that log-file serves to Specify the path to the input log file.

Am I missing something even more obvious than I thought? (it’s well possible)

[1] GoAccess - Manual Page

Whitestrake · October 18, 2019, 1:17pm

-log /share/Public/caddy/caddy.log is the incorrect type of log. It is a process log, not an access log.

GoAccess can not analyse your process log.

Instead of using -log /share/Public/caddy/caddy.log, use the log directive in your Caddyfile:

https://caddyserver.com/v1/docs/log

For example:

example.com {
  log /path/to/access.log
}

With a line in your goaccess.conf:

log-file /path/to/access.log

giopas · October 18, 2019, 1:35pm

Ahhhhhh!

This is an excellent point!

Should I have a different *.log file for each site (and therefore have multiple log-file /path/to/access.log) or can I make each site pointing to the same *.log file?

Thanks a lot!

Whitestrake · October 18, 2019, 2:00pm

Both options will work.

Either way, GoAccess will analyse all the log lines that you’re putting in all the specified log files.

giopas · October 18, 2019, 4:25pm

Thank you Matthew.

I have now configured the file, but I cannot access to the index.html file from Caddy as I do not know how can I point to the famous index.html.

I do not know if is counts, but currently Caddy is set to use the following -root /home/Qhttpd

The caddy.conf (for accessing GoAccess) is the following:

sub.domain.com {
       import gzipconf
       import addheader
       log /share/Public/GoAccessLogs/goaccess-access.log
       proxy / http://192.168.10.37:7890 {
                keepalive 32
                transparent
                websocket
        }
}

The goaccess.conf is the following (in this example I want to see stats on the access to GoAccess done via Caddy):

port 7890
daemonize true
log-format COMMON
log-file /share/Public/GoAccessLogs/goaccess-access.log
output index.html
real-time-html true

Under /home/Qhttpd/Web/goaccess/ I can see index.html (I only created the goaccess folder and then goaccess created the index.html file alone).

However I have problems to access the file index.html (it has permission 666), whatever I type:

https://sub.domain.com
https://sub.domain.com/goaccess
https://sub.domain.com/goaccess/index.html

Once again I am surely missing something really simple…

Thanks!

giopas · October 19, 2019, 9:13am

Ok, I solved it: it seems that if you use the HTML layout you have to use an existing webserver. In my case I have one on port 82, so just I configured Caddy to that port and it works:

https://sub.domain.com/goaccess/index.html
http://192.168.10.37:82/goaccess/index.html

I am just not sure what is then the purpose of port 7890…

giopas · October 21, 2019, 6:44am

It is not a Caddy problem, but probably someone here knows why the index.html file is static and not live updating. Thanks!

Whitestrake · October 21, 2019, 11:16pm

The index.html file is generated once at GoAccess startup and stays the same after that point. For live updating, the site connects you to a websocket which supplies the new information. If the websocket isn’t enabled or can’t connect, you’ll only ever get first-run data.

giopas · October 22, 2019, 5:28am

Thanks Matthew,

How can I then connect it, or check if it is?

Whitestrake · October 22, 2019, 6:32am

Your browser’s developer console - network tab - should give you some insight as to the status of the websocket connection.

giopas · October 22, 2019, 3:06pm

I checked and I receive the following error on the Google Chrome Dev Tool:

WebSocket connection to 'wss://sub.domain.com:7890/' failed: Establishing a tunnel via proxy server failed.
setWebSocket @ (index):28
initialize @ (index):28
window.onload @ (index):28

The problem could be originated by the fact that Caddy is set to look at http://192.168.10.37:82/goaccess/index.html (which is the port where my Apache Server is running, i.e. the sole location that I managed to make it accessible for reading the index.html file).

How can I solve it? Thanks!

Whitestrake · October 22, 2019, 11:37pm

Well, you need:

An externally-accessible reverse proxy to the GoAccess websocket
To have GoAccess point your browser at that externally-accessible URI

Caddy can handle the former quite easily (looks like you’re already doing this in your latest Caddyfile). The latter you’ll need to look into GoAccess’ own configuration options.

giopas · October 23, 2019, 5:17am

In practice you are saying to point the reverse proxy to port 7890 and to just look at the goaccess config file to find a directory which is accessible by the websocket to put the index.html, right?

Whitestrake · October 23, 2019, 5:32am

Not 100% sure I understand what you mean by that, sorry.

The index file and websocket are handled very separately. One is a file served from the disk, the other is an upstream server you can reverse proxy to.

You need to configure GoAccess to announce the correct URI through which an external client can reach the websocket. Currently it’s announcing port 7890, which isn’t working.