Trouble with access logs

[cyrus (Cyrus Javan)]

1. Caddy version (caddy version):

v2.0.0

2. How I run Caddy:

a. System environment:

Ubuntu 16.04 instance on AWS
Using systemd to run caddy

b. Command:

sudo systemctl reload caddy

c. Service/unit/compose file:

[Unit]
Description=Caddy
Documentation=https://caddyserver.com/docs/
After=network.target

[Service]
User=caddy
Group=caddy
ExecStart=/usr/bin/caddy run --environ --config /etc/caddy/Caddyfile
ExecReload=/usr/bin/caddy reload --config /etc/caddy/Caddyfile
TimeoutStopSec=5s
LimitNOFILE=1048576
LimitNPROC=512
PrivateTmp=true
ProtectSystem=full
AmbientCapabilities=CAP_NET_BIND_SERVICE

[Install]
WantedBy=multi-user.target

d. My complete Caddyfile or JSON config:

:80 {
  reverse_proxy srv+http://hello-webs-frontend.service.consul.

  log {
    output file /home/ubuntu/caddy.log
  }
}

3. The problem I’m having:

I have consul running as a local DNS server for service discovery. Consul has SRV records for my service so I am using srv+ in my reverse_proxy directive so that Caddy will find the SRV record and forward to the correct host:port. I am getting a 500 response so I wanted to debug using Caddy’s logs. However, even with the log directive caddy does not seem to be writing any log file. Does anyone know how to debug Caddy itself not writing a log file?

4. Error messages and/or full log output:

N/A

5. What I already tried:

I thought maybe caddy does not have permission to write the log file so instead of /var/log/caddy.log I changed it to my home directory /home/ubuntu/caddy.log however still no log file is being written.

6. Links to relevant resources:

N/A

[francislavoie]

You can add this to the top of your Caddyfile to get some additional debug information about proxy requests:

{
	debug
}

This forces all debug level logs to be emitted to stdout. You can see these using journalctl -u caddy when running as a systemd service.

Global options (Caddyfile) - Caddy Documentation

[Tracy-P (Tracy Pearson)]

Referencing: Logging snippet using {host} placeholder for log filename - #2 by francislavoie
The Log will not start until Caddy is stopped and started.

[matt (Matt Holt)]

@Tracy-P

The Log will not start until Caddy is stopped and started.

I don’t believe that is accurate. Logs are started any time new config is loaded. In fact, when Caddy is first started, no config is loaded anyway.

[Tracy-P (Tracy Pearson)]

I didn’t see that behavior with Caddy V2 and mentioned here.

Intermittent SSL handshake errors

Best as I can determine, this problem has gone away once I have corrected the Caddyfile and actually completely restarted Caddy. I do not know what iterations of incorrectly structured Caddy files I had reloaded. I feel something during reloads did not clear correctly and caused this intermittent problem. I added a log directive. However, that directive will only take affect on the start of Caddy, not a reload.

[matt (Matt Holt)]

How can I exactly reproduce the behavior you experienced?

[Tracy-P (Tracy Pearson)]

Apparently you are correct.
The problem must have been related to the unstable Caddy instance I had due to loading invalid Caddyfiles prior to adding the log entry. I just added a log entry to my Caddyfile and reloaded and it worked.

[matt (Matt Holt)]

Cool, thanks for following-up!

[cyrus (Cyrus Javan)]

Thanks for the help everyone!

I was able to debug using systemctl status caddy and saw this can't open new logfile: open /home/ubuntu/caddy.log: permission denied. So I made that file and opened up it’s permissions and that worked.