Can't serve static files - how to debug path/permissions errors?

Help
1

1. Caddy version (caddy version):

2.3.0-1.fc34

2. How I run Caddy:

Caddy runs on ports 80/443 on a hetzner dedicated server.

It’s intended to proxy most traffic back to another local application, and to serve a directory of static files.

a. System environment:

$ grep PRETTY_NAME /etc/os-release
PRETTY_NAME="Fedora 34 (Thirty Four)"
$ uname --kernel-name --kernel-release
Linux 5.12.8-300.fc34.x86_64

b. Command:

$ dnf -y install caddy
$ systemctl restart caddy

c. Service/unit/compose file:

cat /usr/lib/systemd/system/caddy.service | grep -v ^$ | grep -v ^#
[Unit]
Description=Caddy web server
Documentation=https://caddyserver.com/docs/
After=network.target
[Service]
User=caddy
Group=caddy
ExecStartPre=/usr/bin/caddy validate --config /etc/caddy/Caddyfile
ExecStart=/usr/bin/caddy run --environ --config /etc/caddy/Caddyfile
ExecReload=/usr/bin/caddy reload --config /etc/caddy/Caddyfile
TimeoutStopSec=5s
LimitNOFILE=1048576
LimitNPROC=512
PrivateTmp=true
ProtectHome=true
ProtectSystem=full
AmbientCapabilities=CAP_NET_BIND_SERVICE
[Install]
WantedBy=multi-user.target

d. My complete Caddyfile or JSON config:

downloads.ddd.co.za:443 {
    log {
        output file /var/log/caddy/downloads.ddd.co.za.log
        level DEBUG
    }
    root * /home/downloads/data
    handle /completed/* {
        file_server browse
    }
    handle {
        reverse_proxy localhost:9999
    }
}

3. The problem I’m having:

I want caddy to pass every request that doesn’t match /completed/* to a local app running on port 9999. This works.

I also want caddy to serve up a directory of static files for requests matching /completed/FILENAME. I can’t get this to work. I receive a 403 when I curl things, and in the browser.

I have verified that caddy can access the files by running commands like:
sudo -u caddy cat /home/downloads/data/completed/hello.txt

How can I view any errors for mismatched paths or file permissions etc?

The JSON logs only show the 403 errors, but not what caused the actual error.

"common_log": "133.122.212.24 - downloads [08/Jun/2021:22:31:05 +1200] \"GET /completed/hello.txt HTTP/2.0\" 403 0",

4. Error messages and/or full log output:

5. What I already tried:

I’ve tried changing the root declaration paths, plus set up files in parent and subdirectories in case I had mis-alligned the root with the paths in the requests.

Is there a way to get caddy to be more informative about why it’s decided to serve a 403? I tried a log level of TRACE just in case, but that doesn’t exits.

I’ve also tried http2 vs http1.1.

6. Links to relevant resources:

2

Please upgrade to v2.4.1!

Since v2.4.0, some additional debug logging is done to help determine file server issues.

You can enable the debug global option to set the log level to DEBUG globally. Add this at the top of your Caddyfile:

{
	debug
}

Note that the logs are emitted to stdout/stderr, so your system’s log journal should have the logs. This should probably do the trick (I typically use debian based system so I’m not 100% sure if this is the same on fedora):

journalctl -u caddy --no-pager | less

Note that this only configures access logs when using the log directive inside of a site block (also there’s no DEBUG level access logs, so lowering the level there is ineffectual). For other logs, since v2.4.0, you may configure the log global option to write your logs elsewhere than stdout/stderr.

1 Like
3

Many thanks for the suggestions @francislavoie

Unfortunately, I am still having issues!

I upgraded to v2.4.1 h1:kAJ0JB5Xk5gPdTH/27S5cyoMGqD5lBAe9yZ8zTjVJa0= by just writing the caddy binary over the top of the one installed by dnf.

I simplifed my Caddyfile into the following, with just the file_server directive, and no reverse proxy.

{
	debug
}
download.ddd.co.za:443 {
    log {
        output stdout
        format console
        level  debug
    }
    root * /tmp/download/data
    file_server browse
}

I set up a new directory under /tmp to hold the static files and ran chmod -R ugo+rwX /tmp/download/ to ensure that caddy had access to all the files.

I then restarted with systemctl restart caddy to get the new configuration.

When I curled /completed/hello.txt I got a 404.

The debug from systemctl status caddy.service --no-pager -l | tail showed it composing the correct full filepath.

Jun 09 21:31:28 server.local caddy[11963]: {"level":"debug","ts":1623231088.457257,"logger":"http.handlers.file_server","msg":"sanitized path join","site_root":"/tmp/download/data","request_path":"/completed/hello.txt","result":"/tmp/download/data/completed/hello.txt"}

The “result” filepath was /tmp/download/data/completed/hello.txt which existed, and was readable by caddy.

The access log (in common_log format) was:

151.222.111.222 - - [09/Jun/2021:21:31:28 +1200] "GET /completed/hello.txt HTTP/2.0" 404 0
4

The contents of /var/lib/caddy/.config/caddy/autosave.json is:

{
  "apps": {
    "http": {
      "servers": {
        "srv0": {
          "listen": [
            ":443"
          ],
          "logs": {
            "logger_names": {
              "downloads.ddd.co.za": "log0"
            }
          },
          "routes": [
            {
              "handle": [
                {
                  "handler": "subroute",
                  "routes": [
                    {
                      "handle": [
                        {
                          "handler": "vars",
                          "root": "/tmp/downloads/data"
                        },
                        {
                          "handler": "file_server",
                          "hide": [
                            "/etc/caddy/Caddyfile"
                          ]
                        }
                      ]
                    }
                  ]
                }
              ],
              "match": [
                {
                  "host": [
                    "downloads.ddd.co.za"
                  ]
                }
              ],
              "terminal": true
            }
          ]
        }
      }
    }
  },
  "logging": {
    "logs": {
      "default": {
        "level": "DEBUG"
      },
      "log0": {
        "encoder": {
          "format": "console"
        },
        "include": [
          "http.log.access.log0"
        ],
        "level": "debug",
        "writer": {
          "output": "stdout"
        }
      }
    }
  }
}
5

When running as a systemd service, the systemd config has PrivateTmp=true, so the files written to /tmp actually end up in a location like /tmp/systemd-private-<some-hash>-caddy.service-<some-hash>/tmp, so each service gets their own isolated /tmp.

This means that using /tmp as a location for Caddy to read from is not a good idea, you should put your files elsewhere.

6

TL;DR ProtectHome=true in caddy.service was preventing caddy from seeing the static files in /home/downloads/data - moving the files outside /home/ got everything working without reducing security.

Thanks for that tip - I’ve not used systemd before.

PrivateTmp=true would explain why my tests using files under /tmp/ were returning 404s and but my original issue was returning 403s.

I took another look at caddy.service and saw ProtectHome=true about which the docs say:

Makes /home, /root, and /run/user appear empty. An additional option is read-only, which does exactly what it says. Also new to RHEL 8, tmpfs will overlay a writeable, ephemeral file system at these points. Because these directories are used to store SSH keys and other potentially sensitive information, it’s a good idea to prohibit applications from having access.

When I set ProtectHome=false and bounced systemd and caddy, then caddy was able to properly serve the files from /home/downloads/data.

I ended up moving the static files out of /home/ and into a new directory that isn’t affected by the caddy systemd security settings, so I could turn ProtectHome back on.

It’s all working as expected now, thanks again @francislavoie!

1 Like
7

I would generally recommend not trying to use /home though, best to put your site in /srv or /var/www. That way you don’t need to turn off a piece of protection.

8

Yep, I’m just going to move the static files outside /home/ so I don’t need to change caddy’s systemd

1 Like
9

This topic was automatically closed after 30 days. New replies are no longer allowed.