NFS share as root directory for files

1. The problem I’m having:

I am unable to use an NFS share as the root for sites. I can use the exact same configuration if I move the files over to the local machine. This is on a local dev environment, so not web accessible.

[sergio@samara:/var/www]$  curl -vL https://nerdpress.localhost
* Host nerdpress.localhost:443 was resolved.
* IPv6: ::1
* IPv4: 127.0.0.1
*   Trying [::1]:443...
* Connected to nerdpress.localhost (::1) port 443
* ALPN: curl offers h2,http/1.1
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* TLSv1.3 (IN), TLS handshake, Server hello (2):
* TLSv1.3 (IN), TLS handshake, Encrypted Extensions (8):
* TLSv1.3 (IN), TLS handshake, Certificate (11):
* TLSv1.3 (OUT), TLS alert, unknown CA (560):
* OpenSSL/3.0.13: error:16000069:STORE routines::unregistered scheme
* Closing connection
curl: (35) OpenSSL/3.0.13: error:16000069:STORE routines::unregistered scheme

I suspected it was something to do with file permissions, since I also couldn’t get this to work from my ~/www directory. I’ve since mounted the NFS share with the GID of the local caddy user (239) and chowned the /var/www/nerdpress directory to sergio:239. This changed the error message from a 403 to a 500… Getting a stale NFS file handle from the logs (shown on step 2 below) which I tried to address, but does not seem to be the problem? I reset the server/guest and unmounted/mounted the NFS share with -l option…

This system has been installed through a Nix file on NixOS. Specifics in the NixOS forum post I created:
Setting up a WordPress dev environment. AKA... Trying to run before I can walk - #21 by Seedsca - Help - NixOS Discourse

Output of file permissions showing caddy group:

[sergio@samara:/var/www]$ ls -l nerdpress/
total 9520
-rw-rw-r-- 1 sergio caddy    2574 Sep  1  2021 db_check
-rw-rw-r-- 1 sergio caddy     900 Jun 25 23:51 devenv.nix
-rw-r--r-- 1 sergio caddy       0 Jul  3 13:49 favicon.ico
-rw-r--r-- 1 sergio caddy     534 Jun 21 14:01 flake.lock
-rw-r--r-- 1 sergio caddy    3652 Jun 22 15:20 flake.nix
-rw-r--r-- 1 sergio caddy    2832 Jun 21 13:44 flake.nix.pause
-rw-rw-r-- 1 sergio caddy     405 Jun  4  2020 index.php
-rw-rw-r-- 1 sergio caddy    1545 Sep  5  2021 integrity.php
-rw-rw-r-- 1 sergio caddy   19915 Sep 28  2023 license.txt
-rw-rw-r-- 1 sergio caddy     201 Dec 23  2021 nano.save
-rw-rw-r-- 1 sergio caddy 2376922 Dec 15  2021 nerdpress-2021-12-15-efd1283.sql
-rw-rw-r-- 1 sergio caddy 3407305 Sep 28  2023 nerdpress-2023-09-28-b1c9be3.sql
-rw-rw-r-- 1 sergio caddy 3702713 Mar 11 09:33 nerdpress-2024-03-11-a9e003e.sql
-rw-rw-r-- 1 sergio caddy      73 Apr 28  2023 php.ini
-rw-rw-r-- 1 sergio caddy    7399 Sep 28  2023 readme.html
-rw-r--r-- 1 sergio caddy     150 Jun 20 14:37 shell.nix
-rw-rw-r-- 1 sergio caddy     207 Jun 30 14:22 test.php
-rw-r--r-- 1 sergio caddy       0 Jun 14 23:26 wordpress.db
-rw-rw-r-- 1 sergio caddy    7211 Sep 28  2023 wp-activate.php
drwxr-xr-x 1 sergio caddy    2804 Sep 28  2023 wp-admin
-rw-rw-r-- 1 sergio caddy     351 Jun  4  2020 wp-blog-header.php
-rw-rw-r-- 1 sergio caddy    2323 Sep 28  2023 wp-comments-post.php
-rw-rw-r-- 1 sergio caddy    3535 Apr 28  2023 wp-config.php
-rw-rw-r-- 1 sergio caddy    3013 Sep 28  2023 wp-config-sample.php
drwxr-xr-x 1 sergio caddy     286 Jun 21 14:36 wp-content
-rw-rw-r-- 1 sergio caddy    5638 Sep 28  2023 wp-cron.php
drwxr-xr-x 1 sergio caddy    9744 Sep 28  2023 wp-includes
-rw-rw-r-- 1 sergio caddy    2502 Sep 28  2023 wp-links-opml.php
-rw-rw-r-- 1 sergio caddy    3927 Sep 28  2023 wp-load.php
-rw-rw-r-- 1 sergio caddy   49441 Sep 28  2023 wp-login.php
-rw-rw-r-- 1 sergio caddy    8537 Sep 28  2023 wp-mail.php
-rw-rw-r-- 1 sergio caddy   25602 Sep 28  2023 wp-settings.php
-rw-rw-r-- 1 sergio caddy   34385 Sep 28  2023 wp-signup.php
-rw-rw-r-- 1 sergio caddy    4885 Sep 28  2023 wp-trackback.php
-rw-rw-r-- 1 sergio caddy    3236 Sep 28  2023 xmlrpc.php

NFS exportfs -v output from server:

/mnt/btr-vault/vault/www 192.168.0.0/16(sync,wdelay,hide,no_subtree_check,anonuid=1000,anongid=239,sec=sys,rw,secure,root_squash,all_squash)

mount output from local:

192.168.1.10:/mnt/btr-vault/vault/www on /var/www type nfs4 (rw,relatime,vers=4.2,rsize=1048576,wsize=1048576,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,clientaddr=192.168.1.128,local_lock=none,addr=192.168.1.10,x-systemd.automount,x-systemd.after=network-online.target,x-systemd.mount-timeout=90)

I think the server can read all the directories fine:

<?php
$dirs = [
  "/var",
  "/var/www",
  "/var/www/nerdpress",
];
foreach ( $dirs as $dir ) {
  if ( is_array( scandir($dir ) ) ){
    echo "<p> $dir is readable</p>";
  }
}
phpinfo();

image

2. Error messages and/or full log output:

[sergio@samara:~/nixos-config]$ sudo journalctl -f -u caddy.service -u mysql.service -u phpfpm-nerdpress.localhost.service 

Jul 03 16:53:13 samara caddy[1087]: {"level":"error","ts":1720050793.9880104,"logger":"http.log.error.log0","msg":"stat /var/www/nerdpress: stale NFS file handle","request":{"remote_ip":"127.0.0.1","remote_port":"42830","client_ip":"127.0.0.1","proto":"HTTP/2.0","method":"GET","host":"nerdpress.localhost","uri":"/","headers":{"Upgrade-Insecure-Requests":["1"],"Sec-Fetch-User":["?1"],"Pragma":["no-cache"],"Accept-Encoding":["gzip, deflate, br, zstd"],"Sec-Fetch-Mode":["navigate"],"Dnt":["1"],"Priority":["u=1"],"Cache-Control":["no-cache"],"Te":["trailers"],"Accept":["text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,*/*;q=0.8"],"Sec-Fetch-Site":["none"],"Sec-Gpc":["1"],"Cookie":[],"Accept-Language":["en-US,en;q=0.5"],"Sec-Fetch-Dest":["document"],"User-Agent":["Mozilla/5.0 (X11; Linux x86_64; rv:127.0) Gecko/20100101 Firefox/127.0"]},"tls":{"resumed":false,"version":772,"cipher_suite":4865,"proto":"h2","server_name":"nerdpress.localhost"}},"duration":0.067520311,"status":500,"err_id":"ybc9vpats","err_trace":"fileserver.(*FileServer).ServeHTTP (staticfiles.go:284)"}

3. Caddy version:

2.7.6

● caddy.service - Caddy
     Loaded: loaded (/etc/systemd/system/caddy.service; enabled; preset: enabled)
    Drop-In: /nix/store/k6m7mfl2aqs9qlx7xx7sziqclkn4c6f2-system-units/caddy.service.d
             └─overrides.conf
     Active: active (running) since Wed 2024-07-03 16:26:24 PDT; 1h 50min ago
       Docs: https://caddyserver.com/docs/
   Main PID: 1087 (caddy)
         IP: 13.7K in, 12.1K out
         IO: 35.2M read, 52.0K written
      Tasks: 14 (limit: 38313)
     Memory: 45.6M (peak: 48.2M)
        CPU: 718ms
     CGroup: /system.slice/caddy.service
             └─1087 /nix/store/c496kw0yni8wjg8636dwvnjb4hcn21c7-caddy-2.7.6/bin/caddy run --config /etc/caddy/caddy_config --adapter caddyfile

4. How I installed and ran Caddy:

a. System environment:

NixOS 24.05, x86, systemd

b. Command:

Systemd service is used to run Caddy. The command to run Caddy is:

/nix/store/c496kw0yni8wjg8636dwvnjb4hcn21c7-caddy-2.7.6/bin/caddy run --config /etc/caddy/caddy_config --adapter caddyfile

c. Service/unit/compose file:

[sergio@samara:/var/www]$  systemctl cat caddy.service
# /etc/systemd/system/caddy.service
# caddy.service
#
# For using Caddy with a config file.
#
# Make sure the ExecStart and ExecReload commands are correct
# for your installation.
#
# See https://caddyserver.com/docs/install for instructions.
#
# WARNING: This service does not use the --resume flag, so if you
# use the API to make changes, they will be overwritten by the
# Caddyfile next time the service is restarted. If you intend to
# use Caddy's API to configure it, add the --resume flag to the
# `caddy run` command or use the caddy-api.service file instead.

[Unit]
Description=Caddy
Documentation=https://caddyserver.com/docs/
After=network.target network-online.target
Requires=network-online.target

[Service]
Type=notify
User=caddy
Group=caddy
ExecStart=/nix/store/c496kw0yni8wjg8636dwvnjb4hcn21c7-caddy-2.7.6/bin/caddy run --environ --config /etc/caddy/Caddyfi>
ExecReload=/nix/store/c496kw0yni8wjg8636dwvnjb4hcn21c7-caddy-2.7.6/bin/caddy reload --config /etc/caddy/Caddyfile --f>
TimeoutStopSec=5s
LimitNOFILE=1048576
LimitNPROC=512
PrivateTmp=true
ProtectSystem=full
AmbientCapabilities=CAP_NET_ADMIN CAP_NET_BIND_SERVICE

[Install]
WantedBy=multi-user.target

# /nix/store/k6m7mfl2aqs9qlx7xx7sziqclkn4c6f2-system-units/caddy.service.d/overrides.conf
[Unit]
StartLimitBurst=10
StartLimitIntervalSec=14400
X-Reload-Triggers=/nix/store/18l4b9c4fx49xvqivgv8wl2r00dmjkm4-X-Reload-Triggers-caddy

[Service]
Environment="LOCALE_ARCHIVE=/nix/store/naa6s54b4hc55740lblbs9phv7xk317h-glibc-locales-2.39-52/lib/locale/locale-archi>
Environment="PATH=/nix/store/php4qidg2bxzmm79vpri025bqi0fa889-coreutils-9.5/bin:/nix/store/jjcsr5gs4qanf7ln5c6wgcq4sn>
Environment="TZDIR=/nix/store/y6hmqbmbwq0rmx1fzix5c5jszla2pzmp-tzdata-2024a/share/zoneinfo"
ExecReload=
ExecReload=/nix/store/c496kw0yni8wjg8636dwvnjb4hcn21c7-caddy-2.7.6/bin/caddy reload --config /etc/caddy/caddy_config >
ExecStart=
ExecStart=/nix/store/c496kw0yni8wjg8636dwvnjb4hcn21c7-caddy-2.7.6/bin/caddy run --config /etc/caddy/caddy_config --ad>
Group=caddy
LogsDirectory=caddy
NoNewPrivileges=true
PrivateDevices=false
PrivateTmp=false
ProtectHome=false
ProtectSystem=off
ReadWritePaths=/var/lib/caddy
Restart=on-failure
RestartPreventExitStatus=1
RestartSec=5s
StateDirectory=caddy
User=caddy

[Install]
WantedBy=multi-user.target

d. My complete Caddy config:

[sergio@samara:/var/www]$ cat /etc/caddy/caddy_config 
{
        log {
                level ERROR
        }
}

https://nerdpress.localhost:443 {
        log {
                output file /var/log/caddy/access-https://nerdpress.localhost:443.log
        }

        root * /var/www/nerdpress
        php_fastcgi unix//run/phpfpm/nerdpress.localhost.sock
        file_server
}

Created by this lamp.nix file:

[sergio@samara:~/nixos-config]$ cat modules/lamp.nix
{ config, pkgs, lib, ... }:
let

  php' = pkgs.php83.buildEnv {
    extensions = ({ enabled, all }: enabled ++ (with all; [
      xdebug
      imagick
    ]));
    # any customizations to your `php.ini` go here
    extraConfig = ''
      memory_limit = 1024M
      xdebug.mode = debug
      xdebug.start_with_request = yes
      xdebug.idekey = gdbp
    '';
  };
  webPath = "/var/www";

  sites = [
    "nerdpress.localhost"
  ];
in
{
  networking.hosts = {
    # convenient if you're going to work on multiple sites
    "127.0.0.1" = [
      "nerdpress.localhost"
    ];
  };

  services.mysql.enable = true;
  services.mysql.package = pkgs.mariadb;
  services.mysql.ensureDatabases = [
    # list a database for every site you want and they will be automatically created
    "nerdpress"
  ];
  services.mysql.ensureUsers = [
    # NOTE: it is important that `name` matches your `$USER` name, this allows us to avoid password authentication
    { name = "sergio";
      ensurePermissions = {
        "*.*" = "ALL PRIVILEGES";
      };
    }
  ];

  services.phpfpm.pools."nerdpress.localhost" = {
    user = "sergio";
    group = "users";
    phpPackage = php';
    settings = {
      "listen.owner" = config.services.caddy.user;
      "listen.group" = config.services.caddy.group;
      "pm" = "dynamic";
      "pm.max_children" = 5;
      "pm.start_servers" = 2;
      "pm.min_spare_servers" = 1;
      "pm.max_spare_servers" = 5;
      "php_admin_value[error_log]" = "stderr";
      "php_admin_flag[log_errors]" = true;
      "catch_workers_output" = true;
    };
  };

  services.caddy.enable = true;

  services.caddy.virtualHosts."https://nerdpress.localhost:443".extraConfig = ''
    root * ${webPath}/nerdpress
    php_fastcgi unix/${config.services.phpfpm.pools."nerdpress.localhost".socket}
    file_server
  '';

  # automatically create a directory for each site you will work on with appropriate ownership+permissions
  systemd.tmpfiles.rules = [
    "d ${webPath}/nerdpress 0755 sergio users"
  ];

  systemd.services."phpfpm-nerdpress.localhost".serviceConfig = {
    PrivateDevices = lib.mkForce false;
    PrivateTmp = lib.mkForce false;
    ProtectSystem = lib.mkForce "off";
    ProtectHome = lib.mkForce false;
  };
  systemd.services.mysql.serviceConfig = {
    PrivateDevices = lib.mkForce false;
    PrivateTmp = lib.mkForce false;
    ProtectSystem = lib.mkForce "off";
    ProtectHome = lib.mkForce false;
  };
  systemd.services.caddy.serviceConfig = {
    PrivateDevices = lib.mkForce false;
    PrivateTmp = lib.mkForce false;
    ProtectSystem = lib.mkForce "off";
    ProtectHome = lib.mkForce false;
  };
}

5. Links to relevant resources:

Where I started to create this setup and some relevant information:

Well, this is a bit of a sticky situation. An intersection of NFS, systemd, and Caddy. Smells like the kind of problem that usually needs someone quite familiar with the workings of all systems involved to put their finger on the nuance here.

Might sound like a crazy suggestion, but are you able to export the files over CIFS instead and mount that just to try? Just to maybe help narrow down if it is indeed something NFS-specific? If it works over CIFS then you’ll have a better idea of where to focus your scrutiny.

1 Like

I started wondering why the test.php file would load, but none of the other URLs worked. So I then thought to try one of the WordPress PHP files and it sort of works. I’m getting many 500 errors still, but the pages are kinda loading… So there is a db connection happening and PHP is rendering what it needs.

Still getting a tonne of stale NFS file handle from journalctl

So the only files that do load are php ones. JS and CSS get the 500 code. This would seem to be a setting within Caddy that can be changed. Any pointers as to where it may be?

I’d focus on the stale file handles. The fact you had to lazily unmount the share (umount -l) says there is a problem you need to attend to.

First, remove the systemd automount option. Also, are you sure you need a hard mount? A hard mount requires quite a well defined nfs server, or you will end up with stale file handles.

Can you describe how you have setup the filesyatem and directories that you are sharing from the nfs server?

Nfs4 uses a unified root, and if you share different mounts under it, there can be issues unless you take some extra steps.

Are you using btrfs (btrr-vault)? You may need to explicitly define a filesystem ID for each export. This is because NFS might not be able to determine the correct fsid and separate subvolumes.

Try something like this

/mnt/btr-vault/vault/www 192.168.0.0/16(fsid=111111,rw,no_root_squash,async,no_subtree_check)

It is important that you do not expose snapshots of subvols inside the same export. This is because inode numbers are not unique between snapshots and nfs cannot yet handle this. Samba, for example, deals with this by always generating unique inodes.

2 Likes

Thanks for the input!

I’d like to automount these so as to keep them as seamless as possible. This is the only way I figured to do so. I guess I can manually add them to the fstab file instead?

I have this running in a Proxmox server. The drive is indeed a btrfs formatted drive in a RAID1 configurations, so there is another mirrored drive (not really sure that’s the best way to set that bit up… for another day) I have the NFS server running on the Proxmox host directly, no VM or LXC (trying to make things simple may be making them more complex). The btr-vault share is where I have Proxmox keep it’s data (so backups, ISOs, etc). vault is where all my files are. There, then, are shares per sub-directory…

I’m thinking I may want to separate all of these differently. So, have:
/mnt/btr-vault/proxmox
/mnt/btr-vault/documents
/mnt/btr-vault/www
etc…

Currently this is my working(ish) configuration:

root@pod:~# exportfs -v
/mnt/btr-vault/vault
                192.168.1.0/16(async,wdelay,hide,no_subtree_check,anonuid=1000,anongid=100,sec=sys,rw,secure,root_squash,all_squash)
/mnt/btr-vault/vault/Documents
                192.168.0.0/16(async,wdelay,hide,no_subtree_check,anonuid=1000,anongid=100,sec=sys,rw,secure,root_squash,all_squash)
/mnt/btr-vault/vault/Downloads
                192.168.0.0/16(async,wdelay,hide,no_subtree_check,anonuid=1000,anongid=100,sec=sys,rw,secure,root_squash,all_squash)
/mnt/btr-vault/vault/Music
                192.168.0.0/16(async,wdelay,hide,no_subtree_check,anonuid=1000,anongid=100,sec=sys,rw,secure,root_squash,all_squash)
/mnt/btr-vault/vault/Pictures
                192.168.0.0/16(async,wdelay,hide,no_subtree_check,anonuid=1000,anongid=100,sec=sys,rw,secure,root_squash,all_squash)
/mnt/btr-vault/vault/Videos
                192.168.0.0/16(async,wdelay,hide,no_subtree_check,anonuid=1000,anongid=100,sec=sys,rw,secure,root_squash,all_squash)
/mnt/btr-vault/vault/Web
                192.168.0.0/16(async,wdelay,hide,no_subtree_check,anonuid=1000,anongid=100,sec=sys,rw,secure,root_squash,all_squash)
/mnt/btr-vault/vault/www
                192.168.0.0/16(async,wdelay,hide,no_subtree_check,anonuid=1000,anongid=239,sec=sys,rw,secure,root_squash,all_squash)
/mnt/btr-vault  192.168.1.0/24(sync,wdelay,hide,sec=sys,rw,secure,no_root_squash,no_all_squash)
/mnt/btr-vault  192.168.10.0/24(sync,wdelay,hide,no_subtree_check,sec=sys,rw,secure,no_root_squash,no_all_squash)

This is how I’m mounting the drive:
/dev/nvme3n1 on /mnt/btr-vault type btrfs (rw,relatime,ssd,discard=async,space_cache=v2,subvolid=5,subvol=/)

I tried adding the fsids but ended up not being able to mount the shares anymore. Something about the path not existing. Reading more into it now. I’m guessing the fsid can be any int as long as it’s unique and not 0.

So setting up the shares in a “horizontal” fashion, so no directory exists in any other share is what you mean here?

You shouldn’t edit /etc/fstab directly on NixOS; I think their suggestion was simply to remove the x-systemd.automount option from the filesystems stanza in your configuration.nix.

Removing x-systemd.automount will have the effect of mounting the filesystem when invoked and keeping it there (at boot/mount -a if auto is set) rather than only mounting automatically when a process attempts to interact within the mounted directory.

2 Likes

There is some discussion on fsid here filesystems - Stale NFS File Handle why does fsid resolve it? - Unix & Linux Stack Exchange

In your case, you have the same filesystem under different mounts. Subvolumes can confuse nfs too, so using fsid should helps the situation.

Do not export /mnt/btr-vault/ as well as other directories under it at the same time. Try removing that export. Using fsid should not prevent clients from mounting the export, though you must make sure that no clients are mounted when you add the fsid option as otherwise they will use the wrong fsid for the mount.

@Whitestrake is correct in that i mean to remove the automount feature an keep mounts permanently enabled.

What exactly were your exports and mount options when you tested fsid?

1 Like