Help tracking down a permissions issue?

1. The problem I’m having:

TL:DR; We are seeing some sub-directories and files created in a PHP CMS’s storage/runtime/compiled_templates directory owned by the caddy user, but the CMS then does not have permission to access those files (which causes problems).

We can’t work out how to solve this “properly”.

In the screenshot, the CMS trying to use the 9f directory throws permission errors.

The temporary solution is to run a fix-permissions.sh script which does this:

sudo chown -R ${BASH_USER}:www-data .
sudo find . -type d -exec chmod ug+rwx {} \;
sudo find . -type f -exec chmod ug+rw {} \;
sudo find . -type d -exec chmod g+s {} \;

BASH_USER is admin, but does also work as www-data.

This solves things until the CMS next generates a new directory or file in there, and then… that seems to be owned by the caddy user again.

2. Error messages and/or full log output:

This isn’t strictly a Caddy error, more a “we have some configuration issue we can’t work out how to solve, and hope someone here has some insight”.

3. Caddy version:

v2.7.6 h1:w0NymbG2m9PcvKWsrXO6EEkY9Ru4FJK8uQbYcev1p3A=

4. How I installed and ran Caddy:

Via the official guide, adding a Debian repo, using apt.

a. System environment:

Debian Linux 12, PHP 8.2, Systemd.

Further details:

The system has a “limited user” called admin we use instead of root.

We created a directory out of which to serve multiple sites as follows:

sudo mkdir /websites
sudo chown admin:www-data websites

The Caddyfile is symlinked from in there:

cd /websites
ln -s /etc/caddy/Caddyfile Caddyfile
sudo chown admin:admin /etc/caddy/Caddyfile

All sites are checkout out as sub-directories inside that /websites, and all folders and files are owned by admin:www-data (except the Caddyfile itself).

PHP was installed via apt and we are using it via FPM.

Configs for Caddy and PHP are essentially default, but to clarify:

/etc/php/8.2/fpm/pool.d

user = www-data
group = www-data

/lib/systemd/system/caddy.service

User=caddy
Group=caddy

System users and the groups they belong to:

# groups caddy
caddy : caddy www-data

# groups admin
admin : admin sudo users

b. Command:

It just runs via systemd on system start.

c. Service/unit/compose file:

n/a

d. My complete Caddy config:

# Snippets (see: https://caddyserver.com/docs/caddyfile/concepts)

(staticFileCache) {
        @static {
                file
                path *.ico *.css *.js *.gif *.jpg *.jpeg *.webp *.png *.svg *.woff2
        }
        header @static Cache-Control max-age=5184000
}

(blitzNoQueryString) {
        @blitzCache {
                method GET
                not expression {query} != ''
        }
        route @blitzCache {
                try_files /cache/blitz/{host}{uri}/index.html {path} {path}/index.php?{query}
        }
}

(wordpressBruteForce) {
        @blockList {
                path /wp-*
        }
        abort @blockList
        @block2 {
                path *wp-includes*
        }
        abort @block2
}

thewebsite.url {
        root * /websites/thewebsite/public
        encode gzip

        file_server

        templates {
                between <!--#caddy -->
        }

        import staticFileCache
        import wordpressBruteForce
        import blitzNoQueryString

        log {
                output file /websites/_logs/thewebsite.log
        }

        php_fastcgi unix//run/php/php8.2-fpm.sock
}

5. Links to relevant resources:

We’re using CraftCMS - which does have a CLI so I’m also asking over there to see if there’s some mechanism whereby this might happen because of the CLI functionality and how that might be being called in some manner that results in caddy being the user.

That’s pretty wonky. Caddy itself doesn’t write the files, presumably the PHP-FPM process does, so I’d expect www-data to own new stuff. u+s doesn’t even work on Debian, I don’t think, so it wouldn’t be that… Do you have any ACLs set at all on the files/folders?

2 Likes

Yeah, its so odd - Brad from the Craft CMS side of things thinks it looks set up right too.

It’s just default Debian, no ACLs or anything. It’s set up as literally just a Debian install, then Caddy installed, MySQL installed, firewall and fail2ban, and that’s it.

The only thing I can think of atm that might be “unusual” is that we have CRON scripts that run to trigger the CMS’s job system, and I’m wondering if that does some different path of some sort - but it’s very confusing. The CRON is owned by admin and all it does is call the CraftCLI to trigger job processing:

3 1 * * * /usr/bin/php /websites/thewebsite/craft blitz/cache/refresh-expired

Can you replicate this on a validated fresh install?

i.e. do you think you could give us a strict set of instructions from zero to error in, say, a Debian VM with fresh Craft CMS?

At the moment no :confused: We’re seeing this on a small handful of projects, but its one of those things that shows up only when we’re on production, and the only reason we’re noticing it is because the CMS is failing to send out emails when visitors fill out forms - and most projects don’t have that functionality.

I’ll see if I can get a more reduced test case, and a reproducable trigger, but I’m not understanding how any folder or file would be created under the server user’s ownership.

Ooo, there is one other thing that might be the cause!

We use a Daemonized Queue Runner for job control (as recommended), so we have:
/etc/systemd/system/REPO-queue-worker.service

# /etc/systemd/system/REPO-queue-worker.service

[Unit]
Description=REPO Craft CMS Queue
After=network.target
After=mysql.service
Requires=mysql.service
# (...or postgresql.service!)

[Service]
# User + Group should agree with HTTP processes:
User=caddy
Group=www-data
ExecStart=/usr/bin/nice -n 10 /usr/bin/php /websites/REPO/craft queue/listen --verbose=1 --color=0
# Only restart after unexpected failures:
Restart=on-failure
# Extend time between restart attempts after a failure:
RestartSec=120

[Install]
WantedBy=multi-user.target

Which is enabled. That’s set up “correctly” I think, but it does specify that the User is caddy (because it should agree with the http process, and that’s caddy from my understanding - or does “http process” in this case mean what PHP would be running as, not what Caddy would be running as)… is that wrong in my set up? Should that user be admin or www-data for the User?

Here are a few of the things Craft uses the queue for:

  • Updating search indexes;
  • Resaving elements in bulk;
  • Generating image transforms;
  • Propagating elements between sites;
  • Executing a Find and Replace operation across all site content (started via Utilities);

https://craftcms.com/docs/5.x/system/queue.html

Seems like it could easily be the culprit if it’s creating these directories as part of these long-running async services.

Craft CMS runs as www-data:www-data (by virtue of being executed by the FPM pool) and therefore so should the queue runner.

I’d say “User + Group should agree with HTTP processes” is a bit of a mislead, here, because the HTTP components are handled entirely by Caddy as a separate concern to the dynamic application services handled by Craft CMS. The Queue Runner wants to act in concert with Craft CMS, therefore it should be executed with the same UID/GID as Craft CMS, not Caddy, which you can treat exactly like a reverse proxy in this instance. Caddy itself doesn’t touch your files, it just talks to the FPM interface to get a response to web requests.

This would be a different story if you were running Apache (or maybe FrankenPHP) because under that deployment, the dynamic scripts are executed by the web server process itself (rather than FPM).

2 Likes

Great, thanks - I’ll try changing the Service user to www-data and see if that stops the problem.

That’s my mis-understanding of the queue-runner comment then - it’d have been clearer to me if it was “the same as the PHP process”, but as you say it’s a slightly different model depending on how things are set up! Ok, I suspect that will turn out to have been the problem all along.

Thanks for running through this with me, much appreciated!

I’ll impement the change and then give it a day or two and see if we’ve stopped getting the odd folder with the wrong owners, then confirm the solution if that’s sorted.

2 Likes

Ok, as summary in case anyone else stumbles on this:

It does look like the issue was the configuration of the SystemD service (the daemonized queue runner), which CraftCMS recommends setting up for more robust queue/job management.

When using Caddy, the User and Group for the Service should match the user and group used by PHP itself, and not that of Caddy.

So, assuming you’re running the defaults for Caddy and PHP-FPM, this would be correct:

# /etc/systemd/system/REPO-queue-worker.service

[Unit]
Description=REPO Craft CMS Queue
After=network.target
After=mysql.service
Requires=mysql.service
# (...or postgresql.service!)

[Service]
# User + Group should agree with what PHP-FPM is configured for:
User=www-data
Group=www-data
ExecStart=/usr/bin/nice -n 10 /usr/bin/php /websites/REPO/craft queue/listen --verbose=1 --color=0
# Only restart after unexpected failures:
Restart=on-failure
# Extend time between restart attempts after a failure:
RestartSec=120

[Install]
WantedBy=multi-user.target
1 Like