Help configuring PHP-FPM Status for Caddy2?

MattVCA · April 24, 2023, 10:10am

1. The problem I’m having:

Minor background:

We have a server where PHP is “falling over” from time to time. I’ve been able to see that something is causing PHP-FPM to spin up child processes until it hits the max_children, at which point PHP stops serving anything to any site until PHP is restarted.

The problem I’m trying to solve:

As all the sites share the same PHP-FPM pool (www), there’s no useful information for me to track down where the problem is stemming from. So I am trying to get the PHP-FPM Status page configured, which I am hoping will give me more information so I can at least see which site hosted on the server and possibly what script might be the root cause of the behaviour.

I can’t seem to work out how to configure that in Caddy2, and must admit that this may be as much “not having a clear understanding of the guts of how this works” as it is about “specifically Caddy’s configuration”. The only things I’ve found relating to this are seemingly for “old Caddy”.

For clarity, I can confirm I have edited /etc/php/8.1/fpm/pool.d/www.conf to enable the line pm.status_path = /status and have restarted PHP-FPM via systemctl restart php8.1-fpm. I have also restarted Caddy via sudo systemctl daemon-reload && systemctl reload caddy && systemctl start caddy

2. Error messages and/or full log output:

No ‘Caddy’ errors, I just get a 404 for https://staging.domain.ext/status?html&full

(URL anonymised, but “correct”)

3. Caddy version:

v2.6.4 h1:2hwYqiRwk1tf3VruhMpLcYTg+11fCdr8S3jhNAdnPy8=

4. How I installed and ran Caddy:

Installed Ubuntu 22.04LTS
Set up UFW
Set up Fail2Ban
sudo apt install -y debian-keyring debian-archive-keyring apt-transport-https
curl -1sLf 'https://dl.cloudsmith.io/public/caddy/stable/gpg.key' | sudo gpg --dearmor -o /usr/share/keyrings/caddy-stable-archive-keyring.gpg
curl -1sLf 'https://dl.cloudsmith.io/public/caddy/stable/debian.deb.txt' | sudo tee /etc/apt/sources.list.d/caddy-stable.list
sudo apt update && sudo apt install caddy git

a. System environment:

Ubuntu 22.04LTS

b. Command:

It runs itself on boot?

c. Service/unit/compose file:

Not relevant

d. My complete Caddy config:

# The Caddyfile is an easy way to configure your Caddy web server.
#
# Unless the file starts with a global options block, the first
# uncommented line is always the address of your site.
#
# To use your own domain name (with automatic HTTPS), first make
# sure your domain's A/AAAA DNS records are properly pointed to
# this machine's public IP, then replace ":80" below with your
# domain name.
#

# Snippets (see: https://caddyserver.com/docs/caddyfile/concepts)
(staticfilecache) {
        @static {
                file
                path *.ico *.css *.js *.gif *.jpg *.jpeg *.webp *.png *.svg *.woff2
        }
        header @static Cache-Control max-age=5184000
}

staging.domain.ext {
        root * /websites/staging
        encode gzip zstd

        log {
                output file /websites/_logs/staging.log
        }

        php_fastcgi /status unix//run/php/php8.1-fpm.sock {
                env SCRIPT_NAME /status
        }

        file_server
}

other.website.url {
        root * /websites/craft4-vca/web
        encode gzip zstd
        php_fastcgi unix//run/php/php8.1-fpm.sock

        file_server
        import staticfilecache

        log {
                output file /websites/_logs/craft4-vca.log
        }
}

Above, the domains are not “real” just for security - but in reality both sites work. The first, staging.domain.ext doesn’t have any PHP scripts but it the one I want to be able to access the FPM Status from. The second isn’t related to this problem but I can confirm that it hosts PHP pages and runs fine.

5. Links to relevant resources:

PHP: Status Page - Manual PHP’s FPM-Status docs
An nGinx set up for it How to Enable and Monitor PHP-FPM Status in Nginx

francislavoie · April 25, 2023, 12:21am

The problem is Caddy is looking for a file on disk that matches the script you’re trying to run, and it will also only pass requests to fastcgi if the rewritten file path ends with .php

To work around this, you could do this, i.e. manually setting up a proxy with the fastcgi transport to skip all the custom behaviour of the php_fastcgi directive for the /status path specifically:

staging.domain.ext {
	root * /websites/staging
	encode gzip zstd

	log {
		output file /websites/_logs/staging.log
	}

	reverse_proxy /status unix//run/php/php8.1-fpm.sock {
		transport fastcgi
	}

	php_fastcgi unix//run/php/php8.1-fpm.sock
	file_server
}

MattVCA · April 25, 2023, 7:56am

Thank you - that’s getting me somewhere! Though now I have a “Access denied.” response from it… I’ll see what I can find out about what’s causing that.

[edit]
Nope, I can’t work out why that’s resulting in a 403 error. Would seem to be permissions of some sort but I can not track down why the fastcgi / reverse_proxy is being returned a 403. I did try adding capture_stderr as a flag, but that broke the Caddyfile on that line and dropped the follwing in the jounalctl log Error: adapting config using caddyfile: parsing caddyfile tokens for 'reverse_proxy'.

francislavoie · April 25, 2023, 5:21pm

You need to add it within transport, so like this:

	reverse_proxy /status unix//run/php/php8.1-fpm.sock {
		transport fastcgi {
			capture_stderr
		}
	}

MattVCA · April 25, 2023, 6:09pm

Agh, I’d not put that in braces, thought it must go as a sibling. Ok, that’s validating now - thanks.

Was hoping that I’d see more in the /websites/_logs/staging.log by adding this, but apparently not. And nothing in journalctl -e -u caddy that seems to be of help either. Hmm.

Not sure why I’m getting Access Denied for /status requests.

Interestingly, with the configuration as follows:

staging.website.ext {
        root * /websites/staging
        encode gzip zstd

        log {
                output file /websites/_logs/staging.log
        }

        reverse_proxy /status unix//run/php/php8.1-fpm.sock {
                transport fastcgi {
                        capture_stderr
                }
        }

        php_fastcgi unix//run/php/php8.1-fpm.sock

        file_server
}

accessing https://staging.website.ext/index.html loads as expected, but not accessing https://staging.website.ext/phpinfo.php which is a test script that just contains:

<?php phpinfo(); ?>

That’s being prompted as a download instead of actually showing the page - which would seem to me as though it’s not hooking into PHP-FPM.

Regardless, requesting https://staging.website.ext/status in a browser is a 403 Access denied (which is my core problem atm).

PHP is definitely running and working as there are other declarations for sites in this Caddyfile that use the same php_fastcgi unix//run/php/php8.1-fpm.sock line, and those are running OK.

francislavoie · April 25, 2023, 6:21pm

Turn on the debug global option, see what rewrites are happening, what’s being sent to php-fpm.

MattVCA · May 4, 2023, 10:10am

I finally have this working - here’s how to get the PHP: Status Page - Manual functionality working in Caddy2, so that you’re able to see more details about what the PHP-FPM processes are actually doing.

SOLUTION:

staging.website.ext {
        root * /websites/staging
        encode gzip zstd

        log {
                output file /websites/_logs/staging.log
        }

        # Handle the /status URL specifically, which has been configured following
        # https://www.php.net/manual/en/fpm.status.php
        reverse_proxy /status unix//run/php/php8.1-fpm.sock {
                transport fastcgi {
                        env SCRIPT_NAME /status
                }
        }

        # Handle normal PHP stuff by passing it to PHP-FMP
        php_fastcgi unix//run/php/php8.1-fpm.sock

        # Serve files that exist straight
        file_server
}

francislavoie · May 4, 2023, 6:27pm

So, turns out capture_stderr was broken

Sorry about that. It’s been fixed in this PR though:

github.com/caddyserver/caddy

fastcgi: Fix `capture_stderr`

caddyserver:master ← eanavitarte:patch-1

opened 01:07AM - 03 May 23 UTC

eanavitarte

+1 -0

Hello, first of all I love Caddy! I hope this helps. # THE PROBLEM: In curre…nt 83b2697, the option "capture_stderr" is not working on the directive "php_fastcgi". # HOW TO TEST THE PROBLEM: If you want to test it, just create a php file with syntax error of any type and pass it over php_fastcgi. # WHEN STARTED THE PROBLEM: The problem started in the transition between the f2a7e7c commit, and the current 83b2697. # WHY STARTED THE PROBLEM: On 2022 Sept 2, it was launched the commit 83b2697 which changed some files on fastcgi.go. These changes was titled: fastcgi: Optimize FastCGI transport (#4978). It was changed a lot of internal functions and they were replaced with a more distributed logic into separate files (which is good). On those changes was removed a conditional that evaluates "t.CaptureStderr" (line 164, f2a7e7c, [fastcgi.go](https://github.com/caddyserver/caddy/compare/f2a7e7c966c42b3a09bc8414136af0439fd7b630...83b26975bd9330c4cfc44d52b106da739240e72b#diff-3a7a275287ccf8af168eb64afaff75c8b96a1d231569609c35f2e5db488c1c8fL164)). This conditional set the correct logger to be passed to the client (fcgiBackend in f2a7e7c, [fastcgi.go](https://github.com/caddyserver/caddy/compare/f2a7e7c966c42b3a09bc8414136af0439fd7b630...83b26975bd9330c4cfc44d52b106da739240e72b#diff-3a7a275287ccf8af168eb64afaff75c8b96a1d231569609c35f2e5db488c1c8fL165)) instance. So the logger was always set to "noopLogger" (83b2697, [client.go](https://github.com/caddyserver/caddy/blob/master/modules/caddyhttp/reverseproxy/fastcgi/client.go#L239)) and no matters if you set capture_stderr, it was never registered into log. # HOW TECHNICALLY OCCURRED THE PROBLEM: Those changes on the "t.CaptureStderr" (line 164, f2a7e7c) behaviour, probably were done because it was assumed that the evaluation of "c.stderr" (line 244, 83b2697, [client.go](https://github.com/caddyserver/caddy/blob/master/modules/caddyhttp/reverseproxy/fastcgi/client.go#L244)) on client.go (another change introduced on 83b2697) would have the same behaviour, and it is right in some way. I mean, if "c.stderr" was set, it will work as expected. **The problem** is that nowhere it was passed the value of "t.CaptureStderr" to the client instance created (line 170, 83b2697, [fastgci.go](https://github.com/caddyserver/caddy/blob/master/modules/caddyhttp/reverseproxy/fastcgi/fastcgi.go#L170)), and so, it was never evaluated on "c.stderr" (line 244, 83b2697). # THE SOLUTION: I must to say that, although it has taken me quite some time to find the solution, it is actually quite simple. For fixing it I just pass the value in the creation of the client instance, as show in this pull request. I hope this little enhancement could help a bit to us. If you can cite me on the next commit 🥹

MattVCA · May 5, 2023, 7:44am

Lol! I did wonder, as nothing seemed to happen, but I also couldn’t work out exactly where it was supposed to output anything from the docs either (I assumed it’d be in the standard log, but nothing seemed to differ).

PS: Thanks for all the help! Much appreciated.

system · June 4, 2023, 7:44am

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.