How to use the expanded form of php_fastcgi?

basil · July 27, 2020, 7:30am

1. Caddy version (`caddy version`):

v2.1.1 h1:X9k1+ehZPYYrSqBvf/ocUgdLSRIuiNiMo7CvyGUQKeA=

2. How I run Caddy:

a. System environment:

Within a jail under FreeNAS 11.3-U4

b. Command:

service caddy start

c. Service/unit/compose file:

The startup script used during jail initialisation as provided by the Caddy V2 port maintainer at 246623 – www/caddy: update 1.0.4 -> 2.1.1. The FreeBSD Caddy port is awaiting approval for general release.

#!/bin/sh
#
# $FreeBSD$
#

# PROVIDE: caddy
# REQUIRE: LOGIN DAEMON NETWORKING
# KEYWORD: shutdown

# Add the following lines to /etc/rc.conf.local or /etc/rc.conf
# to enable this service:
# caddy_enable (bool):   Set to NO by default. Set it to YES to enable caddy.
#
# caddy_config (string): Optional full path for caddy config file
# caddy_adapter (string):  Optional adapter type if the configuration is not in caddyfile format
# caddy_extra_flags (string):  Optional flags passed to caddy start

. /etc/rc.subr

name=caddy
rcvar=caddy_enable
desc="Caddy 2 is a powerful, enterprise-ready, open source web server with automatic HTTPS written in Go"

load_rc_config $name

# Defaults
: ${caddy_enable:=NO}
: ${caddy_config:="/usr/local/etc/Caddyfile"}
: ${caddy_adapter:=caddyfile}
: ${caddy_extra_flags:=""}

command="/usr/local/bin/${name}"
caddy_flags="--config ${caddy_config} --adapter ${caddy_adapter}"
pidfile=/var/run/${name}.pid

# Extra Commands
extra_commands="validate reload"
start_cmd="${command} start ${caddy_flags} ${caddy_extra_flags} --pidfile /var/run/${name}.pid"
validate_cmd="${command} validate ${caddy_flags}"
reload_cmd="${command} reload ${caddy_flags}"
stop_cmd="${command} stop"
restart_cmd="${stop_cmd} && ${start_cmd}"

run_rc_command "$1"

d. My complete Caddyfile or JSON config:

This is pretty much the solution provided at Help to migrate Caddyfile V1 to V2 for Nextcloud - #30 by Dougy

{
        debug
        acme_ca https://acme-staging-v02.api.letsencrypt.org/directory
        email stu.pitt@dumas.com
}

cloud2.udance.com.au {

        root    * /usr/local/www/nextcloud
        file_server
        log {
                output file     /var/log/cloud2.udance.com.au.log
                format single_field common_log
        }

        php_fastcgi 127.0.0.1:9000

        header {
                # enable HSTS
                Strict-Transport-Security max-age=31536000;
        }

        redir /.well-known/carddav /remote.php/dav 301
        redir /.well-known/caldav /remote.php/dav 301

        # .htaccess / data / config / ... shouldn't be accessible from outside
        @forbidden {
                path    /.htaccess
                path    /data/*
                path    /config/*
                path    /db_structure
                path    /.xml
                path    /README
                path    /3rdparty/*
                path    /lib/*
                path    /templates/*
                path    /occ
                path    /console.php
        }

        respond @forbidden 404

}

3. The problem I’m having:

Caddy starts automatically when the jail is started, but then aborts abruptly when the Nextcloud service is accessed the first time. If Caddy is then manually started, the Nextcloud service can be accessed without any further issue. The pattern is reproducible as follows:

Start or restart the jail
From within the jail, service caddy status shows Caddy is running.
From a browser, an attempt to connect to the cloud service is refused.
From within the jail, service caddy status shows Caddy is no longer running.
Start Caddy manually service caddy start .
From a browser, connecting to the cloud service succeeds this time.
From within the jail, service caddy status shows Caddy is still running.

4. Error messages and/or full log output:

When Caddy aborts when first accessing the Nextcloud service after jail initialisation, cloud2.udance.com.au.log does not exist.
When Caddy is then started manually and the Nextcloud service accessed, cloud2.udance.com.au contents are:

10.1.1.149 - - [27/Jul/2020:14:50:50 +0800] "GET /index.php/login HTTP/2.0" 200 12189
10.1.1.149 - - [27/Jul/2020:14:50:50 +0800] "GET /apps/files_pdfviewer/css/style.css?v=4fda350b-0 HTTP/2.0" 200 346
10.1.1.149 - - [27/Jul/2020:14:50:50 +0800] "GET /core/css/guest.css?v=c439af14-0 HTTP/2.0" 200 20949
10.1.1.149 - - [27/Jul/2020:14:50:51 +0800] "GET /apps/files_sharing/js/dist/main.js?v=c439af14-0 HTTP/2.0" 200 1178
10.1.1.149 - - [27/Jul/2020:14:50:51 +0800] "GET /core/search/js/search.js?v=c439af14-0 HTTP/2.0" 200 5560
10.1.1.149 - - [27/Jul/2020:14:50:51 +0800] "GET /core/js/files/fileinfo.js?v=c439af14-0 HTTP/2.0" 200 2142
10.1.1.149 - - [27/Jul/2020:14:50:51 +0800] "GET /apps/files_pdfviewer/js/previewplugin.js?v=c439af14-0 HTTP/2.0" 200 5929
10.1.1.149 - - [27/Jul/2020:14:50:51 +0800] "GET /apps/files_videoplayer/js/main.js?v=c439af14-0 HTTP/2.0" 200 6350
10.1.1.149 - - [27/Jul/2020:14:50:51 +0800] "GET /core/search/js/searchprovider.js?v=c439af14-0 HTTP/2.0" 200 12444
10.1.1.149 - - [27/Jul/2020:14:50:51 +0800] "GET /core/js/files/client.js?v=c439af14-0 HTTP/2.0" 200 23878
10.1.1.149 - - [27/Jul/2020:14:50:51 +0800] "GET /core/js/dist/main.js?v=c439af14-0 HTTP/2.0" 200 1397155
10.1.1.149 - - [27/Jul/2020:14:50:51 +0800] "GET /core/js/dist/login.js?v=c439af14-0 HTTP/2.0" 200 800906
10.1.1.149 - - [27/Jul/2020:14:50:51 +0800] "GET /index.php/apps/theming/styles?v=0 HTTP/2.0" 200 1275
10.1.1.149 - - [27/Jul/2020:14:50:51 +0800] "GET /index.php/js/core/merged-template-prepend.js?v=c439af14-0 HTTP/2.0" 200 2982
10.1.1.149 - - [27/Jul/2020:14:50:51 +0800] "GET /index.php/apps/theming/js/theming?v=0 HTTP/2.0" 200 232
10.1.1.149 - - [27/Jul/2020:14:50:51 +0800] "GET /index.php/apps/accessibility/js/accessibility?v=0 HTTP/2.0" 200 85
10.1.1.149 - - [27/Jul/2020:14:50:51 +0800] "GET /core/img/background.png?v=0 HTTP/2.0" 200 35361
10.1.1.149 - - [27/Jul/2020:14:50:51 +0800] "GET /index.php/svg/core/logo/logo?color=fff&v=1 HTTP/2.0" 200 321
10.1.1.149 - - [27/Jul/2020:14:50:51 +0800] "GET /core/img/actions/confirm-white.svg?v=2 HTTP/2.0" 200 405
10.1.1.149 - - [27/Jul/2020:14:50:51 +0800] "GET /core/img/actions/toggle.svg HTTP/2.0" 200 308
10.1.1.149 - - [27/Jul/2020:14:50:51 +0800] "GET /core/img/loading-dark.gif HTTP/2.0" 200 4683
10.1.1.149 - - [27/Jul/2020:14:50:52 +0800] "GET /index.php/apps/theming/favicon?v=0 HTTP/2.0" 200 0

5. What I already tried:

If I replace the Caddyfile contents with a V2 code block that I know works on my network e.g.

office.udance.com.au {
  encode gzip

  reverse_proxy https://10.1.1.17 {
    transport http {
      tls_insecure_skip_verify
    }
  }
}

…and then repeat the steps above (step 3), Caddy does not abort! It seems there is some issue hidden with the Nextcloud Caddyfile code block.

At this point, I’ll pause and see if there is something obvious, that you can see, that I may have overlooked, or if there is something you would like me to try out first.

5a. What I would like to try:

When I look more closely at the thread Help to migrate Caddyfile V1 to V2 for Nextcloud, the culprit is suspiciously beginning to look like the php_fastcgi directive. It’s a focal point of that thread with a fix that I believe has recently been merged into Caddy 2.1.

However, the issue that I and others in the FreeNAS community are observing is quite subtle, only occurring on jail initialisation. It may not be something that the Caddy developers would have readily picked up.

There may be a way to definitely pinpoint the issue to php_fastcgi and possibly the only way to confirm the existence of a bug. There is an expanded form of the php_fastcgi directive. If the php_fastcgi directive is replaced with the expanded form in the Caddyfile, and if Caddy then works with it, it would establish the existence of a bug in the directive.

If this seems like a reasonable approach, then I would really appreciate some help ‘filling in the blanks’ for the expanded form as it does not make a lot of sense to me:

route {
	# Add trailing slash for directory requests
	@canonicalPath {
		file {
			try_files {path}/index.php
		}
		not path */
	}
	redir @canonicalPath {path}/ 308

	# If the requested file does not exist, try index files
	@indexFiles {
		file {
			try_files {path} {path}/index.php index.php
			split_path .php
		}
	}
	rewrite @indexFiles {http.matchers.file.relative}

	# Proxy PHP files to the FastCGI responder
	@phpFiles {
		path *.php
	}
	reverse_proxy @phpFiles <php-fpm_gateway> {
		transport fastcgi {
			split .php
		}
	}
}

At this stage, the only other bit of useful information I can provide to assist in this endeavour is that index.php and other php files reside in /usr/local/www/nextcloud i.e.

6. Links to relevant resources:

francislavoie · July 27, 2020, 7:42am

Those logs you’ve shown are only the access logs. Caddy prints out its errors to stdout and stderr. I’m not familiar with BSD, so I’m not sure how those are handled by the service manager, but that should be the first place to look if you’re seeing Caddy die unexpectedly.

P.S. I took a look at that bugzilla thread, near the end someone asked about environment variables. If you’d like to, you could reply that it’s possible to specify the --envfile flag to the caddy run command to have Caddy read additional env information.

basil · July 27, 2020, 8:30am

I had no idea, so I went looking and found nothing useful so I made some adjustments to the service file (step 2c above) to direct stdout and stderr to a file. I restarted the jail and now can’t make Caddy fall over!

It seems that when Caddy started in the background, such as when a jail starts or restarts, there was nowhere for stdout and stderr to go so Caddy would cough, splutter and die. It wasn’t an issue when Caddy was started manually as output was directed to the terminal. By directing Caddy output to a file when the jail restarts, Caddy now happily hums along blissfully unaware that there is any issue at all!

@francislavoie Thank you for that genius tip! I wasn’t looking forward to using the expanded form of php_fastcgi.

I’ll provide some feedback to the FreeBSD Caddy port maintainer to suggest an amendment to the service initialisation file.

Afterthought: My logic may not be completely bulletproof. It doesn’t explain why output redirection was not necessary for step 5 above. Any ideas? Is there still an issue lurking somewhere in the shadows?

francislavoie · July 27, 2020, 1:19pm

I find that pretty strange frankly. I’m just gonna chalk this up to “I don’t know enough about BSD”

Glad it’s working for you now!

danb35 · July 27, 2020, 2:11pm

I’d expect it’s because when you run service caddy start from the shell prompt, there’s a place for the output to go (both stdout and stderr default to your terminal), but when it’s launched non-interactively, there isn’t. It tries to send the output to stdout, stdout is undefined, and that’s when it dies.

Though why Caddy doesn’t have its own logging facility is a little curious.

francislavoie · July 27, 2020, 2:17pm

Caddy does have full configuration of logging in JSON, but only access log config in the Caddyfile:

This is just a situation of “we haven’t gotten there yet”. Designing the right approach for the global logging is tricky.

basil · July 27, 2020, 2:19pm

I wasn’t clear enough. I can’t explain what I’m seeing under 5. What I already tried:

basil · July 29, 2020, 3:10pm

On a lighter note, did anyone pick up on the ‘hidden egg’ in my Caddyfile?

Hint: Check the email address

system · August 26, 2020, 7:30am

This topic was automatically closed after 30 days. New replies are no longer allowed.