Process check: Spreading the Caddy Word

Whitestrake · July 28, 2021, 8:31am

Depends entirely on whether or not the web admin wants them to be accessible!

Like, with not file present, you could manually request /2020/ and get the 2020 index, but if you request / you’ll get 2021’s index.

Ultimately I don’t think people go around manually prefixing possible version numbers to their URIs, so the risk of a user mistakenly finding your other versions are low; unless you need to keep them inaccessible as a security measure, I’d leave not file present for the sheer accessibility of the web admin / maintainer being able to easily see them.

basil · July 29, 2021, 6:41am

Right, time to test out the theory. I set up a test WordPress site. By default, the installation method I use placed the WordPress files in the webroot. Before moving the files to a subdirectory, I made sure I could access various parts of the site. No problems here.

It’s worth noting what appears in the address bar in the second screenshot.

Next, I move the WordPress files into the subdirectory my_subdir. I confirm that I no longer have access to the site
.

I update the site block in the Caddyfile with the code below and reload Caddy.

@subdir {
  not path /my_subdir/*
  not file
}
rewrite @subdir /my_subdir{uri}

Refreshing the browser, the site bursts into life. However, I now have a problem accessing subpages of the site.

There’s some other issue going on. After further research, I find the issue has to do with permalinks. This extract from the WordPress support article Using Permalinks.

When you create or update a “pretty” permalink structure, WordPress will generate rewrite rules and attempt to insert them into the proper .htaccess file. If it can’t, it will say something like You should update your .htaccess now and print out the rules for you to copy and paste into the file (put them at the end).

You’ll probably need to do this only once, because WordPress does the rewriting internally. If you ever move your WordPress home directory (Site address), you’ll need to repeat this step.

I have a look in the permalinks section of the test site and note the following.

A closer look at the rewrite rules…

<IfModule mod_rewrite.c>
RewriteEngine On
RewriteRule .* - [E=HTTP_AUTHORIZATION:%{HTTP:Authorization}]
RewriteBase /
RewriteRule ^index\.php$ - [L]
RewriteCond %{REQUEST_FILENAME} !-f
RewriteCond %{REQUEST_FILENAME} !-d
RewriteRule . /index.php [L]
</IfModule>

Some bits I recognise; some bits I don’t.

I also notice there is the (one and only!) WordPress reference to Caddy in the permalinks article. This extract…

Pretty permalinks are available under:

Apache web server with the mod_rewrite module

Nginx using try-files, e.g. according to this tutorial

Hiawatha web server with UrlToolkit support enabled.

Lighttpd using a 404 handler or mod_rewrite

Caddy using rewrite, e.g. according to this tutorial

Clicking the link to the tutorial takes me to a Caddy V1 article. (Looks like the WordPress permalinks article is the next one I should try to convert). This is the V1 rewrite rule: I don’t really understand what it’s telling me (e.g. what is this {query} bit?). I’m now stuck again. Help!

    # Routing for WordPress
    rewrite /{
        to {path} {path}/ /index.php?{query}
    }

EDIT: I’ve just made the test site https://xxx.udance.com.au accessible online.

francislavoie · July 29, 2021, 2:17pm

Essentially that’s the try_files pattern.

The php_fastcgi directive should already be doing that for you though. See the expanded form (the rewrite @indexFiles is the “long form” of the try_files directive… which uses the file matcher with the try_files option)

The {query} part is to preserve the request query (the bit after ? in a URL) after the rewrite, because otherwise it would be lost (i.e. copy it from the original URL into the rewritten URL)

basil · July 29, 2021, 10:03pm

@francislavoie Thanks for responding

Does this mean I shouldn’t have to take any further action i.e. the permalinks issue should inherently be addressed in Caddy V2? If so, ideas on how to proceed from here e.g. debug?

OK. I’m still getting my head around this. Sometimes {URI} is used as in…

rewrite @subdir /my_subdir{uri}

I’ve recently become comfortable with this after you explained it to me in Www handling - Use the same multiple site definition or use redirection? - #10 by francislavoie
Now you’re indicating sometimes I need to be more specific and use a subset of the URI. I need to think about this a bit more. How do I decide when I should use one or the other?

francislavoie · July 29, 2021, 11:21pm

{uri} is the entire URI (path + query). Use that if you want to prefix it with something. {path} is just the path. {query} is just the query.

The way the fastcgi transport works, Caddy sends the entire original URI as requested by the client (browser) as an environment variable to php-fpm, among many other environment variables.

But index.php is the actual script that runs. That script is the entrypoint to the PHP app. It will do routing based on multiple different factors, some apps do it differently than others. Sometimes the apps want the query to be preserved in the rewritten path, others will just look at the original request URI.

I don’t think {query} will make a difference in this case; I was just explaining what that Caddyfile snippet from v1 did.

Hopefully. I don’t use WordPress so I can’t say for sure. You’ll have to try it to find out. (IMO WordPress is badsadbad, I very much prefer frameworks like Laravel, or just rolling my own from well maintained, modern libraries)

Whitestrake · July 29, 2021, 11:37pm

This part:

	# If the requested file does not exist, try index files
	@indexFiles file {
		try_files {path} {path}/index.php index.php
		split_path .php
	}
	rewrite @indexFiles {http.matchers.file.relative}

Might need to be overwritten in this case as we’ve put the main site in an arbitrary subdirectory. e.g.

try_files {path} {path}/index.php /my_subdir/index.php

basil · July 29, 2021, 11:46pm

OK. I’ve learnt something new.

You may very well be right, but if this infografic is to be believed:

WordPress powers 40% of the internet.
Around 64% of CMS sites are WordPress.
Around 28% of WordPress sites run e-commerce.
Around 75% of hacked CMS sites were built on WordPress

On that last point, I’d like to add that I’m so glad I have my WP sites sitting behind Caddy!

I think the reason WP is so popular is that it’s accessible to the average person (like myself) who isn’t comfortable rolling their own site.

Whitestrake · July 29, 2021, 11:48pm

To follow up, I think you might be most grateful people who have no business coding are using WordPress instead of trying to roll their own site

francislavoie · July 30, 2021, 12:00am

Yeah. I realize why it’s popular… but I just wish it wasn’t. It’s pretty terribly maintained and is full of holes.

I’ll just uh, recommend https://statamic.com/ as an alternative. See Statamic vs WordPress

basil · July 30, 2021, 12:10am

Right, time to refocus on the primary objective of this exercise and that is to try to figure out a way to get the Caddy word out there (for good and bad apps!) . I’m going to try Mathew’s suggestion and I’ll report back shortly, but first, an early morning cuppa

This is what I love about you. You’re such a purist. Thank goodness for this trait! Without it, Caddy wouldn’t be the amazing product that it is.

Not only is the WP app less than ideal, but the support documentation is woeful as I’ve indicated in an earlier post Process check: Spreading the Caddy Word - #6 by basil. There’s been a recent and positive development though. I’m one step closer to the WP documentation team, which should make it easier to sell the idea of linking to Caddy wiki articles. Check out the WordPress forum thread Giving WordPress Its Own Directory.

basil · July 30, 2021, 2:38am

I’m assuming I have to use the expanded form of php_fastcgi?

I’ve created a snippet to hide the detail

(php_fastcgi2) {
  route {
    # Add trailing slash for directory requests
    @canonicalPath {
      file {path}/index.php
      not path */
    }
    redir @canonicalPath {path}/ 308

    # If the requested file does not exist, try index files
#    @indexFiles file {
#    try_files {path} {path}/index.php index.php
#      split_path .PHP
#    }
#    rewrite @indexFiles {http.matchers.file.relative}
    try_files {path} {path}/index.php /my_subdir/index.php

    # Proxy PHP files to the FastCGI responder
    @phpFiles path *.php
    reverse_proxy @phpFiles {arg.0} {
      transport fastcgi {
    	split .PHP
      }
    }
  }
}

I’m not sure how to include environmental variables within this structure? e.g.

  php_fastcgi 127.0.0.1:9000 {
    env SERVER_PORT 80
  }

francislavoie · July 30, 2021, 4:20am

They go inside transport fastcgi.

Yes, but make sure you keep the same structure as the expanded form (long form of try_files). The split_path .php is important for it to actually work properly.

basil · July 30, 2021, 4:41am

I believe you mean like this?

(php_fastcgi2) {
  route {
    # Add trailing slash for directory requests
    @canonicalPath {
      file {path}/index.php
      not path */
    }
    redir @canonicalPath {path}/ 308

    # If the requested file does not exist, try index files
    @indexFiles file {
#    try_files {path} {path}/index.php index.php
      try_files {path} {path}/index.php /my_subdir/index.php
      split_path .PHP
    }
    rewrite @indexFiles {http.matchers.file.relative}

    # Proxy PHP files to the FastCGI responder
    @phpFiles path *.php
    reverse_proxy @phpFiles {args.0} {
      transport fastcgi {
        env SERVER_PORT 80
    	split .PHP
      }
    }
  }
}

francislavoie · July 30, 2021, 12:43pm

Yeah probably. I think you need lowercase .php though for both split parts. Linux filesystems are case sensitive.

basil · July 31, 2021, 12:15am

Reviewing post Process check: Spreading the Caddy Word - #15 by basil

When the WordPress files were relocated from the webroot to a subdirectory, and the site block in the Caddyfile updated to do an internal redirect…

@subdir {
  not path /my_subdir/*
  not file
}
rewrite @subdir /my_subdir{uri}

…the home page of the site remained accessible, but sub-pages became inaccessible. The issue wasn’t with permalinks. php_fastcgi handles permalinks intrinsically. For the sub-pages to become accessible again, the rewrite required an expanded version of php_fastcgi to be used.

This ‘hard-coding’ of ‘my_subdir’ causes a further problem though. Returning to an earlier discussion…

This breaks as a consequence. Can this be accommodated as well given that a reason for relocating WordPress files out of the webroot was to enable versioning? It seems to me like the hard-coding needs to be replaced with something relative, but I don’t know what that might be.

basil · July 31, 2021, 5:44am

These are tests I undertook to confirm the issue:

My webroot is defined as root * /usr/local/www/wordpress. From the root, I set up the test environment.

cp -r my_subdir 2020
chown -R www:www 2020
echo "Hello from root!" >> sample.txt
echo "Hello from 2020!" >> 2020/sample.txt
echo "Hello from my_subdir!" >> my_subdir/sample.txt
echo "Hello from default!" >> my_subdir/default.txt

Test 1

Confirm that this Caddy structure worked as intended:

@subdir {
  not path /my_subdir/*
  not file
}
rewrite @subdir /my_subdir{uri}

c14

c15

Yep, that looks good.

Test 2

Check that the modified php_fastcgi works for my_subdir.

That looks good too.

Test 3

Check that versioning works.

Oops!

At the beginning of this post, I created a ‘version’ of the WP site.

cp -r my_subdir 2020
chown -R www:www 2020

This was a gross oversimplification of what WP versioning entails. Refer to this independent article Moving a WordPress Root Install to a Subdirectory Install and Vice Versa for a sense of what might be involved… For this exercise, I don’t really want to go any further down the rabbit’s hole. From tests 1 and 2, I’m satisfied the Caddy equivalent code for the WP article is working correctly.

Next step: Writing the wiki article.

basil · August 1, 2021, 4:45pm

The wiki article has been drafted for review Using Caddy to give WordPress its own directory.

The final step is to lobby the WordPress documentation team to have it referenced on a WordPress support page.

basil · August 2, 2021, 3:20am

A request has been submitted to the WP doc team. See Using Caddy to Give WordPress Its Own Directory · Issue #332 · WordPress/HelpHub · GitHub

A related issue…

As the link refers to a Caddy V1 article, and as Caddy V1 is now obsolete, I’d like to get this bullet point changed This is what I’m proposing as the replacement text:

Pretty permalinks are available under:

Caddy automatically in a default WordPress installation.

Just further down in the WP article, in the section Using “Pretty” permalinks, add the following bullet point:

For Caddy, a web server that supports HTTPS natively, the rewrite rule may need to be adjusted if WordPress files are moved to a subdirectory. Refer to Using Caddy to give WordPress its own directory for details.

If you can improve on this wording, please have a go. I’ll submit a change request with issue #332.once we have agreement on the wording.

The draft wording is based on these follow-up comments from @francislavoie and @Whitestrake…

Whitestrake:

francislavoie:

The php_fastcgi directive should already be doing that for you though.

This part:
	# If the requested file does not exist, try index files
	@indexFiles file {
		try_files {path} {path}/index.php index.php
		split_path .php
	}
	rewrite @indexFiles {http.matchers.file.relative}
Might need to be overwritten in this case as we’ve put the main site in an arbitrary subdirectory. e.g.
try_files {path} {path}/index.php /my_subdir/index.php

basil · August 2, 2021, 1:26pm

I’m at the tail end of the process check. I followed these four steps:

Pick a page to transform - Find an application support page to do a Caddy makeover on.
Track the transformation - Create a Caddy forum thread for tracking the conversion, testing, and for community and Caddy core maintainer input.
Wikify the result - Summarise the result in a Caddy wiki article
Lobby for inclusion - Contact the application document team to include a reference to the Caddy wiki article on the original page.

However, I realise after reviewing a few other application pages, step 3 may not be appropriate in many situations. For example, consider the WP article Brute Force Attacks. On this page, several proxy servers are mentioned already. The idea here would be to add Caddy to that list. In light of this, a revised process might look like the following:

Pick a page - Find an application page to make Caddy ready.
Discuss the approach - Create a Caddy forum thread for reviewing the approach, Caddy code to use, community and Caddy core maintainer input, and testing
Optionally, wikify the result - Summarise the result in a Caddy wiki article. This step is appropriate when the application page centres on a single proxy server.
Lobby for inclusion - Contact the application document team to have Caddy referenced on the application page.

basil · August 3, 2021, 3:48pm

I’ve just come out of my (first-ever!) Slack meeting with the WordPress documentation team.

The documentation team holds weekly office hours on Tuesdays from 14:00-15:00 UTC in the #docs Slack channel.

Leading up to the meeting, things were a little bit touch and go regarding the request put through to link the Caddy wiki article to an existing WP article.

There’s a big discussion on whether the articles contain external links or not. I don’t know if it got resolved.

This is generally disallowed, but it appears that Caddy forum wiki articles are considered a credible source. The WP documentation team are short-staffed so it will take some time before the Caddy reference appears on the said article. Still, it’s in the pipeline

The next step I guess is to reflect on this thread and maybe try to understand the benefits/downsides of the process. It’s really a checkpoint on whether or not to continue down this path.