"file" "not file" testing question

1. Caddy version (caddy version):

/srv # caddy version
v2.1.1 h1:X9k1+ehZPYYrSqBvf/ocUgdLSRIuiNiMo7CvyGUQKeA=

2. How I run Caddy:

caddy run -adapter caddyfile -config caddyfile 

a. System environment:

MasOS 10.15.6

b. Command:

caddy run -adapter caddyfile -config caddyfile 

c. Service/unit/compose file:

n/a

d. My complete Caddyfile or JSON config:

{
	admin off
	auto_https disable_redirects
	http_port 1314
}

http://localhost:1314 {
	file_server
	root * {env.PWD}/www

	# --- Method 1
	@post {
		file /blog/{path}index.html /blog/{path}/index.html
		not file {path} {path}/
		path_regexp post ^/([^/]+)/?$
	}

	# --- Method 2
	#@post {
	#	file /blog/{path} /blog/{path}/
	#	not file {path} {path}/
	#	path_regexp post ^/([^/]+)/?$
	#}

	redir @post /blog/{re.post.1}/
	templates
}

3. The problem I’m having:

Please refer to end of post.

4. Error messages and/or full log output:

N/A

5. What I already tried:

2 different methods give unexpected results for different test cases :sweat_smile:

6. Links to relevant resources:

https://github.com/J-Siu/caddy-test.git

Test

The whole dir tree with caddyfile is in github:

All text/html files are single line text stating file path with no html tags. You can check on github before cloning.

git clone https://github.com/J-Siu/caddy-test.git
cd caddy-test
caddy run -adapter caddyfile -config caddyfile

Want to do

www/
β”œβ”€β”€ blog/
β”‚ β”œβ”€β”€ blogdir/
β”‚ β”‚ └── index.html
β”‚ β”œβ”€β”€ blogfile
β”‚ β”œβ”€β”€ dir2dir/
β”‚ β”‚ └── index.html
β”‚ β”œβ”€β”€ dir2file
β”‚ β”œβ”€β”€ file2dir/
β”‚ β”‚ └── index.html
β”‚ └── file2file
β”œβ”€β”€ dir/
β”‚ └── index.html
β”œβ”€β”€ dir2dir/
β”‚ └── index.html
β”œβ”€β”€ dir2file/
β”‚ └── index.html
β”œβ”€β”€ file
β”œβ”€β”€ file2dir
└── file2file

I want:

Request Redirect
if /X(file/dir) does not exist if /blog/X(file/dir) exist

Base on above dir tree, following should happen

Request Redirect
http://localhost:1314/dir no redirect (does not exit in /blog)
http://localhost:1314/dir2dir no redirect (exist at /)
http://localhost:1314/dir2file no redirect (exist at /)
http://localhost:1314/file no redirect (does not exit in /blog)
http://localhost:1314/file2dir no redirect (exist at /)
http://localhost:1314/file2file no redirect (exist at /)
http://localhost:1314/blogdir http://localhost:1314/blog/blogdir
http://localhost:1314/blogfile http://localhost:1314/blog/blogfile

Method 1

caddyfile

{
admin off
auto_https disable_redirects
http_port 1314
}

http://http://localhost:1314 {
file_server
root * {env.PWD}/www

# --- Method 1
@post {
 file /blog/{path}index.html /blog/{path}/index.html
 not file {path} {path}/
 path_regexp post ^/([^/]+)/?$
}

redir @post /blog/{re.post.1}/

}

Result:

Test# Request Expected Actual
M1T1 http://localhost:1314/dir no redirect (does not exit in /blog) no redirect
M1T2 http://localhost:1314/dir2dir no redirect (exist at /) http://localhost:1314/blog/dir2dir/
M1T3 http://localhost:1314/dir2file no redirect (exist at /) no redirect
M1T4 http://localhost:1314/file no redirect (does not exit in /blog) no redirect
M1T5 http://localhost:1314/file2dir no redirect (exist at /) no redirect
M1T6 http://localhost:1314/file2file no redirect (exist at /) no redirect
M1T7 http://localhost:1314/blogdir http://localhost:1314/blog/blogdir http://localhost:1314/blog/blogdir
M1T8 http://localhost:1314/blogfile error 404(no /blog/blogfile/index.html) error 404

Method 2

caddyfile (removed index.html in file matching)

{
admin off
auto_https disable_redirects
http_port 1314
}

http://http://localhost:1314 {
file_server
root * {env.PWD}/www

# --- Method 2
@post {
file /blog/{path} /blog/{path}/
not file {path} {path}/
path_regexp post ^/([^/]+)/?$
}

redir @post /blog/{re.post.1}/
}

Result:

Test# Request Expected Actual
M2T1 http://localhost:1314/dir no redirect (does not exist in /blog) no redirect
M2T2 http://localhost:1314/dir2dir no redirect (exist at /) no redirect
M2T3 http://localhost:1314/dir2file no redirect (exist at /) http://localhost:1314/blog/dir2file
M2T4 http://localhost:1314/file no redirect (does not exist in /blog) no redirect
M2T5 http://localhost:1314/file2dir no redirect (exist at /) no redirect
M2T6 http://localhost:1314/file2file no redirect (exist at /) no redirect
M2T7 http://localhost:1314/blogdir http://localhost:1314/blog/blogdir no redirect
M2T8 http://localhost:1314/blogfile http://localhost:1314/blog/blogfile http://localhost:1314/blog/blogfile

Issue

Why:

Method 1 test M1T2 failed?

Method 2 test M2T3, M2T7 failed?

2 Likes

Excellent! Thanks for putting in the work to set up a test case like this, it’s greatly appreciated, and I wish more people would put in this level of effort :grin:

I’ll take a closer look as soon as I can, I’ll try to set up a build of Caddy with some additional debug tracing to determine the result of each individual matcher to figure out what’s going on.

Okay so trying first M1T2, these are the logs I have it spit out:

DEBUG: /blog/dir2dirindex.html | /blog/dir2dirindex.html | /blog/dir2dirindex.html
FileNotExists: /blog/dir2dirindex.html
DEBUG: /blog/dir2dir/index.html | /blog/dir2dir/index.html | /blog/dir2dir/index.html
FileExists: /blog/dir2dir/index.html
MatchFile: files: [/blog{http.request.uri.path}index.html /blog{http.request.uri.path}/index.html] | rel: /blog/dir2dir/index.html | match: true
DEBUG: /dir2dir | /dir2dir | /dir2dir
FileNotExists: /dir2dir
DEBUG: /dir2dir/ | /dir2dir | /dir2dir
FileNotExists: /dir2dir
MatchFile: files: [{http.request.uri.path} {http.request.uri.path}/] | rel:  | match: false
MatchPathRE: true
2020/08/24 06:36:40.553 INFO    http.log.access handled request {"request": {"remote_addr": "[::1]:60618", "proto": "HTTP/1.1", "method": "GET", "host": "localhost:1314", "uri": "/dir2dir", "headers": {"User-Agent": ["curl/7.55.1"], "Accept": ["*/*"]}}, "common_log": "::1 - - [24/Aug/2020:02:36:40 -0400] \"GET /dir2dir HTTP/1.1\" 302 0", "duration": 0.0010016, "size": 0, "status": 302, "resp_headers": {"Location": ["/blog/dir2dir/"], "Content-Type": [], "Server": ["Caddy"]}}
DEBUG: /blog/blog/dir2dir/index.html | /blog/blog/dir2dir/index.html | /blog/blog/dir2dir/index.html
FileNotExists: /blog/blog/dir2dir/index.html
DEBUG: /blog/blog/dir2dir//index.html | /blog/blog/dir2dir/index.html | /blog/blog/dir2dir/index.html
FileNotExists: /blog/blog/dir2dir/index.html
MatchFile: files: [/blog{http.request.uri.path}index.html /blog{http.request.uri.path}/index.html] | rel:  | match: false
2020/08/24 06:36:40.590 INFO    http.log.access handled request {"request": {"remote_addr": "[::1]:60618", "proto": "HTTP/1.1", "method": "GET", "host": "localhost:1314", "uri": "/blog/dir2dir/", "headers": {"Accept": ["*/*"], "User-Agent": ["curl/7.55.1"]}}, "common_log": "::1 - - [24/Aug/2020:02:36:40 -0400] \"GET /blog/dir2dir/ HTTP/1.1\" 200 32", "duration": 0.0239999, "size": 32, "status": 200, "resp_headers": {"Server": ["Caddy"], "Content-Type": ["text/html; charset=utf-8"], "Content-Length": ["32"]}}

So basically what I’m seeing is that the / from {path}/ is stripped just before it gets tested, and it happens because of the call to path.Clean right here:

So I think if you want to test for a directory specifically, you need to add the index.html at the end. So I think the fix for method 1 is to use this instead:

not file {path} {path}/index.html

@John_Siu could you try to see if that change does what you need? I didn’t spend the time trying all the permutations to confirm (about to go to bed, stayed up too late lol) but I hope that reveals the problem.


The documentation for path.Clean states the following:

The returned path ends in a slash only if it is the root β€œ/”.

@matt do we want to re-add the / after path.Clean if it came with one initially? I’m not sure how we want to handle that. Am I misunderstanding something obvious? I’m looking into this way too late at night and I’ll need to come back to this later with fresh eyes.

Using not file {path} {path}/index.html, M3T2 passed

Method 3

caddyfile

{
admin off
auto_https disable_redirects
http_port 1314
}

http://http://localhost:1314 {
file_server
root * {env.PWD}/www

# --- Method 3
@post {
file /blog/{path}index.html /blog/{path}/index.html
not file {path} {path}/index.html
path_regexp post ^/([^/]+)/?$
}

redir @post /blog/{re.post.1}/
}

Result:

Test# Request Expected Actual
M3T1 http://localhost:1314/dir no redirect (does not exit in /blog) no redirect
M3T2 http://localhost:1314/dir2dir no redirect (exist at /) no redirect
M3T3 http://localhost:1314/dir2file no redirect (exist at /) no redirect
M3T4 http://localhost:1314/file no redirect (does not exit in /blog) no redirect
M3T5 http://localhost:1314/file2dir no redirect (exist at /) no redirect
M3T6 http://localhost:1314/file2file no redirect (exist at /) no redirect
M3T7 http://localhost:1314/blogdir http://localhost:1314/blog/blogdir http://localhost:1314/blog/blogdir
M3T8 http://localhost:1314/blogfile error 404(no /blog/blogfile/index.html) error 404
1 Like

To run the test case super fast, just click the links in this page. Or inside vscode or any editor has md preview the README.md.

So it looks like everything works as expected with that last config. Good! Solved?

For the specific case with index.html it work.

But if I want to implement a more generic form, that’s without the index.html, it is not. And according to try_files doc page, it should just work. Seems like trailing / is broken.

I tried with path_regexp like following:

	@post {
		not path_regexp p1 ^{path}/?$
		path_regexp p2 ^/blog/{path}/?$
		path_regexp post ^/([^/]+)/?$
	}

That failed miserably, everything redirected, LOL.

I wrote a PR that should fix the try_files behaviour.

https://github.com/caddyserver/caddy/pull/3684

If you have time, could you download one of the CI artifacts and give it a spin? fileserver: Fix try_files for directories, windows fix Β· caddyserver/caddy@db4828b Β· GitHub

I’m targeting v2.3 for this change because it’s probably too late to get this into v2.2 since we’re already in RCs for v2.2.

1 Like

I updated my https://github.com/J-Siu/caddy-test

Test result is in https://github.com/J-Siu/caddy-test/blob/master/M2-result.md

Not going to paste it here this time. The extra lines inside code block and tables rows after pasting is driving me nuts.

Sorry, I didn’t see the comment earlier for some reason (might’ve missed the notification or something).

I added a PR + comments on your repo:

https://github.com/J-Siu/caddy-test/pull/1

Finally this come to close!

Short

M5 passed all test (T1-T8) vs your caddy version.

https://github.com/J-Siu/caddy-test/blob/master/M5-result.md

Long

  1. try_file use rewrite, so I created M5 base on original M2 which use redir

  2. I notice there was a mistake on my original M2, the redir has a trailing /, which interfere with M2T8.

    redir @post /blog/{re.post.1}/
    

Following is method 5:

{
	admin off
	auto_https disable_redirects
	http_port 1314
}

http://localhost:1314 {
	file_server
	root www

	# --- Method 5
	@post {
		file /blog/{path} /blog/{path}/
		not file {path} {path}/
		path_regexp post ^/([^/]+)/?$
	}
	# remove trailing /
	redir @post /blog/{re.post.1}

}

Right, my bad about try_files :stuck_out_tongue:

So is this done now? Does it work the way you wanted?

Yes. This is done. Waiting for your fix to get in next release. :laughing:

Maybe your PR should get in 2.2, as file directory matching is actually broken.

1 Like

The test repo is moved to GitHub - J-Siu/test-caddy

The fix has been merged into master and will be released soon. Thanks @francislavoie !

2 Likes

This topic was automatically closed after 30 days. New replies are no longer allowed.