How to rewrite if and only if file exists with specific extension?

neezer · November 10, 2022, 7:20pm

1. Output of `caddy version`:

v2.6.2 h1:wKoFIxpmOJLGl3QXoo6PNbYvGW4xLEgo32GPBEjWL8o=

2. How I run Caddy:

a. System environment:

macOS Ventura 13.0

b. Command:

caddy run

c. Service/unit/compose file:

N/A

d. My complete Caddy config:

http://localhost:8773

@age {
	file
	path_regexp \.age$
}

rewrite @age /entry.html

root * ./srv

file_server {
	pass_thru
}

file_server {
	root ./entries
	browse
}

templates

3. The problem I’m having:

I want to only rewrite a request to

a file that exists
a file that ends in a specific extension

… otherwise I want to 404.

With the Caddyfile above, the file resolves (prints out the file contents) instead of being rewritten to /entry.html.

4. Error messages and/or full log output:

None.

5. What I already tried:

I can get the rewrite working with just

@age `{path}.endsWith("age")`

… but that does not 404 if the file doesn’t exist.

I’ve also tried handling the 404 logic in entry.html (since it’s a template), using something like this:

{{if not (fileExists .OriginalReq.URL.Path)}}{{httpError 404}}{{end}}

<!-- rest of entry.html -->

… but fileExists seems to always return false even when I know my path is correct, so this always 404s.

6. Links to relevant resources:

None that I know of.

francislavoie · November 10, 2022, 8:39pm

I’m not sure why the regexp doesn’t work. It should, I think.

But either way, you can do this, I think:

@age `file() && {path}.endsWith(".age")`

neezer · November 10, 2022, 8:47pm

That gives me this error… ?

expression: compiling CEL program: ERROR: <input>:1:5: matcher requires at least one argument
 | file() && caddyPlaceholder(request, "http.request.uri.path").endsWith(".age")
 | ....^

francislavoie · November 10, 2022, 8:59pm

Ah - I guess we didn’t implement an overload for no params. Maybe file({}) will work.

If not, do file("{path}") I think. Although that might also not work for other reasons.

neezer · November 10, 2022, 9:02pm

The former resolves the file (prints file contents) and doesn’t perform the rewrite.

The latter yields this error:

expression: compiling CEL program: ERROR: <input>:1:34: Syntax error: mismatched input 'http' expecting ')'
 | file("caddyPlaceholder(request, "http.request.uri.path")") && caddyPlaceholder(request, "http.request.uri.path").endsWith(".age")
 | .................................^
ERROR: <input>:1:55: Syntax error: mismatched input '")"' expecting <EOF>
 | file("caddyPlaceholder(request, "http.request.uri.path")") && caddyPlaceholder(request, "http.request.uri.path").endsWith(".age")
 | ......................................................^

I’m guessing because of the quotes.

francislavoie · November 10, 2022, 9:16pm

Yeah. The expression parsing logic is… funky.

I think file({path}) should work, actually. Our tests cover that one.

neezer · November 10, 2022, 9:27pm

@age `file({path}) && {path}.endsWith(".age")`

… resolves the file (no rewrite).

matt · November 10, 2022, 9:28pm

Just curious as to the usecase, are you perchance serving age-encrypted-at-rest files from disk over TLS and then the client is decrypting them? Or decrypting them on-the-fly for clients?

Either way, I could see some interesting plugin functionality here

neezer · November 10, 2022, 9:35pm

@matt Yup. I’m serving age-encrypted-at-rest files to the client. Decryption & re-encryption happens in the client application only (not on-the-fly).

@francislavoie I manage to get this working by moving the top-level root directive inside the first file_server directive, so now I have this:

@age `{path}.endsWith(".age")`

rewrite @age /entry.html

file_server {
	root ./srv
	pass_thru
}

file_server {
	root ./entries
	browse
}

That makes my fileExists detection work inside my template:

{{$filepath := printf "/entries%s" .OriginalReq.URL.Path}}
{{if not (fileExists $filepath)}}{{httpError 404}}{{end}}

It feels a bit weird to have to rely on the template for the 404 handling, though–I’d love to find a solution that captures this logic cleanly & directly in the Caddyfile.

matt · November 11, 2022, 3:57am

Does this work?

@age {
    file
    path *.age
}
rewrite @age /entry.html
...

The regular path matcher can match a suffix easily enough. I would think that should work. ^ I really don’t think CEL or regexp are required… if so, I’m interested in looking into this more because I don’t think that should be the case, with my current understanding.

Anyway, once that works, then I want to come back to the 404’ing because I bet it can probably be done in the Caddyfile too.

francislavoie · November 11, 2022, 4:24am

The file matcher uses your root that you defined. If you don’t use root then it will look for files in Caddy’s working directory (that depends on how you ran Caddy).

neezer · November 11, 2022, 4:29am

@matt Nope, that doesn’t work. It returns the file contents instead of performing the rewrite.

matt · November 11, 2022, 5:16am

I’m not able to reproduce that behavior.

With a file called hello.txt in a new directory, this config:

localhost

@age {
	file
	path *.txt
}
respond @age "Oh it's txt!"
respond "Not txt"

works for me:

$ curl -v "https://localhost/hello.txt"
*   Trying 127.0.0.1:443...
* Connected to localhost (127.0.0.1) port 443 (#0)
* successfully set certificate verify locations:
*  CAfile: /etc/ssl/certs/ca-certificates.crt
*  CApath: none
* ALPN: offers http/1.1
* ALPN: server accepted http/1.1
* SSL connection using TLSv1.3 / TLS13-AES128-GCM-SHA256
> GET /hello.txt HTTP/1.1
> Host: localhost
> User-Agent: curl/7.86.0-DEV
> Accept: */*
> 
* Mark bundle as not supporting multiuse
< HTTP/1.1 200 OK
< Alt-Svc: h3=":443"; ma=2592000
< Content-Type: text/plain; charset=utf-8
< Server: Caddy
< Date: Fri, 11 Nov 2022 05:15:13 GMT
< Content-Length: 12
< 
* Connection #0 to host localhost left intact
Oh it's txt!

$ curl -v "https://localhost/test.txt"
*   Trying 127.0.0.1:443...
* Connected to localhost (127.0.0.1) port 443 (#0)
* successfully set certificate verify locations:
*  CAfile: /etc/ssl/certs/ca-certificates.crt
*  CApath: none
* ALPN: offers http/1.1
* ALPN: server accepted http/1.1
* SSL connection using TLSv1.3 / TLS13-AES128-GCM-SHA256
> GET /test.txt HTTP/1.1
> Host: localhost
> User-Agent: curl/7.86.0-DEV
> Accept: */*
> 
* Mark bundle as not supporting multiuse
< HTTP/1.1 200 OK
< Alt-Svc: h3=":443"; ma=2592000
< Content-Type: text/plain; charset=utf-8
< Server: Caddy
< Date: Fri, 11 Nov 2022 05:15:16 GMT
< Content-Length: 7
< 
* Connection #0 to host localhost left intact
Not txt

Can you please verify by posting directory contents and curl output?

neezer · November 11, 2022, 5:26am

@matt Using your proposed change:

Directory contents:

 .
├──  Caddyfile
├──  entries
│   └──  a
│       └──  b
│           └──  test.tar.gz.b64.age
└──  srv
    └──  entry.html

Caddyfile:

http://localhost:8773

@age {
	file
	path *.age
}

rewrite @age /entry.html

file_server {
	root ./srv
	pass_thru
}

file_server {
	root ./entries
	browse
}

templates

Curl output from curl -v http://localhost:8773/a/b/test.tar.gz.b64.age:

*   Trying 127.0.0.1:8773...
* Connected to localhost (127.0.0.1) port 8773 (#0)
> GET /a/b/test.tar.gz.b64.age HTTP/1.1
> Host: localhost:8773
> User-Agent: curl/7.84.0
> Accept: */*
>
* Mark bundle as not supporting multiuse
< HTTP/1.1 200 OK
< Accept-Ranges: bytes
< Content-Length: 580
< Etag: "rl5je6g4"
< Last-Modified: Thu, 10 Nov 2022 21:51:42 GMT
< Server: Caddy
< Date: Fri, 11 Nov 2022 05:19:05 GMT
<
-----BEGIN AGE ENCRYPTED FILE-----
YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSBKb3lETUI4M0hCZ3ZvN1dW
alhLOWFHQ2lwMGI4TjVKd3RIaUhrOFBrRjFnCmNhUVJrSE9aVnlJUVdUMXhNUXFO
SjQydTg1aHlHOVF3OWl5V0xjOEcwNEkKLS0tIFFtd2crSVVWUGZqZDh5NzlyamdR
QXZhT3FiZGd2a1lrV2d5VHI5RFBFdHcKeguNwom6rdk94Xw+T2dz8KFW3bjLZteX
Cxc+0aX57NNOm8GcEYe502aIpkD3KV8oagnZPgs0CO9ntA3uBpqA+ToTH0d6OFQd
YxTkDHGsbjFpeNPyV5nxIzJoeysWCTr5IHeOudfkD4p+8QrcjLetPkukvf2zlMze
MSRwRhoDg956flVTiqH/nBcxAnmr5c98KSnybye9olAdg7knitftmvw1jMF6kF8+
UL1whiDPEtFoCn/7MfDDqXZX9kzaBoSikLsBeaXLRDgNUDJlj/qYCMY=
-----END AGE ENCRYPTED FILE-----
* Connection #0 to host localhost left intact

This is not desired behavior. What I want is the content from /entry.html, which should look like this:

*   Trying 127.0.0.1:8773...
* Connected to localhost (127.0.0.1) port 8773 (#0)
> GET /a/b/test.tar.gz.b64.age HTTP/1.1
> Host: localhost:8773
> User-Agent: curl/7.84.0
> Accept: */*
>
* Mark bundle as not supporting multiuse
< HTTP/1.1 200 OK
< Content-Length: 2245
< Content-Type: text/html; charset=utf-8
< Server: Caddy
< Date: Fri, 11 Nov 2022 05:22:59 GMT
<


<!DOCTYPE html>
<html>
  <head>
    <!-- ... truncated ... -->
    <script>
      globalThis.entryEnc = `-----BEGIN AGE ENCRYPTED FILE-----
YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSBKb3lETUI4M0hCZ3ZvN1dW
alhLOWFHQ2lwMGI4TjVKd3RIaUhrOFBrRjFnCmNhUVJrSE9aVnlJUVdUMXhNUXFO
SjQydTg1aHlHOVF3OWl5V0xjOEcwNEkKLS0tIFFtd2crSVVWUGZqZDh5NzlyamdR
QXZhT3FiZGd2a1lrV2d5VHI5RFBFdHcKeguNwom6rdk94Xw+T2dz8KFW3bjLZteX
Cxc+0aX57NNOm8GcEYe502aIpkD3KV8oagnZPgs0CO9ntA3uBpqA+ToTH0d6OFQd
YxTkDHGsbjFpeNPyV5nxIzJoeysWCTr5IHeOudfkD4p+8QrcjLetPkukvf2zlMze
MSRwRhoDg956flVTiqH/nBcxAnmr5c98KSnybye9olAdg7knitftmvw1jMF6kF8+
UL1whiDPEtFoCn/7MfDDqXZX9kzaBoSikLsBeaXLRDgNUDJlj/qYCMY=
-----END AGE ENCRYPTED FILE-----
`;
    </script>
  </head>

  <body>
    <!-- ... truncated ... -->
  </body>
</html>
* Connection #0 to host localhost left intact

The file contents of a/b/test.tar.gz.b64.age are injected into the HTML since entry.html is a template.

Does that clarify?

matt · November 11, 2022, 4:00pm

I think I see what you’re saying, but I’m still able to get what I think is the desired behavior with this Caddyfile:

localhost

@age {
        file
        path *.txt
}
rewrite @age /entry.html
templates
file_server {
        pass_thru
}
file_server browse

Requesting hello.txt (which exists) serve entry.html. Requesting / serves a file listing (browse). Requesting noexist.txt serves a 404.

neezer · November 11, 2022, 4:45pm

Where on your file system is entry.html located in your example? Your file_server directives don’t specify different roots: mine do. Would that make a difference?

neezer · November 11, 2022, 4:52pm

Ahh! That was actually it. Modifying your example to this works:

@age {
	file {
		root ./entries
	}
	path *.age
}

… which makes sense, since it was looking in wrong root for the .age file and thus couldn’t find it on the file system, so the matcher missed.

matt · November 11, 2022, 5:06pm

Yeah, it was just in the same dir (current working folder).

So, in summary, you shouldn’t need regex or CEL. Standard matcher/directive pattern should do it just fine.

Thanks for following up!

system · December 10, 2022, 7:20pm

This topic was automatically closed after 30 days. New replies are no longer allowed.