Help to migrate Caddyfile V1 to V2 for Nextcloud

francislavoie · April 22, 2020, 9:18pm

Thought so… is there a way we can guarantee that?

matt · April 22, 2020, 9:19pm

Probably use a route or handle directive so that the outer matcher is guaranteed to run first.

francislavoie · April 22, 2020, 9:22pm

Dang, I don’t love that… we’d need to wrap the entire config in a route to do it… there must be a better way

Dougy · April 22, 2020, 9:22pm

Thank you for your help.

To understand better what was the needed rewrite request, I switched back to Caddy v1 and modified this rewrite block from Caddyfile :

rewrite / {
        if {path} not /core/img/favicon.ico
        if {path} not /core/img/manifest.json
        if {path} not_starts_with /remote.php
        if {path} not_starts_with /public.php
        if {path} not_starts_with /cron.php
        if {path} not_starts_with /core/ajax/update.php
        if {path} not_starts_with /status.php
        if {path} not_starts_with /ocs/v1.php
        if {path} not_starts_with /ocs/v2.php
        if {path} not /robots.txt
        if {path} not_starts_with /updater/
        if {path} not_starts_with /ocs-provider/
        if {path} not_starts_with /ocm-provider/
        if {path} not_starts_with /.well-known/
        to /index.php{uri}
}

I replaced it with the one below

rewrite / {
        to /index.php{uri}
}

Now I get the exact same behaviour as Caddy V2 :
"PROPFIND /remote.php/dav/files/jacques/ HTTP/1.1" 405 0

Thus, I just need to understand how to prevent Caddyfile v2 to rewrite a request such as /remote.php/* to be redirected to index.php

Could it be due to the new V2 php_fastcgi directive which redirects everything to index.php ?

francislavoie · April 22, 2020, 9:24pm

Yep that’s exactly why @Dougy, I’m just trying to think of a generalized solution so we can solve it without relying on specifically dealing with each path that involves a non-index .php file.

Once we figure that out I’ll probably look into baking it into the php_fastcgi directive so it’s generally available.

The crux of the issue is that we made the assumption that an app would only have one PHP file that would act as a router (index.php typically) but NextCloud has more than one (i.e. remote.php as well)

To clarify further, if the requested URI was something like /remote.php?p=/dav/files/jacques it would work fine because {path} would only be /remote.php because the query is not included, but in this case the actual requested URI is /remote.php/dav/files/jacques/ and Caddy doesn’t detect that the .php part denotes the end of the filename.

francislavoie · April 22, 2020, 9:51pm

Okay I think I’ve thought through a solution – the general idea is to add a split feature to the file matcher, it would look something like this:

file {
    split .php
    try_files {path} {path}/index.php index.php
}

I’ll work on implementing that soon, we’ll see if we can get that into Caddy ASAP, but no promises.

That said, here’s a potential temporary solution for you:

@phpFiles {
    path_regexp phpfile ^/(remote|public|cron|core/ajax/update|status|ocs/v1|ocs/v2)\.php
}
rewrite @phpFiles {http.regexp.phpfile.0}

Essentially this should act as a whitelist for those PHP files that shouldn’t be rewritten to index.php.

Dougy · April 22, 2020, 10:58pm

@francislavoie, your solution is working, thank you !

This is my current Caddyfile V2 :

   mydomain.com {

            root    * /usr/local/www/nextcloud
            file_server
            log {
                    output file     /var/log/nextcloud_access.log
                    format single_field common_log
            }

            tls {
                    ca https://acme-v02.api.letsencrypt.org/directory
            }

            php_fastcgi 127.0.0.1:9000

            header {
                    Strict-Transport-Security               "max-age=15768000;"
                    X-Content-Type-Options                  "nosniff"
                    X-XSS-Protection                        "1; mode=block"
                    X-Robots-Tag                            "none"
                    X-Download-Options                      "noopen"
                    X-Permitted-Cross-Domain-Policies       "none"
                    X-Frame-Options "SAMEORIGIN"
                    Referrer-Policy                         "no-referrer"
            }

            header /core/fonts {
                    Cache-Control                           "max-age=604800"
            }

            @phpFiles {
                    path_regexp phpfile ^/(remote|public|cron|core/ajax/update|status|ocs/v1|ocs/v2)\.php
            }
            rewrite @phpFiles {http.regexp.phpfile.0}

            redir /.well-known/cardav /remote.php/dav 301
            redir /.well-known/caldav /remote.php/dav 301

            # .htaccess / data / config / ... shouldn't be accessible from outside
            @forbidden {
                    path    /.htaccess
                    path    /data/*
                    path    /config/*
                    path    /db_structure
                    path    /.xml
                    path    /README
                    path    /3rdparty/*
                    path    /lib/*
                    path    /templates/*
                    path    /occ
                    path    /console.php
            }

            respond @forbidden 404

    }

francislavoie · April 23, 2020, 2:05am

@Dougy are you willing to try out an early build to see if this works for you?

github.com/caddyserver/caddy

matcher: Add `split_path` option to file matcher; used in php_fastcgi

caddyserver:master ← francislavoie:try-files-split-path

opened 01:00AM - 23 Apr 20 UTC

francislavoie

+342 -188

Discussion: https://caddy.community/t/help-to-migrate-caddyfile-v1-to-v2-for-nex…tcloud/7647/18 A common pattern in PHP apps is to have a PHP file that acts as a router entrypoint to the application. This is typically the `index.php` file at the webroot or some other subpath of the application. These entrypoints typically accept requests that have a path appended to the filename, for example `/index.php/blog/123`. Some PHP apps happen to have more than one router entrypoint, for example, `/router.php/some/path`, in addition to the `index.php` file. NextCloud is one of those applications; it has `remote.php`, `public.php` and more that act as entrypoints. With the default `php_fastcgi` Caddyfile directive, many of those pages didn't work correctly in NextCloud. The problem has to do with the `try_files` matcher and rewrite that is included as part of the `php_fastcgi` shortcut: ``` try_files {path} {path}/index.php index.php ``` The issue is that on a request to the path `/remote.php/dav/files`, Caddy would check if that exact path exists as a file on disk (i.e. the first argument, `{path}`), but this doesn't work because there's extra stuff after the actual filename. We already have an option in the reverse_proxy's fastcgi transport to allow splitting the path into two parts, with the left-hand side being the filename (including the split value), and the right-hand side being the path, i.e. `split .php`. The fix we landed on for this issue is to add a new `split` option to the file matcher (`split_path` in JSON) that allows for splitting the path the same way, before testing if the file exists on disk. It looks like this: ``` @phpFiles { file { split .php try_files {path} {path}/index.php index.php } } ``` So now for a request to `/remote.php/dav/files`, Caddy will instead look for `/remote.php` on disk while matching. The `php_fastcgi` shortcut is also updated to internally add this new split option for its `try_files` matcher. This should not be a breaking change, because we assume that any path that involved `.php` in the middle of it was assumed to be the filename boundary, but this wasn't working as expected.

Seeing as you’re on FreeBSD you’d probably need to compile it yourself, let me know if you need any pointers for how to do that!

Build instructions are here:

Basically, just comment out or remove the rewrite block I had you add, and hopefully it should work with the new build. The patch changes how php_fastcgi behaves to handle your situation.

matt · April 23, 2020, 2:06am

@Dougy If you can confirm that the fix works for you within the next couple days, I think we can get the fix merged in before the 2.0 release.

Dougy · April 23, 2020, 11:19am

@francislavoie I need your help as I do not know how to download your fix source code with git, here is what I tried :

root@caddytest:~ # git clone "https://github.com/caddyserver/caddy.git"
Cloning into 'caddy'...
remote: Enumerating objects: 77, done.
remote: Counting objects: 100% (77/77), done.
remote: Compressing objects: 100% (52/52), done.
remote: Total 22243 (delta 42), reused 43 (delta 24), pack-reused 22166
Receiving objects: 100% (22243/22243), 12.75 MiB | 2.00 MiB/s, done.
Resolving deltas: 100% (13939/13939), done.
root@caddytest:~ # cd caddy/
root@caddytest:~/caddy # git checkout ca1df03a4a7a4896cd0a90db1dd7557b0d198816
fatal: reference is not a tree: ca1df03a4a7a4896cd0a90db1dd7557b0d198816

francislavoie · April 23, 2020, 12:01pm

Yeah - the code is on my fork. Did you manage to check it out now?

$ git clone https://github.com/francislavoie/caddy.git
$ git checkout origin/try-files-split-path

Dougy · April 23, 2020, 12:21pm

Yes, I compiled your fix, removed the @phpFiles rewrite in my Caddyfile, and it still works

Dougy · April 23, 2020, 3:25pm

I found another bug, my header directive is not working properly because all header values are set twice.

Currently I have this in my Caddyfile for headers :

header {
        Strict-Transport-Security               max-age=15768000;
        X-Content-Type-Options                  nosniff
        X-XSS-Protection                        1; mode=block
        X-Robots-Tag                            none
        X-Download-Options                      noopen
        X-Permitted-Cross-Domain-Policies       none
        X-Frame-Options                         SAMEORIGIN
        Referrer-Policy                         no-referrer
}

But if you look at this sample request, some headers value are set twice such as “X-Frame-Options”: [“SAMEORIGIN”, “SAMEORIGIN”]

2020/04/23 15:18:33.447 INFO http.log.access.log0 handled request {"request": {"method": "PROPFIND", "uri": "/remote.php/dav/files/jacques/", "proto": "HTTP/1.1", "remote_addr": "192.168.1.115:53773", "host": "mydomain.com", "headers": {"Depth": ["0"], "Accept": ["*/*"], "X-Request-Id": ["4b32add6-36b7-4586-a52f-c0c43952f01a"], "Accept-Encoding": ["gzip, deflate"], "Accept-Language": ["en-US,*"], "Authorization": ["Basic amFjcXVlczo3VEQ1VkVob0k0M05qeXV5Qnh6Y1czYXRKY1lLTlM1WFZHODk4enVJeExTbmk3d25ZaTVzYTFWeEs1MFNyT1R4RGZOMTRlNng="], "User-Agent": ["Mozilla/5.0 (Macintosh) mirall/2.6.4stable (build 20200303) (Nextcloud)"], "Content-Type": ["text/xml; charset=utf-8"], "Cookie": ["oc_sessionPassphrase=zQUNkYPxfX530eWK7G9uMimSjjyB%2BTHng1V1m4%2B9uxCf3Dh%2Fri8sZ9VmzBq2hKRhOBzZ2Yn6W6fmZrIu%2BJ4Wow3JwFrcL4GWX2cR8OjFZ1%2B4aL42oidyXZrxvMWEd4Xc; __Host-nc_sameSiteCookielax=true; __Host-nc_sameSiteCookiestrict=true; ocih7olqwfvd=hijer9sh22bonifu30n7plh4c4"], "Content-Length": ["114"], "Connection": ["Keep-Alive"]}, "tls": {"resumed": false, "version": 772, "ciphersuite": 4865, "proto": "", "proto_mutual": true, "server_name": "mydomain.com"}}, "common_log": "192.168.1.115 - - [23/Apr/2020:17:18:33 +0200] \"PROPFIND /remote.php/dav/files/jacques/ HTTP/1.1\" 207 388", "latency": 0.115401907, "size": 388, "status": 207, "resp_headers": {"X-Xss-Protection": ["1; mode=block"], "Dav": ["1, 3, extended-mkcol, access-control, calendarserver-principal-property-search, nc-calendar-search, nc-enable-birthday-calendar"], "Referrer-Policy": ["no-referrer", "no-referrer"], "X-Permitted-Cross-Domain-Policies": ["none", "none"], "Expires": ["Thu, 19 Nov 1981 08:52:00 GMT"], "X-Robots-Tag": ["none", "none"], "Vary": ["Brief,Prefer"], "X-Download-Options": ["noopen", "noopen"], "X-Frame-Options": ["SAMEORIGIN", "SAMEORIGIN"], "Status": ["207 Multi-Status"], "Content-Type": ["application/xml; charset=utf-8"], "Content-Security-Policy": ["default-src 'none';"], "Server": ["Caddy"], "Strict-Transport-Security": ["max-age=15768000;"], "X-Content-Type-Options": ["nosniff", "nosniff"], "X-Powered-By": ["PHP/7.4.5"], "Pragma": ["no-cache"], "Cache-Control": ["no-store, no-cache, must-revalidate"]}}

Hence I get errors in NextCloud security check :

matt · April 23, 2020, 3:29pm

@Dougy If you remove your header directive and try again, do the headers only appear once or not at all?

Dougy · April 23, 2020, 3:41pm

If I remove all header directives, I only get this warning :

2020/04/23 15:39:28.478 INFO http.log.access.log0 handled request {"request": {"method": "PROPFIND", "uri": "/remote.php/dav/files/jacques/", "proto": "HTTP/1.1", "remote_addr": "192.168.1.115:53918", "host": "mydomain.com", "headers": {"Content-Type": ["text/xml; charset=utf-8"], "X-Request-Id": ["8afac727-8da7-425c-a78c-0395919604f2"], "Content-Length": ["105"], "Connection": ["Keep-Alive"], "Depth": ["0"], "User-Agent": ["Mozilla/5.0 (Macintosh) mirall/2.6.4stable (build 20200303) (Nextcloud)"], "Accept-Encoding": ["gzip, deflate"], "Accept-Language": ["en-US,*"], "Authorization": ["Basic amFjcXVlczo3VEQ1VkVob0k0M05qeXV5Qnh6Y1czYXRKY1lLTlM1WFZHODk4enVJeExTbmk3d25ZaTVzYTFWeEs1MFNyT1R4RGZOMTRlNng="], "Accept": ["*/*"], "Cookie": ["oc_sessionPassphrase=zQUNkYPxfX530eWK7G9uMimSjjyB%2BTHng1V1m4%2B9uxCf3Dh%2Fri8sZ9VmzBq2hKRhOBzZ2Yn6W6fmZrIu%2BJ4Wow3JwFrcL4GWX2cR8OjFZ1%2B4aL42oidyXZrxvMWEd4Xc; __Host-nc_sameSiteCookielax=true; __Host-nc_sameSiteCookiestrict=true; ocih7olqwfvd=hijer9sh22bonifu30n7plh4c4"]}, "tls": {"resumed": false, "version": 772, "ciphersuite": 4865, "proto": "", "proto_mutual": true, "server_name": "mydomain.com"}}, "common_log": "192.168.1.115 - - [23/Apr/2020:17:39:28 +0200] \"PROPFIND /remote.php/dav/files/jacques/ HTTP/1.1\" 207 387", "latency": 0.11005796, "size": 387, "status": 207, "resp_headers": {"Content-Type": ["application/xml; charset=utf-8"], "X-Robots-Tag": ["none"], "Referrer-Policy": ["no-referrer"], "Dav": ["1, 3, extended-mkcol, access-control, calendarserver-principal-property-search, nc-calendar-search, nc-enable-birthday-calendar"], "Server": ["Caddy"], "Status": ["207 Multi-Status"], "Expires": ["Thu, 19 Nov 1981 08:52:00 GMT"], "Pragma": ["no-cache"], "X-Frame-Options": ["SAMEORIGIN"], "Content-Security-Policy": ["default-src 'none';"], "X-Powered-By": ["PHP/7.4.5"], "Vary": ["Brief,Prefer"], "X-Xss-Protection": ["1; mode=block"], "X-Download-Options": ["noopen"], "X-Content-Type-Options": ["nosniff"], "X-Permitted-Cross-Domain-Policies": ["none"], "Cache-Control": ["no-store, no-cache, must-revalidate"]}}

matt · April 23, 2020, 3:57pm

Perfect, so I think all you need is to set that one header (Strict-Transport-Security) as it seems that the backend is probably setting all the other necessary headers. You can verify by adding this to the top of your Caddyfile:

{
    debug
}

which will enable debug logging, and you can see what response headers the backend is setting.

Dougy · April 23, 2020, 4:47pm

Thank you @matt, everything works like it’s supposed to be now.

Here is my latest Caddyfile v2 for NextCloud provided as a reference (it’s only working with @francislavoie fix which hopefully will be merged into v2 release) :

mydomain.com {

        root    * /usr/local/www/nextcloud
        file_server
        log {
                output file     /var/log/mydomain.com.log
                format single_field common_log
        }

        php_fastcgi 127.0.0.1:9000

        header {
                # enable HSTS
                Strict-Transport-Security max-age=31536000;
        }

        redir /.well-known/carddav /remote.php/dav 301
        redir /.well-known/caldav /remote.php/dav 301

        # .htaccess / data / config / ... shouldn't be accessible from outside
        @forbidden {
                path    /.htaccess
                path    /data/*
                path    /config/*
                path    /db_structure
                path    /.xml
                path    /README
                path    /3rdparty/*
                path    /lib/*
                path    /templates/*
                path    /occ
                path    /console.php
        }

        respond @forbidden 404

}

lexpierce · April 24, 2020, 3:17am

This one is easy… I had the same issue with the v1 config. Nextcloud already provides all theose headers, except for HSTS. that is the only one you need to set in your Caddyfile. The rest are returned by reverse_proxy.

Thanks @francislavoie for this. Nextcloud was my next conversion to v2 after my little MTA-STS issue ealier with not understanding matchers. I had a similar issue with another PHP app as well that I will try again with your @phpFiles fix.

PopeRigby · May 23, 2020, 9:48pm

Hey @Dougy, I think I’m having a similar issue but I can’t get your Caddyfile to work. Do you mind sharing your docker-compose.yml or docker run command?

kunago · June 22, 2020, 7:02am

I would like to add one more modification. With Caddy v2.1 (in beta) and in order to get rid of “index.php” in the url path, you need to set one environmental variable:

php_fastcgi 127.0.0.1:9000 {
  env front_controller_active true
}