How to exclude specific files or dirs with Caddy2 as reverse proxy?

1. Caddy version (caddy version): Caddy 2, latest as of writing

2. How I run Caddy:

LXC container

a. System environment:

Debian 10 LXC, xcaddy install

b. Command:

caddy run --config /root/CF2

d. My complete Caddyfile or JSON config:

my.domain.com {
header {
    Strict-Transport-Security "max-age=31536000; includeSubDomains; preload"
    X-Xss-Protection "1; mode=block"
    X-Content-Type-Options "nosniff"
    X-Frame-Options "SAMEORIGIN"
    Content-Security-Policy "upgrade-insecure-requests"
    Referrer-Policy "strict-origin-when-cross-origin"
    Cache-Control "public, max-age=15, must-revalidate"
    Feature-Policy "accelerometer 'none'; ambient-light-sensor 'none'; autoplay 'self'; camera 'none'; encrypted-media 'none'; fullscreen 'self'; geolocation 'none'; gyroscope 'none'; magnetometer 'none'; microphone 'none'; midi 'none'; payment 'none'; picture-in-picture *; speaker 'none'; sync-xhr 'none'; usb 'none'; vr 'none'"
    }
reverse_proxy 192.168.1.2:3456 {
    header_up Authorization {>Authorization} 
    except /local/dir1/ /local/dir2 /localdir3/ /local/file.db
    }
    tls {
        dns cloudflare token_here
    }
}

3. The problem I’m having:

I’m trying to exclude only a few directories and files from a reverse proxy. I used to be able to use ‘except’ for this in the Caddyfile in Caddy v1 but now in v2, it doesn’t recognize this anymore, as the above Caddyfile doesn’t parse (Error during parsing: unrecognized subdirective except)

4. What I already tried:

I have no idea how to achieve the same thing. Please help!

In Caddy v2, you now use request matchers to conditionally exclude a directive from matching a request

Specifically, I think you’re looking to set up a named matcher with the not and path matchers, to exclude the reverse_proxy from matching those paths.

Note that in Caddy v2, path matching is exact, so if you want to exclude an entire directory, make sure to append * to the path.

Also, in Caddy v2, you’ll need to enable a file server explicitly, it’s no longer enabled by default like it was in v1.

And finally, the reverse_proxy in v2 automatically passes through all headers, so you don’t need that header_up directive to forward the Authorization header anymore (and that syntax is no longer valid, you would need to use {http.request.header.Authorization} to get the value of the header in the request).

See the upgrade guide, it explains most of these things:

This seems a lot more complicated than it was in v1. I dont understand how the named matcher syntax would work in this Caddyfile. Can you provide an example on how to achieve this with the few local folders I want to exclude? It doesn’t make sense like it used to with the exclude directive.

It’s not – it only looks complicated because, from what I can tell, you went into v2 assuming it was the same as v1 (for example, assuming that reverse_proxy was the same and had an except subdirective in v2 and that placeholders were all the same). Our upgrade guide can help, and it suggests basically starting with a fresh slate (i.e. not carrying over past assumptions): Upgrading to Caddy 2 — Caddy Documentation

The changes you need to make probably amount to about 2 lines. :smiley:

@notThese not path /local/dir1* /local/dir2* /localdir3/* local/file.db
reverse_proxy @notThese 192.168.1.2:3456

Something like that.

1 Like

@matt that doesn’t seem to work, I can still access those directories. This is the current code:

my.domain.com {
header {
    Strict-Transport-Security "max-age=31536000; includeSubDomains; preload"
    X-Xss-Protection "1; mode=block"
    X-Content-Type-Options "nosniff"
    X-Frame-Options "SAMEORIGIN"
    Content-Security-Policy "upgrade-insecure-requests"
    Referrer-Policy "strict-origin-when-cross-origin"
    Cache-Control "public, max-age=15, must-revalidate"
    Feature-Policy "accelerometer 'none'; ambient-light-sensor 'none'; autoplay 'self'; camera 'none'; encrypted-media 'none'; fullscreen 'self'; geolocation 'none'; gyroscope 'none'; magnetometer 'none'; microphone 'none'; midi 'none'; payment 'none'; picture-in-picture *; speaker 'none'; sync-xhr 'none'; usb 'none'; vr 'none'"
    }
@notTheseDirs /local/dir1* /local/dir2/*
reverse_proxy @notTheseDirs 192.168.1.2:3456
    tls {
        dns cloudflare token_here
    }
}

I can still access both of these dirs and all their contents after caddy runs…

I’m far from knowing how this program works but I believe the named matcher should be identified like so:

@notTheseDirs {
     not path <path to dir>
}

Then once you have it made you can call upon it where needed.

One of you experts correct me if I’m wrong please.

1 Like
@notTheseDirs /local/dir1* /local/dir2/*

Isn’t what I posted:

@notThese not path /local/dir1* /local/dir2* /localdir3/* local/file.db

@jfirestorm44 might be right about needing a block, I know in v2.1 (not yet released) you should be able to single-line it. Anyway, make sure not to lose those crucial tokens.

This is still not working… this really ought to be simpler. Current Caddyfile config:

my.domain.name {
header {
    Strict-Transport-Security "max-age=31536000; includeSubDomains; preload"
    X-Xss-Protection "1; mode=block"
    X-Content-Type-Options "nosniff"
    X-Frame-Options "SAMEORIGIN"
    Content-Security-Policy "upgrade-insecure-requests"
    Referrer-Policy "strict-origin-when-cross-origin"
    Cache-Control "public, max-age=15, must-revalidate"
    Feature-Policy "accelerometer 'none'; ambient-light-sensor 'none'; autoplay 'self'; camera 'none'; encrypted-media 'none'; fullscreen 'self'; geolocation 'none'; gyroscope 'none'; magnetometer 'none'; microphone 'none'; midi 'none'; payment 'none'; picture-in-picture *; speaker 'none'; sync-xhr 'none'; usb 'none'; vr 'none'"
    }
@excludeDirs {
    not path /local/dir1* /local/dir2* /local/private/* /local/file.db
    }
reverse_proxy @excludeDirs 192.168.1.2:3456 
    tls {
        dns cloudflare token_here
    }
}

I can still easily access a file on https://my.domain.name/local/dir1/script.js when it really shouldn’t allow it to proxy that. Caddy V1 had no issues with this when using the exclude directive and that js file would be inaccessible in my browser. Are there more typos in my Caddyfile above?

Just for fun, replace your Caddyfile with:

:80

lkajsdflkj

and reload your config. What happens? (Please paste the full output / logs in your reply.)

(Gonna start at square zero.)

I can also recommend using the caddy fmt command, it’ll fix the tabbing in your Caddyfile and make it easier to follow.

If you’re trying to exclude those paths to serve files that exist on disk, then you’ll need a file_server directive. Caddy v1 implicitly enabled a file server, but in Caddy v2 you need to enable it explicitly. You might also need to specify the site root with root so that Caddy knows where to look on disk for the files.

1 Like

Using your exact Caddyfile example, this is the result:

root@caddy:~# /root/caddy run --config /root/caddytest
2020/05/30 04:51:43.519 INFO    using provided configuration    {"config_file": "/root/caddytest", "config_adapter": ""}
run: loading initial config: decoding request body: invalid character ':' looking for beginning of value

Well, it thinks your input is JSON because you haven’t specified a config adapter.

Fair enough, now I get this error:

root@caddy:~# /root/caddy run --config /root/caddytest --adapter caddyfile
2020/05/30 05:36:04.835 INFO    using provided configuration    {"config_file": "/root/caddytest", "config_adapter": "caddyfile"}
run: adapting config using caddyfile: /root/caddytest:3: unrecognized directive: lkajsdflkj
1 Like

Good, that’s the expected result. Now try with your Caddyfile as you had above!

1 Like

So now trying with exactly this in my Caddyfile (after using caddy fmt command to clean up the formatting even more:

my.domain.name {

        header {

                Strict-Transport-Security "max-age=31536000; includeSubDomains; preload"

                X-Xss-Protection "1; mode=block"

                X-Content-Type-Options "nosniff"

                X-Frame-Options "SAMEORIGIN"

                Content-Security-Policy "upgrade-insecure-requests"

                Referrer-Policy "strict-origin-when-cross-origin"

                Cache-Control "public, max-age=15, must-revalidate"

                Feature-Policy "accelerometer 'none'; ambient-light-sensor 'none'; autoplay 'self'; camera 'none'; encrypted-media 'none'; fullscreen 'self'; geolocation 'none'; gyroscope 'none'; magnetometer 'none'; microphone 'none'; midi 'none'; payment 'none'; picture-in-picture *; speaker 'none'; sync-xhr 'none'; usb 'none'; vr 'none'"

        }

        @excludeDirs {

                not path /local/dir1* /local/dir2* /local/dir3/* /local/file.db

        }

        reverse_proxy @excludeDirs 192.168.1.2:3456

        tls {

                dns cloudflare token_here

        }

}

Which results in the following output:

root@caddy:~# /root/caddy run --config /root/Caddyfile2 --adapter caddyfile
2020/05/30 18:50:20.133 INFO    using provided configuration    {"config_file": "/root/Caddyfile2", "config_adapter": "caddyfile"}
2020/05/30 18:50:20.135 INFO    admin   admin endpoint started  {"address": "tcp/localhost:2019", "enforce_origin": false, "origins": ["[::1]:2019", "127.0.0.1:2019", "localhost:2019"]}
2020/05/30 18:50:20.136 INFO    http    enabling automatic HTTP->HTTPS redirects        {"server_name": "srv0"}
2020/05/30 13:50:20 [INFO][cache:0xc0005597c0] Started certificate maintenance routine
2020/05/30 18:50:20.137 INFO    tls     cleaned up storage units
2020/05/30 18:50:20.137 INFO    http    enabling automatic TLS certificate management   {"domains": ["my.domain.name"]}
2020/05/30 18:50:20.145 INFO    autosaved config        {"file": "/root/.config/caddy/autosave.json"}
2020/05/30 18:50:20.145 INFO    serving initial configuration

The reverse proxy works just fine, but again, I can STILL access files that should have been excluded. There’s a .js file that I can view in plaintext that includes private information. In Caddy V1, using the except directive as described in OP, this was impossible as the file would not be accessible.
The test that I do is type into my browser (in private mode so that nothing gets cached): https://my.domain.name/local/dir1/script.js
What I expect and WANT to happen is that I either get a 403 Forbidden error or it just won’t load anything. Instead, currently I get to see the full contents of this .js file, which includes information that shouldn’t be public, despite putting it in the @excludeDirs.

Why is this so hard in V2 now when it was so simple and straightforward in V1? This seems like a regression if I’m being honest. I’d love to be proven wrong however, and I really just want these folders to not be proxied recursively.

Going to try to help here. As @francislavoie said above you’ll need to enable file_server. That may be part of the problem. I don’t know v1 but from what I gather it was automatically enabled. In v2 you have to enable it. I don’t see it in your file.

1 Like

I added it and tested it, same result, I can still access that .js file that should be excluded. Doesn’t seem to make a difference.

I can only give you what works for me.

nas.mydomain.duckdns.org {
	log {
		output file /var/log/caddy/nas.log
		format console
	}
	root * /srv/dev-disk-by-label-HomeDrive/
	@noAccess {
                not path "*/My Pictures/*"
                not path */RESTRICTED/*
                not path "/config files/Caddy2/caddyfilenas.txt"
	}
	file_server @noAccess browse {
                hide RESRICTED "My Pictures"
}
	encode gzip
	basicauth {
        username (hashbrown64)
	}
}

Be sure if you folders/filenames have a space in them that you use quotes. As you see in mine I had to completely wrap the 1st and 3rd ‘not path’ in quotes because of white space contained in them. The 2nd one didn’t need quotes.

Also I chose to not only deny access but hide two of the folders also. This does however hide them from me also when accessing through a browser. I can still get them if I go directly to the hard drive from my windows PC.

I’m still trying to figure out exactly how the hide module works in regards to hiding only one instance of a specified file/folder name. See post here Help hiding only one named folder - #5 by jfirestorm44

Let me know if that works for you.

EDIT: Be sure to clear you browser cache after reloading your new config. I had an issue where the caddyfilenas.txt kept allowing me to see if even though I shouldn’t have been able to. Once I cleared the cookies/cache I could no longer access it.

2 Likes

I re-did some tests and it looks like it’s actually working now. I used a line for each intro for not path, like @jfirestorm44 did. Not sure if that’s what the issue was but it works now! Thanks for the help everyone.

For documentation sake, I think it would be helpful to add this in the docs, a short tutorial on how to exclude directories and files for a reverse-proxy.

1 Like

We’re adding plenty of examples for request matching to the docs soon:

2 Likes