Cache-Control control

Help
1

1. Caddy version (caddy version):

latest

2. How I run Caddy:

docker

a. System environment:

ubuntu

b. Command:

paste command here

c. Service/unit/compose file:

paste full file contents here

d. My complete Caddyfile or JSON config:

# GLOBAL
    {
        # Global options block. Entirely optional, https is on by default
        # Optional email key for lets encrypt
        email mail@example.com
        # Optional staging lets encrypt for testing.
        acme_ca https://acme-staging-v02.api.letsencrypt.org/directory
        
        servers {
    		timeouts {
    			read_body   10s
    			read_header 10s
    			write       10s
    			idle        2m
    		}
    		max_header_size 16384
    	}

    }

    # SNIPPETS

    (mustheaders) {
            header {
                    Strict-Transport-Security "max-age=31536000; includesubdomains; preload"
                    Content-Security-Policy "default-src https: 'unsafe-inline' 'unsafe-eval'"
                    X-Content-Type-Options "nosniff"
                    X-Frame-Options "SAMEORIGIN"
                    Referrer-Policy "strict-origin-when-cross-origin"
                    X-Xss-Protection "1; mode=block"
                    Feature-Policy "accelerometer 'none'; ambient-light-sensor 'none'; autoplay 'none'; camera 'none'; encrypted-media 'none'; fullscreen 'self'; geolocation 'none'; gyroscope 'none'; magnetometer 'none'; microphone 'none'; midi 'none'; payment 'none'; picture-in-picture *; speaker 'none'; sync-xhr 'none'; usb 'none'; vr 'none'"
                    Expect-CT "max-age=604800"
                    -Server
            }
    }
    (offlinewebsite) {
            header {
                    X-Robots-Tag "noindex, nofollow, noarchive, nosnippet, notranslate, noimageindex"
            }
            basicauth * {
                    admin hashed-password
            }
    }
    (onlinewebsite) {
            header {
                    X-Robots-Tag "noarchive, notranslate"
            }
    }

    (compression) {
            encode zstd gzip
    }

    (caching) {
            header {
                    Cache-Control "public, max-age=604800, must-revalidate"
            }
    }

    (security) {
     
            # Unusual URL rewrite
            try_files {path} {path}/ /index.*

            # deny all access to these folders
            @denied_folders path_regexp /(\.github|cache|bin|logs|backup.*|test.*|content|core|image.*|js|css|php|config|lib|assets|rel|priv|tracker)/.*$
            respond @denied_folders "Access denied" 403
           
            # deny running scripts inside core system folders
            @denied_system_scripts path_regexp /(core|content|test|system|vendor)/.*\.(txt|xml|md|html|yaml|php|pl|py|cgi|twig|sh|bat|yml|js)$
            respond @denied_system_scripts "Access denied" 403

            # deny running scripts inside user folder
            @denied_user_folder path_regexp /user/.*\.(txt|md|yaml|php|pl|py|cgi|twig|sh|bat|yml|js)$
            respond @denied_user_folder "Access denied" 403

            # deny access to specific files in the root folder
            @denied_root_folder path_regexp /(index.php.*|wp-admin.php|wp-login.php|wp-config.php.*|xmlrpc.php|config.production.json|config.development.json|index.js|package.json|renovate.json|.*lock|mix.*|ghost.js|startup.js|\.editorconfig|\.eslintignore|\.eslintrc.json|\.gitattributes|\.gitignore|\.gitmodules|\.npmignore|Gruntfile.js|LICENSE|MigratorConfig.js|LICENSE.txt|composer.lock|composer.json|nginx.conf|web.config|htaccess.txt|\.htaccess)
            respond @denied_root_folder "Access denied" 403

            # block bad crawlers
            @badbots header User-Agent "aesop_com_spiderman, alexibot, backweb, batchftp, bigfoot, blackwidow, blowfish, botalot, buddy, builtbottough, bullseye, cheesebot, chinaclaw, cosmos, crescent, curl, custo, da, diibot, disco, dittospyder, dragonfly, drip, easydl, ebingbong, erocrawler, exabot, eyenetie, filehound, flashget, flunky, frontpage, getright, getweb, go-ahead-got-it, gotit, grabnet, grafula, harvest, hloader, hmview, httplib, humanlinks, ilsebot, infonavirobot, infotekies, intelliseek, interget, iria, jennybot, jetcar, joc, justview, jyxobot, kenjin, keyword, larbin, leechftp, lexibot, lftp, libweb, likse, linkscan, linkwalker, lnspiderguy, lwp, magnet, mag-net, markwatch, memo, miixpc, mirror, missigua, moget, nameprotect, navroad, backdoorbot, nearsite, netants, netcraft, netmechanic, netspider, nextgensearchbot, attach, nicerspro, nimblecrawler, npbot, openfind, outfoxbot, pagegrabber, papa, pavuk, pcbrowser, pockey, propowerbot, prowebwalker, psbot, pump, queryn, recorder, realdownload, reaper, reget, true_robot, repomonkey, rma, internetseer, sitesnagger, siphon, slysearch, smartdownload, snake, snapbot, snoopy, sogou, spacebison, spankbot, spanner, sqworm, superbot, superhttp, surfbot, asterias, suzuran, szukacz, takeout, teleport, telesoft, thenomad, tighttwatbot, titan, urldispatcher, turingos, turnitinbot, *vacuum*, vci, voideye, libwww-perl, widow, wisenutbot, wwwoffle, xaldon, xenu, zeus, zyborg, anonymouse, *zip*, *mail*, *enhanc*, *fetch*, *auto*, *bandit*, *clip*, *copier*, *master*, *reaper*, *sauger*, *quester*, *whack*, *picker*, *catch*, *vampire*, *hari*, *offline*, *track*, *craftbot*, *download*, *extract*, *stripper*, *sucker*, *ninja*, *clshttp*, *webspider*, *leacher*, *collector*, *grabber*, *webpictures*, *seo*, *hole*, *copyright*, *check*"
            respond @badbots "Access denied" 403
    }

    (proxy) {
            header_up X-Forwarded-Proto {scheme}
            header_up X-Forwarded-For {remote}
            header_up X-Real-IP {remote}
            header_down X-Powered-By "the Holy Spirit"
            header_down Server "CERN httpd"
    }

    (logs) {
            log {
                output file /var/log/caddy/caddy.log
                format single_field common_log
            }
    }

    # STRIP WWW PREFIX

    www.example.com {
            redir * https://{http.request.host.labels.1}.{http.request.host.labels.0}{path} permanent
    }

    # WEBSITES

    example.com {
            import mustheaders
            import offlinewebsite
            import security
            import caching
            header /wp-dmin/* /folder/* {
                Cache-Control: no-cache, no-store, must-revalidate
            }
            reverse_proxy internal_IP:2351 {
                    import proxy
            }

            import logs
    }

3. The problem I’m having:

Can I do this by inserting caching by snippet and adding another cache rule (no caching for 2 directories and its subdirectories)? please, check example.com section.

Also, I am thinking about modifing the caching snippet into this. does it make sense?:

(caching) {
        header {
                Cache-Control "public, max-age=604800, must-revalidate"
        }
        header / {
                Cache-Control "public, max-age=3600, must-revalidate"
        }
        header /sitemap* {
                Cache-Control: no-cache, no-store, must-revalidate
        }
}

4. Error messages and/or full log output:

5. What I already tried:

6. Links to relevant resources:

1 Like
2

This would only match exactly /, just FYI.

This header isn’t right, you need to remove the : and wrap your value in quotes like the other two.

That said, I don’t understand your question. Could you elaborate? Caching is entirely a business decision.

3

I want to use common caching rules for all websites thanks to “caching” snippet and e.g. add more extra under example.com as shown below.

What I want to accomplish:

(caching) {
        #cache rule for all pages except these listed with other headers
        header {
                Cache-Control "public, max-age=604800, must-revalidate"
        }
        # short  caching of the homepage
        header / {
                Cache-Control "public, max-age=3600, must-revalidate"
        }
        # no cache for sitemaps
        header /sitemap* {
                Cache-Control "no-cache, no-store, must-revalidate"
        }
}

No cache for admin Wordpress pages e.g.

example.com {
            import mustheaders
            import offlinewebsite
            import security
            import caching
            header /wp-dmin/* /otherdir/* {
                Cache-Control "no-cache, no-store, must-revalidate"
            }
            reverse_proxy internal_IP:2351 {
                    import proxy
            }

Do you think it is overkill? :wink:

4

I can’t answer that, like I said it’s entirely dependent on your business decisions.

You can’t have two path matchers inline. You must use a named matcher in that case:

5
  1. is “caching” snippet code ok?
    (caching) {
            #cache rule for all pages except these listed with other headers
            header {
                    Cache-Control "public, max-age=604800, must-revalidate"
            }
            # short  caching of the homepage
            header / {
                    Cache-Control "public, max-age=3600, must-revalidate"
            }
            # no cache for sitemaps
            header /sitemap* {
                    Cache-Control "no-cache, no-store, must-revalidate"
            }
    }
  1. name macher for " header /wp-dmin/* /otherdir/* like this:
    @nocacheadmin {
           /wp-dmin/*
           /otherdir/*
    }

    header @nocacheadmin {
           Cache-Control "no-cache, no-store, must-revalidate"
    }
6

No, please read the docs I linked. You need to specify a matcher to use. A “named matcher” is just a grouping of matchers. You didn’t tell it to use a path matcher.

Run caddy adapt and/or caddy validate after making changes to your Caddyfile to make sure the syntax is right.

7
  1. I ask again, is “caching” snippet code, syntax ok?
  2. You said I should use “named matcher”, so I did. What’s wrong? Instead giving me a link to docs, perhaps you could re-edit the code, please?
8

We can answer yes/no questions all night long, but it’s best if you’re equipped to discern if the config is right or not. Caddy already provides a tool to validate the syntax as syntax only via caddy validate, but there’s still the matter of semantic check which requires understanding what the designed config is doing. This requires someone who knows what their own business requirements are, which happens to be you.

Again, we can fix the snippet for you, but this will ignore the larger issue of clearly having a knowledge gap of the syntax of named matchers. This is why linking to the docs is important. We’re trying empower the user to be independent rather than being dependent. Not only will this help us as maintainers to reduce our load, it will also help you reduce the feedback cycle and not have to wait on someone in the forums who may or may not have the time to answer you every time.

4 Likes
9

Is it ok now where some directories have different caching?

(caching) {
        header Cache-Control "public, max-age=604800, must-revalidate"

        @short_cache {
                path /
                path /author/*
                path /tag/*
                path /category/*

        }
        handle @short_cache {
                header Cache-Control "public, max-age=86400, must-revalidate"
        }
        @no_cache {
                path /sitemap*
                path */rss/
        }
        handle @no_cache {
                header Cache-Control "no-cache, no-store, must-revalidate"
        }
}

and website some cache rules

example.com {
        import mustheaders
        import offlinewebsite
        import caching
        @ghost_cache {
                path /ghost/*
                path /p/*
        }
        handle @ghost_cache {
                header Cache-Control "no-cache, no-store, must-revalidate"
        }
        import security
        reverse_proxy Internal_IP:2351 {
                import proxy
        }

        import logs
}
10

Syntax-wise, this looks fine. Again, behavior-wise depends on how your website/CMS operates and what you’d like to be cached and how often the content is updated.

1 Like
11

Cache-Control not working as it expected. I checked this by clicking a file in browser console to find out the headers.

My current cache settings are:

(caching) {

        @static {
                file
                path *.css *.js *.ico *.gif *.jpg *.jpeg *.png *.svg *.woff
        }
        handle @static {
	        header Cache-Control "public, max-age=5184000, must-revalidate"
        }
        @nocache {
                path /sitemap*
                path /rss/
        }
        handle @nocache {
	        header Cache-Control "no-cache, no-store, must-revalidate"
        }
        handle {
                header Cache-Control "public, max-age=604800, must-revalidate"
        }
}

and

example.com {
        import mustheaders
        import offlinewebsite
        import caching
        @ghost_nocache {
                path /ghost/*
                path /p/*
                path /membership/*
                path /account/*
        }
        handle @ghost_nocache {
	        header Cache-Control "no-cache, no-store, must-revalidate"
        }
        import security
        reverse_proxy IP:2351 {
                import proxy
        }

        import logs
}

For instance, when I click on a image in browser console to check Response Headers, I should see max-age “5184000”, but I can see “604800”.

12

I can’t replicate it. This Caddyfile:

localhost {
	@static {
		file
		path *.css *.js *.ico *.gif *.jpg *.jpeg *.png *.svg *.woff
	}
	handle @static {
		header Cache-Control "public, max-age=5184000, must-revalidate"
	}
	handle {
		header Cache-Control "public, max-age=604800, must-revalidate"
	}
	file_server browse
}

returns these responses:

~ curl --insecure -sSL -D - 'https://localhost/' -o /dev/null
HTTP/2 200
cache-control: public, max-age=604800, must-revalidate
content-type: text/html; charset=utf-8
server: Caddy
date: Thu, 18 Mar 2021 18:42:43 GMT

 ~ curl --insecure -sSL -D - 'https://localhost/fresh.gif' -o /dev/null
HTTP/2 200
accept-ranges: bytes
cache-control: public, max-age=5184000, must-revalidate
content-type: image/gif
etag: "qja74l3ie5e"
last-modified: Wed, 04 Nov 2020 16:57:57 GMT
server: Caddy
content-length: 5896994
date: Thu, 18 Mar 2021 18:42:52 GMT

Which is exactly what’s expected.

1 Like
13

This what I got for the image file:

HTTP/2 200
accept-ranges: bytes
cache-control: public, max-age=604800, must-revalidate #### That's wrong Caddy setting
cache-control: public, max-age=31536000 ##### It was added by Ghost CMS
content-security-policy: default-src https: 'unsafe-inline' 'unsafe-eval'
content-type: image/jpeg
date: Thu, 18 Mar 2021 19:47:23 GMT
etag: W/"19d00-178268f9512"
expect-ct: max-age=604800
feature-policy: accelerometer 'none'; autoplay 'none'; camera 'none'; encrypted-media 'none'; fullscreen 'self'; geolocation 'none'; gyroscope 'none'; magnetometer 'none'; microphone 'none'; midi 'none'; payment 'none'; picture-in-picture *; sync-xhr 'none'; usb 'none'
last-modified: Fri, 12 Mar 2021 13:10:47 GMT
referrer-policy: same-origin
strict-transport-security: max-age=31536000; includesubdomains; preload
x-content-type-options: nosniff
x-frame-options: SAMEORIGIN
x-powered-by: the Holy Spirit
x-robots-tag: noarchive, notranslate
x-xss-protection: 1; mode=block
content-length: 105728
14

What’s the URI/path of the image?

15

Ah, I think you might need to use the defer option of the header directive if your upstream is setting its own headers. This is because Caddy will set its headers before proxying, by default, so nothing stops the proxy from adding its own.

header {
	Cache-Control "public, max-age=604800, must-revalidate"
	defer
}

Or if you want to only set the header if the upstream didn’t define one, then you can use the ? operator (which I believe will implicitly enable defer, but I’m not certain, it’s a new feature that I haven’t played with yet):

header ?Cache-Control "public, max-age=604800, must-revalidate"
3 Likes
16 
validate: adapting config using caddyfile: parsing caddyfile tokens for 'handle': Caddyfile:73 - Error during parsing: unrecognized directive: defer
17

Please re-read what I wrote – defer is an option for header, not a directive.

1 Like
18

This topic was automatically closed after 30 days. New replies are no longer allowed.