Header directive is not mutually exclusive

koledi · May 14, 2020, 5:53am

1. Caddy version (`caddy version`):

v2.0.0 h1:pQSaIJGFluFvu8KDGDODV8u4/QRED/OPyIR+MWYYse8=

2. How I run Caddy:

CLI

a. System environment:

Solus 4.1

b. Command:

./caddy adapt --pretty --validate --config caddy.conf --adapter caddyfile > caddy.json
./caddy run --config caddy.json

~~#### c. Service/unit/compose file:~~

d. My complete Caddyfile or JSON config:

koledi.com:4430 www.koledi.com:4430 {
  bind 127.0.0.1

  ## Self-signed
  tls cert.pem cert.key

  @www {
    host www.koledi.com
  }

  redir @www https://koledi.com:4430{uri} permanent

  route {
    header {
      -server
      Cache-Control "max-age=604800, public"
    }

    header /libs/* {
      Cache-Control "public, max-age=31536000, immutable"
    }
  }

  reverse_proxy localhost:8080
}

3. The problem I’m having:

curl -IL https://koledi.com/libs/foo.js

Cache-Control response header is “max-age=604800, public”, expects “public, max-age=31536000, immutable”

4. Error messages and/or full log output:

5. What I already tried:

Reorder header directives.
Change path pattern /libs/*, /libs*, /libs/, /libs
Remove route {}

6. Links to relevant resources:

Header directive adapted from header (Caddyfile directive) — Caddy Documentation

route {
	header           Cache-Control max=age=3600
	header /static/* Cache-Control max=age=31536000
}

francislavoie · May 14, 2020, 6:00am

Luckily there’s a tool just for this! The handle directive:

  handle /libs/* {
    header Cache-Control "public, max-age=31536000, immutable"
  }

  handle {
    header Cache-Control "max-age=604800, public"
  }

Interesting. That might be a bug then. We might need to look into that @matt

Whitestrake · May 14, 2020, 6:09am

Had a similar issue over here:

I might have to look closer at this one…

koledi · May 14, 2020, 6:17am

Using the handle example results in two Cache-Control response headers, instead of replacing the original header,

$ curl -IL https://koledi.com/libs/foo.js
cache-control: public, max-age=31536000, immutable
cache-control: max-age=86400,public

The second header is from the backend (hence the header rewrite).

Whitestrake · May 14, 2020, 6:19am

That’s some weird behaviour.

Your initial configuration works for me.

~/Projects/test
➜ cat Caddyfile
http://:8080 {
  route {
    header {
      Cache-Control "max-age=604800, public"
    }
    header /libs/* {
      Cache-Control "public, max-age=31536000, immutable"
    }
  }
}

~/Projects/test
➜ curl -I localhost:8080/
HTTP/1.1 200 OK
Cache-Control: max-age=604800, public
Server: Caddy
Date: Thu, 14 May 2020 06:17:38 GMT

~/Projects/test
➜ curl -I localhost:8080/libs/
HTTP/1.1 200 OK
Cache-Control: public, max-age=31536000, immutable
Server: Caddy
Date: Thu, 14 May 2020 06:17:43 GMT

Whitestrake · May 14, 2020, 6:23am

Simplified variant as well behaved as expected:

~/Projects/test
➜ cat Caddyfile
http://:8080 {
  route {
    header Cache-Control "max-age=604800, public"
    header /libs/* Cache-Control "public, max-age=31536000, immutable"
  }
}

~/Projects/test
➜ curl -I localhost:8080/
HTTP/1.1 200 OK
Cache-Control: max-age=604800, public
Server: Caddy
Date: Thu, 14 May 2020 06:20:24 GMT


~/Projects/test
➜ curl -I localhost:8080/libs/
HTTP/1.1 200 OK
Cache-Control: public, max-age=31536000, immutable
Server: Caddy
Date: Thu, 14 May 2020 06:20:28 GMT

@francislavoie’s handle method also worked.

~/Projects/test
➜ cat Caddyfile
http://:8080 {
  handle /libs/* {
    header Cache-Control "public, max-age=31536000, immutable"
  }
  handle {
    header Cache-Control "max-age=604800, public"
  }
}

~/Projects/test
➜ curl -I localhost:8080/
HTTP/1.1 200 OK
Cache-Control: max-age=604800, public
Server: Caddy
Date: Thu, 14 May 2020 06:22:36 GMT


~/Projects/test
➜ curl -I localhost:8080/libs/
HTTP/1.1 200 OK
Cache-Control: public, max-age=31536000, immutable
Server: Caddy
Date: Thu, 14 May 2020 06:22:38 GMT

koledi · May 14, 2020, 6:48am

I forgot to mention that my config has -server (updated my first post),

  route {
    header {
      -server
      Cache-Control "max-age=604800, public"
    }

    header /libs/* {
      Cache-Control "public, max-age=31536000, immutable"
    }
  }

If I access https://koledi.com/libs/foo.js, the response header is Cache-Control: max-age=604800, public, instead of 31536000.

Here’s the weird thing, if I comment out the -server,

  route {
    header {
      #-server
      Cache-Control "max-age=604800, public"
    }

    header /libs/* {
      Cache-Control "public, max-age=31536000, immutable"
    }
  }

This will result in duplicate “Cache-Control”, just like my previous comment,

$ curl -IL https://koledi.com/libs/foo.js
cache-control: public, max-age=31536000, immutable
cache-control: max-age=86400,public

I think you should test with reverse_proxy with “cache-control: max-age=86400,public” header.

matt · May 14, 2020, 6:50am

That might be a bug in our docs… why would I write a route for mutually-exclusive header directives…

Whitestrake · May 14, 2020, 6:56am

See here @matt: Header directive is not mutually exclusive - #3 by Whitestrake

Specifically the last comment I left in that thread:

matt · May 14, 2020, 6:58am

@francislavoie and I figured that one out and it is already fixed on master; but I think this is a separate issue because the route block does not change the order of its directives. Our docs currently recommend using route with two header directives inside it for mutual-exclusivity, but I wonder if I wrote that before I had finished designing the route and handle directives… I just think our docs are wrong.

francislavoie · May 14, 2020, 11:29am

I don’t remember us doing a fix for this, are you thinking of the not matcher thing? That was separate.

matt · May 14, 2020, 5:11pm

No, I was referring to the route-ordering bug where ordering a directive’s handler based on its path matcher length was wrong. We fixed that.

Also, the example in the docs are correct, it turns out. So, I was wrong about that.

Just so you know, the reverse proxy appends headers from the backend, it doesn’t replace them. Header operations on the response are applied immediately, unless explicitly deferred until later: Modules - Caddy Documentation

If true, header operations will be deferred until they are written out. Superceded if Require is set. Usually you will need to set this to true if any fields are being deleted.

So if you have a header directive like above, it will set the Cache-Control header right away, then the reverse proxy will append the one from the backend. That’s why the reverse proxy has its own header controls, so you can manipulate headers to/from the backend independently of to/from the client.

francislavoie · May 14, 2020, 5:39pm

Okay so I think this is what we need?

  route {
    header {
      -server
      Cache-Control "max-age=604800, public"
      defer
    }

    header /libs/* {
      Cache-Control "public, max-age=31536000, immutable"
      defer
    }
  }

Read the note at the top:

By default, header operations are performed immediately unless any of the headers are being deleted, in which case the header operations are automatically deferred until the time they are being written to the client.

That would explain why it behaved that way when you had -server in there, because it switches modes automatically if you’re also deleting a header.

Whitestrake · May 14, 2020, 11:18pm

Also, you could tell your reverse proxy to stop adding cache control headers. Or tell Caddy to strip them out:

  reverse_proxy localhost:8080 {
    header_down -Cache-Control
  }

matt · May 14, 2020, 11:21pm

Perhaps the header directive should overwrite changes resulting from reverse_proxy? But I don’t know how to make that work well. Feel free to open an issue on GitHub so we can discuss it. It’d be interesting to see how other servers handle this. I feel like it’s a complicated issue.

koledi · May 15, 2020, 1:30am

http://localhost:8080 {
  file_server

  header Cache-Control "max-age=86400, public"
}

http://localhost:8081 {
  header {
    Cache-Control "max-age=604800, public"
  }

  reverse_proxy localhost:8080 {
    header_up Host localhost
    header_down -Cache-Control
  }
}

$ curl -IL http://localhost:8081

Response header has no Cache-Control.

http://localhost:8080 ...

http://localhost:8081 {
  header {
    Cache-Control "max-age=604800, public"
  }

  reverse_proxy localhost:8080 {
    header_up Host localhost
  }
}

Response header has two Cache-Control.

  reverse_proxy localhost:8080 {
    header_up Host localhost
    header_down "max-age=604800, public"
  }

This works, but I have multiple backends and prefer to use a single header directive, instead of duplicating header_down.

koledi · May 15, 2020, 1:39am

That’s v1 behaviour, if the header matches any response from backend, the response header is replaced, while still forwarding other (unmatched) response headers; v2 appends to it instead (as you mentioned), resulting in duplicate response headers, unless there is header removal or defer.

koledi · May 15, 2020, 1:49am

defer does work, but I need to switch the order (probably related to “Can't get simple alias to work - #9 by matt”, and may have been fixed).

  route {
    header /libs/* {
      Cache-Control "public, max-age=31536000, immutable"
      defer
    }
    header {
      -server
      Cache-Control "max-age=604800, public"
      defer
    }
  }

I also tried comment out the -server and it still works. Removing route also works and regardless of header order.

system · June 14, 2020, 1:49am

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.