Repeated Server Header

[borjapazr (Borja Paz Rodríguez)]

1. The problem I’m having:

I am trying to replace the Server response header with a custom value. The problem is that with the following configuration the Server header is being sent twice, once with the value I set in the Caddy configuration and once with the value set by the upstream service.

What am I doing wrong, should I use > to defer writing the header?

Thanks in advance

2. Error messages and/or full log output:

3. Caddy version:

v2.8.4 h1:q3pe0wpBj1OcHFZ3n/1nl4V4bxBrYoSoab7rL9BMYNk=

4. How I installed and ran Caddy:

Using Docker and Docker Compose.

a. Service/unit/compose file:

services:
  caddy:
    build:
      context: .
      dockerfile: Dockerfile
    image: example/caddy
    container_name: caddy
    hostname: caddy
    restart: unless-stopped
    environment:
      - TLS__EMAIL=${TLS__EMAIL}
      - TLS__CLOUDFLARE_API_TOKEN=${TLS__CLOUDFLARE_API_TOKEN}
      - DOMAINS__001=${DOMAINS__001}
      - TZ=${GLOBAL__TIMEZONE}
    volumes:
      - ./Caddyfile:/etc/caddy/Caddyfile
      - ./site:/srv
      - ${GLOBAL__VOLUME_DIR}/data:/data
      - ${GLOBAL__VOLUME_DIR}/config:/config
      - ${GLOBAL__VOLUME_DIR}/logs:/logs
    networks:
      - default
      - reverse-proxy
    ports:
      - 80:80
      - 443:443
      - 443:443/udp

networks:
  default:
    name: caddy
  reverse-proxy:
    name: reverse-proxy
    external: true

b. My complete Caddy config:

# Global Configuration
## Global options block. Entirely optional, HTTPS is ON by default.
{
	### TLS configuration
	email {$TLS__EMAIL}

	### Enable debug mode
	debug
}

# Snippets
## Snippets are reusable configuration blocks that can be included in multiple sites.
(security) {
	header {
		### Replace Server header
		Server "Example Server"

		### Disable FLoC tracking
		Permissions-Policy "interest-cohort=()"

		### Enable HSTS
		Strict-Transport-Security "max-age=31536000; includeSubDomains; preload"

		### Upgrade insecure requests to HTTPS
		Content-Security-Policy "upgrade-insecure-requests"

		### Disable clients from sniffing the media type
		X-Content-Type-Options "nosniff"
		X-XSS-Protection "1; mode=block"
		Referrer-Policy "strict-origin-when-cross-origin"

		### Clickjacking protection
		?X-Frame-Options "SAMEORIGIN"
	}
}

(compression) {
	### Enable compression
	encode zstd gzip
}

(log) {
	log {args[0]} {
		hostnames {args[0]}
		level INFO
		output file /logs/{args[0]}.log {
			roll_size 3MiB
			roll_keep 5
			roll_keep_for 48h
		}
		format json
	}
}

(service) {
	import log {args[0]}

	@{args[1]} host {args[0]}

	handle @{args[1]} {
		import security
		import compression

		reverse_proxy {args[1]}:{args[2]}
	}
}

# Sites
## Configuration for defining sites.
*.{$DOMAINS__001}, {$DOMAINS__001} {
	tls {$TLS__EMAIL} {
		dns cloudflare {$TLS__CLOUDFLARE_API_TOKEN}
		resolvers 1.1.1.1 8.8.8.8 8.8.4.4
	}

	import service wordpress.{$DOMAINS__001} wordpress 80

	handle {
		abort
	}
}

[matt (Matt Holt)]

That’s normal and expected, as it shows you that the request hit both servers.

You can override headers from the upstream by using header_down inside the reverse_proxy directive. It works the same way header directive does, but it applies specifically to headers going toward the downstream (client).

[borjapazr (Borja Paz Rodríguez)]

Thank you, @matt. Understood.

I have a couple of additional questions.

Would using the > (defer) option with the header directive be equivalent to using header_down inside the reverse_proxy directive? Making the change I see that it overwrites the header Server from the upstream server with the one I specify and the client only gets one.

# Snippets
## Snippets are reusable configuration blocks that can be included in multiple sites.
(security) {
	header {
		>Server "Example Server"

		...
	}
}

On the other hand, could this header Server be set globally without having to specify it in each reverse_proxy/header_down directive?

Thanks again!

[matt (Matt Holt)]

Yes, you can instead defer in the header directive to apply the changes after the reverse proxy sets the header from the upstream.

Caddy doesn’t really support “global” configuration (there are “global options” but those mainly pertain to things that don’t fit in individual site blocks).

[borjapazr (Borja Paz Rodríguez)]

Great!

And the last question, is there any difference to consider between using defer in the header directive and using the reverse_proxy/header_down directive for this purpose?

Thanks again!

[matt (Matt Holt)]

header with defer overwrites pretty much any changes to the header while handling the request, whereas header_down in the reverse proxy affects only the proxied headers.

[borjapazr (Borja Paz Rodríguez)]

Hello again, @matt .

Sorry for asking another question. Going back to the use of the header directive, I see from the documentation that it says the following:

header (Caddyfile directive) - Caddy Documentation

Based on that documentation, shouldn’t the header set by the upstream service be overridden? You said no, that the expected behavior is to keep both headers (in this case the Server header), but why? Because the directive is evaluated immediately?

Thanks again.

[matt (Matt Holt)]

It’s overwritten at the time the operation is evaluated, which is earlier in the handler chain than the proxy, unless it’s deferred to the very end.

[borjapazr (Borja Paz Rodríguez)]

Hello, @matt .

Great, got it!

I’ve been doing some more testing with the header directive these days and, going back to the original example, isn’t defer supposed to be enabled when using the ? prefix to set a default value for a header if it doesn’t exist?

...
(security) {
	header {
		### Replace Server header
		Server "Example Server"

		...

		### Clickjacking protection
		?X-Frame-Options "SAMEORIGIN"
	}
}
...

In this example, by setting a default value for the X-Frame-Options header, the defer should be activated and, therefore, the Server header should overwrite all other set values since operations on the headers are deferred until the time when the response is written to the client. But that’s not really the case. The above example causes two Server headers to be sent to the client, one with the value Example Server and one with the value set by the upstream service (as in the original question).

Instead, this works if I use - or > in any other header, for example:

...
(security) {
	header {
		### Replace Server header
		Server "Example Server"

		...

		### Clickjacking protection
		-X-Frame-Options "SAMEORIGIN"
	}
}
...

This results in the Server header value being set to Example Server overwriting the one that might be set by the upstream service.

Is there a bug with this or is the documentation not correct?

Thanks in advance

[matt (Matt Holt)]

Hmm, you’re right, that doesn’t seem to be setting deferred: true in the JSON.

I found in the commit history that the deferred logic was removed because:

It removes the need to specify the header as “deferred” because it is
already implicitely deferred by the use of the require handler. This
should be less confusing to the user.

Implement default header values in Caddyfile

I wonder if we need to revisit this.

@borjapazr Would you mind opening an issue and link to this thread? Doesn’t have to be super detailed since you’ve done all that here. Just so we don’t forget about this.

[francislavoie]

In that case, you should use multiple header directives, not just one big one. So like:

header Server "Example Server"
header ?X-Frame-Options "SAMEORIGIN"

Or something like that, so that they have their modes separated.

[matt (Matt Holt)]

That’s probably not a bad idea anyway.

[borjapazr (Borja Paz Rodríguez)]

Hello, @matt .

Sorry for the delay in replying. It makes sense to me that this is not the expected behavior, at least with respect to what the documentation says. It would be great if you can review it to clarify what should be the expected behavior in the case I describe or if the documentation needs to be updated.

I’ll open the issue on GitHub gladly, of course.

EDIT: This is the GitHub issue: Defer not activated when using the `?` prefix for default header value inside a header block · Issue #6473 · caddyserver/caddy · GitHub

Thanks again.

Best regards.

[borjapazr (Borja Paz Rodríguez)]

Hello, @francislavoie .

I understand what you are saying. I guess you are commenting this as a recommendation, right? In fact in the documentation it mentions the following regarding the use of the ? prefix.

Anyway, I think this has nothing to do with the behavior I describe, since what is also said in the documentation is that if the ? prefix is used in a block of headers the operations on them will be deferred to the time when the response is to be written to the client. But this is not happening when the ? prefix is used.

Thanks!

[francislavoie]

I did a bit of testing, I don’t see a problem here with how Caddy behaves. Maybe the problem statement needs to be restated in simple terms, I kinda lost the plot with this thread.

I tried this:

:8881 {
    header ?X-Frame-Options "downstream"
    header >Foo bar
    reverse_proxy :8882
}

:8882 {
    header X-Frame-Options "upstream"
}

Then made a request curl -v http://localhost:8881:

$ curl -v http://localhost:8881
...
< HTTP/1.1 200 OK
< Content-Length: 0
< Date: Thu, 25 Jul 2024 14:58:27 GMT
< Foo: bar
< Server: Caddy
< Server: Caddy
< X-Frame-Options: upstream

This is the expected outcome:

• Two server headers because it went through Caddy twice with reverse_proxy
• Foo got set normally with defer (defer wouldn’t matter here though)
• We got the upstream’s value for X-Frame-Options and the default value did _not_override it, as expected

If you adapt your Caddyfile to JSON (with caddy adapt -c /path/to/Caddyfile -p) you’ll see that actually whether you put ? inside a large header block or have it on its own, either way the Caddyfile adapter actually yoinks it out of the block and makes a separate handler for it so that the "require" rule for it is kept separate from the rest of the header operations.

When you use ? it sets require. If you use > it sets deferred. In code, if either of those options are used, then it has the effect of deferring the operation until on the way out. Setting deferred at the same time as require is unnecessary because they mean the same thing except that require lets you declare a condition and isn’t just a simple true/false like deferred is.

[borjapazr (Borja Paz Rodríguez)]

Hello, @francislavoie

I understand the case you comment, but I think it is different from what I raise. In my case I don’t want to use several header directives separately, I want to use a single header directive to configure all the header fields. I will try to recapitulate my use case to evaluate if it is a bug or it is really the expected behavior.

In the documentation related to the header directive the following is specified:

From what I understand in that paragraph it says that defer is going to be enabled for that header block if the prefixes ?, - or > are used in any of the header fields. So, based on this documentation, I did the following test:

:8881 {
	header {
		Server "Example"
		?Downstream-Header "downstream"
	}
	reverse_proxy :8882
}

:8882 {
	header Upstream-Header "upstream"
}

Based on the above configuration and as the documentation says, I expect defer to be activated for the whole block of headers and, therefore, the Server header to be overwritten just before sending the response to the client. That said, the Server header of the upstream service (port :8882) should be overwritten by Example. But actually the response I get is as follows:

caddy on main !2 ❯ curl -v http://127.0.0.1:8881
...
< HTTP/1.1 200 OK
< Content-Length: 0
< Date: Thu, 25 Jul 2024 21:11:56 GMT
< Downstream-Header: downstream
< Server: Example
< Server: Caddy
< Upstream-Header: upstream
...

As you can see, two Server headers are being sent. It appears that the defer is not being done and, therefore, the Server header value is not overwritten to send only Server: Example.

On the other hand, doing the same test by changing the prefix ? for -, the behavior is as expected and as indicated in the documentation.

:8881 {
	header {
		Server "Example"
		-Downstream-Header "downstream"
	}
	reverse_proxy :8882
}

:8882 {
	header Upstream-Header "upstream"
}

caddy on main !2 ❯ curl -v http://127.0.0.1:8881
...
< HTTP/1.1 200 OK
< Content-Length: 0
< Date: Thu, 25 Jul 2024 21:15:05 GMT
< Server: Example
< Upstream-Header: upstream
...

In this case it seems that defer is active and the header Server of the upstream service is overwritten by the value Example and only one is sent to the client. This is in line with what is said in the documentation.

Finally, I ran the following test. I explicitly added defer to the block of headers in which there is a field of a header with the prefix ?.

:8881 {
	header {
		Server "Example"
		?Downstream-Header "downstream"
		defer
	}
	reverse_proxy :8882
}

:8882 {
	header Upstream-Header "upstream"
}

caddy on main !2 ❯ curl -v http://127.0.0.1:8881
...
< HTTP/1.1 200 OK
< Content-Length: 0
< Date: Thu, 25 Jul 2024 21:17:22 GMT
< Downstream-Header: downstream
< Server: Example
< Upstream-Header: upstream
...

In this case the result is as expected, since only one header Server with the value Example is sent. With what is specified in the documentation and, under my interpretation, this should work this way also in the first case I mention, where the prefix ? is used in a header field inside a header block and without explicitly activating defer.

I hope that you have understood the problem I am detailing. Please do not hesitate to tell me if something is not understood correctly. If this is the expected behavior, please could you explain what the reasoning is? From what I can read in the documentation, the behavior of the prefixes ?, - and > should be the same in terms of defer activation. As a user I think it is important to clarify this, as it can lead to unwanted behavior.

Once again, thank you for your patience and hard work

Best regards.

[francislavoie]

@borjapazr

From what I understand in that paragraph it says that defer is going to be enabled for that header block if the prefixes ?, - or > are used in any of the header fields. So, based on this documentation, I did the following test:

@borjapazr

	header {
		Server "Example"
		?Downstream-Header "downstream"
	}

Right, okay. So how ? actually works doesn’t 100% match that documentation. What it effectively does is produce this in JSON config:

	header Server "Example"
	header ?Downstream-Header "downstream"

In other words, if a ? appears, it gets taken out of that block and a new header handler is created with the condition on there not having been another Downstream-Header on the response.

And similarly, this:

	header {
		Foo Bar
		Bar Baz
		?Down-A a
		?Down-B b
	}

would become:

	header {
		Foo Bar
		Bar Baz
	}
	header ?Down-A a
	header ?Down-B b

Hopefully that clarifies that.

If you want the Server to be deferred, just put > in front of it:

	header {
		>Server "Example"
		?Downstream-Header "downstream"
	}

But really, just split it up anyway, less lines:

	header >Server "Example"
	header ?Downstream-Header "downstream"

[borjapazr (Borja Paz Rodríguez)]

Hello again, @francislavoie .

Ok, with this you comment now it is clear to me

With the issue now clarified, don’t you think it would be convenient to update the documentation to reflect the actual behavior of the ? prefix in a block of headers? As it is now, it could lead to confusion.

Thank you

[francislavoie]

Yes, I’ll update the docs re ?

[matt (Matt Holt)]

Ah wow, thank you for clarifying that I didn’t even notice that.