Headers defined in a directive in the site block are not able to be overwritten by header directives in the handlers

1. The problem I’m having:

I have security headers in a file /etc/caddy/security_headers.caddy and i import the file into my wild card site blocks then i use handlers for each of the subdomains, and whenever it is necessary for the upstream service i have setup header directives to find and replace the parts of the headers so that everything works, but recently caddy stopped running the find and replace operations on headers so the parts of the security headers that need to be changed are not getting change

2. Error messages and/or full log output:

request to the server:

$ curl -vL https://pve.sniper.cat
* Host pve.sniper.cat:443 was resolved.
* IPv6: (none)
* IPv4: 192.168.1.1
*   Trying 192.168.1.1:443...
* ALPN: curl offers h2,http/1.1
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
*  CAfile: /etc/ssl/certs/ca-certificates.crt
*  CApath: none
* TLSv1.3 (IN), TLS handshake, Server hello (2):
* TLSv1.3 (IN), TLS change cipher, Change cipher spec (1):
* TLSv1.3 (IN), TLS handshake, Encrypted Extensions (8):
* TLSv1.3 (IN), TLS handshake, Certificate (11):
* TLSv1.3 (IN), TLS handshake, CERT verify (15):
* TLSv1.3 (IN), TLS handshake, Finished (20):
* TLSv1.3 (OUT), TLS change cipher, Change cipher spec (1):
* TLSv1.3 (OUT), TLS handshake, Finished (20):
* SSL connection using TLSv1.3 / TLS_AES_128_GCM_SHA256 / x25519 / id-ecPublicKey
* ALPN: server accepted h2
* Server certificate:
*  subject: CN=*.sniper.cat
*  start date: Mar 22 18:43:13 2025 GMT
*  expire date: Jun 20 18:43:12 2025 GMT
*  subjectAltName: host "pve.sniper.cat" matched cert's "*.sniper.cat"
*  issuer: C=US; O=Let's Encrypt; CN=E5
*  SSL certificate verify ok.
*   Certificate level 0: Public key type EC/prime256v1 (256/128 Bits/secBits), signed using ecdsa-with-SHA384
*   Certificate level 1: Public key type EC/secp384r1 (384/192 Bits/secBits), signed using sha256WithRSAEncryption
*   Certificate level 2: Public key type RSA (4096/152 Bits/secBits), signed using sha256WithRSAEncryption
* Connected to pve.sniper.cat (192.168.1.1) port 443
* using HTTP/2
* [HTTP/2] [1] OPENED stream for https://pve.sniper.cat/
* [HTTP/2] [1] [:method: GET]
* [HTTP/2] [1] [:scheme: https]
* [HTTP/2] [1] [:authority: pve.sniper.cat]
* [HTTP/2] [1] [:path: /]
* [HTTP/2] [1] [user-agent: curl/8.12.1]
* [HTTP/2] [1] [accept: */*]
> GET / HTTP/2
> Host: pve.sniper.cat
> User-Agent: curl/8.12.1
> Accept: */*
> 
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
* Request completely sent off
< HTTP/2 200 
< alt-svc: h3=":443"; ma=2592000
< cache-control: max-age=0
< content-security-policy: default-src 'self'; script-src 'self'; style-src 'self' 'unsafe-inline' https://fonts.googleapis.com; img-src 'self' data:; connect-src 'self'; font-src 'self' data: https://fonts.gstatic.com; object-src 'none'; media-src 'self'; child-src 'self'; frame-ancestors 'none';
< content-type: text/html; charset=utf-8
< cross-origin-embedder-policy: require-corp
< cross-origin-opener-policy: same-origin
< cross-origin-resource-policy: same-site
< date: Sun, 06 Apr 2025 12:43:21 GMT
< expires: Sun, 06 Apr 2025 12:43:21 GMT
< permissions-policy: interest-cohort=()
< pragma: no-cache
< referrer-policy: no-referrer
< strict-transport-security: max-age=31536000; includeSubDomains; preload
< x-content-type-options: nosniff
< 
<!DOCTYPE html>
<html>
  <head>
    <meta http-equiv="Content-Type" content="text/html; charset=utf-8" />
    <meta http-equiv="X-UA-Compatible" content="IE=edge">
    <meta name="viewport" content="width=device-width, initial-scale=1, maximum-scale=1, user-scalable=no">
    <title>proxmox - Proxmox Virtual Environment</title>
    <link rel="icon" sizes="128x128" href="/pve2/images/logo-128.png" />
    <link rel="apple-touch-icon" sizes="128x128" href="/pve2/images/logo-128.png" />
    <link rel="stylesheet" type="text/css" href="/pve2/ext6/theme-crisp/resources/theme-crisp-all.css?ver=7.0.0" />
    <link rel="stylesheet" type="text/css" href="/pve2/ext6/crisp/resources/charts-all.css?ver=7.0.0" />
    <link rel="stylesheet" type="text/css" href="/pve2/fa/css/font-awesome.css" />
    <link rel="stylesheet" type="text/css" href="/pve2/font-logos/css/font-logos.css" />
    <link rel="stylesheet" type="text/css" href="/pve2/css/ext6-pve.css?ver=8.3.5" />
    <link rel="stylesheet" type="text/css" href="/pwt/css/ext6-pmx.css?ver=v4.3.6-t1740503330" />
    <link rel="stylesheet" type="text/css" media="(prefers-color-scheme: dark)" href="/pwt/themes/theme-proxmox-dark.css?ver=v4.3.6-t1740503330" />
    
    <script type='text/javascript'>function gettext(buf) { return buf; }</script>
    
    <script type="text/javascript" src="/pve2/ext6/ext-all.js?ver=7.0.0"></script>
    <script type="text/javascript" src="/pve2/ext6/charts.js?ver=7.0.0"></script>
    
    <script type="text/javascript" src="/pve2/js/u2f-api.js"></script>
    <script type="text/javascript" src="/qrcode.min.js"></script>
    <script type="text/javascript">
    Proxmox = {
	Setup: { auth_cookie_name: 'PVEAuthCookie' },
	defaultLang: 'en',
	NodeName: 'proxmox',
	UserName: '',
	CSRFPreventionToken: 'null'
    };
    </script>
    <script type="text/javascript" src="/proxmoxlib.js?ver=v4.3.6-t1740503330"></script>
    <script type="text/javascript" src="/pve2/js/pvemanagerlib.js?ver=8.3.5"></script>
    <script type="text/javascript" src="/pve2/ext6/locale/locale-en.js?ver=7.0.0"></script>

    <script type="text/javascript">
    if (typeof(PVE) === 'undefined') PVE = {};
    Ext.History.fieldid = 'x-history-field';
    Ext.onReady(function() { Ext.create('PVE.StdWorkspace');});
    </script>

  </head>
  <body>
    <!-- Fields required for history management -->
    <form id="history-form" class="x-hidden">
    <input type="hidden" id="x-history-field"/>
    </form>
  </body>
</html>
* Connection #0 to host pve.sniper.cat left intact

entries from log:

Apr 06 08:43:21 proxmox caddy[3051]: {"level":"debug","ts":1743943401.601594,"logger":"events","msg":"event","name":"tls_get_certificate","id":"a3e94f30-3f5f-4afa-809a-78a77579bd8d","origin":"tls","data":{"client_hello":{"CipherSuites":[4866,4867,4865,49196,49200,159,52393,52392,52394,49195,49199,158,49188,49192,107,49187,49191,103,49162,49172,57,49161,49171,51,157,156,61,60,53,47],"ServerName":"pve.sniper.cat","SupportedCurves":[29,23,30,25,24,256,257,258,259,260],"SupportedPoints":"AAEC","SignatureSchemes":[1027,1283,1539,2055,2056,2074,2075,2076,2057,2058,2059,2052,2053,2054,1025,1281,1537,771,769,770,1026,1282,1538],"SupportedProtos":["h2","http/1.1"],"SupportedVersions":[772,771],"RemoteAddr":{"IP":"192.168.1.62","Port":59234,"Zone":""},"LocalAddr":{"IP":"192.168.1.1","Port":443,"Zone":""}}}}
Apr 06 08:43:21 proxmox caddy[3051]: {"level":"debug","ts":1743943401.6017113,"logger":"tls.handshake","msg":"no matching certificates and no custom selection logic","identifier":"pve.sniper.cat"}
Apr 06 08:43:21 proxmox caddy[3051]: {"level":"debug","ts":1743943401.601719,"logger":"tls.handshake","msg":"choosing certificate","identifier":"*.sniper.cat","num_choices":1}
Apr 06 08:43:21 proxmox caddy[3051]: {"level":"debug","ts":1743943401.601724,"logger":"tls.handshake","msg":"default certificate selection results","identifier":"*.sniper.cat","subjects":["*.sniper.cat"],"managed":true,"issuer_key":"acme-v02.api.letsencrypt.org-directory","hash":"873a0a0f54d486ab966000e2df7d3de9f98ce6807990623bf93d78fd1c8f29e7"}
Apr 06 08:43:21 proxmox caddy[3051]: {"level":"debug","ts":1743943401.6017308,"logger":"tls.handshake","msg":"matched certificate in cache","remote_ip":"192.168.1.62","remote_port":"59234","subjects":["*.sniper.cat"],"managed":true,"expiration":1750444993,"hash":"873a0a0f54d486ab966000e2df7d3de9f98ce6807990623bf93d78fd1c8f29e7"}
Apr 06 08:43:21 proxmox caddy[3051]: {"level":"debug","ts":1743943401.6060371,"logger":"http.handlers.reverse_proxy","msg":"selected upstream","dial":"pve.sniper.cat:8006","total_upstreams":1}
Apr 06 08:43:21 proxmox caddy[3051]: {"level":"debug","ts":1743943401.612066,"logger":"http.handlers.reverse_proxy","msg":"upstream roundtrip","upstream":"pve.sniper.cat:8006","duration":0.005995244,"request":{"remote_ip":"192.168.1.62","remote_port":"59234","client_ip":"192.168.1.62","proto":"HTTP/2.0","method":"GET","host":"pve.sniper.cat","uri":"/","headers":{"User-Agent":["curl/8.12.1"],"Accept":["*/*"],"X-Forwarded-For":["192.168.1.62"],"X-Forwarded-Proto":["https"],"X-Forwarded-Host":["pve.sniper.cat"]},"tls":{"resumed":false,"version":772,"cipher_suite":4865,"proto":"h2","server_name":"pve.sniper.cat"}},"headers":{"Date":["Sun, 06 Apr 2025 12:43:21 GMT"],"Server":["pve-api-daemon/3.0"],"Content-Type":["text/html; charset=utf-8"],"Expires":["Sun, 06 Apr 2025 12:43:21 GMT"],"Cache-Control":["max-age=0"],"Pragma":["no-cache"],"Connection":["Keep-Alive"]},"status":200}

3. Caddy version: v2.9.1 h1:OEYiZ7DbCzAWVb6TNEkjRcSCRGHVoZsJinoDR/n9oaY=

4. How I installed and ran Caddy:

xcaddy installed using cloudsmith debian repos and a systemd service that builds the caddy binary on system startup and a systemd service that runs caddy

$ xcaddy version
v0.4.4 h1:xstmMhsxrDF/1rnncpi+CLbBfxEP0PjjdtXe3YFwnuo=

i run caddy validate --config /etc/caddy/Caddyfile && systemctl reload caddy to reload my caddy configs

a. System environment:

$ uname -a
Linux proxmox 6.8.12-9-pve #1 SMP PREEMPT_DYNAMIC PMX 6.8.12-9 (2025-03-16T19:18Z) x86_64 GNU/Linux

$ systemd --version
systemd 252 (252.36-1~deb12u1)
+PAM +AUDIT +SELINUX +APPARMOR +IMA +SMACK +SECCOMP +GCRYPT -GNUTLS +OPENSSL +ACL +BLKID +CURL +ELFUTILS +FIDO2 +IDN2 -IDN +IPTC +KMOD +LIBCRYPTSETUP +LIBFDISK +PCRE2 -PWQUALITY +P11KIT +QRENCODE +TPM2 +BZIP2 +LZ4 +XZ +ZLIB +ZSTD -BPF_FRAMEWORK -XKBCOMMON +UTMP +SYSVINIT default-hierarchy=unified

b. Command:

the systemd service runs:

/usr/bin/caddy run --environ --config /etc/caddy/Caddyfile

c. Service/unit/compose file:

/etc/systemd/system/build-caddy.service:

[Unit]
Description=Builds a caddy binary with the clodflare module, this has to be seperate because it needs to run as root
After=network.target network-online.target
Requires=network-online.target
Before=caddy.service
PartOf=caddy.service

[Service]
Type=oneshot
Environment="HOME=/root/"
ExecStart=xcaddy build --with github.com/caddy-dns/cloudflare --output /usr/bin/caddy
ExecStartPost=bash -c "caddy completion fish > /etc/fish/completions/caddy.fish"
TimeoutStartSec=30s
RemainAfterExit=true

[Install]
WantedBy=multi-user.target

/etc/systemd/system/caddy.service:

[Unit]
Description=Caddy
Documentation=https://caddyserver.com/docs/
After=network.target network-online.target
Requires=network-online.target

[Service]
Type=notify
User=caddy
Group=caddy
ExecStart=/usr/bin/caddy run --environ --config /etc/caddy/Caddyfile
ExecReload=/usr/bin/caddy reload --config /etc/caddy/Caddyfile --force
TimeoutStopSec=5s
LimitNOFILE=1048576
PrivateTmp=true
ProtectSystem=full
AmbientCapabilities=CAP_NET_ADMIN CAP_NET_BIND_SERVICE

[Install]
WantedBy=multi-user.target

d. My complete Caddy config:

all of the follwing are direct copy/pastes of the output of running cat <file name>

/etc/caddy/Caddyfile

{
        acme_dns cloudflare ***redacted api key***
        metrics {
                per_host
        }
        log access {
                output file /var/log/caddy/access {
                        mode 644
                        hostnames *

                        roll_size 2MiB
                        roll_uncompressed

                        format console
                        level info
                }
        }
        debug
}

import /etc/caddy/sniper.cat.caddy

import /etc/caddy/willsey.cloud.caddy

import /etc/caddy/drav.co.caddy

import /etc/caddy/equilibrium.gay.caddy

/etc/caddy/security_headers.caddy

header {
	-Server

	Cross-Origin-Resource-Policy "same-site"
	Cross-Origin-Embedder-Policy "require-corp"
	Cross-Origin-Opener-Policy   "same-origin"

	Content-Security-Policy      "default-src 'self'; script-src 'self'; style-src 'self' 'unsafe-inline' https://fonts.googleapis.com; img-src 'self' data:; connect-src 'self'; font-src 'self' data: https://fonts.gstatic.com; object-src 'none'; media-src 'self'; child-src 'self'; frame-ancestors 'none';"
	Permissions-Policy           "interest-cohort=()"
	Strict-Transport-Security    "max-age=31536000; includeSubDomains; preload"
	Referrer-Policy              "no-referrer"
	X-Content-Type-Options       "nosniff"
}

/etc/caddy/sniper.cat.caddy

sniper.cat *.sniper.cat *.*.sniper.cat *.*.*.sniper.cat *.*.*.*.sniper.cat {
	import /etc/caddy/security_headers.caddy

	@www host www.sniper.cat
	handle @www {
		redir https://sniper.cat{uri} permanent
	}

	@sniper.cat host sniper.cat
	handle @sniper.cat {
		# Set this path to your site's directory.
		root * /usr/share/caddy

		# Enable the static file server.
		file_server
	}

	@pve host pve.sniper.cat
	handle @pve {
		header {
			Content-Security-Policy "script-src" "script-src 'unsafe-inline' 'unsafe-eval'"
		}

		reverse_proxy https://pve.sniper.cat:8006
	}

	@amp host amp.sniper.cat
	handle @amp {
		reverse_proxy http://192.168.1.2:8080
	}

	@plex host plex.sniper.cat
	handle @plex {
		header {
			Content-Security-Policy "script-src" "script-src 'unsafe-inline' https://www.gstatic.com"
			Content-Security-Policy "connect-src" "connect-src https://*.plex.direct:32400 wss://*.plex.direct:32400 https://*.plex.tv wss://*.plex.tv"
			Content-Security-Policy "img-src" "img-src https://*.plex.direct:32400 https://*.plex.tv"
		}

		reverse_proxy http://192.168.1.3:32400
	}

	@bitwarden host bitwarden.sniper.cat
	handle @bitwarden {
		reverse_proxy http://192.168.1.6:80
	}

	@mox_hosts host mox.sniper.cat mail.sniper.cat mta-sts.sniper.cat autoconfig.sniper.cat
	handle @mox_hosts {
		reverse_proxy http://192.168.1.8:80 {
			header_up Host "192.168.1.8"
		}
	}

	@router host router.sniper.cat
	handle @router {
		respond /* 403
		reverse_proxy http://192.168.1.254:80
	}

	error * 404

	handle_errors {
		rewrite * /{err.status_code}.html
		root * /www/errors
		file_server
	}
}

/etc/caddy/willsey.cloud.caddy

willsey.cloud *.willsey.cloud *.*.willsey.cloud *.*.*.willsey.cloud *.*.*.*.willsey.cloud {
	import /etc/caddy/security_headers.caddy

        @www.willsey.cloud host www.willsey.cloud
        handle @www.willsey.cloud {
                redir https://willsey.cloud{uri} permanent
        }

	@willsey.cloud host willsey.cloud
	handle @willsey.cloud {
		header {
			Content-Security-Policy "connect-src" "connect-src willsey.cloud:3478 wss://willsey.cloud"
			Content-Security-Policy "img-src" "img-src https://usercontent.apps.nextcloud.com"
		}
		reverse_proxy http://192.168.1.4:11000
	}
}

willsey.cloud:8443 {
	import /etc/caddy/security_headers.caddy

	reverse_proxy https://willsey.cloud:8443
}

/etc/caddy/drav.co.caddy

drav.co *.drav.co *.*.drav.co *.*.*.drav.co *.*.*.*.drav.co {
        import /etc/caddy/security_headers.caddy

        @www.drav.co host www.drav.co
        handle @www.drav.co {
                redir https://drav.co{uri} permanent
        }
}

/etc/caddy/equilibrium.gay.caddy

equilibrium.gay *.equilibrium.gay *.*.equilibrium.gay *.*.*.equilibrium.gay *.*.*.*.equilibrium.gay {
        import /etc/caddy/security_headers.caddy

        @www.equilibrium.gay host www.equilibrium.gay
        handle @www.equilibrium.gay {
                redir https://equilibrium.gay{uri} permanent
        }
}

5. Links to relevant resources:

I guess one workaround is to use the ? in your /etc/caddy/security_headers.caddy definitions so they become conditional paired with the use of header_down in reverse-proxy.