1. The problem I’m having:
I have security headers in a file /etc/caddy/security_headers.caddy and i import the file into my wild card site blocks then i use handlers for each of the subdomains, and whenever it is necessary for the upstream service i have setup header directives to find and replace the parts of the headers so that everything works, but recently caddy stopped running the find and replace operations on headers so the parts of the security headers that need to be changed are not getting change
2. Error messages and/or full log output:
request to the server:
$ curl -vL https://pve.sniper.cat
* Host pve.sniper.cat:443 was resolved.
* IPv6: (none)
* IPv4: 192.168.1.1
* Trying 192.168.1.1:443...
* ALPN: curl offers h2,http/1.1
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* CAfile: /etc/ssl/certs/ca-certificates.crt
* CApath: none
* TLSv1.3 (IN), TLS handshake, Server hello (2):
* TLSv1.3 (IN), TLS change cipher, Change cipher spec (1):
* TLSv1.3 (IN), TLS handshake, Encrypted Extensions (8):
* TLSv1.3 (IN), TLS handshake, Certificate (11):
* TLSv1.3 (IN), TLS handshake, CERT verify (15):
* TLSv1.3 (IN), TLS handshake, Finished (20):
* TLSv1.3 (OUT), TLS change cipher, Change cipher spec (1):
* TLSv1.3 (OUT), TLS handshake, Finished (20):
* SSL connection using TLSv1.3 / TLS_AES_128_GCM_SHA256 / x25519 / id-ecPublicKey
* ALPN: server accepted h2
* Server certificate:
* subject: CN=*.sniper.cat
* start date: Mar 22 18:43:13 2025 GMT
* expire date: Jun 20 18:43:12 2025 GMT
* subjectAltName: host "pve.sniper.cat" matched cert's "*.sniper.cat"
* issuer: C=US; O=Let's Encrypt; CN=E5
* SSL certificate verify ok.
* Certificate level 0: Public key type EC/prime256v1 (256/128 Bits/secBits), signed using ecdsa-with-SHA384
* Certificate level 1: Public key type EC/secp384r1 (384/192 Bits/secBits), signed using sha256WithRSAEncryption
* Certificate level 2: Public key type RSA (4096/152 Bits/secBits), signed using sha256WithRSAEncryption
* Connected to pve.sniper.cat (192.168.1.1) port 443
* using HTTP/2
* [HTTP/2] [1] OPENED stream for https://pve.sniper.cat/
* [HTTP/2] [1] [:method: GET]
* [HTTP/2] [1] [:scheme: https]
* [HTTP/2] [1] [:authority: pve.sniper.cat]
* [HTTP/2] [1] [:path: /]
* [HTTP/2] [1] [user-agent: curl/8.12.1]
* [HTTP/2] [1] [accept: */*]
> GET / HTTP/2
> Host: pve.sniper.cat
> User-Agent: curl/8.12.1
> Accept: */*
>
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
* Request completely sent off
< HTTP/2 200
< alt-svc: h3=":443"; ma=2592000
< cache-control: max-age=0
< content-security-policy: default-src 'self'; script-src 'self'; style-src 'self' 'unsafe-inline' https://fonts.googleapis.com; img-src 'self' data:; connect-src 'self'; font-src 'self' data: https://fonts.gstatic.com; object-src 'none'; media-src 'self'; child-src 'self'; frame-ancestors 'none';
< content-type: text/html; charset=utf-8
< cross-origin-embedder-policy: require-corp
< cross-origin-opener-policy: same-origin
< cross-origin-resource-policy: same-site
< date: Sun, 06 Apr 2025 12:43:21 GMT
< expires: Sun, 06 Apr 2025 12:43:21 GMT
< permissions-policy: interest-cohort=()
< pragma: no-cache
< referrer-policy: no-referrer
< strict-transport-security: max-age=31536000; includeSubDomains; preload
< x-content-type-options: nosniff
<
<!DOCTYPE html>
<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8" />
<meta http-equiv="X-UA-Compatible" content="IE=edge">
<meta name="viewport" content="width=device-width, initial-scale=1, maximum-scale=1, user-scalable=no">
<title>proxmox - Proxmox Virtual Environment</title>
<link rel="icon" sizes="128x128" href="/pve2/images/logo-128.png" />
<link rel="apple-touch-icon" sizes="128x128" href="/pve2/images/logo-128.png" />
<link rel="stylesheet" type="text/css" href="/pve2/ext6/theme-crisp/resources/theme-crisp-all.css?ver=7.0.0" />
<link rel="stylesheet" type="text/css" href="/pve2/ext6/crisp/resources/charts-all.css?ver=7.0.0" />
<link rel="stylesheet" type="text/css" href="/pve2/fa/css/font-awesome.css" />
<link rel="stylesheet" type="text/css" href="/pve2/font-logos/css/font-logos.css" />
<link rel="stylesheet" type="text/css" href="/pve2/css/ext6-pve.css?ver=8.3.5" />
<link rel="stylesheet" type="text/css" href="/pwt/css/ext6-pmx.css?ver=v4.3.6-t1740503330" />
<link rel="stylesheet" type="text/css" media="(prefers-color-scheme: dark)" href="/pwt/themes/theme-proxmox-dark.css?ver=v4.3.6-t1740503330" />
<script type='text/javascript'>function gettext(buf) { return buf; }</script>
<script type="text/javascript" src="/pve2/ext6/ext-all.js?ver=7.0.0"></script>
<script type="text/javascript" src="/pve2/ext6/charts.js?ver=7.0.0"></script>
<script type="text/javascript" src="/pve2/js/u2f-api.js"></script>
<script type="text/javascript" src="/qrcode.min.js"></script>
<script type="text/javascript">
Proxmox = {
Setup: { auth_cookie_name: 'PVEAuthCookie' },
defaultLang: 'en',
NodeName: 'proxmox',
UserName: '',
CSRFPreventionToken: 'null'
};
</script>
<script type="text/javascript" src="/proxmoxlib.js?ver=v4.3.6-t1740503330"></script>
<script type="text/javascript" src="/pve2/js/pvemanagerlib.js?ver=8.3.5"></script>
<script type="text/javascript" src="/pve2/ext6/locale/locale-en.js?ver=7.0.0"></script>
<script type="text/javascript">
if (typeof(PVE) === 'undefined') PVE = {};
Ext.History.fieldid = 'x-history-field';
Ext.onReady(function() { Ext.create('PVE.StdWorkspace');});
</script>
</head>
<body>
<!-- Fields required for history management -->
<form id="history-form" class="x-hidden">
<input type="hidden" id="x-history-field"/>
</form>
</body>
</html>
* Connection #0 to host pve.sniper.cat left intact
entries from log:
Apr 06 08:43:21 proxmox caddy[3051]: {"level":"debug","ts":1743943401.601594,"logger":"events","msg":"event","name":"tls_get_certificate","id":"a3e94f30-3f5f-4afa-809a-78a77579bd8d","origin":"tls","data":{"client_hello":{"CipherSuites":[4866,4867,4865,49196,49200,159,52393,52392,52394,49195,49199,158,49188,49192,107,49187,49191,103,49162,49172,57,49161,49171,51,157,156,61,60,53,47],"ServerName":"pve.sniper.cat","SupportedCurves":[29,23,30,25,24,256,257,258,259,260],"SupportedPoints":"AAEC","SignatureSchemes":[1027,1283,1539,2055,2056,2074,2075,2076,2057,2058,2059,2052,2053,2054,1025,1281,1537,771,769,770,1026,1282,1538],"SupportedProtos":["h2","http/1.1"],"SupportedVersions":[772,771],"RemoteAddr":{"IP":"192.168.1.62","Port":59234,"Zone":""},"LocalAddr":{"IP":"192.168.1.1","Port":443,"Zone":""}}}}
Apr 06 08:43:21 proxmox caddy[3051]: {"level":"debug","ts":1743943401.6017113,"logger":"tls.handshake","msg":"no matching certificates and no custom selection logic","identifier":"pve.sniper.cat"}
Apr 06 08:43:21 proxmox caddy[3051]: {"level":"debug","ts":1743943401.601719,"logger":"tls.handshake","msg":"choosing certificate","identifier":"*.sniper.cat","num_choices":1}
Apr 06 08:43:21 proxmox caddy[3051]: {"level":"debug","ts":1743943401.601724,"logger":"tls.handshake","msg":"default certificate selection results","identifier":"*.sniper.cat","subjects":["*.sniper.cat"],"managed":true,"issuer_key":"acme-v02.api.letsencrypt.org-directory","hash":"873a0a0f54d486ab966000e2df7d3de9f98ce6807990623bf93d78fd1c8f29e7"}
Apr 06 08:43:21 proxmox caddy[3051]: {"level":"debug","ts":1743943401.6017308,"logger":"tls.handshake","msg":"matched certificate in cache","remote_ip":"192.168.1.62","remote_port":"59234","subjects":["*.sniper.cat"],"managed":true,"expiration":1750444993,"hash":"873a0a0f54d486ab966000e2df7d3de9f98ce6807990623bf93d78fd1c8f29e7"}
Apr 06 08:43:21 proxmox caddy[3051]: {"level":"debug","ts":1743943401.6060371,"logger":"http.handlers.reverse_proxy","msg":"selected upstream","dial":"pve.sniper.cat:8006","total_upstreams":1}
Apr 06 08:43:21 proxmox caddy[3051]: {"level":"debug","ts":1743943401.612066,"logger":"http.handlers.reverse_proxy","msg":"upstream roundtrip","upstream":"pve.sniper.cat:8006","duration":0.005995244,"request":{"remote_ip":"192.168.1.62","remote_port":"59234","client_ip":"192.168.1.62","proto":"HTTP/2.0","method":"GET","host":"pve.sniper.cat","uri":"/","headers":{"User-Agent":["curl/8.12.1"],"Accept":["*/*"],"X-Forwarded-For":["192.168.1.62"],"X-Forwarded-Proto":["https"],"X-Forwarded-Host":["pve.sniper.cat"]},"tls":{"resumed":false,"version":772,"cipher_suite":4865,"proto":"h2","server_name":"pve.sniper.cat"}},"headers":{"Date":["Sun, 06 Apr 2025 12:43:21 GMT"],"Server":["pve-api-daemon/3.0"],"Content-Type":["text/html; charset=utf-8"],"Expires":["Sun, 06 Apr 2025 12:43:21 GMT"],"Cache-Control":["max-age=0"],"Pragma":["no-cache"],"Connection":["Keep-Alive"]},"status":200}
3. Caddy version: v2.9.1 h1:OEYiZ7DbCzAWVb6TNEkjRcSCRGHVoZsJinoDR/n9oaY=
4. How I installed and ran Caddy:
xcaddy installed using cloudsmith debian repos and a systemd service that builds the caddy binary on system startup and a systemd service that runs caddy
$ xcaddy version
v0.4.4 h1:xstmMhsxrDF/1rnncpi+CLbBfxEP0PjjdtXe3YFwnuo=
i run caddy validate --config /etc/caddy/Caddyfile && systemctl reload caddy to reload my caddy configs
a. System environment:
$ uname -a
Linux proxmox 6.8.12-9-pve #1 SMP PREEMPT_DYNAMIC PMX 6.8.12-9 (2025-03-16T19:18Z) x86_64 GNU/Linux
$ systemd --version
systemd 252 (252.36-1~deb12u1)
+PAM +AUDIT +SELINUX +APPARMOR +IMA +SMACK +SECCOMP +GCRYPT -GNUTLS +OPENSSL +ACL +BLKID +CURL +ELFUTILS +FIDO2 +IDN2 -IDN +IPTC +KMOD +LIBCRYPTSETUP +LIBFDISK +PCRE2 -PWQUALITY +P11KIT +QRENCODE +TPM2 +BZIP2 +LZ4 +XZ +ZLIB +ZSTD -BPF_FRAMEWORK -XKBCOMMON +UTMP +SYSVINIT default-hierarchy=unified
b. Command:
the systemd service runs:
/usr/bin/caddy run --environ --config /etc/caddy/Caddyfile
c. Service/unit/compose file:
/etc/systemd/system/build-caddy.service:
[Unit]
Description=Builds a caddy binary with the clodflare module, this has to be seperate because it needs to run as root
After=network.target network-online.target
Requires=network-online.target
Before=caddy.service
PartOf=caddy.service
[Service]
Type=oneshot
Environment="HOME=/root/"
ExecStart=xcaddy build --with github.com/caddy-dns/cloudflare --output /usr/bin/caddy
ExecStartPost=bash -c "caddy completion fish > /etc/fish/completions/caddy.fish"
TimeoutStartSec=30s
RemainAfterExit=true
[Install]
WantedBy=multi-user.target
/etc/systemd/system/caddy.service:
[Unit]
Description=Caddy
Documentation=https://caddyserver.com/docs/
After=network.target network-online.target
Requires=network-online.target
[Service]
Type=notify
User=caddy
Group=caddy
ExecStart=/usr/bin/caddy run --environ --config /etc/caddy/Caddyfile
ExecReload=/usr/bin/caddy reload --config /etc/caddy/Caddyfile --force
TimeoutStopSec=5s
LimitNOFILE=1048576
PrivateTmp=true
ProtectSystem=full
AmbientCapabilities=CAP_NET_ADMIN CAP_NET_BIND_SERVICE
[Install]
WantedBy=multi-user.target
d. My complete Caddy config:
all of the follwing are direct copy/pastes of the output of running cat <file name>
/etc/caddy/Caddyfile
{
acme_dns cloudflare ***redacted api key***
metrics {
per_host
}
log access {
output file /var/log/caddy/access {
mode 644
hostnames *
roll_size 2MiB
roll_uncompressed
format console
level info
}
}
debug
}
import /etc/caddy/sniper.cat.caddy
import /etc/caddy/willsey.cloud.caddy
import /etc/caddy/drav.co.caddy
import /etc/caddy/equilibrium.gay.caddy
/etc/caddy/security_headers.caddy
header {
-Server
Cross-Origin-Resource-Policy "same-site"
Cross-Origin-Embedder-Policy "require-corp"
Cross-Origin-Opener-Policy "same-origin"
Content-Security-Policy "default-src 'self'; script-src 'self'; style-src 'self' 'unsafe-inline' https://fonts.googleapis.com; img-src 'self' data:; connect-src 'self'; font-src 'self' data: https://fonts.gstatic.com; object-src 'none'; media-src 'self'; child-src 'self'; frame-ancestors 'none';"
Permissions-Policy "interest-cohort=()"
Strict-Transport-Security "max-age=31536000; includeSubDomains; preload"
Referrer-Policy "no-referrer"
X-Content-Type-Options "nosniff"
}
/etc/caddy/sniper.cat.caddy
sniper.cat *.sniper.cat *.*.sniper.cat *.*.*.sniper.cat *.*.*.*.sniper.cat {
import /etc/caddy/security_headers.caddy
@www host www.sniper.cat
handle @www {
redir https://sniper.cat{uri} permanent
}
@sniper.cat host sniper.cat
handle @sniper.cat {
# Set this path to your site's directory.
root * /usr/share/caddy
# Enable the static file server.
file_server
}
@pve host pve.sniper.cat
handle @pve {
header {
Content-Security-Policy "script-src" "script-src 'unsafe-inline' 'unsafe-eval'"
}
reverse_proxy https://pve.sniper.cat:8006
}
@amp host amp.sniper.cat
handle @amp {
reverse_proxy http://192.168.1.2:8080
}
@plex host plex.sniper.cat
handle @plex {
header {
Content-Security-Policy "script-src" "script-src 'unsafe-inline' https://www.gstatic.com"
Content-Security-Policy "connect-src" "connect-src https://*.plex.direct:32400 wss://*.plex.direct:32400 https://*.plex.tv wss://*.plex.tv"
Content-Security-Policy "img-src" "img-src https://*.plex.direct:32400 https://*.plex.tv"
}
reverse_proxy http://192.168.1.3:32400
}
@bitwarden host bitwarden.sniper.cat
handle @bitwarden {
reverse_proxy http://192.168.1.6:80
}
@mox_hosts host mox.sniper.cat mail.sniper.cat mta-sts.sniper.cat autoconfig.sniper.cat
handle @mox_hosts {
reverse_proxy http://192.168.1.8:80 {
header_up Host "192.168.1.8"
}
}
@router host router.sniper.cat
handle @router {
respond /* 403
reverse_proxy http://192.168.1.254:80
}
error * 404
handle_errors {
rewrite * /{err.status_code}.html
root * /www/errors
file_server
}
}
/etc/caddy/willsey.cloud.caddy
willsey.cloud *.willsey.cloud *.*.willsey.cloud *.*.*.willsey.cloud *.*.*.*.willsey.cloud {
import /etc/caddy/security_headers.caddy
@www.willsey.cloud host www.willsey.cloud
handle @www.willsey.cloud {
redir https://willsey.cloud{uri} permanent
}
@willsey.cloud host willsey.cloud
handle @willsey.cloud {
header {
Content-Security-Policy "connect-src" "connect-src willsey.cloud:3478 wss://willsey.cloud"
Content-Security-Policy "img-src" "img-src https://usercontent.apps.nextcloud.com"
}
reverse_proxy http://192.168.1.4:11000
}
}
willsey.cloud:8443 {
import /etc/caddy/security_headers.caddy
reverse_proxy https://willsey.cloud:8443
}
/etc/caddy/drav.co.caddy
drav.co *.drav.co *.*.drav.co *.*.*.drav.co *.*.*.*.drav.co {
import /etc/caddy/security_headers.caddy
@www.drav.co host www.drav.co
handle @www.drav.co {
redir https://drav.co{uri} permanent
}
}
/etc/caddy/equilibrium.gay.caddy
equilibrium.gay *.equilibrium.gay *.*.equilibrium.gay *.*.*.equilibrium.gay *.*.*.*.equilibrium.gay {
import /etc/caddy/security_headers.caddy
@www.equilibrium.gay host www.equilibrium.gay
handle @www.equilibrium.gay {
redir https://equilibrium.gay{uri} permanent
}
}