1. The problem I’m having:
We are running a caddy with frankenphp in a Digital Ocean kubernetes cluster.
Entrypoint is a load balancer service which distributes all request to two concurrent running instance of our application. Load Balancer is configured with SSL passthrough so SSL Termination is Happening in Caddy. Both Deployment share a SSL certificate.
If we log in our symfony hosts etc, it is sometimes correct (www.bibleserver.com), but sometimes incorrect (http-service).
When we activate our varnish and using ESI, symfony errors if host is not www.bibleserver.com.
Logs are with disabled varnish-service.
Http-service targets :80 section of the caddyfile.
How can we prevent this “random” change of host header ?
How can we fix http2 errors?
2. Error messages and/or full log output:
Symfony logs:
{"scheme":"https","baseurl":"","port":443,"host":"www.bibleserver.com","path":"/_fragment","query":"_path=XXX"}
{"scheme":"http","baseurl":"","port":80,"host":"http-service","path":"/_fragment","query":"_path=XXXX"}
Caddy logs:
{"duration":4.876707258,"error":"writing: http2: stream closed","level":"error","logger":"http.handlers.reverse_proxy","msg":"aborting with incomplete response","request":{"client_ip":"2001:4860:7:603::ff","headers":{"Accept":["text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.7"],"Accept-Encoding":["gzip, br"],"Accept-Language":["de-DE,de;q=0.9,en-US;q=0.8,en;q=0.7"],"Cdn-Loop":["cloudflare; loops=1"],"Cf-Connecting-Ip":["2001:4860:7:603::ff"],"Cf-Ipcountry":["AT"],"Cf-Ray":["8ec3ed6d2dad7900-CDG"],"Cf-Visitor":["{\"scheme\":\"https\"}"],"Priority":["u=4, i"],"Purpose":["prefetch"],"Referer":["https://www.google.com/"],"Sec-Ch-Ua":["\"Google Chrome\";v=\"131\", \"Chromium\";v=\"131\", \"Not_A Brand\";v=\"24\""],"Sec-Ch-Ua-Mobile":["?0"],"Sec-Ch-Ua-Platform":["\"Windows\""],"Sec-Fetch-Dest":["empty"],"Sec-Fetch-Mode":["no-cors"],"Sec-Fetch-Site":["none"],"Sec-Purpose":["prefetch;anonymous-client-ip"],"Upgrade-Insecure-Requests":["1"],"User-Agent":["Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/131.0.0.0 Safari/537.36"],"X-Caddy-Forwarded":["1"],"X-Forwarded-For":["2001:4860:7:603::ff"],"X-Forwarded-Host":["www.bibleserver.com"],"X-Forwarded-Port":["443"],"X-Forwarded-Proto":["https"]},"host":"http-service","method":"GET","proto":"HTTP/2.0","remote_ip":"10.114.0.22","remote_port":"40808","tls":{"cipher_suite":4865,"proto":"h2","resumed":false,"server_name":"www.bibleserver.com","version":772},"uri":"/de/verse/Matth%C3%A4us6,13"},"ts":1733232800.649357,"upstream":"http-service:80"}
{"level":"error","ts":1733230290.3546765,"logger":"http.log.error","msg":"dial tcp 172.16.18.150:8080: connect: connection refused","request":{"remote_ip":"10.114.0.20","remote_port":"60094","client_ip":"10.114.0.20","proto":"HTTP/2.0","method":"GET","host":"www.bibleserver.com","uri":"/KJV/Psalm23%3A4","headers":{"Cdn-Loop":["cloudflare; loops=1"],"Cf-Ipcountry":["US"],"Cf-Visitor":["{\"scheme\":\"https\"}"],"User-Agent":["Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/127.0.0 Safari/537.36"],"Cf-Connecting-Ip":["134.199.76.78"],"Accept-Encoding":["gzip, br"],"X-Forwarded-For":["134.199.76.78"],"Cf-Ray":["8ec3b0412ff00f63-EWR"],"X-Forwarded-Proto":["https"]},"tls":{"resumed":false,"version":772,"cipher_suite":4865,"proto":"h2","server_name":"www.bibleserver.com"}},"duration":0.002403112,"status":502,"err_id":"mkv0q1q26","err_trace":"reverseproxy.statusError (reverseproxy.go:1269)"}
{"error":"http2: stream closed","level":"error","msg":"write error","ts":1733232796.9338708}
{"duration":5.785179248,"error":"writing: http2: stream closed","level":"error","logger":"http.handlers.reverse_proxy","msg":"aborting with incomplete response","request":{"client_ip":"134.147.21.216","headers":{"Accept":["*/*"],"Accept-Encoding":["gzip, br"],"Accept-Language":["de,de-DE;q=0.9,en;q=0.8,en-GB;q=0.7,en-US;q=0.6"],"Cdn-Loop":["cloudflare; loops=1"],"Cf-Connecting-Ip":["134.147.21.216"],"Cf-Ipcountry":["DE"],"Cf-Ray":["8ec3f2213a41f964-DUS"],"Cf-Visitor":["{\"scheme\":\"https\"}"],"Cookie":["REDACTED"],"Mode":["same-origin"],"Priority":["u=1, i"],"Referer":["https://www.bibleserver.com/de/verse/1.Korinther1,30"],"Sec-Ch-Ua":["\"Microsoft Edge\";v=\"131\", \"Chromium\";v=\"131\", \"Not_A Brand\";v=\"24\""],"Sec-Ch-Ua-Mobile":["?0"],"Sec-Ch-Ua-Platform":["\"Windows\""],"Sec-Fetch-Dest":["empty"],"Sec-Fetch-Mode":["cors"],"Sec-Fetch-Site":["same-origin"],"User-Agent":["Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/131.0.0.0 Safari/537.36 Edg/131.0.0.0"],"X-Caddy-Forwarded":["1"],"X-Forwarded-For":["134.147.21.216"],"X-Forwarded-Host":["www.bibleserver.com"],"X-Forwarded-Port":["443"],"X-Forwarded-Proto":["https"],"X-Locale":["de"],"X-Requested-With":["XMLHttpRequest"]},"host":"http-service","method":"GET","proto":"HTTP/2.0","remote_ip":"10.114.0.22","remote_port":"39226","tls":{"cipher_suite":4865,"proto":"h2","resumed":false,"server_name":"www.bibleserver.com","version":772},"uri":"/api/w53CmsKVwpfClcKYwpfClsOZw4zDmsOWw5zDmcOKw4zClsKowrXCusK7wrbCusK6wpbCnMKfwpvCnw=="},"ts":1733232994.1759555,"upstream":"http-service:80"}
3. Caddy version:
v2.8.4 / frankenphp
4. How I installed and ran Caddy:
We are using an offical frankenphp docker image.
a. System environment
Kubernetes Cluster
LoadBalancer(SSL Passthrough) → Deployment(Frankenphp, 2 Replicas)
Reverse Proxy ist actually configured to use a Varnish Cache. That is Not working at the Moment because of the changes in x-forwarded-host headers.
b. Command:
exec frankenphp run --config /etc/caddy/Caddyfile
d. My complete Caddy config:
{
log {
level warn
output file /var/log/caddy.log
}
# Enable FrankenPHP
frankenphp {
worker /var/www/public/index.php 12
}
# Configure when the directive must be executed
order mercure after encode
order vulcain after reverse_proxy
order php_server before file_server
order php before file_server
# cloudflare proxy
servers {
trusted_proxies static private_ranges 173.245.48.0/20 103.21.244.0/22 103.22.200.0/22 103.31.4.0/22 141.101.64.0/18 108.162.192.0/18 190.93.240.0/20 188.114.96.0/20 197.234.240.0/22 198.41.128.0/17 162.158.0.0/15 104.16.0.0/13 104.24.0.0/14 172.64.0.0/13 131.0.72.0/22
}
servers :80 {
protocols h1 h2c
}
}
:80 {
root * /var/www/public
handle /ping {
respond "pong"
}
php_server
}
*.bibleserver.com {
root * /var/www/public
@notWww not host www.bibleserver.com
redir @notWww https://www.bibleserver.com{uri}
tls produktion.online@erf.de {
dns cloudflare {$CLOUDFLARE_API_TOKEN}
}
@cloudflare header CF-Connecting-IP *
request_header @cloudflare X-Forwarded-For {header.CF-Connecting-IP}
request_header !@cloudflare X-Forwarded-For {header.Remote-Address}
@buildFile path /build/*
@noBuildFile not path /build/*
handle @buildFile {
header {
Cache-Control "public, max-age=31536000, immutable"
-Via
-Server
-Vary
-X-Debug-Token
-X-Locale
-X-Varnish
-X-Powered-By
}
file_server
}
@maintenance {
file "maintenance/active.txt"
not path /favicon.ico
not path /build/roboto*
}
handle @maintenance {
try_files maintenance/maintenance.html
file_server {
status 503
}
}
header @noBuildFile {
Strict-Transport-Security "max-age=31536000; includeSubDomains; preload"
X-Frame-Options "DENY"
X-Content-Type-Options "nosniff"
X-XSS-Protection "1; mode=block"
-Via
-Server
-Vary
-X-Debug-Token
-X-Locale
-X-Varnish
-X-Powered-By
}
@useVarnish {
header !X-Caddy-Forwarded
not file
not path /api/chat/subscribe/*
not path /ping
}
reverse_proxy @useVarnish {
to h2c://varnish-service:8080 h2c://http-service
# use first upstream, if healthy
lb_policy first
# configure health check
health_uri /ping
health_fails 2
health_timeout 11s
health_interval 5s
health_status 200
# remember failed upstreams for 30s
fail_duration 30s
header_up Host http-service
header_up X-Forwarded-For {header.X-Forwarded-For}
header_up X-Forwarded-Host www.bibleserver.com
header_up X-Forwarded-Proto https
header_up X-Forwarded-Port 443
header_up X-Caddy-Forwarded 1
transport http {
versions 1.1 h2c
}
}
# enable http3 push
push
php_server
encode gzip
file_server
}
