X-Forwarded-Proto keeps being http

Help
1

1. The problem I’m having:

I’m trying to get X-Forwarded-Proto to be https. From the docs it seems like I shouldn’t need to do anything. Caddy is NOT behind a proxy so the requests it gets should contain no X-Anything

2. Error messages and/or full log output:

There are no log messages, it proxies the request as expected, all the X-Forwarded headers look right, EXCEPT for -Proto and it is always http

3. Caddy version:

v2.6.4 h1:2hwYqiRwk1tf3VruhMpLcYTg+11fCdr8S3jhNAdnPy8=

4. How I installed and ran Caddy:

Co-worker installed Caddy, not sure how, and not sure if that matters for this question. He’ll be back from vacation soon.

a. System environment:

Ubuntu 22.04.2 LTS

b. Command:

/usr/local/bin/caddy run --environ --config /etc/caddy/Caddyfile

c. Service/unit/compose file:

service file

;
; Ansible managed
;
; source: https://github.com/mholt/caddy/blob/master/dist/init/linux-systemd/caddy.service
; version: 6be0386
; changes: Set variables via Ansible

[Unit]
Description=Caddy HTTP/2 web server
Documentation=https://caddyserver.com/docs
After=network-online.target
Wants=network-online.target systemd-networkd-wait-online.service
StartLimitIntervalSec=86400
StartLimitBurst=5

[Service]
Restart=on-failure

; User and group the process will run as.
User=www-data
Group=www-data

; Letsencrypt-issued certificates will be written to this directory.
Environment=CADDYPATH=/etc/ssl/caddy

ExecStart="/usr/local/bin/caddy" run --environ --config "/etc/caddy/Caddyfile"
ExecReload="/usr/local/bin/caddy" reload --config "/etc/caddy/Caddyfile"

; Limit the number of file descriptors; see `man systemd.exec` for more limit settings.
LimitNOFILE=1048576

; Use private /tmp and /var/tmp, which are discarded after caddy stops.
PrivateTmp=true
; Use a minimal /dev
PrivateDevices=true
; Hide /home, /root, and /run/user. Nobody will steal your SSH-keys.
ProtectHome=true
; Make /usr, /boot, /etc and possibly some more folders read-only.
ProtectSystem=full
; ... except /etc/ssl/caddy, because we want Letsencrypt-certificates there.
;   This merely retains r/w access rights, it does not add any new. Must still be writable on the host!
ReadWriteDirectories=/etc/ssl/caddy /var/log/caddy

; The following additional security directives only work with systemd v229 or later.
; They further restrict privileges that can be gained by caddy.
; Note that you may have to add capabilities required by any plugins in use.
CapabilityBoundingSet=CAP_NET_BIND_SERVICE
AmbientCapabilities=CAP_NET_BIND_SERVICE
NoNewPrivileges=true

[Install]
WantedBy=multi-user.target

d. My complete Caddy config:

e2e1.qa.dubclub.win {
	reverse_proxy http://e2e1.rhino-jazz.ts.net
}
e2e2.qa.dubclub.win {
	reverse_proxy http://e2e2.rhino-jazz.ts.net {
		header_up X-Forwarded-Proto https
	}
}
e2e3.qa.dubclub.win {
	reverse_proxy http://e2e3.rhino-jazz.ts.net
}

5. Links to relevant resources:

2

I’ve also tried including https in quotes “https”. But still no change,

3

Please enable the debug global option and show your Caddy logs. Make a request with curl -v and show what that looks like as well.

4

The problem is NOT with caddy. It was with nginx which was behind Caddy. Caddy is the gateway from the internet into our tailnet and then each server has nginx to serve static files and proxy to gunicorn. The problem had nothing to do with Caddy and everything to do with nginx. I’ve solved part of the issue with nginx but not all of it yet. Still having problems with X-Forwarded-For having 2 IPs in it, one from the client and one from the caddy server. Sorry for the trouble.

5

That sounds normal. Proxies are supposed to append the direct connecting client’s IP address to the header, if they trust the downstream proxy. By default, Caddy distrusts all connecting clients (unless you configure trusted_proxies), so it will only contain the IP of the direct connecting client and throw away the incoming XFF header. Your nginx might be adding Caddy’s IP to the header, and that should be fine. It’s up to your application to parse the header and get the IP address it needs from it.

6

Yea. It was nginx that was creating the problems. Caddy was doing it exactly right but nginx had not been setup to sit behind another proxy. I solved the issue by changing the default proxy_params to be

proxy_set_header Host $http_host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $http_x_forwarded_for;
proxy_set_header X-Forwarded-Proto https;
proxy_headers_hash_max_size 2048;
proxy_headers_hash_bucket_size 128;

The last 2 lines were suggestions from sudo nginx -t

7

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.