/metrics endpoint responding 400 Bad Request

1. The problem I’m having:

I’m trying to retrieve prometheus metrics from the /metrics endpoint on a non-default port. I was able to expose the endpoint on port 2018 and bind it to the private IP interface.

Since Caddy, for several reasons that I’ll omit for simplicity, is behind an HAProxy instance, I enabled the PROXY protocol so that Caddy can parse X-Forwarded-For and correctly read the client IP.

The problem is that with the current configuration, when I visit http://192.168.1.0:2018/metrics, I get a “400 Bad Request” error.

Note: I do not go through the Load Balancer to retrieve the metrics but I contact directly the Caddy host.

2. Error messages and/or full log output:

The error message is a “400 Bad Request”. Right now the Caddy is handling hundreds of requests per seconds so the logs are unreadable.

3. Caddy version:

v2.7.4 h1:J8nisjdOxnYHXlorUKXY75Gr6iBfudfoGhrJ8t7/flI=                                                                                                                            

4. How I installed and ran Caddy:

a. System environment:

Ubuntu 20.04.6 LTS, Kernel Linux 5.4.0, go 1.23.4

b. Command:

/path/to/caddy run --environ --config /path/to/Caddyfile

d. My complete Caddy config:

{ 

	default_bind 192.168.1.0

	auto_https off
	order ... some modules (ratelimit, waf, etc.)

	log {
		level INFO
		output file /var/log/caddy/caddy-full.log {
			roll_size 10mb 
			roll_keep 100
			roll_keep_for 720h
		}

		format filter {
			wrap json
			fields {
				request>remote_ip hash
				request>headers>x-forwarded-for hash 
				request>headers>cf-connecting-ip hash
			}
		}
	}

	servers {
		listener_wrappers {
			proxy_protocol {
				timeout 10s
				# allow <cidrs>
			}
			tls
		}

		protocols h1 h2
		metrics
	}
}

:2018 {
	metrics
}

:443 {
	...
}

:80 {
	...
}	

Can you please first try with the latest version? v2.7.4 is ancient

I upgraded Caddy to version v2.10.2, but the error still occurs. I did some further research and discovered the --haproxy-protocol flag.

When I run a standard curl command, it returns a 400 error:
curl http://192.168.0.1:2018/metrics

However, when I use that flag, it works:
curl --haproxy-protocol http://192.168.0.1:2018/metrics

I think this issue is no longer related to Caddy. Do you agree?

The behavior you’re seeing is expected because you configured the proxy_protocol on the server, and you’re using the :2018 site that is configured on this server.

If you don’t want to use the PROXY protocol, you can call the /metrics endpoint on the admin endpoint, i.e. localhost:2019/metrics.

1 Like