Access metrics when using dockerized caddy?

Help
1

1. Caddy version (caddy version):

v2.5.1 h1:bAWwslD1jNeCzDa+jDCNwb8M3UJ2tPa8UZFFzPVmGKs=

2. How I run Caddy:

I run Caddy using the official docker container, Docker Hub, with vanilla settings. I run caddy (and other services) directly via docker run, no additional orchestration layer (no docker-compose etc).

  docker run --name caddy \
    --detach \
    --publish 0.0.0.0:80:80 \
    --publish 0.0.0.0:443:443 \
    --restart always \
    -v ${HERE}/Caddyfile:/etc/caddy/Caddyfile \
    -v ${HERE}/.caddy/:/data \
    --network cirrus-net \
    caddy

a. System environment:

My host machine is a baremetal workstation running Ubuntu Linux 22.04 with current docker, Docker version 20.10.12, build 20.10.12-0ubuntu4.

b. Command:

See above

c. Service/unit/compose file:

see above

d. My complete Caddyfile or JSON config:

Actually my complete Caddyfile is here: servers/Caddyfile at 9be448631cb166db8fb0a06ba6266075a74b99a7 · boettiger-lab/servers · GitHub

it’s a little long and really I don’t think very relevant to this question, as all the services configured there work 100% as expected.

3. The problem I’m having:

I’m trying to see caddy’s metrics/use load, trying to follow the official docs at Monitoring Caddy with Prometheus metrics — Caddy Documentation. docker logs caddy works great to see the default logs (i.e errors), but I cannot access the metrics endpoint (either for basic metrics or prometheus). The official docs do not cover the dockerized case, they suggest trying something like:

 curl http://localhost:2019/metrics

Of course this fails for my setup – caddy isn’t on port 2019 as you can see from my docker command, and caddy is behind the docker network abstraction layer so it’s not on localhost either.

I can run another container that is also attached to the above network, e.g. a vanilla ubuntu container, and then I think based on the usual docker network logic caddy would be visible under the container name, e.g. something like:

curl caddy:2019/metrics

but that errors with:

curl: (7) Failed to connect to caddy port 2019: Connection refused

(as expected I think since caddy isn’t running on port 2019, but on 80 and 443 as indicated in my docker command, right?). Omitting the port seems like it would be correct, but that returns silently:

root@981ba2b05c84:/# curl caddy/metrics

I’m probably missing something very obvious here, but just not sure where to start. I’m not getting any unexpected error, I just can’t figure out how to talk to the caddy service directly (i.e. ‘localhost:2019’ in the demos). I have no trouble with any of the services listed under full https domain names in my caddyfile.

4. Error messages and/or full log output:

none

5. What I already tried:

Please see above

6. Links to relevant resources:

I’d like to do something similar to
Caddy V2 and prometheus metrics, but the case is not quite the same, as it appears in that configuration caddy is already on localhost:2019 in that example, and I just don’t know what to change that to in the case of docker.

1 Like
2

The admin endpoint only listens on localhost:2019 by default, which means it will only accept requests coming from localhost (i.e. 127.0.0.1 or ::1), which means it will only accept connections coming from within the container.

You need to configure the admin endpoint to listen on all IPs if you want it to be accessible from outside the container. Configure the global option admin :2019 to make it listen to all interfaces on port 2019.

You’ll need to publish that port with Docker as well. Make sure to protect access to this port so that only trusted clients/machines can connect to it, otherwise you risk allowing anyone to manipulate Caddy’s config, and they could do some bad things.

Alternatively, you could instead set up a site in your config that publishes the metrics data, with the metrics directive:

Again, you’ll want to make sure only trusted clients have access to this, either by limiting which remote IPs are allowed to see the data. For example:

metrics.example.com {
	@nope not remote_ip private_ranges
	abort @nope

	metrics
}

This way any connection coming from public IP ranges will be aborted, and only private IP ranges will reach the metrics handler.

6 Likes
3

wow, thanks, these answers are super!

Just to be clear, to set the global option admin :2019, I would literally put that at the top of my Caddyfile, right? I think I can do that and expose the port only within the container network and then access metrics from the container.

The internal metrics site idea is also clever! Amazed that @nope not remote_ip private_ranges is actually functional code and not just pseudo-code I’m supposed to fill in some how. Caddy never fails to impress me.

2 Likes
4

It’s a global option, so it needs to go in the global options block:

{
    admin :2019
}

:smiling_face:

1 Like
5

This topic was automatically closed after 30 days. New replies are no longer allowed.