Caddy as reverse proxy in front of checkmk w/o handling special service port 8000

1. Caddy version:

$ docker exec caddy caddy version
v2.6.2 h1:wKoFIxpmOJLGl3QXoo6PNbYvGW4xLEgo32GPBEjWL8o=

2. How I installed, and run Caddy:

Setup using docker compose. I run checkmk as a service and want to use caddy as a reverse proxy to the checkmk web ui in order to handle TLS.

a. System environment:

$ uname -a
Linux sentinel 5.15.0-56-generic #62-Ubuntu SMP Tue Nov 22 19:54:14 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux
$ docker --version
Docker version 20.10.23, build 7155243
$ docker-compose version
docker-compose version 1.29.2, build unknown
docker-py version: <module 'docker.version' from '/usr/local/lib/python3.10/dist-packages/docker/version.py'>
CPython version: 3.10.6
OpenSSL version: OpenSSL 3.0.2 15 Mar 2022

b. Command:

Default command of official caddy docker image.

c. Service/unit/compose file:

docker-compose file
services:
  checkmk:
    image: checkmk/check-mk-free:2.1.0p20
    restart: always
    environment:
      - "CMK_SITE_ID=mysite"
      - "CMK_PASSWORD=mypassword"
    volumes:
      - type: bind
        source: /etc/localtime
        target: /etc/localtime
        read_only: true
      - type: volume
        source: checkmk-data
        target: /omd/sites
    tmpfs:
      - /opt/omd/sites/mysite/tmp:uid=1000,gid=1000  
    expose:
      - 5000
    ports:
      - 8000:8000
  caddy:
    image: caddy:2.6.2
    restart: always
    ports:
      - 80:80
      - 443:443
    volumes:
      - type: volume
        source: caddy-data
        target: /data
      - type: volume
        source: caddy-config
        target: /config
      - type: bind
        source: "/etc/caddy/Caddyfile"
        target: /etc/caddy/Caddyfile
    depends_on:
      - checkmk
volumes:
  caddy-data:
  caddy-config:
  checkmk-data:

d. My complete Caddy config:

Caddyfile
{
    email admin@example.com
}

checkmk.example.com {
    reverse_proxy checkmk:5000
}

3. The problem I’m having:

The checkmk monitoring service provides a frontend to configure and visualize metrics, alarms, etc. In order to secure the connection to the frontend, I want to use caddy with automatic HTTPS as reverse proxy. Additionaly checkmk communicates with client “agents” on other devices trough port 8000. As you can see I opened it up using ports: in the compose file.

With the given caddy config the UI is accessible and gets automatic HTTPS certificates as well as 80->443 redirects.

But when an agent on another device tries to connect to the checkmk server it gets rejected. Probably because caddy blocks this attempt?

4. Error messages and/or full log output:

client:~$ sudo cmk-agent-ctl register --hostname client_hostname --server checkmk.example.com --site p8 --user automation --password "ABCDEFGHIJKLMNOPQRSTUVWXYZ" --verbose
INFO [cmk_agent_ctl] starting
INFO [cmk_agent_ctl] Loaded config from '"/var/lib/cmk-agent/cmk-agent-ctl.toml"', legacy pull 'LegacyPullMarker("/var/lib/cmk-agent/allow-legacy-pull")' exists
INFO [cmk_agent_ctl::site_spec] Failed to discover agent receiver port using https, trying http.
INFO [cmk_agent_ctl::site_spec] Failed to discover agent receiver port using http.
ERROR [cmk_agent_ctl] Failed to discover agent receiver port from Checkmk REST API, both with http and https. Run with verbose output to see errors.

5. What I already tried:

I tried only listening on ports 80 and 443 for caddy, with

{
    email admin@example.com
}

checkmk.example.com:80, checkmk.example.com:443 {
    reverse_proxy checkmk:5000
}

Then the agent can connect to port 8000 and can register, but when I now try to access the checkmk web ui, HTTPS does not work anymore.

So, the question is, can I achieve handling traffic on port 80 and 443 using caddy as reverse proxy, while in the sametime let the checkmk service handle port 8000?

6. Links to relevant resources:

tbd

I don’t think this has anything to do with Caddy. There’s no way for Caddy to interact with traffic on port 8000 when it’s only listening on 80 and 443. Even moreso when you only bind those ports to the host in Docker.

Where are the agents running? Are they in the same private network, or are they connecting over the public internet?

Make sure your firewall and/or port forwarding settings are properly set up to route traffic on port 8000 to your server.

But how does handle Caddy with such a config

checkmk.example.com {
    reverse_proxy checkmk:5000
}

a connection attempt on Port 8000?

In this case the agent (connecting over public internet) are not reaching the service on port 8000.

There’s no way for Caddy to interact with traffic on port 8000 when it’s only listening on 80 and 443. Even moreso when you only bind those ports to the host in Docker.

Yes, in exactly this case, i.e. with

checkmk.example.com:80, checkmk.example.com:443 {
    reverse_proxy checkmk:5000
}

the agents can connect, but automatic HTTPS and the redirect are not working anymore.

So I need to either understand how caddy handles connections regarding the first config or how to “reactivate” auto-HTTPS and redirecting for the later config.

Make sure your firewall and/or port forwarding settings are properly set up to route traffic on port 8000 to your server.

Yes, of course. Maybe I’m missing something, but I think this is configured correctly

$ sudo ufw status
Status: active

To                         Action      From
--                         ------      ----
22/tcp                     ALLOW       Anywhere
80/tcp                     ALLOW       Anywhere
443/tcp                    ALLOW       Anywhere
8000/tcp                   ALLOW       Anywhere
22/tcp (v6)                ALLOW       Anywhere (v6)
80/tcp (v6)                ALLOW       Anywhere (v6)
443/tcp (v6)               ALLOW       Anywhere (v6)
8000/tcp (v6)              ALLOW       Anywhere (v6)

8000/tcp                   ALLOW FWD   Anywhere
443                        ALLOW FWD   Anywhere
80/tcp                     ALLOW FWD   Anywhere
8000/tcp (v6)              ALLOW FWD   Anywhere (v6)
443 (v6)                   ALLOW FWD   Anywhere (v6)
80/tcp (v6)                ALLOW FWD   Anywhere (v6)

Just reread your anweser, and this part sticks. I have no idea. The caddy service is only allowed to bind to 80 an 443 on the host, so why the heck is this not working, but does work with another caddy config… :thinking:

It doesn’t.

Caddy isn’t touching port 8000 at all.

Then you’d need to publish port 8000 on your Caddy container, and add another site to your Caddyfile that listens on port 8000 and proxy that to checkmk.

How are your agents configured? Are you specifying port 8000? Or do you leave out the port? If you leave out the port, it might be trying to connect on port 80 by default, not port 8000. Which would make sense, because port 80 is the default HTTP port.

You could try specifying https:// in your agent config, if it supports HTTPS. But I know nothing about checkmk so you’ll need to look into that.

Thanks for your help so far, I’m totally aware that I have to figure out the checkmk related parts myself.

Yes, they try to connect on 80 or 443 to request the “agent registration” port from the HTTP API. They will then run some custom tcp protocol.

So its not just using a different HTTP port, but they need both.

Then you’d need to publish port 8000 on your Caddy container, and add another site to your Caddyfile that listens on port 8000 and proxy that to checkmk.

I will try and come back once I have an update.

I works now. Not sure what happend, most probably I wasn’t thorough enough when cleaning up all caddy volumes in the docker compose setup. Wiping everything and start from the beginning, solved the problem. The configuration is the same as before.

Caddyfile
{
    email admin@example.com
}

checkmk.example.com {
    reverse_proxy checkmk:5000
}
compose.yaml
services:
  checkmk:
    image: checkmk/check-mk-free:2.1.0p20
    restart: always
    environment:
      - "CMK_SITE_ID=mysite"
      - "CMK_PASSWORD=mypassword"
    volumes:
      - type: bind
        source: /etc/localtime
        target: /etc/localtime
        read_only: true
      - type: volume
        source: checkmk-data
        target: /omd/sites
    tmpfs:
      - /opt/omd/sites/mysite/tmp:uid=1000,gid=1000  
    expose:
      - 5000
    ports:
      - 8000:8000
  caddy:
    image: caddy:2.6.2
    restart: always
    ports:
      - 80:80
      - 443:443
    volumes:
      - type: volume
        source: caddy-data
        target: /data
      - type: volume
        source: caddy-config
        target: /config
      - type: bind
        source: "/etc/caddy/Caddyfile"
        target: /etc/caddy/Caddyfile
    depends_on:
      - checkmk
volumes:
  caddy-data:
  caddy-config:
  checkmk-data:

Sorry for bothering and thanks again for rubberducking at least…

1 Like

No worries!

:rubber_duck:

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.