Help to config metrics on reverse proxy setup

Help
1

1. Caddy version:

v2.6.2 h1:wKoFIxpmOJLGl3QXoo6PNbYvGW4xLEgo32GPBEjWL8o=

2. How I installed, and run Caddy:

a. System environment:

OS = Ubuntu 22.04
Caddy was installed by a 3rd party tool named HatchBox.

b. Command:

/usr/bin/caddy run --environ --resume

c. Service/unit/compose file:

(not sure...)

d. My complete Caddy config:

{
  "apps": {
    "http": {
      "servers": {
        "srv0": {
          "errors": {
            "routes": [
              {
                "handle": [
                  {
                    "handler": "subroute",
                    "routes": [
                      {
                        "handle": [
                          {
                            "handler": "headers",
                            "response": {
                              "add": {
                                "Cache-Control": [
                                  "no-cache"
                                ],
                                "expires": [
                                  "0"
                                ]
                              }
                            }
                          }
                        ]
                      },
                      {
                        "handle": [
                          {
                            "handler": "vars",
                            "root": "/home/deploy/.hatchbox/templates"
                          }
                        ]
                      },
                      {
                        "handle": [
                          {
                            "handler": "rewrite",
                            "uri": "/error.html"
                          }
                        ]
                      },
                      {
                        "handle": [
                          {
                            "handler": "file_server"
                          }
                        ]
                      }
                    ]
                  }
                ],
                "match": [
                  {
                    "host": [
                      "prod.pmrcc.com",
                      "market-monitor.pmrcc.com",
                      "pmr-lims.com",
                      "supplier.nw-cp.com",
                      "mgmt.nw-cp.com",
                      "12w2g.hatchboxapp.com",
                      "y998v.hatchboxapp.com",
                      "64kgp.hatchboxapp.com",
                      "6r2wl.hatchboxapp.com",
                      "6e5nm.hatchboxapp.com"
                    ]
                  }
                ],
                "terminal": true
              }
            ]
          },
          "listen": [
            ":443",
            ":80"
          ],
          "routes": [
            {
              "handle": [
                {
                  "handler": "subroute",
                  "routes": [
                    {
                      "handle": [
                        {
                          "handler": "reverse_proxy",
                          "trusted_proxies": [
                            "173.245.48.0/20",
                            "103.21.244.0/22",
                            "103.22.200.0/22",
                            "103.31.4.0/22",
                            "141.101.64.0/18",
                            "108.162.192.0/18",
                            "190.93.240.0/20",
                            "188.114.96.0/20",
                            "197.234.240.0/22",
                            "198.41.128.0/17",
                            "162.158.0.0/15",
                            "104.16.0.0/13",
                            "104.24.0.0/14",
                            "172.64.0.0/13",
                            "131.0.72.0/22",
                            "2400:cb00::/32",
                            "2606:4700::/32",
                            "2803:f800::/32",
                            "2405:b500::/32",
                            "2405:8100::/32",
                            "2a06:98c0::/29",
                            "2c0f:f248::/32"
                          ],
                          "upstreams": [
                            {
                              "dial": "192.168.203.167:80"
                            }
                          ]
                        }
                      ]
                    }
                  ]
                }
              ],
              "match": [
                {
                  "host": [
                    "prod.pmrcc.com",
                    "12w2g.hatchboxapp.com",
                    "162.216.19.41"
                  ]
                }
              ],
              "terminal": true
            },
            {
              "handle": [
                {
                  "handler": "subroute",
                  "routes": [
                    {
                      "handle": [
                        {
                          "handler": "reverse_proxy",
                          "trusted_proxies": [
                            "173.245.48.0/20",
                            "103.21.244.0/22",
                            "103.22.200.0/22",
                            "103.31.4.0/22",
                            "141.101.64.0/18",
                            "108.162.192.0/18",
                            "190.93.240.0/20",
                            "188.114.96.0/20",
                            "197.234.240.0/22",
                            "198.41.128.0/17",
                            "162.158.0.0/15",
                            "104.16.0.0/13",
                            "104.24.0.0/14",
                            "172.64.0.0/13",
                            "131.0.72.0/22",
                            "2400:cb00::/32",
                            "2606:4700::/32",
                            "2803:f800::/32",
                            "2405:b500::/32",
                            "2405:8100::/32",
                            "2a06:98c0::/29",
                            "2c0f:f248::/32"
                          ],
                          "upstreams": [
                            {
                              "dial": "192.168.203.167:80"
                            }
                          ]
                        }
                      ]
                    }
                  ]
                }
              ],
              "match": [
                {
                  "host": [
                    "market-monitor.pmrcc.com",
                    "y998v.hatchboxapp.com",
                    "162.216.19.41"
                  ]
                }
              ],
              "terminal": true
            },
            {
              "handle": [
                {
                  "handler": "subroute",
                  "routes": [
                    {
                      "handle": [
                        {
                          "handler": "reverse_proxy",
                          "trusted_proxies": [
                            "173.245.48.0/20",
                            "103.21.244.0/22",
                            "103.22.200.0/22",
                            "103.31.4.0/22",
                            "141.101.64.0/18",
                            "108.162.192.0/18",
                            "190.93.240.0/20",
                            "188.114.96.0/20",
                            "197.234.240.0/22",
                            "198.41.128.0/17",
                            "162.158.0.0/15",
                            "104.16.0.0/13",
                            "104.24.0.0/14",
                            "172.64.0.0/13",
                            "131.0.72.0/22",
                            "2400:cb00::/32",
                            "2606:4700::/32",
                            "2803:f800::/32",
                            "2405:b500::/32",
                            "2405:8100::/32",
                            "2a06:98c0::/29",
                            "2c0f:f248::/32"
                          ],
                          "upstreams": [
                            {
                              "dial": "192.168.203.167:80"
                            }
                          ]
                        }
                      ]
                    }
                  ]
                }
              ],
              "match": [
                {
                  "host": [
                    "pmr-lims.com",
                    "64kgp.hatchboxapp.com",
                    "162.216.19.41"
                  ]
                }
              ],
              "terminal": true
            },
            {
              "handle": [
                {
                  "handler": "subroute",
                  "routes": [
                    {
                      "handle": [
                        {
                          "handler": "reverse_proxy",
                          "trusted_proxies": [
                            "173.245.48.0/20",
                            "103.21.244.0/22",
                            "103.22.200.0/22",
                            "103.31.4.0/22",
                            "141.101.64.0/18",
                            "108.162.192.0/18",
                            "190.93.240.0/20",
                            "188.114.96.0/20",
                            "197.234.240.0/22",
                            "198.41.128.0/17",
                            "162.158.0.0/15",
                            "104.16.0.0/13",
                            "104.24.0.0/14",
                            "172.64.0.0/13",
                            "131.0.72.0/22",
                            "2400:cb00::/32",
                            "2606:4700::/32",
                            "2803:f800::/32",
                            "2405:b500::/32",
                            "2405:8100::/32",
                            "2a06:98c0::/29",
                            "2c0f:f248::/32"
                          ],
                          "upstreams": [
                            {
                              "dial": "192.168.203.167:80"
                            }
                          ]
                        }
                      ]
                    }
                  ]
                }
              ],
              "match": [
                {
                  "host": [
                    "supplier.nw-cp.com",
                    "6r2wl.hatchboxapp.com",
                    "162.216.19.41"
                  ]
                }
              ],
              "terminal": true
            },
            {
              "handle": [
                {
                  "handler": "subroute",
                  "routes": [
                    {
                      "handle": [
                        {
                          "handler": "reverse_proxy",
                          "trusted_proxies": [
                            "173.245.48.0/20",
                            "103.21.244.0/22",
                            "103.22.200.0/22",
                            "103.31.4.0/22",
                            "141.101.64.0/18",
                            "108.162.192.0/18",
                            "190.93.240.0/20",
                            "188.114.96.0/20",
                            "197.234.240.0/22",
                            "198.41.128.0/17",
                            "162.158.0.0/15",
                            "104.16.0.0/13",
                            "104.24.0.0/14",
                            "172.64.0.0/13",
                            "131.0.72.0/22",
                            "2400:cb00::/32",
                            "2606:4700::/32",
                            "2803:f800::/32",
                            "2405:b500::/32",
                            "2405:8100::/32",
                            "2a06:98c0::/29",
                            "2c0f:f248::/32"
                          ],
                          "upstreams": [
                            {
                              "dial": "192.168.203.167:80"
                            }
                          ]
                        }
                      ]
                    }
                  ]
                }
              ],
              "match": [
                {
                  "host": [
                    "mgmt.nw-cp.com",
                    "6e5nm.hatchboxapp.com",
                    "162.216.19.41"
                  ]
                }
              ],
              "terminal": true
            }
          ],
          "tls_connection_policies": [
            {}
          ]
        },
        "srv99": {
          "listen": [
            ":2020"
          ],
          "routes": [
            {
              "handle": [
                {
                  "handler": "metrics"
                }
              ]
            }
          ]
        }
      }
    },
    "tls": {
      "automation": {
        "policies": [
          {
            "on_demand": true,
            "subjects": [
              "12w2g.hatchboxapp.com",
              "prod.pmrcc.com"
            ]
          },
          {
            "on_demand": true,
            "subjects": [
              "y998v.hatchboxapp.com",
              "market-monitor.pmrcc.com"
            ]
          },
          {
            "on_demand": true,
            "subjects": [
              "64kgp.hatchboxapp.com",
              "pmr-lims.com"
            ]
          },
          {
            "on_demand": true,
            "subjects": [
              "6r2wl.hatchboxapp.com",
              "supplier.nw-cp.com"
            ]
          },
          {
            "on_demand": true,
            "subjects": [
              "6e5nm.hatchboxapp.com",
              "mgmt.nw-cp.com"
            ]
          }
        ]
      },
      "certificates": {
        "load_files": []
      }
    }
  },
  "logging": {
    "logs": {
      "default": {
        "encoder": {
          "format": "console"
        },
        "level": "info"
      }
    }
  }
}

3. The problem I’m having:

I want to monitor metrics with prometheus and with current config I only have metrics for admin endpoints I think.

For example I only see caddy_admin_http_… endpoints in prometheus.

I think that way my configuration JSON is, I didn’t enable metrics for the actual websites.

By the way, this caddy server is a load balancer for another caddy server.

4. Error messages and/or full log output:

5. What I already tried:

6. Links to relevant resources:

2

Be careful with this – this’ll disable automatic HTTP->HTTPS redirects. You probably want to remove :80 from here and allow Caddy’s Automatic HTTPS set up the HTTP server for you automatically, instead.

Since v2.6.0, you need to enable metrics on the HTTP server, because it has some performance overhead, and many users don’t actually need metrics. See JSON Config Structure - Caddy Documentation and look for "metrics".

3

Thanks for the advice about HTTPS redirect. I don’t think, in my case, this is an issue because the app server force SSL connection. But maybe the author of ht 3rd party provisioning tool will want to know about that.

I did try to add metrics at the server level without any success. Maybe that’s because I am trying to listen metrics on a different port ?

I think I’ll switch to App server monitoring instead. Metrics in Caddy seems a bit too rough for the moment.

4

Well what it also does is allow traffic over HTTP on port 80 as well, without enforcing HTTPS. That’s the problem.

I’m not sure what you mean by “because the app server force SSL connection”, but port 80 should be open and accessible for the ACME HTTP challenge to successfully complete. And if that’s open, then it should only serve HTTP->HTTPS redirects, to ensure users never load your sites over HTTP.

There’s two halves to this. First, you need to enable metrics tracking, which is done via the :443 server, and then you may enable a metrics handler which you can point your metrics scraper to. The handler can be on a different server. Or you could pull metrics from the admin endpoint.

This is all covered in this page in the docs:

5

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.