V2: Health checks are going to the wrong upstream

1. My Caddy version (caddy version):

2.0 beta 14

2. How I run Caddy:

caddy run --config /etc/caddy/config.json

a. System environment:

CentOS 8, fresh vps

d. My complete Caddyfile or JSON config:

...
{
  "handle": [
    {
      "handler": "authentication",
      "providers": {
          "http_basic": {
            "hash": {
              "algorithm": "bcrypt"
            },
            "accounts": [
              {"username": "admin", "password": "....."}
            ]
          }
      }
    }
  ],
  "match": [
    {
      "path_regexp": {
        "name": "cdbadmin", 
        "pattern": "^/cdbadmin(.*)?"
      }
    }
  ]
},
{
  "handle": [
    {
      "handler": "subroute",
      "routes": [
        {
          "handle": [
            {
              "handler": "rewrite",
              "strip_path_prefix": "/cdbadmin"
            }
          ]
        },
        {
          "handle": [
            {
              "handler": "reverse_proxy",
              "headers": {
                "request": {
                  "set": {
                    "Host": [
                      "{http.request.host}"
                    ],
                    "X-Forwarded-For": [
                      "{http.request.remote.host}"
                    ],
                    "X-Forwarded-Proto": [
                      "{http.request.scheme}"
                    ],
                    "X-Real-Ip": [
                      "{http.request.remote.host}"
                    ]
                  }
                }
              },
              "health_checks": {
                "active": {
                  "expect_status": 2,
                  "path": "/_status/nodes"
                }
              },
              "transport": {
                "protocol": "http",
                "read_buffer_size": 4096
              },
              "upstreams": [
                {
                  "dial": "127.0.0.1:8080"
                }
              ]
            }
          ]
        }
      ]
    }
  ],
  "match": [
    {
      "path_regexp": {
        "name": "cdbadmin", 
        "pattern": "^/cdbadmin(.*)?"
      }
    }
  ],
  "terminal": true
},
{
  "handle": [
    {
      "handler": "subroute",
      "routes": [
        {
          "handle": [
            {
              "handler": "rewrite",
              "strip_path_prefix": "/api"
            }
          ]
        },
        {
          "handle": [
            {
              "handler": "reverse_proxy",
              "headers": {
                "request": {
                  "set": {
                    "Host": [
                      "{http.request.host}"
                    ],
                    "X-Forwarded-For": [
                      "{http.request.remote.host}"
                    ],
                    "X-Forwarded-Proto": [
                      "{http.request.scheme}"
                    ],
                    "X-Real-Ip": [
                      "{http.request.remote.host}"
                    ]
                  }
                }
              },
              "health_checks": {
                "active": {
                  "expect_status": 2,
                  "path": "/health-check"
                }
              },
              "transport": {
                "protocol": "http",
                "read_buffer_size": 4096
              },
              "upstreams": [
                {
                  "dial": "127.0.0.1:1325"
                }
              ]
            }
          ]
        }
      ]
    }
  ],
  "match": [
    {
      "path_regexp": {
        "name": "api", 
        "pattern": "^/api(.*)?"
      }
    }
  ],
  "terminal": true
},
...

3. The problem I’m having:

At 127.0.0.1:8080 i have cockroachdb’s admin running and at 127.0.0.1:1325 i have my api.
When i access /cdbadmin i need to get the cockroachdb admin. When i access /api i need to get the api. Thus i made two routes for this as you can see above.
The problem is that caddy queries the wrong upstream for health checks.
Looking at the above configuration, you would expect the health checks to go to 127.0.0.1:8080/_status/nodes for cdb and to 127.0.0.1:1325/health-check for the api, but instead, both checks are made at 127.0.0.1:1325 as you can see in the access logs for the api app:

{"time":"2020-02-23T17:59:55.352354839Z","id":"","remote_ip":"127.0.0.1","host":"127.0.0.1:1325","method":"GET","uri":"/_status/nodes","user_agent":"Go-http-client/1.1","status":404,"error":"code=404, message=Not Found","latency":20305,"latency_human":"20.305µs","bytes_in":0,"bytes_out":24}
{"time":"2020-02-23T17:59:55.352815964Z","id":"","remote_ip":"127.0.0.1","host":"127.0.0.1:1325","method":"GET","uri":"/health-check","user_agent":"Go-http-client/1.1","status":200,"error":"","latency":4184,"latency_human":"4.184µs","bytes_in":0,"bytes_out":0}
{"time":"2020-02-23T18:00:25.348730513Z","id":"","remote_ip":"127.0.0.1","host":"127.0.0.1:1325","method":"GET","uri":"/_status/nodes","user_agent":"Go-http-client/1.1","status":404,"error":"code=404, message=Not Found","latency":21165,"latency_human":"21.165µs","bytes_in":0,"bytes_out":24}
{"time":"2020-02-23T18:00:25.349226829Z","id":"","remote_ip":"127.0.0.1","host":"127.0.0.1:1325","method":"GET","uri":"/health-check","user_agent":"Go-http-client/1.1","status":200,"error":"","latency":4243,"latency_human":"4.243µs","bytes_in":0,"bytes_out":0}

And here’s what caddy has to say about this:

2020/02/23 17:59:55.353	INFO	http.handlers.reverse_proxy.health_checker.active	unexpected status code	{"status_code": 404, "host": "127.0.0.1:8080"}
2020/02/23 18:00:25.348	INFO	http.handlers.reverse_proxy.health_checker.active	unexpected status code	{"status_code": 404, "host": "127.0.0.1:1325"}
2020/02/23 18:00:25.349	INFO	http.handlers.reverse_proxy.health_checker.active	host is up	{"host": "127.0.0.1:1325"}
2020/02/23 18:00:25.349	INFO	http.handlers.reverse_proxy.health_checker.active	unexpected status code	{"status_code": 404, "host": "127.0.0.1:8080"}
2020/02/23 18:00:25.352	INFO	http.handlers.reverse_proxy.health_checker.active	host is up	{"host": "127.0.0.1:8080"}
2020/02/23 18:00:55.350	INFO	http.handlers.reverse_proxy.health_checker.active	unexpected status code	{"status_code": 404, "host": "127.0.0.1:8080"}
2020/02/23 18:00:55.351	INFO	http.handlers.reverse_proxy.health_checker.active	unexpected status code	{"status_code": 404, "host": "127.0.0.1:1325"}
2020/02/23 18:00:55.352	INFO	http.handlers.reverse_proxy.health_checker.active	host is up	{"host": "127.0.0.1:8080"}
2020/02/23 18:01:25.352	INFO	http.handlers.reverse_proxy.health_checker.active	unexpected status code	{"status_code": 404, "host": "127.0.0.1:1325"}
2020/02/23 18:01:25.352	INFO	http.handlers.reverse_proxy.health_checker.active	unexpected status code	{"status_code": 404, "host": "127.0.0.1:8080"}
2020/02/23 18:01:25.352	INFO	http.handlers.reverse_proxy.health_checker.active	host is up	{"host": "127.0.0.1:1325"}

Which makes no sense. Why would caddy query the wrong upstream?

This also happens if i create another server in parallel with the base one and i have it listen on another port and proxy the request to the internal cdb admin:

"servers": {
  "cockroachdbadmin": {
    "listen": [
      ":8181"
    ],
    "routes": [
      {
        "handle": [
          {
            "handler": "authentication",
            "providers": {
                "http_basic": {
                  "hash": {
                    "algorithm": "bcrypt"
                  },
                  "accounts": [
                    {"username": "admin", "password": "....."}
                  ]
                }
            }
          }
        ]
      },
      {
        "handle": [
          {
            "handler": "subroute",
            "routes": [
              {
                "handle": [
                  {
                    "handler": "reverse_proxy",
                    "headers": {
                      "request": {
                        "set": {
                          "Host": [
                            "{http.request.host}"
                          ],
                          "X-Forwarded-For": [
                            "{http.request.remote.host}"
                          ],
                          "X-Forwarded-Proto": [
                            "{http.request.scheme}"
                          ],
                          "X-Real-Ip": [
                            "{http.request.remote.host}"
                          ]
                        }
                      }
                    },
                    "health_checks": {
                      "active": {
                        "expect_status": 2,
                        "path": "/_status/nodes"
                      }
                    },
                    "transport": {
                      "protocol": "http",
                      "read_buffer_size": 4096
                    },
                    "upstreams": [
                      {
                        "dial": "127.0.0.1:8080"
                      }
                    ]
                  }
                ]
              }
            ]
          }
        ],
        "terminal": true
      }
    ]
  },

Crossed the streams.

Important safety tip:

Don't cross the streams

Thanks for the detailed report… I think I’ve fixed this in reverse_proxy: Health checks: Don't cross the streams · caddyserver/caddy@7cca291 · GitHub. (Next time, please provide your whole config, it would have made this a little faster. Most of my time on it was just making a config to run the server.)

If you can build from that commit and try again, it’d be much appreciated!

4 Likes

@matt - i can confirm your fix is spot on, appreciate it, now things seem to work just fine.
I’ll get back to you if i find anything else.
Thank you for building this amazing piece of software.

1 Like

Awesome!! Thanks for confirming!

This topic was automatically closed 90 days after the last reply. New replies are no longer allowed.