Caddy as a default backend for k8s ingress controllers

Showcase
1

Hi,

at my current job, users often get lost when trying to reach k8s deployments for test systems which have been stopped. By default, the k8s nginx-ingress just returns a meaningless error message when no service behind the ingress is available. By defining a default backend (Annotations - Ingress-Nginx Controller), you can provide hints to the users on how to fix the current error.

(Caveat: This is only tested with the nginx-ingress-controller, other ingress controllers might provide similar features)

Caddyfile

{
        admin off
}

http://:8080

root * /srv/www
encode zstd gzip

@custom_err header_regexp X-Code ^\d+$
error @custom_err "Default {header.X-Code}" {header.X-Code}

templates
file_server

handle_errors {
        @custom_accept_json header_regexp Accept \b(application|text)/json\b
        respond @custom_accept_json `\{error: {err.status_code}, msg: "Ingress: {err.status_text}"\}`

        @err_file file /err-{err.status_code}.html /err.html
        handle @err_file {
                rewrite * {file_match.relative}
                templates
                file_server
        }
        respond "{err.status_code} {err.status_text}"
}

The deployment is nothing special, but sets two variables for use in the error pages:

          env:
            - name: RANCHERURL
              value: https://k8s-prod.company.internal/dashboard/c/some-cluster-id/explorer
            - name: STATICSITE
              value: https://default-backend.k8s-prod.company.internal

The webroot contains a bunch of templated HTML files to direct the user to the admin interface, for example:

err.html

<!DOCTYPE html>
<head>
<meta http-equiv="content-type" content="text/html;charset=utf-8">
<link rel="stylesheet" href="{{env "STATICSITE"}}/water-2.1.1.min.css" integrity="sha256-QST90Wzz4PEr5KlclQaOCsjc00FTyf86Wrj41oqZB4w=" crossorigin="anonymous">
<title>Ingress {{placeholder "http.error.status_code"}} - {{placeholder "http.error.status_text"}}</title>
</head>
<body>
	<h2>Error {{placeholder "http.error.status_code"}} - {{placeholder "http.error.status_text"}}</h2>
{{include "upstream-error.html"}}
</body>
</html>

err-503.html

<!DOCTYPE html>
<head>
<meta http-equiv="content-type" content="text/html;charset=utf-8">
<meta http-equiv="refresh" content="600">
<link rel="stylesheet" href="{{env "STATICSITE"}}/water-2.1.1.min.css" integrity="sha256-QST90Wzz4PEr5KlclQaOCsjc00FTyf86Wrj41oqZB4w=" crossorigin="anonymous">
<title>{{.Req.Header.Get "X-Ingress-Name"}} - missing workload error</title>
</head>
<body>
	<h2>Sorry Mario, your workload is in another castle.</h2>
	<p>It seems the system you want to reach (<code>{{.Host}}</code>) isn't
	able to accept your requests right now.</p>
{{include "upstream-error.html"}}
</body>
</html>

upstream-error.html

{{if .Req.Header.Get "X-Ingress-Name"}}
<p>If you are responsible for the system, take a look at Rancher
 (<a href="{{env "RANCHERURL"}}/networking.k8s.io.ingress/{{.Req.Header.Get "X-Namespace"}}/{{.Req.Header.Get "X-Ingress-Name"}}">Ingress</a>,
 <a href="{{env "RANCHERURL"}}/service/{{.Req.Header.Get "X-Namespace"}}/{{.Req.Header.Get "X-Service-Name"}}">Service</a>)
 to find out what the problem is.</p>
{{end}}

By constructing URLs to the Rancher admin interface from the base URL and the “magic” headers the ingress controller uses, we can direct users to the correct please where they can start debugging why the system isn’t reachable.

2 Likes