Extending Caddy by reusing existing reverseproxy module

Help
1

1. The problem I’m having:

I am trying to reuse the reverseproxy module in my Caddy module. I have an authentication scenario which requires each request to a wildcard subdomain domain to be authenticated and then based on the subdomain I would like to reverse proxy to internal services mapped from the subdomain. In a sense i will have a subdomain → internal ip mapping. I am still far from fulfilling that since I am stuck on reusing use reverseproxy because I get an error when I call the ServeHTTP() method from the reverseproxy Handler struct.

module.go file

package shield

import (
	"fmt"
	"net/http"
	"strings"

	"github.com/caddyserver/caddy/v2"
	"github.com/caddyserver/caddy/v2/caddyconfig/caddyfile"
	"github.com/caddyserver/caddy/v2/modules/caddyhttp"
	"github.com/caddyserver/caddy/v2/modules/caddyhttp/reverseproxy"
	"github.com/jackc/pgx/v5/pgxpool"
	"go.uber.org/zap"
)

const (
	DBPoolKey string = "shield_db"
)

var resourcePool *caddy.UsagePool = caddy.NewUsagePool()

func init() {
	caddy.RegisterModule(ShieldMiddleware{})
}

// ShieldMiddleware implements an HTTP handler that authenticates requests and looks up the upstream to which
// the request  should be proxied to
type ShieldMiddleware struct {
	ctx    caddy.Context
	logger *zap.Logger

	pgxPool    *pgxpool.Pool
	authClient *auth

	ReverseProxy *reverseproxy.Handler
}

// CaddyModule returns the Caddy module information.
func (ShieldMiddleware) CaddyModule() caddy.ModuleInfo {
	return caddy.ModuleInfo{
		ID:  "http.handlers.shield",
		New: func() caddy.Module { return new(ShieldMiddleware) },
	}
}

// Provision implements caddy.Provisioner
func (m *ShieldMiddleware) Provision(ctx caddy.Context) error {
	m.ctx = ctx
	m.logger = ctx.Logger()

	m.ReverseProxy = &reverseproxy.Handler{}
	if m.ReverseProxy != nil {
		if err := m.ReverseProxy.Provision(ctx); err != nil {
			return fmt.Errorf("provision reverse proxy, %v", err)
		}
	}

	db, loaded, err := resourcePool.LoadOrNew(DBPoolKey, func() (caddy.Destructor, error) {
		d, err := getDB(ctx)
		if err != nil {
			return nil, err
		}
		return dbDestructor{Pool: d}, nil
	})
	if err != nil {
		m.logger.Error("loading database connections pool", zap.String("db_key", DBPoolKey), zap.Error(err))
		return err
	}

	if loaded {
		m.logger.Info("using loaded db connections pool")
	}

	dbDesctructor, ok := db.(dbDestructor)
	if !ok {
		m.logger.Error("couldn't unmarshal to pgx pool")
	}
	m.pgxPool = dbDesctructor.Pool
	m.authClient = NewAuth()
	return err
}

// ServeHTTP implements caddyhttp.MiddlewareHandler
func (m ShieldMiddleware) ServeHTTP(w http.ResponseWriter, r *http.Request, next caddyhttp.Handler) error {
	token := m.GetToken(r)
	if !m.authClient.Authenticated(token) {
		w.WriteHeader(http.StatusUnauthorized)
		return nil
	}

	m.ReverseProxy.Upstreams = reverseproxy.UpstreamPool{
		&reverseproxy.Upstream{
			Dial: "localhost:8000", // in reality this will not be hardcode but fetched from a database / service
		},
	}
	return m.ReverseProxy.ServeHTTP(w, r, next)
}

func (m ShieldMiddleware) GetToken(r *http.Request) string {
	bearer := r.Header.Get("Authorization")
	if len(bearer) > 7 && strings.ToUpper(bearer[:6]) == "BEARER" {
		return bearer[7:]
	}
	return ""
}

// UnmarshalCaddyfile implements caddyfile.Unmarshaler
func (m *ShieldMiddleware) UnmarshalCaddyfile(d *caddyfile.Dispenser) error {
	d.Next() // consume directive name
	return nil
}

// Cleanup implements caddy.CleanerUpper
func (m *ShieldMiddleware) Cleanup() error {
	deleted, err := resourcePool.Delete(DBPoolKey)
	if deleted {
		m.logger.Debug("unloading unused database", zap.String("db_key", DBPoolKey))
	}
	if err != nil {
		m.logger.Error("closing database", zap.String("db_key", DBPoolKey), zap.Error(err))
	}
	return err
}

// Interface guards
var (
	_ caddy.Provisioner           = (*ShieldMiddleware)(nil)
	_ caddy.CleanerUpper          = (*ShieldMiddleware)(nil)
	_ caddyfile.Unmarshaler       = (*ShieldMiddleware)(nil)
	_ caddyhttp.MiddlewareHandler = (*ShieldMiddleware)(nil)
)

2. Error messages and/or full log output:

For context, on localhost:8000, I just serve a simple message:

❯ curl https://localhost:8000
hello!⏎

So I would have expected that doing the curl request to localhost:8080 would serve the “hello” content as well, because I am calling the ServeHTTP(w, r, next) from the reverseproxy Handler. Instead we get:

❯ curl -vL -H "Authorization Bearer this-is-valid-token" https://localhost:8080

* Host localhost:8080 was resolved.
* IPv6: ::1
* IPv4: 127.0.0.1
*   Trying [::1]:8080...
* Connected to localhost (::1) port 8080
* ALPN: curl offers h2,http/1.1
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
*  CAfile: /etc/ssl/certs/ca-certificates.crt
*  CApath: none
* TLSv1.3 (IN), TLS handshake, Server hello (2):
* TLSv1.3 (IN), TLS handshake, Encrypted Extensions (8):
* TLSv1.3 (IN), TLS handshake, Certificate (11):
* TLSv1.3 (IN), TLS handshake, CERT verify (15):
* TLSv1.3 (IN), TLS handshake, Finished (20):
* TLSv1.3 (OUT), TLS change cipher, Change cipher spec (1):
* TLSv1.3 (OUT), TLS handshake, Finished (20):
* SSL connection using TLSv1.3 / TLS_AES_128_GCM_SHA256 / x25519 / id-ecPublicKey
* ALPN: server accepted h2
* Server certificate:
*  subject: [NONE]
*  start date: Jul  8 08:17:16 2024 GMT
*  expire date: Jul  8 20:17:16 2024 GMT
*  subjectAltName: host "localhost" matched cert's "localhost"
*  issuer: CN=Caddy Local Authority - ECC Intermediate
*  SSL certificate verify ok.
*   Certificate level 0: Public key type EC/prime256v1 (256/128 Bits/secBits), signed using ecdsa-with-SHA256
*   Certificate level 1: Public key type EC/prime256v1 (256/128 Bits/secBits), signed using ecdsa-with-SHA256
*   Certificate level 2: Public key type EC/prime256v1 (256/128 Bits/secBits), signed using ecdsa-with-SHA256
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
* using HTTP/2
* [HTTP/2] [1] OPENED stream for https://localhost:8080/
* [HTTP/2] [1] [:method: GET]
* [HTTP/2] [1] [:scheme: https]
* [HTTP/2] [1] [:authority: localhost:8080]
* [HTTP/2] [1] [:path: /]
* [HTTP/2] [1] [user-agent: curl/8.8.0]
* [HTTP/2] [1] [accept: */*]
> GET / HTTP/2
> Host: localhost:8080
> User-Agent: curl/8.8.0
> Accept: */*
> 
* Request completely sent off
* HTTP/2 stream 1 was not closed cleanly: INTERNAL_ERROR (err 2)
* Connection #0 to host localhost left intact
curl: (92) HTTP/2 stream 1 was not closed cleanly: INTERNAL_ERROR (err 2)

❯ sudo ./caddy run --config Caddyfile
2024/07/08 12:25:23.812 INFO    using config from file  {"file": "Caddyfile"}
2024/07/08 12:25:23.813 INFO    adapted config to JSON  {"adapter": "caddyfile"}
2024/07/08 12:25:23.815 INFO    admin   admin endpoint started  {"address": "localhost:2019", "enforce_origin": false, "origins": ["//[::1]:2019", "//127.0.0.1:2019", "//localhost:2019"]}
2024/07/08 12:25:23.815 INFO    http.auto_https enabling automatic HTTP->HTTPS redirects        {"server_name": "srv0"}
2024/07/08 12:25:23.815 INFO    tls.cache.maintenance   started background certificate maintenance      {"cache": "0xc000458a00"}
2024/07/08 12:25:23.816 DEBUG   http.auto_https adjusted config {"tls": {"automation":{"policies":[{"subjects":["localhost"]},{}]}}, "http": {"servers":{"remaining_auto_https_redirects":{"listen":[":80"],"routes":[{},{}]},"srv0":{"listen":[":8080"],"routes":[{"handle":[{"handler":"subroute","routes":[{"handle":[{"handler":"subroute","routes":[{"handle":[{"ReverseProxy":null,"handler":"shield"}]}]}]}]}],"terminal":true}],"tls_connection_policies":[{}],"automatic_https":{}}}}}
2024/07/08 12:25:23.844 INFO    http    enabling HTTP/3 listener        {"addr": ":8080"}
2024/07/08 12:25:23.844 DEBUG   http    starting server loop    {"address": "[::]:8080", "tls": true, "http3": true}
2024/07/08 12:25:23.845 INFO    http.log        server running  {"name": "srv0", "protocols": ["h1", "h2", "h3"]}
2024/07/08 12:25:23.845 DEBUG   http    starting server loop    {"address": "[::]:80", "tls": false, "http3": false}
2024/07/08 12:25:23.845 INFO    http.log        server running  {"name": "remaining_auto_https_redirects", "protocols": ["h1", "h2", "h3"]}
2024/07/08 12:25:23.845 INFO    http    enabling automatic TLS certificate management   {"domains": ["localhost"]}
2024/07/08 12:25:23.845 WARN    tls     stapling OCSP   {"error": "no OCSP stapling for [localhost]: no OCSP server specified in certificate", "identifiers": ["localhost"]}
2024/07/08 12:25:23.845 DEBUG   tls.cache       added certificate to cache      {"subjects": ["localhost"], "expiration": "2024/07/08 20:17:17.000", "managed": true, "issuer_key": "local", "hash": "72beaffd66fb6d210c9a75eb58f12e722eca63e63fb9d31d2a8d757d000ca391", "cache_size": 1, "cache_capacity": 10000}
2024/07/08 12:25:23.845 DEBUG   events  event   {"name": "cached_managed_cert", "id": "96e1cf70-c89a-4332-83a9-6d303197d7f7", "origin": "tls", "data": {"sans":["localhost"]}}
2024/07/08 12:25:23.845 INFO    pki.ca.local    root certificate is already trusted by system   {"path": "storage:pki/authorities/local/root.crt"}
2024/07/08 12:25:23.846 INFO    autosaved config (load with --resume flag)      {"file": "/root/.config/caddy/autosave.json"}
2024/07/08 12:25:23.846 INFO    serving initial configuration
2024/07/08 12:25:23.848 INFO    tls     storage cleaning happened too recently; skipping for now        {"storage": "FileStorage:/root/.local/share/caddy", "instance": "ecd77583-7859-44f5-bf0b-baf6fe1d24c0", "try_again": "2024/07/09 12:25:23.848", "try_again_in": 86399.999999664}
2024/07/08 12:25:23.848 INFO    tls     finished cleaning storage units
2024/07/08 12:26:02.827 DEBUG   events  event   {"name": "tls_get_certificate", "id": "9372168c-79c8-473f-a03e-d501c47a4284", "origin": "tls", "data": {"client_hello":{"CipherSuites":[4866,4867,4865,49196,49200,159,52393,52392,52394,49195,49199,158,49188,49192,107,49187,49191,103,49162,49172,57,49161,49171,51,157,156,61,60,53,47,255],"ServerName":"localhost","SupportedCurves":[29,23,30,25,24,256,257,258,259,260],"SupportedPoints":"AAEC","SignatureSchemes":[1027,1283,1539,2055,2056,2074,2075,2076,2057,2058,2059,2052,2053,2054,1025,1281,1537,771,769,770,1026,1282,1538],"SupportedProtos":["h2","http/1.1"],"SupportedVersions":[772,771],"RemoteAddr":{"IP":"::1","Port":36416,"Zone":""},"LocalAddr":{"IP":"::1","Port":8080,"Zone":""}}}}
2024/07/08 12:26:02.827 DEBUG   tls.handshake   choosing certificate    {"identifier": "localhost", "num_choices": 1}
2024/07/08 12:26:02.827 DEBUG   tls.handshake   default certificate selection results   {"identifier": "localhost", "subjects": ["localhost"], "managed": true, "issuer_key": "local", "hash": "72beaffd66fb6d210c9a75eb58f12e722eca63e63fb9d31d2a8d757d000ca391"}
2024/07/08 12:26:02.827 DEBUG   tls.handshake   matched certificate in cache    {"remote_ip": "::1", "remote_port": "36416", "subjects": ["localhost"], "managed": true, "expiration": "2024/07/08 20:17:17.000", "hash": "72beaffd66fb6d210c9a75eb58f12e722eca63e63fb9d31d2a8d757d000ca391"}
2024/07/08 12:26:02.843 DEBUG   http.handlers.shield    selected upstream       {"dial": "localhost:8000", "total_upstreams": 1}
2024/07/08 12:26:02.843 DEBUG   http.stdlib     http2: panic serving [::1]:36416: runtime error: invalid memory address or nil pointer dereference
goroutine 56 [running]:
golang.org/x/net/http2.(*serverConn).runHandler.func1()
        golang.org/x/net@v0.25.0/http2/server.go:2363 +0x145
panic({0x18967e0?, 0x2db7ea0?})
        runtime/panic.go:770 +0x132
github.com/caddyserver/caddy/v2/modules/caddyhttp/reverseproxy.(*Host).NumRequests(...)
        github.com/caddyserver/caddy/v2@v2.8.4/modules/caddyhttp/reverseproxy/hosts.go:147
github.com/caddyserver/caddy/v2/modules/caddyhttp/reverseproxy.(*Handler).proxyLoopIteration(0xc0007e6f00, 0xc000818900, 0xc0008187e0, {0x2070120, 0xc0008640c0}, {0x0, 0x0}, {0xc19b1496b242610e, 0x9187a72c9, 0x2e10940}, ...)
        github.com/caddyserver/caddy/v2@v2.8.4/modules/caddyhttp/reverseproxy/reverseproxy.go:516 +0xa60
github.com/caddyserver/caddy/v2/modules/caddyhttp/reverseproxy.(*Handler).ServeHTTP(0xc0007e6f00, {0x2070120, 0xc0008640c0}, 0xc0008187e0, {0x205ec20, 0x1de57a0})
        github.com/caddyserver/caddy/v2@v2.8.4/modules/caddyhttp/reverseproxy/reverseproxy.go:432 +0x56b
github.com/integer-technologies-b-v/caddy-shield.ShieldMiddleware.ServeHTTP({{{0x2074490, 0xc000441c20}, 0xc00041f0b0, 0xc000809a40, {0xc000051b40, 0x4, 0x4}, {0x0, 0x0, 0x0}, ...}, ...}, ...)
        github.com/integer-technologies-b-v/caddy-shield@v0.0.0-00010101000000-000000000000/module.go:102 +0x16d
github.com/caddyserver/caddy/v2/modules/caddyhttp.wrapMiddleware.func1.1({0x2070120, 0xc0008640c0}, 0xc0008187e0)
        github.com/caddyserver/caddy/v2@v2.8.4/modules/caddyhttp/routes.go:333 +0xd2
github.com/caddyserver/caddy/v2/modules/caddyhttp.HandlerFunc.ServeHTTP(0x205ec20?, {0x2070120?, 0xc0008640c0?}, 0xc0008187e0?)
        github.com/caddyserver/caddy/v2@v2.8.4/modules/caddyhttp/caddyhttp.go:58 +0x29
github.com/caddyserver/caddy/v2/modules/caddyhttp.RouteList.Compile.wrapRoute.func1.1({0x2070120, 0xc0008640c0}, 0xc0008187e0)
        github.com/caddyserver/caddy/v2@v2.8.4/modules/caddyhttp/routes.go:300 +0x325
github.com/caddyserver/caddy/v2/modules/caddyhttp.HandlerFunc.ServeHTTP(0xc0005e4360?, {0x2070120?, 0xc0008640c0?}, 0x205ec20?)
        github.com/caddyserver/caddy/v2@v2.8.4/modules/caddyhttp/caddyhttp.go:58 +0x29
github.com/caddyserver/caddy/v2/modules/caddyhttp.(*Subroute).ServeHTTP(0xc000290720, {0x2070120, 0xc0008640c0}, 0xc0008187e0, {0x205ec20, 0x1de57a0})
        github.com/caddyserver/caddy/v2@v2.8.4/modules/caddyhttp/subroute.go:74 +0x67
github.com/caddyserver/caddy/v2/modules/caddyhttp.wrapMiddleware.func1.1({0x2070120, 0xc0008640c0}, 0xc0008187e0)
        github.com/caddyserver/caddy/v2@v2.8.4/modules/caddyhttp/routes.go:333 +0xd2
github.com/caddyserver/caddy/v2/modules/caddyhttp.HandlerFunc.ServeHTTP(0x205ec20?, {0x2070120?, 0xc0008640c0?}, 0xc0008187e0?)
        github.com/caddyserver/caddy/v2@v2.8.4/modules/caddyhttp/caddyhttp.go:58 +0x29
github.com/caddyserver/caddy/v2/modules/caddyhttp.RouteList.Compile.wrapRoute.func1.1({0x2070120, 0xc0008640c0}, 0xc0008187e0)
        github.com/caddyserver/caddy/v2@v2.8.4/modules/caddyhttp/routes.go:300 +0x325
github.com/caddyserver/caddy/v2/modules/caddyhttp.HandlerFunc.ServeHTTP(0xc0005e4240?, {0x2070120?, 0xc0008640c0?}, 0x205ec20?)
        github.com/caddyserver/caddy/v2@v2.8.4/modules/caddyhttp/caddyhttp.go:58 +0x29
github.com/caddyserver/caddy/v2/modules/caddyhttp.(*Subroute).ServeHTTP(0xc0002906a0, {0x2070120, 0xc0008640c0}, 0xc0008187e0, {0x205ec20, 0x1de57a0})
        github.com/caddyserver/caddy/v2@v2.8.4/modules/caddyhttp/subroute.go:74 +0x67
github.com/caddyserver/caddy/v2/modules/caddyhttp.wrapMiddleware.func1.1({0x2070120, 0xc0008640c0}, 0xc0008187e0)
        github.com/caddyserver/caddy/v2@v2.8.4/modules/caddyhttp/routes.go:333 +0xd2
github.com/caddyserver/caddy/v2/modules/caddyhttp.HandlerFunc.ServeHTTP(0x205ec20?, {0x2070120?, 0xc0008640c0?}, 0xc0008187e0?)
        github.com/caddyserver/caddy/v2@v2.8.4/modules/caddyhttp/caddyhttp.go:58 +0x29
github.com/caddyserver/caddy/v2/modules/caddyhttp.RouteList.Compile.wrapRoute.func1.1({0x2070120, 0xc0008640c0}, 0xc0008187e0)
        github.com/caddyserver/caddy/v2@v2.8.4/modules/caddyhttp/routes.go:300 +0x325
github.com/caddyserver/caddy/v2/modules/caddyhttp.HandlerFunc.ServeHTTP(0x1a7e480?, {0x2070120?, 0xc0008640c0?}, 0xc0008666e0?)
        github.com/caddyserver/caddy/v2@v2.8.4/modules/caddyhttp/caddyhttp.go:58 +0x29
github.com/caddyserver/caddy/v2/modules/caddyhttp.(*Server).enforcementHandler(0x10?, {0x2070120?, 0xc0008640c0?}, 0x0?, {0x205ec20?, 0xc0002b2040?})
        github.com/caddyserver/caddy/v2@v2.8.4/modules/caddyhttp/server.go:447 +0x24b
github.com/caddyserver/caddy/v2/modules/caddyhttp.(*App).Provision.(*Server).wrapPrimaryRoute.func1({0x2070120?, 0xc0008640c0?}, 0x4dc08f?)
        github.com/caddyserver/caddy/v2@v2.8.4/modules/caddyhttp/server.go:423 +0x35
github.com/caddyserver/caddy/v2/modules/caddyhttp.HandlerFunc.ServeHTTP(0xc0000c63c0?, {0x2070120?, 0xc0008640c0?}, 0xc0008187e0?)
        github.com/caddyserver/caddy/v2@v2.8.4/modules/caddyhttp/caddyhttp.go:58 +0x29
github.com/caddyserver/caddy/v2/modules/caddyhttp.(*Server).ServeHTTP(0xc0007f3808, {0x2070120, 0xc0008640c0}, 0xc000818480)
        github.com/caddyserver/caddy/v2@v2.8.4/modules/caddyhttp/server.go:353 +0xbbc
net/http.serverHandler.ServeHTTP({0x60?}, {0x2070120?, 0xc0008640c0?}, 0xc00067a5a8?)
        net/http/server.go:3137 +0x8e
net/http.initALPNRequest.ServeHTTP({{0x2074490?, 0xc00051f3e0?}, 0xc00073c008?, {0xc00066c000?}}, {0x2070120, 0xc0008640c0}, 0xc000818480)
        net/http/server.go:3745 +0x231
golang.org/x/net/http2.(*serverConn).runHandler(0x2e73f80?, 0x0?, 0x0?, 0xc00061f040?)
        golang.org/x/net@v0.25.0/http2/server.go:2370 +0xbb
created by golang.org/x/net/http2.(*serverConn).scheduleHandler in goroutine 46
        golang.org/x/net@v0.25.0/http2/server.go:2305 +0x21d

3. Caddy version:

Using xcaddy to build caddy, the compiled binary reports 2.8.4

4. How I installed and ran Caddy:

a. System environment:

OS: EndeavourOS Linux on Windows (under WSL2)

b. Command:

To build

❯ xcaddy build --with <name>=<location>

to run

❯ ./caddy run --config Caddyfile

c. Service/unit/compose file:

PASTE OVER THIS, BETWEEN THE ``` LINES.
Please use the preview pane to ensure it looks nice.

d. My complete Caddy config:

{
	debug
}

localhost:8080 {
	route {
		shield
	}
}

5. Links to relevant resources:

I already read that two maintainers, already proposed a solution about a similar problem, described by @jasonmccallister on this. One of the proposed solutions, which the author went with is setting a custom variable (placeholder) from your own module and then just letting reverse_proxy directive read it. That is not what I am trying to achieve here, If possible I would like to have everything wrapped in my own module which does this, without relying on somebody to know which variable to put for the reverse proxy.

Link to the discussion: Dynamically set reverse_proxy upstreams from custom module - #3 by jasonmccallister

2

Just sounds like you want to use forward_auth (Caddyfile directive) — Caddy Documentation. You can make this set a request header, then you can just do reverse_proxy {header.Some-Header} which contains the upstream address (must be host:port format, no scheme). Alternatively, you could implement a dynamic upstreams module which pulls the upstream to connect to based on the current request.

I think trying to embed handlers inside of others like that isn’t a great approach. It should be driven by config.

1 Like
3

Yes, I saw that one, indeed it looks useful (forward_auth). But I really want to do both things in one module - authenticating the request and reverse proxying to the grabbed upstream from my Subdomain → IP mapping.

But also based off on @matt’s reply in Dynamically set reverse_proxy upstreams from custom module

I am genuinely curious what is wrong with my setup, and what can I do in order to get the reverseproxy module working in the ServeHTTP

❯ curl https://localhost:8080
curl: (92) HTTP/2 stream 1 was not closed cleanly: INTERNAL_ERROR (err 2)
4

Sorry, this fell off my radar.

I’m really not sure. I haven’t tried doing that yet.

It likely has to do with how you’re passing the list of upstreams to the module. Doing so within ServeHTTP is probably too late, it should have been done in Provision I think. If you need the upstreams list to be dynamic, then you should use dynamic upstreams module, not static upstreams.

2 Likes
5

Thanks so much @francislavoie
I ended up implementing a dynamic upstream module and it worked fine.

1 Like
6

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.