Modifying request and response bodies

[k_o (Karsten)]

1. The problem I’m having:

I want to modify the request and response in a flexible way, i.e. also replace it completely. I have to deal with JSON and YAML modifications. I found the GitHub - caddyserver/replace-response: Caddy module that performs replacements in response bodies module but this is only useful for regex replacements. I would like to include my own logic. Is the best and only way currently to write my own module?

3. Caddy version:

v2.7.6 h1:w0NymbG2m9PcvKWsrXO6EEkY9Ru4FJK8uQbYcev1p3A=

4. How I installed and ran Caddy:

sudo apt-get install caddy

a. System environment:

Ubuntu 22.04

[francislavoie]

@k_o

Is the best and only way currently to write my own module?

Probably, yes. It will always be easier to write code to do exactly what you need.

Writing a plugin is quite easy: Extending Caddy — Caddy Documentation

[k_o (Karsten)]

I have written a little test plugin, but now I don’t know how to apply it. I’m extending a caddyhttp.MiddlewareHandler but how to integrate it in my existing file? My logreqresp should act as directive.

My orig config:

{
    debug
    admin   off
    servers 127.0.0.200:7778 {
        protocols h1 h2c
    }
}

:7778 {
    bind 127.0.0.200
    reverse_proxy * h2c://127.0.0.201:7777
    log default
}

When adding logreqresp:

:7778 {
    bind 127.0.0.200
    reverse_proxy * h2c://127.0.0.201:7777
    log default
    logreqresp
}

Adding it on a new line make caddy try to use start a new https server on port 443:

logreqresp

I just want to add a new logging directive to my existing defined :7778 config.

[francislavoie]

See Caddyfile Support — Caddy Documentation which explains how to add Caddyfile support to your plugin.

[k_o (Karsten)]

The referenced documentation is describing how to program a gizmo directive, but does not give a Caddyfile example. From what I see there it seems that a module is completely independent, does not inherit any settings and I cannot use to power of any other existing directives and modules? Is the standard approach to parse the Caddyfile file in UnmarshalCaddyfile and re-implement manually all of bind, reverse_proxy, log etc. behavior? This seems to be a lot of work.

I want to keep the bind, reverse_proxy, log etc directives and just need an additional behavior. What is the best way to extend caddy, i.e. adding a new directive by keeping what I is already provided?

[francislavoie]

@k_o

What is the best way to extend caddy, i.e. adding a new directive by keeping what I is already provided?

Literally those docs explain how to do that.

You call RegisterDirective to register your directive so that your UnmarshalCaddyfile gets called when the Caddyfile parser encounters your directive in your config.

@k_o

:7778 {
    bind 127.0.0.200
    reverse_proxy * h2c://127.0.0.201:7777
    log default
    logreqresp
}

is exactly how your config will look.

You’ll need to add the order global option though to set an order for your directive (plugins don’t get an order automatically, only built-in ones do; see Caddyfile Directives — Caddy Documentation)

[k_o (Karsten)]

I have added the order directive, since my directive is not found:

{
    debug
    admin   off
    servers 127.0.0.200:7778 {
        protocols h1 h2c
    }
    order replace before abort
}

:7778 {
    bind 127.0.0.200
    reverse_proxy * h2c://127.0.0.201:7777
    log default
    logreqresp
}

According to the Global options (Caddyfile) — Caddy Documentation documentation I have tried to add it as a global option, but I get:

2024/01/02 12:55:04.698 INFO using adjacent Caddyfile
Error: adapting config using caddyfile: Caddyfile:10: unrecognized directive: order

[francislavoie]

Your directive is called logreqresp, not replace.

Also is that actually your entire config? If it is, then that error message doesn’t make sense. The error says order was on line 10, but it’s on line 7 in what you pasted.

[k_o (Karsten)]

Right, the error line was incorrect, it must be line 7. My other error was that I was using caddy run instead of xcaddy run and then my work in progress plugin could not be found.

Maybe the final obstacle I have to solve is now:

:7778 {
    bind 127.0.0.200
    reverse_proxy * h2c://127.0.0.201:7777
    log default
    logreqresp {
        output stderr
    }
}

Error: adapting config using caddyfile: parsing caddyfile tokens for ‘logreqresp’: wrong argument count or unexpected line ending after ‘logreqresp’, at Caddyfile:14
2024/01/02 14:54:28 [ERROR] exit status 1

[francislavoie]

Clearly your code wasn’t written correctly to parse the Caddyfile tokens for that directive. That error comes from calling d.ArgErr().

I don’t understand exactly what you’re trying to implement here.

[k_o (Karsten)]

The plugin I’m working on is logging the response body and request body. Basically I was using the example and modified it a bit:

package logreqresp

import (
	"bytes"
	"fmt"
	"io"
	"log"
	"net/http"
	"net/http/httptest"
	"os"

	"github.com/caddyserver/caddy/v2"
	"github.com/caddyserver/caddy/v2/caddyconfig/caddyfile"
	"github.com/caddyserver/caddy/v2/caddyconfig/httpcaddyfile"
	"github.com/caddyserver/caddy/v2/modules/caddyhttp"
)

func init() {
	caddy.RegisterModule(ReqRespLogger{})
	httpcaddyfile.RegisterHandlerDirective("logreqresp", parseCaddyfile)
}

// ReqRespLogger implements an HTTP handler that writes the
// visitor's IP address to a file or stream.
type ReqRespLogger struct {
	// The file or stream to write to. Can be "stdout"
	// or "stderr".
	Output string `json:"output,omitempty"`

	w io.Writer
}

// CaddyModule returns the Caddy module information.
func (ReqRespLogger) CaddyModule() caddy.ModuleInfo {
	return caddy.ModuleInfo{
		ID:  "http.handlers.logreqresp",
		New: func() caddy.Module { return new(ReqRespLogger) },
	}
}

// Provision implements caddy.Provisioner.
func (m *ReqRespLogger) Provision(ctx caddy.Context) error {
	switch m.Output {
	case "stdout":
		m.w = os.Stdout
	case "stderr":
		m.w = os.Stderr
	default:
		return fmt.Errorf("an output stream is required")
	}
	return nil
}

// Validate implements caddy.Validator.
func (m *ReqRespLogger) Validate() error {
	if m.w == nil {
		return fmt.Errorf("no writer")
	}
	return nil
}

// ServeHTTP implements caddyhttp.MiddlewareHandler.
func (m ReqRespLogger) ServeHTTP(w http.ResponseWriter, r *http.Request, next caddyhttp.Handler) error {
	if r.Body != nil {
		defer r.Body.Close()
		buf := new(bytes.Buffer)
		_, err := io.Copy(buf, r.Body)
		if err != nil {
			log.Printf("error copying request body: %v", err)
		} else {
			m.w.Write([]byte("request:\n"))
			_, err = m.w.Write(buf.Bytes())
			if err != nil {
				log.Printf("error writing request body: %v", err)
			}
			m.w.Write([]byte("\n"))
			r.Body = io.NopCloser(buf)
		}
	} else {
		log.Println("no request body")
	}
	wrec := httptest.NewRecorder()
	err := next.ServeHTTP(wrec, r)
	if err != nil {
		log.Printf("error calling next module: %v", err)
	}
	m.w.Write([]byte("response:\n"))
	_, err = m.w.Write(wrec.Body.Bytes())
	if err != nil {
		log.Printf("error logging response body: %v", err)
	}
	m.w.Write([]byte("\n"))
	for k, v := range wrec.Header() {
		w.Header().Add(k, v[0])
	}
	w.WriteHeader(wrec.Result().StatusCode)
	_, err = w.Write(wrec.Body.Bytes())
	if err != nil {
		log.Printf("error writing response body: %v", err)
	}
	return nil
}

// UnmarshalCaddyfile implements caddyfile.Unmarshaler.
func (m *ReqRespLogger) UnmarshalCaddyfile(d *caddyfile.Dispenser) error {
	for d.Next() {
		if !d.Args(&m.Output) {
			return d.ArgErr()
		}
	}
	return nil
}

// parseCaddyfile unmarshals tokens from h into a new ReqRespLogger.
func parseCaddyfile(h httpcaddyfile.Helper) (caddyhttp.MiddlewareHandler, error) {
	var m ReqRespLogger
	err := m.UnmarshalCaddyfile(h.Dispenser)
	return m, err
}

// Interface guards
var (
	_ caddy.Provisioner           = (*ReqRespLogger)(nil)
	_ caddy.Validator             = (*ReqRespLogger)(nil)
	_ caddyhttp.MiddlewareHandler = (*ReqRespLogger)(nil)
	_ caddyfile.Unmarshaler       = (*ReqRespLogger)(nil)
)

[francislavoie]

@k_o

	for d.Next() {
		if !d.Args(&m.Output) {
			return d.ArgErr()
		}
	}
	return nil

This is trying to parse something like logreqresp <output> (arg on the same line as the directive) not a block with { }. If you wanted to parse a block you’d need to use d.NextBlock() and loop through the lines.

See the Caddy source for more directive examples.

[k_o (Karsten)]

Great, understood. Here is the working configuration solution.

{
    debug
    admin   off
    servers 127.0.0.200:7778 {
        protocols h1 h2c
    }
    order logreqresp before abort
}

:7778 {
    bind 127.0.0.200
    reverse_proxy * h2c://127.0.0.201:7777
    log default
    logreqresp stderr
}

[k_o (Karsten)]

Is this plugin of general use if it works? Then I would publish it. I have searched if something like this exists, but right now it does not seem to be the case.

[k_o (Karsten)]

I have updated above my plugin which is working fine now, in case one is needing it.

[matt (Matt Holt)]

Congrats on a working plugin!

[stbu (Steffen)]

@k_o

Is this plugin of general use if it works? Then I would publish it.

Yes, it would be great if you could publish the plugin.