Filesystems module for Azure Blob

Plugins
1

I would like to add a filesystem plugin which makes it possible to get data from Azure Blob store Azure Blob Storage REST API. I plan to use the Azure SDK for Go

I haven’t seen any doc for the Module caddy.filesystems therefore have I looked into the source caddy/filesystem.go at master · caddyserver/caddy · GitHub .

Now my questions.

  • Is there a doc what a filesystem module need to work besides the Extending Caddy.
  • What filesystem module can I use as a good example for the filesystem implementation?

OT:
As I have already implemented a caddy module GitHub - git001/caddyv2-upload: This repo holds a simple caddyserver v2 upload handler I hope that I’m able to create the FS module as my second module for caddy :slight_smile:

2

Howdy, always great to see people looking to make new Caddy modules.

I don’t think we’ve got a doc specific to filesystem modules. That said, both the S3 and the Git FS modules already developed, registered, and listed in https://caddyserver.com/docs/modules/caddy.filesystems seem like great places to start.

2 Likes
3

Thank you

4

I have started to add the azure blob and have seen that the Open() function have no Request context, is that right?

How can I add something like request ID into the open call?
I assume via ctx but the fs have another lifecycle then a normal caddy module.
here my fs open snipplet


func (azbfs AZBlobFS) Open(name string) (fs.File, error) {

	azbfs.logger.Info("Call Open",
		zap.String("requuid", "requuid"),
		zap.String("name", name))

	credential, err_ndac := azidentity.NewDefaultAzureCredential(nil)

	if err_ndac != nil {
		azbfs.logger.Error("Provision NewDefaultAzureCredential",
			zap.String("requuid", requuid),
			zap.String("msg", "Error at NewDefaultAzureCredential"),
			zap.Error(err_ndac))
		return nil, err_ndac
	}

	azbfs.logger.Debug("NewDefaultAzureCredential",
		zap.Bool("requuiderr", requuiderr))

	client, err_nc := azblob.NewClient(azbfs.AZURL, credential, nil)

	if err_nc != nil {
		azbfs.logger.Error("Provision NewClient",
			zap.String("requuid", requuid),
			zap.String("msg", "Error at NewClient"),
			zap.Error(err_nc))
		return err_nc
	}

	azbfs.logger.Debug("NewClient",
		zap.Bool("requuiderr", requuiderr),
		zap.String("client", client.URL()))

	return nil, caddyhttp.ErrNotImplemented
}

That’s the debug log which shows the request and directly the fs call which make sense because the fileserver request should get the blob content from azure container.

2024/09/24 20:51:21.937	DEBUG	http.handlers.rewrite	rewrote request	{"request": {"remote_ip": "127.0.0.1", "remote_port": "50068", "client_ip": "127.0.0.1", "proto": "HTTP/1.1", "method": "GET", "host": "localhost:8080", "uri": "/assets/600x400_1.png", "headers": {"User-Agent": ["curl/7.81.0"], "Accept": ["*/*"]}}, "method": "GET", "uri": "/600x400_1.png"}
2024/09/24 20:51:21.937	DEBUG	http.handlers.file_server	sanitized path join	{"site_root": ".", "fs": "my-blob", "request_path": "/600x400_1.png", "result": "600x400_1.png"}
2024/09/24 20:51:21.937	INFO	caddy.fs.azureblobfs	Call Open	{"requuid": "requuid", "name": "600x400_1.png"}
2024/09/24 20:51:21.937	INFO	caddy.fs.azureblobfs	Call Open	{"requuid": "requuid", "name": "600x400_1.png"}

What’s the best way to add some information to the fs module?
Here the full file.

package azureblobfs

import (
	"bytes"
	"fmt"
	"io/fs"
	"net/http"

	"github.com/Azure/azure-sdk-for-go/sdk/azidentity"
	"github.com/Azure/azure-sdk-for-go/sdk/storage/azblob"
	"github.com/caddyserver/caddy/v2"
	"github.com/caddyserver/caddy/v2/caddyconfig/caddyfile"
	"github.com/caddyserver/caddy/v2/caddyconfig/httpcaddyfile"
	"github.com/caddyserver/caddy/v2/modules/caddyhttp"
	"go.uber.org/zap"
)

const (
	Version = "0.1"
)

func init() {
	caddy.RegisterModule(AZBlobFS{})
	httpcaddyfile.RegisterHandlerDirective("azureblobfs", parseCaddyfile)
}

// Middleware implements an HTTP handler that writes the
// uploaded file  to a file on the disk.
type AZBlobFS struct {
	fs.StatFS `json:"-"`
	AzSA      string `json:"azure_storage_account,omitempty"`
	AZURL     string `json:"azure_url,omitempty"`
	AZCont    string `json:"azure_container,omitempty"`

	AZCred azidentity.DefaultAzureCredential

	ctx    caddy.Context
	logger *zap.Logger
}

// CaddyModule returns the Caddy module information.
func (AZBlobFS) CaddyModule() caddy.ModuleInfo {
	return caddy.ModuleInfo{
		ID:  "caddy.fs.azureblobfs",
		New: func() caddy.Module { return new(AZBlobFS) },
	}
}

// Provision implements caddy.Provisioner.
func (azbfs *AZBlobFS) Provision(ctx caddy.Context) error {
	azbfs.ctx = ctx
	azbfs.logger = ctx.Logger(azbfs)

	if azbfs.AzSA == "" {
		azbfs.logger.Error("Provision",
			zap.String("msg", "no Azure Storage Account specified (azure_storage_account)"))
		//return fmt.Errorf("no Azure Storage Account specified (azure_storage_account)")
	}

	if azbfs.AZURL == "" {
		azbfs.logger.Info("Provision",
			zap.String("msg", "no Azure Storage Account URL specified (azure_url), will use https://{azure_storage_account}.blob.core.windows.net/"))
	}

	if azbfs.AZCont == "" {
		azbfs.logger.Info("Provision",
			zap.String("msg", "no Azure Blob Container specified (azure_container)"))
		return fmt.Errorf("no Azure Blob Container specified (azure_container)")
	}

	if (azbfs.AzSA == "") || (azbfs.AZURL == "") {
		return fmt.Errorf("no Azure Storage Account (azure_storage_account) and no Azure Storage Account URL (azure_url) specified")
	}

	azbfs.logger.Info("Current Config",
		zap.String("Version", Version),
		zap.String("azure_storage_account", azbfs.AzSA),
		zap.String("azure_url", azbfs.AZURL),
		zap.String("azure_container", azbfs.AZCont),
	)

	return nil
}

// Validate implements caddy.Validator.
func (azbfs *AZBlobFS) Validate() error {
	// TODO: Do I need this func
	return nil
}

func (azbfs AZBlobFS) Open(name string) (fs.File, error) {

	azbfs.logger.Info("Call Open",
		zap.String("requuid", "requuid"),
		zap.String("name", name))

	credential, err_ndac := azidentity.NewDefaultAzureCredential(nil)

	if err_ndac != nil {
		azbfs.logger.Error("Provision NewDefaultAzureCredential",
			zap.String("requuid", requuid),
			zap.String("msg", "Error at NewDefaultAzureCredential"),
			zap.Error(err_ndac))
		return nil, err_ndac
	}

	azbfs.logger.Debug("NewDefaultAzureCredential",
		zap.Bool("requuiderr", requuiderr))

	client, err_nc := azblob.NewClient(azbfs.AZURL, credential, nil)

	if err_nc != nil {
		azbfs.logger.Error("Provision NewClient",
			zap.String("requuid", requuid),
			zap.String("msg", "Error at NewClient"),
			zap.Error(err_nc))
		return err_nc
	}

	azbfs.logger.Debug("NewClient",
		zap.Bool("requuiderr", requuiderr),
		zap.String("client", client.URL()))

	return nil, caddyhttp.ErrNotImplemented
}

// ServeHTTP implements caddyhttp.MiddlewareHandler.
func (azbfs AZBlobFS) ServeHTTP(w http.ResponseWriter, r *http.Request, next caddyhttp.Handler) error {

	repl := r.Context().Value(caddy.ReplacerCtxKey).(*caddy.Replacer)

	requuid, requuiderr := repl.GetString("http.request.uuid")
	if !requuiderr {
		requuid = "0"
		azbfs.logger.Error("http.request.uuid",
			zap.Bool("requuiderr", requuiderr),
			zap.Object("request", caddyhttp.LoggableHTTPRequest{Request: r}))
	}

	azbfs.logger.Info("Successful Blob Access",
		zap.String("requuid", requuid),
		zap.Object("request", caddyhttp.LoggableHTTPRequest{Request: r}))

	// Download the blob
	// Container: craftcmsdev2
	// Blob: 600x400.png
	blobName := "600x400.png"
	get, err_ds := client.DownloadStream(azbfs.ctx, azbfs.AZCont, blobName, nil)

	if err_ds != nil {
		azbfs.logger.Error("Provision DownloadStream",
			zap.String("requuid", requuid),
			zap.String("msg", "Error at DownloadStream"),
			zap.Error(err_ds))
		return err_ds
	}

	downloadedData := bytes.Buffer{}
	retryReader := get.NewRetryReader(azbfs.ctx, &azblob.RetryReaderOptions{})
	_, err_dd := downloadedData.ReadFrom(retryReader)

	if err_dd != nil {
		azbfs.logger.Error("Provision downloadedData",
			zap.String("requuid", requuid),
			zap.String("msg", "Error at downloadedData"),
			zap.Error(err_dd))
		return err_dd
	}

	err_rr := retryReader.Close()
	if err_rr != nil {
		azbfs.logger.Error("Provision retryReader",
			zap.String("requuid", requuid),
			zap.String("msg", "Error at retryReader"),
			zap.Error(err_rr))
		return err_rr
	}

	// Print the content of the blob we created
	azbfs.logger.Info("Blob contents",
		zap.String("requuid", requuid),
		zap.String("downloadedData-String", downloadedData.String()))

	return nil
}

// Interface guards
var (
	_ caddy.Provisioner           = (*AZBlobFS)(nil)
	_ caddy.Validator             = (*AZBlobFS)(nil)
	_ caddyhttp.MiddlewareHandler = (*AZBlobFS)(nil)
	_ caddyfile.Unmarshaler       = (*AZBlobFS)(nil)
)

that’s my Caddyfile

{
	debug
	auto_https off

	filesystem my-blob azureblobfs {
		azure_storage_account "default"
		azure_url "us-east-1"
		azure_container "http://localhost:9000"
	}
	servers :8080 {
		name main
		metrics
		protocols h1 h2
		#trusted_proxies static private_ranges
	}
}

:8080

log default {
	level DEBUG
	output stdout
	format console {
		time_format iso8601
		duration_format ms
	}
}

route /assets/* {
	uri strip_prefix /assets

	file_server {
		fs my-blob
		pass_thru
	}
}
5

Filesystem modules can be used in non-request contexts (even non-HTTP contexts), so it doesn’t make sense to have a request context being passed to it. I don’t really understand what your goal is with that.

This doesn’t make sense, it’s not a Caddyfile directive.

2 Likes
6

Copy + paste mistake

7

This topic was automatically closed after 30 days. New replies are no longer allowed.