State of HTTP Caching in Caddy 2

Forest_Johnson · February 18, 2021, 9:12pm

1. Caddy version (`caddy version`):

v2.4.0-beta.1

2. How I run Caddy:

Download binary from github releases page and caddy run

a. System environment:

  ___      _           _     _  __  ___   _ _  _   
 / _ \  __| |_ __ ___ (_) __| | \ \/ / | | | || |  
| | | |/ _` | '__/ _ \| |/ _` |  \  /| | | | || |_ 
| |_| | (_| | | | (_) | | (_| |  /  \| |_| |__   _|
 \___/ \__,_|_|  \___/|_|\__,_| /_/\_\\___/   |_|  
                                                   
Welcome to Armbian 21.02.1 Buster with Linux 5.4.94-odroidxu4

System load:   1%           	Up time:       23:10		Local users:   2            	
Memory usage:  10% of 1.94G  	IP:            192.168.0.24
CPU temp:      37°C           	Usage of /:    2% of 234G

b. Command:

caddy run

d. My complete Caddyfile or JSON config:

Caddyfile

http://localhost:8080 {
    reverse_proxy localhost:80
}

As you can see im up to quite some complex stuff

3. The problem I’m having:

I want to use Caddy as a web server because I like how its easily extensible, portable, and it manages the ACME renewals for me

However for my use case, I also need 3 capabilities:

Active health checks against upstream servers & upstream server health states (not 100% needed, but nice to have)
Ability to handle requests differently depending on whether the upstream server is “healthy” or not.
Ability to cache HTTP responses from upstream servers, store them on disk, and respond with cached version.

I’m pretty sure Caddy already has 1. and 2. However it looks like 3. is in progress: GitHub - caddyserver/cache-handler: Distributed HTTP caching module for Caddy

4. Error messages and/or full log output:

N/A

5. What I already tried:

First I tried doing this with nginx but I was unable to get it to work perfectly.

I already tried building with the 3rd party module: GitHub - sillygod/cdp-cache: a caddy 2 proxy cache plugin but it had a lot of issues. I had to update some dependencies and fix a couple bugs to get it to run at all without crashing. It was often failing to serve responses or panic-ing on nil pointers, and I never observed it actually serving from cache when the upstream was down. Nail in the coffin was when I tried to add debug logging, I was unable to get any visible output from the program, even after trying to manually open a file and append lines to it. Maybe this is because I’m impatient and unfamiliar with xcaddy build process or because I made a stupid mistake or something but I just gave up cuz I don’t want to spend another whole day on this.

In the end I shoddily hacked together my own custom solution which works fine but I feel like it should not be required for this.

6. Links to relevant resources:

I cant post links because im a new user RIP ME

Basically I just wanted to ask, is caching out of scope for Caddy? I realize caching is complex and difficult so that might explain why its not present in Caddy after all these years.

Yes I realize my proposed solution of utilizing a cache to serve static files & separate static from dynamic content as a part of a web application (like CDN+API) is kinda silly. Basically I was talking about making the API (dynamic part) serve everything and having the reverse proxy cache only the static files, thus allowing site to stay up when API goes down. This is just because its most convenient for the application I am serving. Yes I realize I can craft some custom set up to manually copy the static files from the app server’s webroot folder to the reverse proxy server and set up different routes for serving static files + reverse proxying the dynamic content. It’s just more of a pain in the butt and caching would feel more clean and nice. So this is just my 2c in the jar for caching support.

Yeas I realize the irony of asking for something wile at the same time stating that im too lazy to figure it out myself…

francislavoie · February 18, 2021, 9:37pm

Yep, this can all be done by configuring active health checking for the reverse_proxy directive:

If the upstream is healthy, then the request will be proxied as normal. If it’s not healthy, then you can configure handle_errors to handle 502 errors which Caddy emits when it doesn’t have an available upstream and do whatever with it.

handle_errors {
	@502 expression `{http.error.status_code} == 502`
	handle @502 {
		# Do whatever
	}
}

The cache-handler module is probably what you’ll want to experiment with. Yes it’s still WIP, but as far as I know it should have basic functionality available:

Forest_Johnson · February 18, 2021, 10:31pm

Thanks for the detailed response! I tried to give the a try but I was unable to figure out how to build it, as there is nothing on the ReadMe which describes how to build caddy with it.

I tried building this:

package main

import (
	caddycmd "github.com/caddyserver/caddy/v2/cmd"

	// plug in Caddy modules here
	_ "github.com/caddyserver/cache-handler"

	_ "github.com/caddyserver/caddy/v2/modules/standard"
)

func main() {
	caddycmd.Main()
}

and when I ran it with

http://localhost:8080 {

    cache
  
    reverse_proxy 192.168.0.45:8080 {
      health_path     /
      health_port     8080
      health_interval 1s
      health_timeout  5s
      health_status   200
    }

    handle_errors {
	@502 expression `{http.error.status_code} == 502`
	handle @502 {
           respond "test" 200
	}
    }

}

I got run: adapting config using caddyfile: Caddyfile:4: unrecognized directive: cache

I also tried manually introducing it into the source tree and building caddy from source : https://git.sequentialread.com/forest/caddy/commits/master

However I got the same results (builds without including the module & caddy says unrecognized directive: cache . I can’t figure out how to coax go build to actuall build the source code sitting in front of it, seems it always builds something cached or something it downloaded from the internet, or some combination of both.

Forest_Johnson · February 18, 2021, 10:39pm

Never mind, I am a dummy and I missed a spot, mis-attributed my mistake to go build cache:

https colon slash slash git dot sequentialread.com/forest/caddy/commit/7e81959db810666abfc9df5b7fd301351adbc609

Why does this web site not allow links?? seems counter productive

Forest_Johnson · February 18, 2021, 10:45pm

Oh, I see, all my posts got “flagged” because they contained links to my git server? I realize you folks probably deal with a lot of spam but this is really dystopic… links to git repository will get your post removed UNLESS you link to “approved” tech empire sites ? Eww…

francislavoie · February 18, 2021, 10:50pm

Yeah sorry about that. I’ve unflagged your post. Not sure why the forum is being so aggressive with your posts.

The best way to make custom builds of Caddy is with xcaddy:

$ xcaddy build --with github.com/caddyserver/cache-handler

It looks like you’re working on touching up the cache-handler lib, if so then you can do a replacement like this:

xcaddy build --with github.com/caddyserver/cache-handler=/path/to/your/cache-handler/repo

And if you need to replace Caddy itself then you can do:

xcaddy build --with github.com/caddyserver/caddy/v2=/path/to/caddy/repo --with <other plugins>

Forest_Johnson · February 18, 2021, 11:13pm

Thanks! After fixing my mistake, when I compiled it for arm using my janky&ham fisted method of just forking the entire repo, it worked, and it did produce an ARM binary, but the binary crashes on a nil pointer inside the olric app when I run it with the Caddyfile from the test fixture:

root@odroidxu4:~# ./caddy2 run
2021/02/18 22:51:21.269	INFO	using adjacent Caddyfile
[WARNING][caddyfile] Caddyfile:1: input is not formatted with 'caddy fmt'
2021/02/18 22:51:21.276	INFO	admin	admin endpoint started	{"address": "tcp/localhost:2019", "enforce_origin": false, "origins": ["localhost:2019", "[::1]:2019", "127.0.0.1:2019"]}
2021/02/18 22:51:21.276	INFO	http	server is listening only on the HTTPS port but has no TLS connection policies; adding one to enable TLS	{"server_name": "srv0", "https_port": 443}
2021/02/18 22:51:21.276	INFO	http	enabling automatic HTTP->HTTPS redirects	{"server_name": "srv0"}
2021/02/18 22:51:21.276	INFO	tls.cache.maintenance	started background certificate maintenance	{"cache": "0x2459180"}
2021/02/18 16:51:21 [INFO] Join completed. Synced with 0 initial nodes => olric.go:304
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x0 pc=0x11db8]

goroutine 85 [running]:
runtime/internal/atomic.goStore64(0x25ac014, 0xd, 0x0)
	/usr/local/go/src/runtime/internal/atomic/atomic_arm.go:144 +0x1c
github.com/buraksezer/olric.(*Olric).setOwnedPartitionCount(0x25ac000)
	/home/forest/go/pkg/mod/github.com/buraksezer/olric@v0.3.5/routing.go:452 +0x15c
github.com/buraksezer/olric.(*Olric).updateRoutingOperation(0x25ac000, 0x13915e8, 0x22281b0, 0x13915e8, 0x2228180)
	/home/forest/go/pkg/mod/github.com/buraksezer/olric@v0.3.5/routing.go:496 +0x33c
github.com/buraksezer/olric.(*Olric).requestDispatcher(0x25ac000, 0x13915e8, 0x22281b0, 0x13915e8, 0x2228180)
	/home/forest/go/pkg/mod/github.com/buraksezer/olric@v0.3.5/olric.go:265 +0xac
github.com/buraksezer/olric/internal/transport.(*Server).processMessage(0x22d8a80, 0xa694d568, 0x26cc690, 0x25fa6d4, 0x20cc540, 0x0, 0x0)
	/home/forest/go/pkg/mod/github.com/buraksezer/olric@v0.3.5/internal/transport/server.go:194 +0x294
github.com/buraksezer/olric/internal/transport.(*Server).processConn(0x22d8a80, 0xa694d568, 0x26cc690)
	/home/forest/go/pkg/mod/github.com/buraksezer/olric@v0.3.5/internal/transport/server.go:218 +0x108
created by github.com/buraksezer/olric/internal/transport.(*Server).listenAndServe
	/home/forest/go/pkg/mod/github.com/buraksezer/olric@v0.3.5/internal/transport/server.go:257 +0xd8

EDIT: for your viewing pleasure, here is /home/forest/go/pkg/mod/github.com/buraksezer/olric@v0.3.5/internal/transport/server.go annotated with line numbers

Honestly I can’t make heads or tails of how it would throw a nil pointer there

When I build it with xcaddy , I think I’m getting a cross compilation issue where GOARCH=arm is making it throw a compile-time numerical overflow on this line:

/home/forest/go/pkg/mod/github.com/dgraph-io/badger/v2@v2.2007.2/value.go

// maxVlogFileSize is the maximum size of the vlog file which can be created. Vlog Offset is of
// uint32, so limiting at max uint32.
var maxVlogFileSize = math.MaxUint32

Here is the log where xcaddy build failed:

forest@thingpad:~/Desktop/git/caddy/cmd/caddy$ GOARCH=arm ~/Desktop/programs/xcaddy/xcaddy build --with github.com/caddyserver/cache-handler
2021/02/18 17:04:16 [INFO] Temporary folder: /tmp/buildenv_2021-02-18-1704.413007958
2021/02/18 17:04:16 [INFO] Writing main module: /tmp/buildenv_2021-02-18-1704.413007958/main.go
2021/02/18 17:04:16 [INFO] Initializing Go module
2021/02/18 17:04:16 [INFO] exec (timeout=10s): /usr/local/go/bin/go mod init caddy 
go: creating new go.mod: module caddy
2021/02/18 17:04:16 [INFO] Pinning versions
2021/02/18 17:04:16 [INFO] exec (timeout=0s): /usr/local/go/bin/go get -d -v github.com/caddyserver/caddy/v2 
go: downloading github.com/caddyserver/caddy v1.0.5
go: downloading github.com/caddyserver/caddy/v2 v2.3.0
.........blahblah.........
go: downloading github.com/golang/snappy v0.0.2
2021/02/18 17:04:26 [INFO] Build environment ready
2021/02/18 17:04:26 [INFO] Building Caddy
2021/02/18 17:04:26 [INFO] exec (timeout=0s): /usr/local/go/bin/go mod tidy 
go: downloading gopkg.in/natefinch/lumberjack.v2 v2.0.0
.........blahblah.........
go: downloading github.com/jmespath/go-jmespath v0.3.0
2021/02/18 17:04:29 [INFO] exec (timeout=0s): /usr/local/go/bin/go build -o /home/forest/Desktop/git/caddy/cmd/caddy/caddy -ldflags -w -s -trimpath 
# github.com/dgraph-io/badger/v2
/home/forest/go/pkg/mod/github.com/dgraph-io/badger/v2@v2.2007.2/value.go:50:5: constant 4294967295 overflows int
2021/02/18 17:04:58 [INFO] Cleaning up temporary folder: /tmp/buildenv_2021-02-18-1704.413007958
2021/02/18 17:04:58 [FATAL] exit status 2

After a while I was able finagle xcaddy build into working by clearing the go build cache (go clean -r -x -cache -modcache), retrying the build, editing that file /home/forest/go/pkg/mod/github.com/dgraph-io/badger/v2@v2.2007.2/value.go and setting maxVlogFileSize to math.MaxInt32 rather than math.MaxUint32, then finally running xcaddy again.

Unfortunately it still has the same nil pointer inside olric when I run it.

I think the TL;DR is, Caddy serves static files really really well but it doesn’t quite cache things yet, especially not caching them on disk as this isn’t even on the roadmap yet… so for now we have to either build the functionality into caddy ourselves or build a “devops” process to ship the static files out to the Reverse Proxy servers so they can be served directly from disk there.

matt · February 18, 2021, 11:52pm

Might be worth filing an issue for that panic in case it’s not already reported: Sign in to GitHub · GitHub

Forest_Johnson · February 19, 2021, 12:12am

https://github.com/caddyserver/cache-handler/issues/11

system · March 20, 2021, 9:12pm

This topic was automatically closed after 30 days. New replies are no longer allowed.