Caddy behaviour on higher-latency file systems

nh2 · June 18, 2020, 11:03pm

Hi, I’d like to ask how Caddy technically does syscalls (on Linux) and consequently about its behaviour on higher-latency systems, like network-mounted NFS or CephFS.

In https://blog.cloudflare.com/how-we-scaled-nginx-and-saved-the-world-54-years-every-day/, Cloudflare points out that in Nginx read() syscalls can be put on thread pools (using aio threads;) instead of happenning in the epoll() event loop, such that a single slow read() (e.g. blocking for 500 ms) cannot block other syscalls.

However, they also point out that Nginx cannot currently do it for open(), which is a pain point. Cloudflare submitted an Nginx patch for that to be done in threads as well, but it was never merged upstream (last email here).

How does Caddy fare here? How is file system access implemented?

Thanks!

Whitestrake · June 18, 2020, 11:31pm

Hi @nh2, welcome to the Caddy community.

I’m not able to speak to the finer points of optimization here but I might be able to point you in the right direction to inspect how it’s handled yourself.

In the static file server module, Caddy relies pretty heavily on the Go standard library.

Files are opened in this code:

github.com

caddyserver/caddy/blob/3af15c0725605162dc8dad0c6ab241e2dc4553c6/modules/caddyhttp/fileserver/staticfiles.go#L244-L264

    
      
          // openFile opens the file at the given filename. If there was an error,
          // the response is configured to inform the client how to best handle it
          // and a well-described handler error is returned (do not wrap the
          // returned error value).
          func (fsrv *FileServer) openFile(filename string, w http.ResponseWriter) (*os.File, error) {
          	file, err := os.Open(filename)
          	if err != nil {
          		err = mapDirOpenError(err, filename)
          		if os.IsNotExist(err) {
          			return nil, caddyhttp.Error(http.StatusNotFound, err)
          		} else if os.IsPermission(err) {
          			return nil, caddyhttp.Error(http.StatusForbidden, err)
          		}
          		// maybe the server is under load and ran out of file descriptors?
          		// have client wait arbitrary seconds to help prevent a stampede
          		backoff := weakrand.Intn(maxBackoff-minBackoff) + minBackoff
          		w.Header().Set("Retry-After", strconv.Itoa(backoff))
          		return nil, caddyhttp.Error(http.StatusServiceUnavailable, err)
          	}
          	return file, nil
          }

Later, after some manipulation they are served to the client as appropriate:

github.com

caddyserver/caddy/blob/3af15c0725605162dc8dad0c6ab241e2dc4553c6/modules/caddyhttp/fileserver/staticfiles.go#L235-L239

    
      
          	// let the standard library do what it does best; note, however,
          	// that errors generated by ServeContent are written immediately
          	// to the response, so we cannot handle them (but errors there
          	// are rare)
          	http.ServeContent(w, r, info.Name(), info.ModTime(), file)

All the magic happens in the ServeHTTP function (caddy/staticfiles.go at 3af15c0725605162dc8dad0c6ab241e2dc4553c6 · caddyserver/caddy · GitHub).

The Golang os.Open functionality is here: os package - os - pkg.go.dev

Its implementation varies depending on the operating system.

system · September 16, 2020, 11:46pm

This topic was automatically closed 90 days after the last reply. New replies are no longer allowed.