Idea: python plugin

Plugins
1

Caddy works great using proxy to dispatch requests to uwsgi, but I think it would be super if there was a plugin that could directly talk to the python process. Something like:

example.com {
	python / {
		module wsgi:app
		venv # to use $root/venv/bin/python
		except /static # same as proxy
		# Other options, maybe:
		#processes-max 5
		#threads-per-process 2
	}
	root /var/www/example.com
}

I’m going to give implementing this a try (I see there are some WSGI implementations in go already).

Would anyone be interested in using such a plugin?

1 Like
2

Well I think you’re not alone in wanting a direct connection to the Python code, and that would be cool, but you should probably read this issue first: https://github.com/mholt/caddy/issues/176

Also, before we add more plugins that do proxying, I think it would be super-useful to generalize the proxy middleware’s behavior that’s not specific to the “kind” of proxy (HTTP, FastCGI, WSGI, etc.) so that they can all share the load balancing features, etc, for their own protocols.

1 Like
3

Yeah, I read that two days ago when my friend told me about Caddy and I immediately started migrating my sites :).

Thinking of implementing the plugin as korylprince’s option 1.

Since this is my first time programming in Go, I’m constantly looking at other plugins/caddy itself for guidance. If I have ideas about generalizing proxy, I’ll be sure to let you know! Thanks for being open to the idea.

4

Finished prototype, example here. Please excuse the code. The tests are out of date and don’t pass.

I mostly copied/used the proxy code, and realized this plugin doesn’t need to be python-specific. It’s just a directive that spins up processes, then proxies to them. I.e., I feel it should be generalized to:

example.com {
	<some name> / {
		# %i replaced with port
		# pwd is set to root
		cmd "venv/bin/python app.py %i"
		except /static
		processes-max 5
	}
	root /var/www/example.com
}
5

So how is this different from, say, CGI? (Nice job hacking on it, by the way, this was just the first question that came to mind!)

6

Yeah, it’s CGI, but doesn’t start a new process for each request, it reuses them like FastCGI.

More general project is here: caddy-spawn

A few remarks/questions:

  1. Spawner.Select uses a mutex to prevent concurrent writes to s.Processes, which is probably bad to do for every request? It could be removed in the steady state by only locking if len(s.Processes) < s.MaxProcesses.
  2. It uses some private funcs from Caddy. Can I do a PR to make these public?
  3. It does rudimentary load balancing. This is what proxy.Policy does and it might be possible to combine them.
  4. Currently doesn’t retry, but otherwise handler.ServeHTTP is very similar to proxy.Handler.ServerHTTP and should be generalized. Likewise, spawn.Process → proxy.UpstreamHost and spawn.Spawner → proxy.Upstream. However, because number of processes is dynamic (starting new processes, checking/restarting those that have died…), a generic implementation would have some overhead that proxy does not need.