Enabling "encode gzip" interferes with reverse proxy rewrites

Help
1

1. Caddy version (caddy version):

v2.4.6 h1:HGkGICFGvyrodcqOOclHKfvJC0qTU7vny/7FhYp9hNw=

2. How I run Caddy:

a. System environment:

Ubuntu 20.04

b. Command:

caddy2 run --watch --config Caddyfile

d. My complete Caddyfile or JSON config:

{
	order realip first
}

http:// {
	root * /var/www/webapp

	uri /api/* strip_prefix /api
	reverse_proxy /api/* http://httpbin.org {
		header_up X-Forwarded-For {remote_host}
	}

	@assets path *.js *.css *.png *.svg
	header @assets Cache-Control "public; max-age=604800"

	php_fastcgi unix//run/php/php7.4-fpm.sock
	file_server
}

3. The problem I’m having:

I’m trying to write a Caddyfile for a setup that serves content from an API server under /api and a typical PHP webapp with certain static assets otherwise. The Caddyfile contents are as above.

This work as expected; until I decided to gzip responses by adding the encode directive as shown below, at which point caddy decides to handle all traffic using the php_fastcgi directive.

	root * /var/www/webapp
	encode {
		gzip 4
	}

Without the encode directive, this is what happens. (I’m developing on a remote server, so I’ve used port 51541 to the server’s port 80 through SSH port forwarding):

$ curl -i http://localhost:51541/api/get
HTTP/1.1 200 OK
Access-Control-Allow-Credentials: true
Access-Control-Allow-Origin: *
Content-Length: 295
Content-Type: application/json
Date: Wed, 19 Jan 2022 05:23:59 GMT
Server: Caddy
Server: gunicorn/19.9.0

{
  "args": {}, 
  "headers": {
    "Accept": "*/*", 
    "Accept-Encoding": "gzip", 
    "Host": "localhost", 
    "User-Agent": "curl/7.77.0", 
    "X-Amzn-Trace-Id": "Root=1-61e7a06f-178c5eb151b4f82b669408e1"
  }, 
  "origin": "127.0.0.1, 103.217.221.153", 
  "url": "http://localhost/get"
}

These are the debug logs generated for the request:

2022/01/19 05:30:00.518 DEBUG   http.handlers.rewrite   rewrote request {"request": {"remote_addr": "127.0.0.1:60390", "proto": "HTTP/1.1", "method": "GET", "host": "localhost:51541", "uri": "/api/get", "headers": {"User-Agent": ["curl/7.77.0"], "Accept": ["*/*"]}}, "method": "GET", "uri": "/get"}
2022/01/19 05:30:01.777 DEBUG   http.handlers.reverse_proxy     upstream roundtrip      {"upstream": "httpbin.org:80", "duration": 1.258933302, "request": {"remote_addr": "127.0.0.1:60390", "proto": "HTTP/1.1", "method": "GET", "host": "localhost:51541", "uri": "/get", "headers": {"User-Agent": ["curl/7.77.0"], "Accept": ["*/*"], "X-Forwarded-Proto": ["http"], "X-Forwarded-For": ["127.0.0.1"]}}, "headers": {"Server": ["gunicorn/19.9.0"], "Access-Control-Allow-Origin": ["*"], "Access-Control-Allow-Credentials": ["true"], "Date": ["Wed, 19 Jan 2022 05:30:01 GMT"], "Content-Type": ["application/json"], "Content-Length": ["295"], "Connection": ["keep-alive"]}, "status": 200}

With encode enabled, this is what happens instead. Notice the PHPSESSID cookie indicating that the request was handled by the PHP-FPM server instead:

$ curl -i http://localhost:51541/api/get
HTTP/1.1 404 Not Found
Cache-Control: max-age=0, private, must-revalidate
Cache-Control: no-cache, private
Content-Security-Policy: frame-ancestors 'self'
Content-Type: text/html; charset=UTF-8
Date: Wed, 19 Jan 2022 05:31:49 GMT
Server: Caddy
Set-Cookie: PHPSESSID=gk1p5d4nlbm11tac5fn7i93ha8anqqht; path=/; HttpOnly; SameSite=lax
Status: 404 Not Found
X-Frame-Options: sameorigin
Content-Length: 9

Not Found

These are the corresponding debug logs generated with encode enabled, which apparently indicate that the /get request is now being passed to the php_fastcgi handler:

2022/01/19 05:31:49.492 DEBUG   http.handlers.rewrite   rewrote request {"request": {"remote_addr": "127.0.0.1:44002", "proto": "HTTP/1.1", "method": "GET", "host": "localhost:51541", "uri": "/api/get", "headers": {"User-Agent": ["curl/7.77.0"], "Accept": ["*/*"]}}, "method": "GET", "uri": "/get"}
2022/01/19 05:31:49.492 DEBUG   http.handlers.rewrite   rewrote request {"request": {"remote_addr": "127.0.0.1:44002", "proto": "HTTP/1.1", "method": "GET", "host": "localhost:51541", "uri": "/get", "headers": {"User-Agent": ["curl/7.77.0"], "Accept": ["*/*"]}}, "method": "GET", "uri": "index.php"}
2022/01/19 05:31:49.492 DEBUG   http.reverse_proxy.transport.fastcgi    roundtrip       {"request": {"remote_addr": "127.0.0.1:44002", "proto": "HTTP/1.1", "method": "GET", "host": "localhost:51541", "uri": "index.php", "headers": {"Accept": ["*/*"], "X-Forwarded-Proto": ["http"], "X-Forwarded-For": ["127.0.0.1"], "User-Agent": ["curl/7.77.0"]}}, "dial": "/run/php/php7.4-fpm.sock", "env": {"AUTH_TYPE":"","CONTENT_LENGTH":"","CONTENT_TYPE":"","DOCUMENT_ROOT":"/var/www/webapp","DOCUMENT_URI":"index.php","GATEWAY_INTERFACE":"CGI/1.1","HTTP_ACCEPT":"*/*","HTTP_HOST":"localhost:51541","HTTP_USER_AGENT":"curl/7.77.0","HTTP_X_FORWARDED_FOR":"127.0.0.1","HTTP_X_FORWARDED_PROTO":"http","PATH_INFO":"","QUERY_STRING":"","REMOTE_ADDR":"127.0.0.1","REMOTE_HOST":"127.0.0.1","REMOTE_IDENT":"","REMOTE_PORT":"44002","REMOTE_USER":"","REQUEST_METHOD":"GET","REQUEST_SCHEME":"http","REQUEST_URI":"/api/get","SCRIPT_FILENAME":"/var/www/webapp/index.php","SCRIPT_NAME":"/index.php","SERVER_NAME":"localhost","SERVER_PORT":"51541","SERVER_PROTOCOL":"HTTP/1.1","SERVER_SOFTWARE":"Caddy/v2.4.6"}}
2022/01/19 05:31:49.500 DEBUG   http.handlers.reverse_proxy     upstream roundtrip      {"upstream": "unix//run/php/php7.4-fpm.sock", "duration": 0.007257121, "request": {"remote_addr": "127.0.0.1:44002", "proto": "HTTP/1.1", "method": "GET", "host": "localhost:51541", "uri": "index.php", "headers": {"X-Forwarded-For": ["127.0.0.1"], "User-Agent": ["curl/7.77.0"], "Accept": ["*/*"], "X-Forwarded-Proto": ["http"]}}, "headers": {"Content-Type": ["text/html; charset=UTF-8"], "Status": ["404 Not Found"], "Cache-Control": ["max-age=0, private, must-revalidate", "no-cache, private"], "Set-Cookie": ["PHPSESSID=gk1p5d4nlbm11tac5fn7i93ha8anqqht; path=/; HttpOnly; SameSite=lax"], "Date": ["Wed, 19 Jan 2022 05:31:49 GMT"], "X-Frame-Options": ["sameorigin"], "Content-Security-Policy": ["frame-ancestors 'self'"]}, "status": 404}

4. Error messages and/or full log output:

Shared in section 3.

5. What I already tried:

I’m not sure what else to try, since encode shouldn’t have anything to do with rewrite, although I have a suspicion if directive orders have anything to do with it.

6. Links to relevant resources:

n/a.

2

I don’t think this has anything to do with encode, and it probably just happened to align with the fact Caddy only reloaded after making that change.

The problem is actually with your uri and reverse_proxy matchers. Path matching applies to the current path of the request. With uri, you’re modifying the current path, so the reverse_proxy that follows will no longer match because the path no longer starts with /api.

What you should do instead is use handle_path (which has built-in strip_prefix logic), and handle, to set up mutually exclusive routes.

http:// {

	handle_path /api/* {
		reverse_proxy http://httpbin.org
	}

	handle {
		root * /var/www/webapp

		@assets path *.js *.css *.png *.svg
		header @assets Cache-Control "public; max-age=604800"

		encode gzip
		php_fastcgi unix//run/php/php7.4-fpm.sock
		file_server
	}
}

Of note – don’t use header_up X-Forwarded-For, the reverse_proxy handler already sets this header correctly before sending the request upstream.

Also, you don’t need to change the gzip level, the default is tuned to be the best balance of performance to compression. The option is more there for experts to play with if they want to experiment, but most users don’t need to touch it.

Since you’re running on Ubuntu, I strongly recommend running as a systemd service, rather than running Caddy directly. If you install it with the apt repo, then you get the systemd service set up for you automatically (also read the note at the bottom of that section if you need to use a custom build of Caddy with plugins):

2 Likes
3

This topic was automatically closed after 30 days. New replies are no longer allowed.