Preserving query string during rewrite?

Help
1

1. The problem I’m having:

I’m on week 1 of caddy so please forgive me if this is obvious.

I have a handful of PHP files that comprise a small API. Params are passed on the URL, due to some old software I can’t change.

  • $ curl -vL https://api.gingerbeardman.com/v1/posts/add?url=https://www.example.com&description=Example

But the URL params are being stripped by the time I check in the PHP, both the following are blank:

print_r($_SERVER['QUERY_STRING'], true);
print_r($_GET, true);

Thanks to MaxGhost on Reddit, I see that Caddy sets query. But as we can see below from logs they disappear.

I think I need to keep them somehow during the rewrite? I’m not doing any of that right now.

2. Error messages and/or full log output:


Nov 27 10:20:48 phoenix caddy[2443]: {"level":"debug","ts":1732702848.8315847,"logger":"http.handlers.rewrite","msg":"rewrote request","request":{"remote_ip":"144.24.224.205","remote_port":"60016","client_ip":"144.24.224.205","proto":"HTTP/2.0","method":"GET","host":"api.gingerbeardman.com","uri":"/v1/posts/add/","headers":{"User-Agent":["curl/7.61.1"],"Accept":["*/*"]},"tls":{"resumed":false,"version":772,"cipher_suite":4865,"proto":"h2","server_name":"api.gingerbeardman.com"}},"method":"GET","uri":"/v1/posts/add/index.php"}
Nov 27 10:20:48 phoenix caddy[2443]: {"level":"debug","ts":1732702848.8316858,"logger":"http.reverse_proxy.transport.fastcgi","msg":"roundtrip","request":{"remote_ip":"144.24.224.205","remote_port":"60016","client_ip":"144.24.224.205","proto":"HTTP/2.0","method":"GET","host":"api.gingerbeardman.com","uri":"/v1/posts/add/index.php","headers":{"X-Forwarded-Host":["api.gingerbeardman.com"],"User-Agent":["curl/7.61.1"],"Accept":["*/*"],"X-Forwarded-For":["144.24.224.205"],"X-Forwarded-Proto":["https"]},"tls":{"resumed":false,"version":772,"cipher_suite":4865,"proto":"h2","server_name":"api.gingerbeardman.com"}},"env":{"REMOTE_ADDR":"144.24.224.205","HTTP_HOST":"api.gingerbeardman.com","HTTP_USER_AGENT":"curl/7.61.1","REMOTE_PORT":"60016","DOCUMENT_URI":"/v1/posts/add/index.php","HTTP_X_FORWARDED_HOST":"api.gingerbeardman.com","REQUEST_URI":"/v1/posts/add/","HTTPS":"on","REMOTE_HOST":"144.24.224.205","REQUEST_METHOD":"GET","DOCUMENT_ROOT":"/var/www/api.gingerbeardman.com/htdocs","SSL_CIPHER":"TLS_AES_128_GCM_SHA256","AUTH_TYPE":"","REQUEST_SCHEME":"https","SERVER_PROTOCOL":"HTTP/2.0","HTTP_X_FORWARDED_PROTO":"https","CONTENT_LENGTH":"","CONTENT_TYPE":"","SERVER_PORT":"443","HTTP_X_FORWARDED_FOR":"144.24.224.205","PATH_INFO":"","SCRIPT_NAME":"/v1/posts/add/index.php","SSL_PROTOCOL":"TLSv1.3","HTTP_ACCEPT":"*/*","REMOTE_IDENT":"","QUERY_STRING":"","REMOTE_USER":"","SCRIPT_FILENAME":"/var/www/api.gingerbeardman.com/htdocs/v1/posts/add/index.php","GATEWAY_INTERFACE":"CGI/1.1","SERVER_NAME":"api.gingerbeardman.com","SERVER_SOFTWARE":"Caddy/v2.8.4"},"dial":"/run/php-fpm/www.sock","env":{"SERVER_NAME":"api.gingerbeardman.com","SERVER_SOFTWARE":"Caddy/v2.8.4","SCRIPT_FILENAME":"/var/www/api.gingerbeardman.com/htdocs/v1/posts/add/index.php","GATEWAY_INTERFACE":"CGI/1.1","HTTP_HOST":"api.gingerbeardman.com","HTTP_USER_AGENT":"curl/7.61.1","REMOTE_ADDR":"144.24.224.205","DOCUMENT_URI":"/v1/posts/add/index.php","HTTP_X_FORWARDED_HOST":"api.gingerbeardman.com","REMOTE_PORT":"60016","REQUEST_METHOD":"GET","DOCUMENT_ROOT":"/var/www/api.gingerbeardman.com/htdocs","REQUEST_URI":"/v1/posts/add/","HTTPS":"on","REMOTE_HOST":"144.24.224.205","REQUEST_SCHEME":"https","SERVER_PROTOCOL":"HTTP/2.0","SSL_CIPHER":"TLS_AES_128_GCM_SHA256","AUTH_TYPE":"","CONTENT_TYPE":"","SERVER_PORT":"443","HTTP_X_FORWARDED_PROTO":"https","CONTENT_LENGTH":"","SCRIPT_NAME":"/v1/posts/add/index.php","SSL_PROTOCOL":"TLSv1.3","HTTP_X_FORWARDED_FOR":"144.24.224.205","PATH_INFO":"","QUERY_STRING":"","REMOTE_USER":"","HTTP_ACCEPT":"*/*","REMOTE_IDENT":""},"request":{"remote_ip":"144.24.224.205","remote_port":"60016","client_ip":"144.24.224.205","proto":"HTTP/2.0","method":"GET","host":"api.gingerbeardman.com","uri":"/v1/posts/add/index.php","headers":{"Accept":["*/*"],"X-Forwarded-For":["144.24.224.205"],"X-Forwarded-Proto":["https"],"X-Forwarded-Host":["api.gingerbeardman.com"],"User-Agent":["curl/7.61.1"]},"tls":{"resumed":false,"version":772,"cipher_suite":4865,"proto":"h2","server_name":"api.gingerbeardman.com"}}}
Nov 27 10:20:49 phoenix caddy[2443]: {"level":"debug","ts":1732702849.244736,"logger":"http.handlers.reverse_proxy","msg":"upstream roundtrip","upstream":"unix//run/php-fpm/www.sock","duration":0.413083204,"request":{"remote_ip":"144.24.224.205","remote_port":"60016","client_ip":"144.24.224.205","proto":"HTTP/2.0","method":"GET","host":"api.gingerbeardman.com","uri":"/v1/posts/add/index.php","headers":{"User-Agent":["curl/7.61.1"],"Accept":["*/*"],"X-Forwarded-For":["144.24.224.205"],"X-Forwarded-Proto":["https"],"X-Forwarded-Host":["api.gingerbeardman.com"]},"tls":{"resumed":false,"version":772,"cipher_suite":4865,"proto":"h2","server_name":"api.gingerbeardman.com"}},"headers":{"X-Powered-By":["PHP/7.4.33"],"Content-Type":["text/xml; charset=utf-8"]},"status":200}

3. Caddy version:

2.8.4

4. How I installed and ran Caddy:

dnf install caddy

a. System environment:

Oracle Linux 8 (RHEL8)

b. Command:

systemctl start caddy

c. Service/unit/compose file:

n/a

d. My complete Caddy config:

api.gingerbeardman.com {
    tls email@example.com
    root * /var/www/api.gingerbeardman.com/htdocs
    php_fastcgi unix//run/php-fpm/www.sock
    log {
        output file /var/log/caddy/api.access.log
    }
}

5. Links to relevant resources:

2

Howdy @gingerbeardman, welcome to the Caddy community.

Are you sure the query is actually being sent to Caddy itself?

I just did a really quick test:

{
  debug
}

localhost {
  php_fastcgi unix//run/php-fpm/www.sock
}

I made a dummy file with touch index.php just so Caddy would pick it up and try to proxy it. I don’t have a www.sock so I expect the request to fail, but that’s okay, because I’m only testing the environment Caddy is formulating to pass to FastCGI.

I ran curl https://localhost?query=foo and checked the debug logs with jq .env:

{
  "SSL_PROTOCOL": "TLSv1.3",
  "SERVER_PORT": "443",
  "CONTENT_TYPE": "",
  "REMOTE_ADDR": "::1",
  "REQUEST_SCHEME": "https",
  "SERVER_NAME": "localhost",
  "DOCUMENT_URI": "index.php",
  "REQUEST_METHOD": "GET",
  "SCRIPT_NAME": "/index.php",
  "HTTP_USER_AGENT": "curl/8.10.1",
  "HTTP_X_FORWARDED_FOR": "::1",
  "GATEWAY_INTERFACE": "CGI/1.1",
  "HTTP_ACCEPT": "*/*",
  "HTTP_X_FORWARDED_HOST": "localhost",
  "REMOTE_HOST": "::1",
  "SERVER_SOFTWARE": "Caddy/v2.8.4",
  "REQUEST_URI": "/?query=foo",
  "HTTP_X_FORWARDED_PROTO": "https",
  "REMOTE_IDENT": "",
  "REMOTE_PORT": "60680",
  "HTTPS": "on",
  "QUERY_STRING": "query=foo",
  "REMOTE_USER": "",
  "HTTP_HOST": "localhost",
  "SCRIPT_FILENAME": "/Users/whitestrake/Projects/caddy/index.php",
  "SSL_CIPHER": "TLS_AES_128_GCM_SHA256",
  "AUTH_TYPE": "",
  "CONTENT_LENGTH": "",
  "PATH_INFO": "",
  "SERVER_PROTOCOL": "HTTP/2.0",
  "DOCUMENT_ROOT": "/Users/whitestrake/Projects/caddy"
}

The QUERY_STRING populated just fine, but also, the query itself was in REQUEST_URI. Compared with the roundtrip from your post, passed to jq .env:

{
  "SERVER_NAME": "api.gingerbeardman.com",
  "SERVER_SOFTWARE": "Caddy/v2.8.4",
  "SCRIPT_FILENAME": "/var/www/api.gingerbeardman.com/htdocs/v1/posts/add/index.php",
  "GATEWAY_INTERFACE": "CGI/1.1",
  "HTTP_HOST": "api.gingerbeardman.com",
  "HTTP_USER_AGENT": "curl/7.61.1",
  "REMOTE_ADDR": "144.24.224.205",
  "DOCUMENT_URI": "/v1/posts/add/index.php",
  "HTTP_X_FORWARDED_HOST": "api.gingerbeardman.com",
  "REMOTE_PORT": "60016",
  "REQUEST_METHOD": "GET",
  "DOCUMENT_ROOT": "/var/www/api.gingerbeardman.com/htdocs",
  "REQUEST_URI": "/v1/posts/add/",
  "HTTPS": "on",
  "REMOTE_HOST": "144.24.224.205",
  "REQUEST_SCHEME": "https",
  "SERVER_PROTOCOL": "HTTP/2.0",
  "SSL_CIPHER": "TLS_AES_128_GCM_SHA256",
  "AUTH_TYPE": "",
  "CONTENT_TYPE": "",
  "SERVER_PORT": "443",
  "HTTP_X_FORWARDED_PROTO": "https",
  "CONTENT_LENGTH": "",
  "SCRIPT_NAME": "/v1/posts/add/index.php",
  "SSL_PROTOCOL": "TLSv1.3",
  "HTTP_X_FORWARDED_FOR": "144.24.224.205",
  "PATH_INFO": "",
  "QUERY_STRING": "",
  "REMOTE_USER": "",
  "HTTP_ACCEPT": "*/*",
  "REMOTE_IDENT": ""
}

We see an empty QUERY_STRING but also no query in REQUEST_URI.

Looking at your first log entry, the rewrite:

{
  "level": "debug",
  "ts": 1732702848.8315847,
  "logger": "http.handlers.rewrite",
  "msg": "rewrote request",
  "request": {
    "remote_ip": "144.24.224.205",
    "remote_port": "60016",
    "client_ip": "144.24.224.205",
    "proto": "HTTP/2.0",
    "method": "GET",
    "host": "api.gingerbeardman.com",
    "uri": "/v1/posts/add/",
    "headers": {
      "User-Agent": [
        "curl/7.61.1"
      ],
      "Accept": [
        "*/*"
      ]
    },
    "tls": {
      "resumed": false,
      "version": 772,
      "cipher_suite": 4865,
      "proto": "h2",
      "server_name": "api.gingerbeardman.com"
    }
  },
  "method": "GET",
  "uri": "/v1/posts/add/index.php"
}

The original URI coming in never had a query attached. Compare with the rewrite log from my test:

{
  "request": {
    "remote_ip": "::1",
    "remote_port": "60680",
    "client_ip": "::1",
    "proto": "HTTP/2.0",
    "method": "GET",
    "host": "localhost",
    "uri": "/?query=foo",
    "headers": {
      "User-Agent": [
        "curl/8.10.1"
      ],
      "Accept": [
        "*/*"
      ]
    },
    "tls": {
      "resumed": false,
      "version": 772,
      "cipher_suite": 4865,
      "proto": "h2",
      "server_name": "localhost"
    }
  },
  "method": "GET",
  "uri": "index.php?query=foo"
}

Not only did the request coming in have a query string in the URI, but the rewrite preserved it just fine.

All of this to say that it seems far more likely to me your client is misbehaving than Caddy is, in this particular instance.

1 Like
3

Thanks for the hint!

I figured out what is happening, at least.

  1. /add = query is stripped
    • $ curl -vL https://api.gingerbeardman.com/v1/posts/add?url=https://www.example.com&description=Example
  2. /add/ = query survives
    • $ curl -vL https://api.gingerbeardman.com/v1/posts/add/?url=https://www.example.com&description=Example
  3. /add/index.php = query survives
    • $ curl -vL https://api.gingerbeardman.com/v1/posts/add/index.php?url=https://www.example.com&description=Example

What’s going on?

How is using just /add different?

And is there a workaround?

4

Huh. Yes, you are right. There is different behaviour at play.

The reason appears to be here:

	# Add trailing slash for directory requests
	@canonicalPath {
		file {path}/index.php
		not path */
	}
	redir @canonicalPath {http.request.orig_uri.path}/ 308
  • The first section deals with canonicalizing the request path. The goal is to ensure that requests that target a directory on disk actually have the trailing slash / added to the request path, so that only a single URL is valid for requests to that directory.
    This is performed by using a request matcher that matches only requests that don’t end in a slash, and which map to a directory on disk which contains an index.php file, and if it matches, performs a HTTP 308 redirect with the trailing slash appended. So for example, it would redirect a request with path /foo to /foo/ (appending a /, to canonicalize the path to the directory), if /foo/index.php exists on disk.

—php_fastcgi (Caddyfile directive) — Caddy Documentation

But that redirect is not query-preserving. See here:

~/Projects/caddy
➜ curl -i 'https://localhost/add?query=foo'
HTTP/2 308
alt-svc: h3=":443"; ma=2592000
location: /add/
server: Caddy
content-length: 0
date: Thu, 28 Nov 2024 02:29:57 GMT

I am not sure why this is the case - possibly there’s a good reason for it, possibly it’s an oversight.

Instead of php_fastcgi you will have to copy and use the entire expanded form so that you can edit the canonical redirect. For your use case it should instead read:

redir @canonicalPath {http.request.orig_uri.path}/?{query} 308

I tested it like so and had a good result from /add?query=foo:

{
  debug
}

localhost {
  route {
    # Add trailing slash for directory requests
    @canonicalPath {
      file {path}/index.php
      not path */
    }
    redir @canonicalPath {http.request.orig_uri.path}/?{query} 308

    # If the requested file does not exist, try index files
    @indexFiles file {
      try_files {path} {path}/index.php index.php
      split_path .php
    }
    rewrite @indexFiles {file_match.relative}

    # Proxy PHP files to the FastCGI responder
    @phpFiles path *.php
    reverse_proxy @phpFiles unix//run/php-fpm/www.sock {
      transport fastcgi {
      	split .php
      }
    }
  }
}

~/Projects/caddy
➜ curl -i 'https://localhost/add?query=foo'
HTTP/2 308
alt-svc: h3=":443"; ma=2592000
location: /add/?query=foo
server: Caddy
content-length: 0
date: Thu, 28 Nov 2024 02:27:41 GMT

{
  "AUTH_TYPE": "",
  "DOCUMENT_URI": "add/index.php",
  "SCRIPT_NAME": "/add/index.php",
  "HTTPS": "on",
  "CONTENT_LENGTH": "",
  "REMOTE_PORT": "62283",
  "SCRIPT_FILENAME": "/Users/whitestrake/Projects/caddy/add/index.php",
  "HTTP_X_FORWARDED_HOST": "localhost",
  "HTTP_HOST": "localhost",
  "SERVER_SOFTWARE": "Caddy/v2.8.4",
  "DOCUMENT_ROOT": "/Users/whitestrake/Projects/caddy",
  "REQUEST_URI": "/add/?query=foo",
  "HTTP_X_FORWARDED_FOR": "::1",
  "QUERY_STRING": "query=foo",
  "REMOTE_USER": "",
  "REQUEST_SCHEME": "https",
  "SERVER_NAME": "localhost",
  "REMOTE_ADDR": "::1",
  "HTTP_X_FORWARDED_PROTO": "https",
  "HTTP_USER_AGENT": "curl/8.10.1",
  "REMOTE_IDENT": "",
  "REQUEST_METHOD": "GET",
  "SERVER_PROTOCOL": "HTTP/2.0",
  "SSL_PROTOCOL": "TLSv1.3",
  "SSL_CIPHER": "TLS_AES_128_GCM_SHA256",
  "CONTENT_TYPE": "",
  "PATH_INFO": "",
  "REMOTE_HOST": "::1",
  "SERVER_PORT": "443",
  "GATEWAY_INTERFACE": "CGI/1.1",
  "HTTP_ACCEPT": "*/*"
}
1 Like
5

Hey @gingerbeardman, good news - it seems like this was an oversight that will likely be fixed in 2.9.0, thanks to the quick work of @francislavoie.

You have the option of the workaround I outlined above (copying and editing the php_fastcgi expanded form) or building Caddy with the included fix. (Simplest way to do that would probably be with xcaddy build prefixed-query (for now) or xcaddy build master once it’s been merged in.)

1 Like
6

Awesome! :sunglasses:

Thanks both.

I’ll wait for the update, just for kicks.