JSON directory listing

Coding
1

The application/json output for file_server doesn’t enclose the array elements in [ ] (so, some things don’t read it as valid JSON).

It would also make the results usable by more programs if the array was named (I used the name “files” as an example).

The following is an example of what the JSON output current looks (no outer braces, no array brackets, and no name for the array).

	{	
		"name": "XXDV-BV-JV-Oaklan.fit",
		"size": 25467,
		"url": "XXDV-BV-JV-Oaklan.fit",
		"mod_time": "2022-01-14T13:36:26.3597617Z",
		"mode": 438,
		"is_dir": false,
		"is_symlink": false
	},
	{	
		"name": "DV-BV-JV-Oaklan.fit",
		"size": 25467,
		"url": "DV-BV-JV-Oaklan.fit",
		"mod_time": "2022-01-14T13:36:26.3597617Z",
		"mode": 438,
		"is_dir": false,
		"is_symlink": false
	}

The new lines below are what I’m suggesting should be added to the JSON directory listing output.

{
	"files":[
	{	
		"name": "XXDV-BV-JV-Oaklan.fit",
		"size": 25467,
		"url": "XXDV-BV-JV-Oaklan.fit",
		"mod_time": "2022-01-14T13:36:26.3597617Z",
		"mode": 438,
		"is_dir": false,
		"is_symlink": false
	},
	{	
		"name": "DV-BV-JV-Oaklan.fit",
		"size": 25467,
		"url": "DV-BV-JV-Oaklan.fit",
		"mod_time": "2022-01-14T13:36:26.3597617Z",
		"mode": 438,
		"is_dir": false,
		"is_symlink": false
	}
]
}
2

That doesn’t sound right. How can we reproduce the bug?

Relevant code:

3

Err. I wasn’t quite correct (how I was looking at it was misleading). It is delimiting the array with braces.

The thing (a Garmin GPS unit) I’m having read the JSON doesn’t like the array not having a name and the braces around everything (which, I believe, is required if there’s a name for the array).

Is there a “simple” way of having a name for the array?

=====================

curl -H "Accept: application/json" -i http://127.0.0.1:22222

Output.

[{“name”:“folder”,“size”:0,“url”:“folder/”,“mod_time”:“2022-01-14T17:28:27.7492227Z”,“mode”:2147484159,“is_dir”:true,“is_symlink”:false},{“name”:“access.log”,“size”:8799,“url”:“access.log”,“mod_time”:“2022-01-14T13:41:27.3539037Z”,“mode”:438,“is_dir”:false,“is_symlink”:false},{“name”:“caddy.cfg”,“size”:1083,“url”:“caddy.cfg”,“mod_time”:“2022-01-14T19:39:43.497098Z”,“mode”:438,“is_dir”:false,“is_symlink”:false},{“name”:“caddy_windows_amd64.exe”,“size”:34601984,“url”:“caddy_windows_amd64.exe”,“mod_time”:“2021-11-22T22:32:55.5283705Z”,“mode”:438,“is_dir”:false,“is_symlink”:false},{“name”:“dir.json”,“size”:146,“u*Preformatted text*rl”:“dir.json”,“mod_time”:“2022-01-14T19:39:44.9550518Z”,“mode”:438,“is_dir”:false,“is_symlink”:false},{“name”:“folder.dir”,“size”:202,“url”:“folder.dir”,“mod_time”:“2022-01-14T19:39:44.9239734Z”,“mode”:438,“is_dir”:false,“is_symlink”:false}]

4

There’s no simple way. You could do this with a plugin, I guess, by wrapping the response written, but that’s probably more effort than you want.

I don’t think we really have anything to suggest here. I don’t think it’s strange to return an array as a response. It’s valid JSON. Having options to control this would be awkward since it would likely only be used by you (I don’t really think many people have a similar usecase).

5

The result is missing the root node.

It seems fairly common that a missing root node is a problem. It’s not “just my problem”.

It would not be unreasonable to have an option to provide a root node. Some deserializers appear to require one. The “include root” option could allow the root name to be specified as an option (the root name could default to something like “files”).

Oh, it’s returning a value that happens to be an array (the missing root issue separate from what type the value is).

JSON is supposed to be objects with name-value pairs recursively. The missing root appears to be incorrect in a strict sense and ambiguous at best.

Given how customizable Caddy is and that one can’t always change the behavior of the data consumer, adding the root node should be an option.

6

What? Sorry, but that’s absolutely not true. Arrays are valid JSON and have a root node, being [. Consult any compliant JSON parser.

That’s also not the case; did you read Francis’ reply?:

So you can do this if you need the JSON structure to be different. You might even be able to use the existing plugin GitHub - caddyserver/replace-response: Caddy module that performs replacements in response bodies for this.

7

That chart doesn’t seem to unambiguously say that arrays can stand alone.

Yes, I read it. It’s what I was replying to.

I’m not sure how this could be enabled just for the JSON result. If it applies to any result, it’s not useful.

8

Did you even read the ~2 sentences on that page?

JSON is built on two structures:

  • A collection of name/value pairs. …
  • An ordered list of values. In most languages, this is realized as an array, vector, list, or sequence.

In JSON, they take on these forms:

So, yes, unequivocally, an array is a valid JSON body.

Please read and understand request matchers so you can apply it only to the requests where the response needs transforming: Request matchers (Caddyfile) — Caddy Documentation

9

As an extra data point, the MDN by Mozilla have a section on that with an explicit example showing plain array is valid JSON: Working with JSON - Learn web development | MDN

1 Like
10

Not just arrays, apparently. Unnamed values too.

11

I’m not getting any changes using a simple example.

replace_reponse doesn’t seem to be called with the file_server handler.

The replacement would need to be applied to just the directory response and not to files being downloaded. I’m not sure if there’s a way of doing that.

12

What have you tried, exactly? What’s your config? What are you seeing in your logs? Turn on the debug global option and try again, before checking your logs.

13

Thanks for your help.

I got it to mostly work. The regex replacement is only performed on the “application/json” accept header and it isn’t applied on the files downloaded.

Using “search_regexp”: “((?s).*)$”, the problem is that there are two wrapped results: one is the directory listing and the other is something “empty”.

Recall the goal was to take the “response” and produce ’ { “files”: response }’.

What I’m getting is ’ { “files”: response" } { “files”: }’.

{ "files" : [{"name":"folder","size":0,"url":"folder/","mod_time":"2022-01-14T17:28:27.7492227Z","mode":2147484159,"is_dir":true,"is_symlink":false},{"name":"DV-BV-JV-Oaklan.fit","size":25467,"url":"DV-BV-JV-Oaklan.fit","mod_time":"2022-01-18T22:09:13.8801593Z","mode":438,"is_dir":false,"is_symlink":false},{"name":"GWB.fit","size":1173,"url":"GWB.fit","mod_time":"2022-01-18T22:09:13.5771337Z","mode":438,"is_dir":false,"is_symlink":false},{"name":"LocationsX.fit","size":5079,"url":"LocationsX.fit","mod_time":"2022-01-15T00:05:02Z","mode":438,"is_dir":false,"is_symlink":false},{"name":"VTLG01-22-Chise.fit","size":13107,"url":"VTLG01-22-Chise.fit","mod_time":"2022-01-18T22:09:14.1246372Z","mode":438,"is_dir":false,"is_symlink":false}]
 }{ "files" :  }

Relevant JSON configuration.

					"routes": [
						{
							"match": [
									{
									  "header": {
										"accept": ["application/json"]
									  }
									}
								  ],
								"handle": [
								{
									"handler": "vars",
									"root": "C:\\Users\\dpawlyk\\Downloads\\GRouteLoaderServer"
								},
								{
									"handler": "replace_response",
									"replacements": [
										{
											"search_regexp": "((?s).*)$",
											"replace": "{ \"files\" : $1 }"
										}
									]
								},
								{
									"handler": "file_server",
									"browse": {},
									"hide": [
										"*.cfg",
										"*.log",
										"*.exe",
										"*.dir",
										"*.json",
										"*.info"
									]
								}
							]
						}
1 Like
14

I think you want to use this:

{
	"search_regexp": "^(.*)$",
	"replace": "{\"files\":$1}"
}

You can use the website regex101 to test it: https://regex101.com/r/CjHXWj/1

1 Like
15

(I messed up the example a bit.) (I was already using that website.)

“^(.*)$” => Doesn’t match (the wrapper isn’t output).

It needs the flag “(?s)” to match (it appears).

“^((?s).*)$” => produces the two wrappers (the second one empty).

“((?s).*)” => produces the same result.

16

This is my complete configuration file

{
	"apps": {
		"http": {
			"http_port": 22222,
			"servers": {
				"srv0": {
					"automatic_https": {
						"disable": true,
						"disable_redirects": true,
						"ignore_loaded_certificates": true
					},
					"listen": [
						":22222"
					],
					"logs": {
						"default_logger_name": "log0"
					},
					"routes": [
						{
							"match": [
									{
									  "header": {
										"accept": ["application/json"]
									  }
									}
								  ],
								"handle": [
								{
									"handler": "vars",
									"root": "C:\\Users\\dpawlyk\\Downloads\\GRouteLoaderServer"
								},
								{
									"handler": "replace_response",
									"replacements": [
										{
											"search_regexp": "((?s).*)",
											"replace": "{ \"files\" : $1 }"
										}
									]
								},
								{
									"handler": "file_server",
									"browse": {},
									"hide": [
										"*.cfg",
										"*.log",
										"*.exe",
										"*.dir",
										"*.json",
										"*.info"
									]
								}
							]
						},
						{
							"handle": [
								{
									"handler": "vars",
									"root": "C:\\Users\\dpawlyk\\Downloads\\GRouteLoaderServer"
								},
								{
									"handler": "file_server",
									"browse": {},
									"hide": [
										"*.cfg",
										"*.log",
										"*.exe",
										"*.dir",
										"*.json",
										"*.info"
									]
								}
							]
						}
					],
					"tls_connection_policies": [
						{}
					]
				}
			}
		}
	},
	"logging": {
		"logs": {
			"default": {
				"exclude": [
					"http.log.access.log0"
				]
			},
			"log0": {
				"include": [
					"http.log.access.log0"
				],
				"writer": {
					"filename": "access.log",
					"output": "file",
					"roll_keep": 3,
					"roll_keep_days": 3,
					"roll_size_mb": 1
				}
			}
		}
	}
}
17

Another search-and-replace pass can clean up the extra stuff.

								{
									"handler": "replace_response",
									"replacements": [
										{
											"search_regexp": "((?s).*)",
											"replace": "{ \"files\" : $1 }"
										},
										{
											"search_regexp": "{ \"files\" *: *}",
											"replace": ""
										}
									]
								},
type or paste code here
18

This topic was automatically closed after 60 days. New replies are no longer allowed.