Bug in respond (newlines)

phrazer · September 18, 2020, 8:26am

Hi, I belive this could be a bug in respond.

This is not working:

respond /robots.txt “User-agent: *\n Disallow: /” 200

You have to write it like this:

respond /robots.txt 200 {
body “User-agent: *
Disallow: /”
close
}

If it is allowed in second case, I guess it could be also allowed in first case, or is there something else in play here?

Thanks for great software.

francislavoie · September 18, 2020, 5:04pm

Works for me:

:80

respond /robots.txt "User-agent: *\n Disallow: /" 200

Adapts to:

{
	"apps": {
		"http": {
			"servers": {
				"srv0": {
					"listen": [
						":80"
					],
					"routes": [
						{
							"match": [
								{
									"path": [
										"/robots.txt"
									]
								}
							],
							"handle": [
								{
									"body": "User-agent: *\\n Disallow: /",
									"handler": "static_response",
									"status_code": 200
								}
							]
						}
					]
				}
			}
		}
	}
}

What exactly isn’t working for you?

phrazer · September 19, 2020, 10:05am

yes it writes OK in json, but response gives you literally \n, not new line if written like this. If you write it the ugly way, newline gets correctly shown in response. try with curl both ways and you will see the problem.

you get litteraly this: “User-agent: *\n Disallow: /”

francislavoie · September 19, 2020, 11:20pm

Currently it seems “intentional” that \n in Caddyfile tokens aren’t interpreted as newlines. See this note in the code:

github.com

caddyserver/caddy/blob/master/caddyconfig/caddyfile/lexer.go#L70-L74

    
      
          // in which case the token goes until the closing
          // quotes (the enclosing quotes are not included).
          // Inside quoted strings, quotes may be escaped
          // with a preceding \ character. No other chars
          // may be escaped. The rest of the line is skipped

I agree with you though, I think it would be nice to support them. I’ll need to see what @matt thinks about it.

matt · September 21, 2020, 4:19pm

The only character that isn’t literally interpreted in a Caddyfile string is the bounding " characters, so there’s no need to escape anything else. You can use literal newlines in your Caddyfile strings. This also eases complexity w.r.t Windows slashes (\).

phrazer · September 23, 2020, 8:40am

You can use literal newlines in your Caddyfile strings.
Then config becomes more messy, since you litteraly have to go to new line, it should support al least \n (newlines).

Just my though. Thanks,

francislavoie · September 23, 2020, 12:13pm

If \n was supported, then paths like C:\Users\nicolas\site would fail on Windows.

francislavoie · October 16, 2020, 8:52pm

FYI I worked on a solution for this yesterday, adding heredoc syntax to the Caddyfile:

github.com/caddyserver/caddy

caddyfile: Implement heredoc support

caddyserver:master ← francislavoie:caddyfile-heredoc

opened 07:21AM - 16 Oct 20 UTC

francislavoie

+166 -10

Adds support for heredoc syntax to the Caddyfile, inspired by HCL (HashiCorp Con…fig Language), and PHP 7.3's flexible syntax (trimming the front of every line based on the indentation of the ending marker: https://wiki.php.net/rfc/flexible_heredoc_nowdoc_syntaxes) Here's an example: ``` example.com { respond <<EOF <html> <head><title>Foo</title> <body>Foo</body> </html> EOF 200 } ``` Adapted JSON: ```json { "apps": { "http": { "servers": { "srv0": { "listen": [ ":443" ], "routes": [ { "match": [ { "host": [ "example.com" ] } ], "handle": [ { "handler": "subroute", "routes": [ { "handle": [ { "body": "\u003chtml\u003e\n \u003chead\u003e\u003ctitle\u003eFoo\u003c/title\u003e\n \u003cbody\u003eFoo\u003c/body\u003e\n\u003c/html\u003e\n", "handler": "static_response", "status_code": 200 } ] } ] } ], "terminal": true } ] } } } } } ``` Notice that the response body has the leading whitespace on each line stripped away. --- The implementation was a bit funky here, I don't like that I added a new field on the Token struct, but I couldn't think of a better way to handle it, frankly. Basically, heredocs don't include the newline that counts as the starting point as part of the token contents. This is ideal, because then you don't have an extra newline at the start of every token, so it reads more like a file. The problem is this: the logic for splitting the tokens into segments involves counting the amount of newlines inside of the token text, and checking whether it matches the line numbers on the tokens. Since we drop one newline from the heredoc, this doesn't match that expectation. So basically my "hack" was to just add a "+1" marker on the token, with a field called "Heredoc" for the purposes of counting line breaks, while keeping the line numbers intact (because they're useful for debugging etc). Can you think of a better way? I also went ahead and commented the shit out of the lexer, because I think it deserves it, since it's getting pretty involved. The heredoc stuff is admittedly quite chunky, and probably non-optimally written cause I'm by no means a Go expert... but I think the logic is sound. I should add some more tests for this to `lexer_test.go` but I'm too tired right now :smile: I'll come back to it soon.

No guarantees it’ll make it in, but it should solve this for you very nicely:

:80 {
	respond /robots.txt <<EOF
	User-agent: *
	Disallow: /
	EOF 200
}

Or if you prefer, this should work too:

:80 {
	respond /robots.txt <<EOF
		User-agent: *
		Disallow: /
		EOF 200
}

system · January 14, 2021, 8:52pm

This topic was automatically closed 90 days after the last reply. New replies are no longer allowed.