Proxy Upstream Placeholders

in regards to this: Proxy upstream depending on hostname · Issue #990 · caddyserver/caddy · GitHub

I put a comment on there asking whether it works in the Caddyfile, cause the Caddyfile does additional checks on that before it translates to the json (based on the code that I saw) and it doesn’t allow you to put placeholders in an upstream on a caddy file.

you’re going to hit this logic:

// TODO: the logic in this function is kind of sensitive, we need
// to write tests before making any more changes to it
upstreamDialAddress := func(upstreamAddr string) (string, error) {
	var network, scheme, host, port string

	if strings.Contains(upstreamAddr, "://") {
		toURL, err := url.Parse(upstreamAddr)
		if err != nil {
			return "", d.Errf("parsing upstream URL: %v", err)
		}

I’m not sure that should be closed unless your intent was that it only worked in the json file.

I’m not seeing anything in your snippet that would prevent it from working.

Seems to function just fine for me:

~/Projects/test
➜ cat Caddyfile
http://:8080
reverse_proxy {host}:8081

~/Projects/test
➜ caddy2 adapt --pretty
2020/05/11 04:06:06.113	INFO	using adjacent Caddyfile
{
	"apps": {
		"http": {
			"servers": {
				"srv0": {
					"listen": [
						":8080"
					],
					"routes": [
						{
							"handle": [
								{
									"handler": "reverse_proxy",
									"upstreams": [
										{
											"dial": "{http.request.host}:8081"
										}
									]
								}
							]
						}
					]
				}
			}
		}
	}
}

Did you test anything before asking? Were there any placeholders you had problems with?

1 Like

Yes, sorry for not clarifying, I could not get it to work at all with the placeholders I was trying, I did stick scheme in the front and did get that to work, but I can’t seem to get a placeholder to work in the middle.

in my circumstance it was a regex placeholder in the middle, but I’m not sure it matters what it is if it’s not at the beginning, you get errors like this: invalid character “{” in host name"}. Like it’s not expanding the placeholder. I am do some additional testing…

Would you mind filling out a help template? One should have been prefilled when you first started typing out a Help category post. With that info, others might be able to help out.

yes, I will do that

1 Like

As an fyi, if you start typing in a message and realize you need to pick a topic, which is what I did, you don’t get the template. While I’m filling that out, here’s an example that doesn’t work. (having the placeholder not at the beginning)

handle * {   
    reverse_proxy {
        to "http://{method}.google.com"
    }
}

Aye, I’m seeing that too.

~/Projects/test
➜ cat Caddyfile
http://:8080
reverse_proxy http://{host}:8081

~/Projects/test
➜ caddy2 adapt --pretty
2020/05/11 04:31:53.894	INFO	using adjacent Caddyfile
adapt: parsing caddyfile tokens for 'reverse_proxy': Caddyfile:2 - Error during parsing: parsing upstream URL: parse "http://{http.request.host}:8081": invalid character "{" in host name

(FYI - please encase your code, logs, configs, etc in preformatted text blocks - triple backticks (```) or single backticks (`) - makes it much easier for people to read as Markdown won’t mess up indentations etc!)

yeah, I’d reopen the feature to say it’s not complete given at the top of it, it mentions the exact example you just did, but it got locked, and it probably should be a bug at this point?

That specific issue I locked ultimately added it as JSON support (locked it due to it being old), but the Caddyfile parsing is failing because it doesn’t make the same assumptions. You can file a new issue as a bug report :+1:

Oof this is a hard problem. I’m looking at the code, and the way the Caddyfile parser works, it would be very tricky to generally integrate placeholders here because currently we parse the address to handle special cases in the scheme (like srv+ to support SRV DNS lookups).

In your case, you specify http://. It seems like you don’t hit the code path that triggers the parsing error if you skip specifying the scheme entirely. If you need https://, then instead set a transport block like this as a reverse_proxy subdirective:

transport http {
    tls
}
1 Like

@francislavoie, thanks for the suggestion, I’ll give that a try in the morning and log the bug repot.

Hmm, yeah. I think this’ll be a “wontfix” situation. You can do what you need as long as you don’t specify the scheme. If you don’t specify it, it defaults to http://, and if you need https then just specify a transport like above.

There is one situation that is broken though, that’s specifically when pairing placeholders with SRV. We might implement a solution for that if someone asks about it, but it’s unlikely we’ll do so unless someone shows they have a specific use-case for it. I just don’t see that being a common enough case to warrant doing anything to support that right now.

We could improve the error message though, i.e. if the address has a scheme + contains { somewhere, then we could print a different error message like don't use a scheme if you need a placeholder.

Edit: Opened a PR to improve the error message:

1 Like

Works like a charm! thanks for your help. I’ve been fighting switching from NGINX for a year or so… This is working very well for me so far.

2 Likes

@francislavoie FYI, it looks like you have to throw a port on the end of that :80 or probably explicitly set the transport even if it’s http

{"level":"error","ts":1589213029.92645,"logger":"http.log.error","msg":"making dial info: upstream cvs-{http.regexp.ns_env.ns}-{http.regexp.ns_env.env}.infocus-cloud-client:: invalid dial address cvs-curr-int.infocus-cloud-client:: invalid start port: strconv.ParseUint: parsing \"\": invalid syntax","request":{"method":"GET","uri":"/cloud/test","proto":"HTTP/1.1","remote_addr":"172.27.0.1:43346","host":"dev-curr.int.infocus.app","headers":{"User-Agent":["curl/7.64.1"],"Accept":["*/*"]}},"duration":0.0001145}
1 Like

yeah, just the port works, which makes sense, just making sure someone else who runs into this adds the port :slight_smile:

1 Like

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.