Trying to reverse proxy on main domain but not on path

[wooshie (Woon Wu Chyi)]

1. Caddy version (caddy version):

v2.2.1

2. How I run Caddy:

Installed via apt based on provided tutorial
using systemctl to have my caddy as a daemon, with config in my Caddyfile

a. System environment:

Ubuntu 20.04.1 LTS

d. My complete Caddyfile or JSON config:

www.foodfix.my {
        redir https://foodfix.my{uri}
}

foodfix.my {
        tls {
                dns cloudflare [redacted]
        }
        @excl {
                not path /survey
        }
        reverse_proxy @excl https://ig.foodfix.my
        root /survey /var/www/html/survey
        encode zstd gzip
        file_server
}

3. The problem I’m having:

I’m not launching my new business just yet, only a small page for a small market research that we are conducting. The page will be located at foodfix.my/survey.

For now, I haven’t had the time to create a homepage, so I’m looking to reverse proxy my main domain to our instagram page, which i’ve created a page rule via Cloudflare.

4. Error messages and/or full log output:

On my web browser:

"This page isn't working at the moment
foodfix.my redirected you too many times."

5. What I already tried:

Tried to use a not directive (?) within a named matcher for the reverse proxy directive but it shows the error above when trying to access both foodfix.my and foodfix.my/survey.

[francislavoie]

Hmm, I don’t think you should reverse_proxy to there, since it just bounces you to instagram. You should probably just use a redir to instagram for now, from Caddy itself.

@excl not path /survey*
redir @excl https://www.instagram.com/foodfixmy/

And then you probably want to use handle_path to serve the static site:

handle_path /survey* {
	root * /var/www/html/survey
	encode gzip
	file_server
}

The handle_path directive takes care of stripping the prefix from the path, which matters because Caddy takes the value of root and appends the path to it when looking for files on disk. So if you didn’t strip it, you’d be looking for files in /var/www/html/survey/survey.

Note the * on the path matchers I used, this is important because path matching is exact in Caddy v2, so /survey would only match exactly /survey and nothing below that.

Also, you put your Cloudflare API key in your post so I think you’ll need to regenerate that… it should stay secret.

[wooshie (Woon Wu Chyi)]

My goodness! Thanks for the heads up, i’ve regenerated my cloudflare API token and redacted the previous one.

Thanks for your pointers! Only have one page at /survey so I thought that wouldn’t be an issue, but i’ve added it in anyway. Here’s my modified my Caddyfile:

www.foodfix.my {
        redir https://foodfix.my{uri}
}

foodfix.my {
        tls {
                dns cloudflare [TOKEN]
        }

        @excl not path /survey*

        redir @excl https://instagram.com/foodfixmy/

        handle_path /survey* {
                root * /var/www/html/survey
                encode gzip
                file_server
        }
}

However, I’m still getting the redirect issue on my browsers. Can you perhaps try it on your side?

[francislavoie]

https://foodfix.my/ doesn’t seem to be reachable for me.

Btw this might be a better approach for the config:

foodfix.my {
	tls {
		dns cloudflare [TOKEN]
	}

	handle_path /survey* {
		root * /var/www/html/survey
		encode gzip
		file_server
	}

	# Fallback for anything else
	handle {
		redir https://instagram.com/foodfixmy/
	}
}

Avoids needing to use that not path matcher. handle and handle_path blocks are mutually exclusive from eachother, and they’re sorted by their path matchers, so this should do the trick.

[wooshie (Woon Wu Chyi)]

I’ve just tried that block of code in my Caddyfile - the redirect for my domain now works, but even the /survey path is redirecting to instagram now. In fact, any path that I type at the end of the URL also seem to be redirecting to the instagram page.

[matt (Matt Holt)]

Is that true when using curl too?

[wooshie (Woon Wu Chyi)]

on curl -L foodfix.my/survey yes, it shows the Instagram page.

[francislavoie]

What’s your full Caddyfile at this point, exactly?

Can you caddy adapt --pretty your config and post what you get? (Don’t forget to omit the cloudflare secret)

When I tried adapting it locally myself, it looked fine, i.e. the order of the handlers was correct.

Also, try with curl -v which will show the headers. When I try it, I see:

$ curl -v https://foodfix.my
...
< HTTP/2 302 
< location: https://instagram.com/foodfixmy
< server: Caddy
< content-length: 0
< date: Thu, 29 Oct 2020 14:50:59 GMT

[wooshie (Woon Wu Chyi)]

Here’s my full Caddyfile

www.foodfix.my {
        redir https://foodfix.my{uri}
}

foodfix.my {
        tls {
                dns cloudflare [CLOUDFLARE API TOKEN]
        }

        handle_path /survey* {
                root * /var/www/html/survey
                encode gzip
                file_server
        }

        #fallback 
        handle {
                redir https://instagram.com/foodfixmy
        }
}

and after caddy adapt --pretty

{
        "apps": {
                "http": {
                        "servers": {
                                "srv0": {
                                        "listen": [
                                                ":443"
                                        ],
                                        "routes": [
                                                {
                                                        "match": [
                                                                {
                                                                        "host": [
                                                                                "www.foodfix.my"
                                                                        ]
                                                                }
                                                        ],
                                                        "handle": [
                                                                {
                                                                        "handler": "subroute",
                                                                        "routes": [
                                                                                {
                                                                                        "handle": [
                                                                                                {
                                                                                                        "handler": "static_response",
                                                                                                        "headers": {
                                                                                                                "Location": [
                                                                                                                       "https://foodfix.my{http.request.uri}"
                                                                                                                ]
                                                                                                        },
                                                                                                        "status_code": 302
                                                                                                }
                                                                                        ]
                                                                                }
                                                                        ]
                                                                }
                                                        ],
                                                        "terminal": true
                                                },
                                                {
                                                        "match": [
                                                                {
                                                                        "host": [
                                                                                "foodfix.my"
                                                                        ]
                                                                }
                                                        ],
                                                        "handle": [
                                                                {
                                                                        "handler": "subroute",
                                                                        "routes": [
                                                                                {
                                                                                        "group": "group3",
                                                                                        "handle": [
                                                                                                {
                                                                                                        "handler": "subroute",
                                                                                                        "routes": [
                                                                                                                {
                                                                                                                       "handle": [

{

        "handler": "rewrite",

        "strip_path_prefix": "/survey"

}
                                                                                                                       ]
                                                                                                                },
                                                                                                                {
                                                                                                                       "handle": [

{

        "handler": "vars",

        "root": "/var/www/html/survey"

},

{

        "encodings": {

                "gzip": {}

        },

        "handler": "encode"

},

{

        "handler": "file_server",

        "hide": [

                "Caddyfile"

        ]

}
                                                                                                                       ]
                                                                                                                }
                                                                                                        ]
                                                                                                }
                                                                                        ],
                                                                                        "match": [
                                                                                                {
                                                                                                        "path": [
                                                                                                                "/survey*"
                                                                                                        ]
                                                                                                }
                                                                                        ]
                                                                                },
                                                                                {
                                                                                        "group": "group3",
                                                                                        "handle": [
                                                                                                {
                                                                                                        "handler": "subroute",
                                                                                                        "routes": [
                                                                                                                {
                                                                                                                       "handle": [

{

        "handler": "static_response",

        "headers": {

                "Location": [

                        "https://instagram.com/foodfixmy"

                ]

        },

        "status_code": 302

}
                                                                                                                       ]
                                                                                                                }
                                                                                                        ]
                                                                                                }
                                                                                        ]
                                                                                }
                                                                        ]
                                                                }
                                                        ],
                                                        "terminal": true
                                                }
                                        ]
                                }
                        }
                },
                "tls": {
                        "automation": {
                                "policies": [
                                        {
                                                "subjects": [
                                                        "foodfix.my"
                                                ],
                                                "issuer": {
                                                        "challenges": {
                                                                "dns": {
                                                                        "provider": {
                                                                                "api_token": "CLOUDFLARE API TOKEN",
                                                                                "name": "cloudflare"
                                                                        }
                                                                }
                                                        },
                                                        "module": "acme"
                                                }
                                        }
                                ]
                        }
                }
        }
}

[wooshie (Woon Wu Chyi)]

Oddly enough when i try to curl -v https://foodfix.my/survey it shows there’s somehow a 308 permanent redirect at that path.

curl -v https://foodfix.my/survey
...
* Connection state changed (MAX_CONCURRENT_STREAMS updated)!
< HTTP/2 308
< content-type: text/html; charset=utf-8
< location: /
< server: Caddy
< content-length: 37
< date: Thu, 29 Oct 2020 19:00:58 GMT
<
<a href="/">Permanent Redirect</a>.

[francislavoie]

Please add this to the top of your config, then look at the logs:

{
	debug
}

This’ll add some additional information about the requests.

Since you’re running as a service, you can run journalctl --no-pager -u caddy | less to find them (hit Shift+G to jump to the bottom)

[francislavoie]

AHA! I figured it out.

The 308 redirect is actually triggered by file_server. The reason is that /survey is actually a directory on disk, so it decides to trigger a redirect to canonicalize the path to /survey/.

This usually works just fine, but since we’re inside of handle_path, the path was stripped so it actually redirects to / instead of /survey/.

So the fix is to add this to your config (you can put it just before the handle_path line, but where you put it doesn’t matter, because the Caddyfile adapter will sort directives):

redir /survey /survey/

/cc ty @Mohammed90 for being my rubber duck

Also note that 308 redirects are cached by browsers, so you’ll probably need to clear your browser cache for your site for the change to take effect.

[wooshie (Woon Wu Chyi)]

Thanks a lot for your help! I’m still trying to understand your explanation but it definitely worked!

[francislavoie]

Basically, it’s ambiguous to a browser whether you’re looking at a directory or a file, if there’s no trailing / in the URL path. The way browsers calculate relative URLs (like in your <a> tags) is to append the link to the current path, if there’s a trailing slash, but if not then it’ll remove the last path segment to load a file that’s a sibling of the current one.

For example say you’re on /foo/ and you have <a href="bar"> on that page. That’ll link to /foo/bar.

If you’re on /foo and you have <a href="bar">, then the browser will link to /bar.

So Caddy’s file server, since it knows that /survey is actually a directory on disk, will want to redirect /survey to /survey/ typically, so that browsers don’t link to the wrong place. In other words we want to “canonicalize” the URL. Canonicalizing can also be thought of as disambiguate between duplicate page content by triggering redirects so that you land on the “right one”. Another example of that is redirecting www. to a non-www domain, or vice versa, so that people always land on the one you actually want them to use.

So the trouble is since we use handle_path, for a URL of /survey, Caddy will remove /survey from the path before file_server sees it, so it just sees an empty string as the path. And since it will canonicalize the URL, it will then trigger a 308 redirect to path + / (which is just / because the path is empty). And that’s what you saw in your curl -v output.

Make sense?