Doesn't work when reverse proxy Windows Admin Center

matt · November 5, 2019, 3:30am

I’ve been adding debug logs to the reverse proxy since this is a prime time to do that!

~~And actually, the upstream appears to be returning a 200 OK. Let me investigate further.~~ Scratch that, I didn’t follow the log line correctly… and it is returning a 400 from upstream.

Edit: Okay, so, I don’t know why this is happening. We’ll need WAC to tell us why the requests are bad.

What config do you use on nginx or whatever other proxy you used before that works?

matt · November 5, 2019, 3:51am

@hez2010 One more idea. By accessing directly, the only difference I can see between requests that succeed on the direct access and the ones that fail with the proxy are the absence of the Cookie request headers.

In my debug logs, the upstream replies with headers like "Set-Cookie": ["XSRF-TOKEN=66f5a97c-8ac9-47cd-8115-b7d3c49ec64e; path=/; secure"]} but I do not see these cookies being set, perhaps because the browser sees that the client is on an HTTP page.

This is just a hunch, and doesn’t explain why some requests fail and others succeed.

Still, you might try this on an HTTPS page and see if it works.

(Edit: I’m now almost certain that is at least one problem, if not the problem. If you go to the Application tab of the web inspector, then go to Storage → Cookies, you will see that no XSRF-TOKEN cookie is set on the proxied HTTP page – probably because it is a secure context cookie only – but it is set on the direct access HTTPS page.)

Dang. I hoped I had it. I cheated a bit and forced the cookie to set on HTTP pages by adding this to the reverse proxy config:

"headers": {
	"response": {
		"replace": {
			"Set-Cookie": [
				{
					"search": "; secure",
					"replace": ""
				}
			]
		}
	}
}

It successfully manipulated the response header so that the cookie was set and is being included on the proxied requests, BUT… still getting 400s back.

What is your working proxy config for whatever you’re using now?

hez2010 · November 5, 2019, 4:09am

nginx:

        upstream wac {
                server test.hez2010.com:1080;
                keepalive 64;
        }

        server {
                listen 80;
                location / {
                        proxy_pass https://wac;
                        proxy_http_version 1.1;
                        proxy_set_header Connection "";
                }
        }

Maybe need to setup https for nginx.

matt · November 5, 2019, 4:19am

Thank you! That is helpful.

Interesting… your nginx config disables keep-alive (by setting Connection header to empty string, i.e. removing it entirely).

Doing that in Caddy yields all 400s. (in the "keep_alive" object of your config, add "enabled": false)

I’m stumped.

matt · November 5, 2019, 4:40am

@hez2010 What’s your full nginx config? Do you have anything else that might be relevant?

hez2010 · November 5, 2019, 4:53am

I used to config my nginx by referencing this: nginx reverse proxy with Windows authentication that uses NTLM - Stack Overflow

WAC is using NTLM authentication method, I suspect this problem may relate to it.
However I don’t know why it works

matt · November 5, 2019, 4:17pm

I’m beginning to suspect you are right.

NTLM appears to violate HTTP’s convention of “stateless” by using HTTP requests to authenticate underlying connections. So, once a connection is authenticated, all requests using that connection have to be aware that the underlying connection is authenticated and set headers accordingly: authentication takes multiple round-trips as well, so Go’s connection pooling doesn’t guarantee that requests to finish authentication happen on the same connection. (Keep-Alive is also required.)

NGINX does not guarantee this either, but it probably does connection pooling very differently in a way that happens to work for you. Based on the comments and what I am reading about nginx’s additional ntlm directive, it sounds like your current config is just working by sheer luck!

Docs for proxying NTLM are sparse, but the best help I have found is from nginx’s docs:

Allows proxying requests with NTLM Authentication. The upstream connection is bound to the client connection once the client sends a request with the “Authorization” header field value starting with “ Negotiate ” or “ NTLM ”. Further client requests will be proxied through the same upstream connection, keeping the authentication context.

In order for NTLM authentication to work, it is necessary to enable keepalive connections to upstream servers. The proxy_http_version directive should be set to “ 1.1 ” and the “Connection” header field should be cleared.

So, we basically need to implement this behavior into Caddy’s proxy, along with a config property to enable it.

matt · November 5, 2019, 7:51pm

@hez2010 I spent just an hour on it this morning because it was REALLY bugging me for some reason.

And…

IT WORKS!!

My implementation is super spikey, i.e. not at all commit-worthy. But now I know how to do it and with some time I can clean things up and push it.

It is effectively what NGINX’s commercial ntlm directive does: Module ngx_http_upstream_module

But ours is free.

I do not know what implications this feature has when enabled. Under the hood, we use a separate http.Transport for every request.RemoteAddr (client connection address) that we see, so it definitely isn’t as fast as your streamlined HTTP/2 pipelines. But that’s NTLM for you.

matt · November 5, 2019, 11:38pm

@hez2010

I have cleaned up my implementation and pushed it (300th commit to v2, too): reverse_proxy: Add support for NTLM · caddyserver/caddy@8e51528 · GitHub

You can build from the v2 branch to use it right away, or wait for beta 10.

Scroll down from here and you’ll see how to use the http_ntlm transport module: Home · caddyserver/caddy Wiki · GitHub

Here’s my config, for example:

{
	"handler": "reverse_proxy",
	"transport": {
		"protocol": "http_ntlm",
		"tls": {
			"insecure_skip_verify": true
		}
	},
	"upstreams": [
		{"dial": "wac:1080"}
	]
}

alexandzors · November 6, 2019, 12:30am

Thanks Matt! I’ve been wanting this for so long and I’m glad to finally see that it’s possible to reverse proxy WAC with NTLM.

hez2010 · November 6, 2019, 12:46am

Thanks a lot!!!

matt · November 6, 2019, 12:52am

Let me know if it works for you too

hez2010 · November 6, 2019, 1:57am

I just tried the new http_ntlm module but still stuck at “Bootstrapping the application” again.

My caddyfile:

{
    "apps": {
        "http": {
            "servers": {
                "srv0": {
                    "listen": [
                        ":80"
                    ],
                    "routes": [{
                        "match": [{
                            "host": [
                                "test.hez2010.com"
                            ]
                        }],
                        "handle": [{
                            "handler": "subroute",
                            "routes": [{
                                "handle": [{
                                    "handler": "reverse_proxy",
                                    "transport": {
                                        "protocol": "http_ntlm",
                                        "tls": {
                                            "insecure_skip_verify": true
                                        }
                                    },
                                    "upstreams": [{
                                        "dial": "localhost:1080"
                                    }]
                                }]
                            }]
                        }]
                    }]
                }
            }
        }
    }
}

This time I saw a lot of 401s in caddy’s log, and lots of 400s in browser’s network request records.
You can try accessing http://test.hez2010.com.

matt · November 6, 2019, 7:16am

D’oh. As part of the cleanup I accidentally read/transferred something wrong:

Anyway, I’ve pushed the fix and tested that it works with your backend. (Although, I’m getting 403 / access denied errors from WAC, different from before – did you change something in your backend WAC configuration? It’s OK if you did but I wasn’t able to get the exact same results as before)

Can you pull the latest on the v2 branch and try again? Thank you!

hez2010 · November 6, 2019, 7:58am

This time it works, but I got access denied too.
In caddy’s log:

2019/11/06 07:49:15.354 e[31mERRORe[0m  http.log.access handled request {"request": {"method": "GET", "uri": "/api/settings/admin", "proto": "HTTP/1.1", "remote_addr": "120.236.174.144:30142", "host": "test.hez2010.com", "headers": {"Accept": ["application/json, text/plain, */*"], "User-Agent": ["Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/80.0.3952.0 Safari/537.36 Edg/80.0.320.3"], "Content-Type": ["application/json; charset=utf-8"], "Accept-Language": ["en-US,en;q=0.9"], "Connection": ["keep-alive"], "Referer": ["http://test.hez2010.com/errors/forbidden"], "Accept-Encoding": ["gzip, deflate"], "Authorization": ["Negotiate TlRMTVNTUAADAAAAGAAYAIQAAABGAUYBnAAAAAAAAABYAAAADgAOAFgAAAAeAB4AZgAAABAAEADiAQAAFYKI4goAukcAAAAPJ1IL8gJrZEnNhEQL3sj3q2gAZQB6ADIAMAAxADAARABFAFMASwBUAE8AUAAtAEMARAAyAEMANQAyAEYAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAEMLsVky5/A4KfcZgtngrywEBAAAAAAAAGUejr3aU1QF7HQSGcCw/xwAAAAACAB4AaABlAHoAMgAwADEAMAAtAHQAZQBzAHQALQB2AG0AAQAeAGgAZQB6ADIAMAAxADAALQB0AGUAcwB0AC0AdgBtAAQAHgBoAGUAegAyADAAMQAwAC0AdABlAHMAdAAtAHYAbQADAB4AaABlAHoAMgAwADEAMAAtAHQAZQBzAHQALQB2AG0ABwAIABlHo692lNUBBgAEAAIAAAAIADAAMAAAAAAAAAABAAAAACAAAOAArFjBDPP4KI0rBfm6YjcorJy/d5/2RdvheUm/+K8CCgAQAAAAAAAAAAAAAAAAAAAAAAAJACoASABUAFQAUAAvAHQAZQBzAHQALgBoAGUAegAyADAAMQAwAC4AYwBvAG0AAAAAAAAAAACURheVy66r6mCKVAN5tk1f"], "Dnt": ["1"]}}, "common_log": "120.236.174.144 - - [06/Nov/2019:07:49:15 +0000] \"GET /api/settings/admin HTTP/1.1\" 403 96", "latency": 0.008572, "size": 96, "status": 403}

Access wac from caddy vs directly access:

I found the x-xsrf-token was missing, may be the reason for forbidden?

Cookies difference:

matt · November 6, 2019, 8:03am

@hez2010 Ah yes Nice find! That’s not a Caddy issue, fortunately.

It’s because you’re on an HTTP page, but the WAC backend is setting secure-only cookies. You have to do one of these things:

Use HTTPS on the front end (recommended)
Configure WAC to set non-secure cookies
Cheat

#3 is the most fun, so let’s do that:

{
	"handler": "reverse_proxy",
	"transport": {
		"protocol": "http_ntlm",
		"tls": {
			"insecure_skip_verify": true
		}
	},
	"headers": {
		"response": {
			"replace": {
				"Set-Cookie": [
					{
						"search": "; secure",
						"replace": ""
					}
				]
			}
		}
	},
	"upstreams": [
		{"dial": "wac:1080"}
	]
}

Notice that we are rewriting the Set-Cookie header so it is no longer a secure cookie.

I did this while I was developing the fix, so I know it works.

hez2010 · November 6, 2019, 8:14am

Wow it works well!
Thank you so much for your work and patience.

matt · November 6, 2019, 8:20am

Likewise, it was fun.

Have fun not paying for nginx

matt · November 19, 2019, 9:47pm

FWIW, Beta 10 has been released, which includes NTLM proxying. For free.

system · February 17, 2020, 9:47pm

This topic was automatically closed 90 days after the last reply. New replies are no longer allowed.