Doesn't work when reverse proxy Windows Admin Center

Dangit.

I wonder if that actually disabled http2, though, since we are manually enabling it… and I actually don’t see any search results for that config in the http2 package.

Can you change some code and try again? Let’s start by removing these lines:

(Sorry for the hassle, it will be worth it in the end!)

If you need me to send you a binary with these modifications, let me know the OS and architecture.

@hez2010 Any chance you can give that a try using the latest on the v2 branch?

Tested but still not working:

1.5729160506655843e+09  e[31mERRORe[0m  http.log.error  net/http: HTTP/1.x transport connection broken: malformed HTTP response "\x00\x00\x12\x04\x00\x00\x00\x00\x00\x00\x03\x00\x00\x00d\x00\x10\x00\x00\x00\x02\x00\x04\x00\x10\x00\x00\x00\x00\x04\b\x00\x00\x00\x00\x00\x00\x0f\x00\x01" {"request": {"method": "GET", "uri": "/", "proto": "HTTP/1.1", "remote_addr": "45.56.155.244:30715", "host": "test.hez2010.com", "headers": {"Connection": ["keep-alive"], "Upgrade-Insecure-Requests": ["1"], "User-Agent": ["Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/79.0.3945.16 Safari/537.36 Edg/79.0.309.11"], "Accept": ["text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.9"], "Accept-Encoding": ["gzip, deflate"], "Cache-Control": ["max-age=0"], "Dnt": ["1"], "Accept-Language": ["en-US,en;q=0.9"]}}, "status": 502, "err_id": "vum6uc06h", "err_trace": "reverseproxy.(*Handler).ServeHTTP (reverseproxy.go:304)"}
1 Like

Ah okay, thanks for trying. I think I know what is going on now, I just need some time to work out the fix. That was very helpful, thank you again! Let me know if you discover anything else in the meantime.

This time the 502 error occurred before the authentication process. (which means that you will get a 502 error immediately while accessing the website)
You can try accessing http://test.hez2010.com
And WAC was directly hosted on https://test.hez2010.com:1080

1 Like

Also you can use this caddyfile to host the reverse proxy on your own machine. I think it may be more convenient for you to workout the fix.

{
        "apps": {
                "http": {
                        "http_port": 1080,
                        "servers": {
                                "srv0": {
                                        "listen": [
                                                ":1080"
                                        ],
                                        "routes": [
                                                {
                                                        "match": [
                                                                {
                                                                              "host": [
                                                                                      "localhost"
                                                                              ]
                                                                }
                                                        ],
                                                        "handle": [
                                                                {
                                                                              "handler": "subroute",
                                                                              "routes": [
                                                                                      {
                                                                                              "handle": [
                                                                                                      {
                                                                                                              "handler": "reverse_proxy",
                                                                                                              "transport": {
                                                                                                                      "keep_alive": {
                                                                                                                              "idle_timeout": "64s"
                                                                                                                      },
                                                                                                                      "protocol": "http",
                                                                                                                      "tls": {
                                                                                                                              "insecure_skip_verify": true
                                                                                                                      }
                                                                                                              },
                                                                                                              "upstreams": [
                                                                                                                      {
                                                                                                                              "dial": "test.hez2010.com:1080"
                                                                                                                      }
                                                                                                              ]
                                                                                                      }
                                                                                              ]
                                                                                      }
                                                                              ]
                                                                }
                                                        ]
                                                }
                                        ]
                                }
                        }
                }
        }
}

To access the reverse proxy, visit http://localhost:1080

1 Like

Thank you! Experimenting now. Please leave the backend up for a while if you could :slight_smile:

@hez2010 Okay, got it working.

You can restore the http2.ConfigureTransport() segment I told you to remove, and instead comment out line 201 in the same file:

cfg.NextProtos = []string{"h2", "http/1.1"} // TODO: ensure that this actually enables HTTP/2

Or remove it entirely. Notice the fun TODO :roll_eyes:

Simply removing that line should make it work. Can you confirm either way?

If so I’ll push the fix.

Still not working after I typed my credentials:

1.5729191763240354e+09  e[31mERRORe[0m  http.log.error  stream error: stream ID 9; HTTP_1_1_REQUIRED    {"request": {"method": "GET", "uri": "/", "proto": "HTTP/1.1", "remote_addr": "[::1]:60458", "host": "localhost:1080", "headers": {"Dnt": ["1"], "User-Agent": ["Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/79.0.3945.16 Safari/537.36 Edg/79.0.309.11"], "Sec-Fetch-Site": ["none"], "Accept-Encoding": ["gzip, deflate, br"], "Accept-Language": ["en-US,en;q=0.9"], "Sec-Fetch-Mode": ["navigate"], "Connection": ["keep-alive"], "Cache-Control": ["max-age=0"], "Authorization": ["Negotiate TlRMTVNTUAABAAAAl4II4gAAAAAAAAAAAAAAAAAAAAAKALpHAAAADw=="], "Upgrade-Insecure-Requests": ["1"], "Sec-Fetch-User": ["?1"], "Accept": ["text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.9"]}}, "status": 502, "err_id": "tj7janqrh", "err_trace": "reverseproxy.(*Handler).ServeHTTP (reverseproxy.go:304)"}

You can try it with below credentials:
username: hez2010
password: hez2010!hez2010

:expressionless: I swear it was totally working:

Try commenting out the http2.ConfigureTransport() lines again.

A little more info… I found this in the HTTP/2 spec:

Alternatively, a server might use an error (Section 5.4) of type HTTP_1_1_REQUIRED to request the client use a protocol that supports renegotiation.

It seems strange to me that a server would advertise that it supports HTTP/2 if it just wants you to use HTTP/1.1.

Er… anyway… I bet if you remove both that NextProtos line I suggested above and the ConfigureTransport line, it will work for you too. Please let me know!

After commenting out the http2.ConfigureTransport() and NextProtos lines, I can open the Windows Admin Center.
However it stuck at the process “Bootstrapping the application” and there were many 400 errors in network requests. (I think it may relate to authentication)

The problem didn’t exist while accessing the WAC directly.

@hez2010 For this one, you’ll need to look at the logs in WAC and see why it thinks the requests are bad, I think, because unfortunately the responses don’t contain any information…

I’ve been adding debug logs to the reverse proxy since this is a prime time to do that!

And actually, the upstream appears to be returning a 200 OK. Let me investigate further. Scratch that, I didn’t follow the log line correctly… and it is returning a 400 from upstream.

Edit: Okay, so, I don’t know why this is happening. We’ll need WAC to tell us why the requests are bad.

What config do you use on nginx or whatever other proxy you used before that works?

@hez2010 One more idea. By accessing directly, the only difference I can see between requests that succeed on the direct access and the ones that fail with the proxy are the absence of the Cookie request headers.

In my debug logs, the upstream replies with headers like "Set-Cookie": ["XSRF-TOKEN=66f5a97c-8ac9-47cd-8115-b7d3c49ec64e; path=/; secure"]} but I do not see these cookies being set, perhaps because the browser sees that the client is on an HTTP page.

This is just a hunch, and doesn’t explain why some requests fail and others succeed.

Still, you might try this on an HTTPS page and see if it works.

(Edit: I’m now almost certain that is at least one problem, if not the problem. If you go to the Application tab of the web inspector, then go to Storage -> Cookies, you will see that no XSRF-TOKEN cookie is set on the proxied HTTP page – probably because it is a secure context cookie only – but it is set on the direct access HTTPS page.)

Dang. I hoped I had it. I cheated a bit and forced the cookie to set on HTTP pages by adding this to the reverse proxy config:

"headers": {
	"response": {
		"replace": {
			"Set-Cookie": [
				{
					"search": "; secure",
					"replace": ""
				}
			]
		}
	}
}

It successfully manipulated the response header so that the cookie was set and is being included on the proxied requests, BUT… still getting 400s back.

What is your working proxy config for whatever you’re using now?

nginx:

        upstream wac {
                server test.hez2010.com:1080;
                keepalive 64;
        }

        server {
                listen 80;
                location / {
                        proxy_pass https://wac;
                        proxy_http_version 1.1;
                        proxy_set_header Connection "";
                }
        }

Maybe need to setup https for nginx.

Thank you! That is helpful.

Interesting… your nginx config disables keep-alive (by setting Connection header to empty string, i.e. removing it entirely).

Doing that in Caddy yields all 400s. (in the "keep_alive" object of your config, add "enabled": false)

I’m stumped.

@hez2010 What’s your full nginx config? Do you have anything else that might be relevant?

I used to config my nginx by referencing this: https://stackoverflow.com/questions/21284935/nginx-reverse-proxy-with-windows-authentication-that-uses-ntlm

WAC is using NTLM authentication method, I suspect this problem may relate to it.
However I don’t know why it works :frowning:

1 Like

I’m beginning to suspect you are right.

NTLM appears to violate HTTP’s convention of “stateless” by using HTTP requests to authenticate underlying connections. So, once a connection is authenticated, all requests using that connection have to be aware that the underlying connection is authenticated and set headers accordingly: authentication takes multiple round-trips as well, so Go’s connection pooling doesn’t guarantee that requests to finish authentication happen on the same connection. (Keep-Alive is also required.)

NGINX does not guarantee this either, but it probably does connection pooling very differently in a way that happens to work for you. Based on the comments and what I am reading about nginx’s additional ntlm directive, it sounds like your current config is just working by sheer luck!

Docs for proxying NTLM are sparse, but the best help I have found is from nginx’s docs:

Allows proxying requests with NTLM Authentication. The upstream connection is bound to the client connection once the client sends a request with the “Authorization” header field value starting with “ Negotiate ” or “ NTLM ”. Further client requests will be proxied through the same upstream connection, keeping the authentication context.

In order for NTLM authentication to work, it is necessary to enable keepalive connections to upstream servers. The proxy_http_version directive should be set to “ 1.1 ” and the “Connection” header field should be cleared.

So, we basically need to implement this behavior into Caddy’s proxy, along with a config property to enable it.