Doesn't work when reverse proxy Windows Admin Center

hez2010 · October 23, 2019, 11:57am

1. My Caddy version (`caddy -version`):

Caddy v1.0.3 (h1:i9gRhBgvc5ifchwWtSe7pDpsdS9+Q0Rw9oYQmYUTw1w=)

2. How I run Caddy:

a. System environment:

Windows Server 2019 Data Center Core

b. Command:

caddy -http-port 80 -https-port 443 -log stdout -agree=true -conf=Caddyfile.txt

c. Service/unit/compose file:

None

d. My complete Caddyfile:

(proxy) {
    websocket
    transparent
}

0.0.0.0:80 {
    proxy / https://localhost:1080 {
        import proxy
        insecure_skip_verify
        keepalive 64
    }
}

3. The problem I’m having:

If I try to access the caddy reverse proxy, it will return a ‘502 bad gateway’ after authorized.

4. Error messages and/or full log output:

Oct 16 02:10:16 mypc caddy[71188]: 16/Oct/2019:02:10:16 +0800 [ERROR 502 /] stream error: stream ID 13; HTTP_1_1_REQUIRED
Oct 16 02:10:16 mypc caddy[71188]: 16/Oct/2019:02:10:16 +0800 [ERROR 502 /favicon.ico] stream error: stream ID 17; HTTP_1_1_REQUIRED

5. Related resources for testing

I created a temporary test server and deployed Windows Admin Center on it. I will delete it after several days.

Access caddy reverse proxy to windows admin center: http://test.hez2010.com/
Access windows admin center directly: https://test.hez2010.com:1080/

username: hez2010, password: hez2010!hez2010
you can also use above credentials to access the server using RDP.

Thanks!

Whitestrake · October 24, 2019, 12:34am

Hi @hez2010, welcome to the Caddy community.

Thanks especially for throwing up that temporary test server. I had a quick look, found pretty much exactly what you’re running into (502s from Caddy).

Just to check, you’re running Caddy on Windows on the same server that’s serving WAC, right? Can you throw up a PowerShell and run Invoke-WebRequest -SkipCertificateCheck -Uri https://localhost:1080 | Select-Object -ExpandProperty Headers just to see what we get back?

hez2010 · October 24, 2019, 12:57am

Yes, Caddy is on the same server that’s serving WAC.

I ran the following script:

# bypass TLS check
add-type @"
    using System.Net;
    using System.Security.Cryptography.X509Certificates;
    public class TrustAllCertsPolicy : ICertificatePolicy {
        public bool CheckValidationResult(
            ServicePoint srvPoint, X509Certificate certificate,
            WebRequest request, int certificateProblem) {
            return true;
        }
    }
"@
[System.Net.ServicePointManager]::CertificatePolicy = New-Object TrustAllCertsPolicy

[Net.ServicePointManager]::SecurityProtocol = [Net.SecurityProtocolType]::Ssl3, [Net.SecurityProtocolType]::Tls, [Net.SecurityProtocolType]::Tls11, [Net.SecurityProtocolType]::Tls12

# get credential
$cred = Get-Credential

# curl
Invoke-WebRequest -Uri https://localhost:1080 -Credential $cred -UseBasicParsing | Select-Object -ExpandProperty Headers

output:

Key             Value
---             -----
Pragma          no-cache
X-Frame-Options sameorigin
Content-Length  6559
Cache-Control   no-cache,no-store,must-revalidate
Content-Type    text/html
Date            Thu, 24 Oct 2019 00:48:25 GMT
Expires         0
ETag            "1d50c1410ead59f"
Last-Modified   Thu, 16 May 2019 18:20:40 GMT
Set-Cookie      XSRF-TOKEN=fb1d8489-4969-41a5-9d40-cfd4319db2b8; path=/; secure
Server          Microsoft-HTTPAPI/2.0

Whitestrake · October 24, 2019, 1:00am

What do you get if you don’t provide a credential (i.e. the user has not authenticated yet)?

hez2010 · October 24, 2019, 1:15am

Will get 401 Unauthorized.

Whitestrake · October 24, 2019, 1:21am

Alright. Well, this is the stage where I advise opening an issue on the Caddy repository.

That said, this is a v1 issue, and development is full steam ahead on the beta of v2. Would you be interested in giving Caddy v2 a try and see if it produces a similar result? Given that it’s being worked on right now, it might see updates much faster if this is an issue with the proxy module.

hez2010 · October 24, 2019, 3:15am

It seems that Caddyfile in v1 is not compatible with caddy v2.

Whitestrake · October 24, 2019, 3:22am

That’s correct, you’d need to adapt to the v2 Caddyfile spec. That said, you have a relatively straightforward proxy, so it should be as simple as the v1 Caddyfile.

hez2010 · October 27, 2019, 8:33am

I’ve tried caddy 2, but it still not working.
This time I cannot even see the authorization process.

Logs:

2019/10/27 08:29:47 Caddy 2 admin endpoint listening on localhost:2019
2019/10/27 08:29:47 [INFO] Server srv0 is only listening on the HTTP port 80, so no automatic HTTPS will be applied to this server
2019/10/27 08:29:47 [INFO] tls: Cleaned up storage unit(s)
2019/10/27 08:29:47 Caddy 2 serving initial configuration
2019/10/27 08:29:47 [INFO][cache:0xc00009efa0] Started certificate maintenance routine
2019/10/27 08:30:05 [ERROR] [GET /] {id=k1qwti3jt} reverseproxy.(*Handler).ServeHTTP (reverseproxy.go:296): HTTP 502: net/http: HTTP/1.x transport connection broken: malformed HTTP response "\x00\x00\x12\x04\x00\x00\x00\x00\x00\x00\x03\x00\x00\x00d\x00\x10\x00\x00\x00\x02\x00\x04\x00\x10\x00\x00\x00\x00\x04\b\x00\x00\x00\x00\x00\x00\x0f\x00\x01"

My Caddyfile:

{
    "apps": {
        "http": {
            "servers": {
                "srv0": {
                    "listen": [
                        ":80"
                    ],
                    "routes": [
                        {
                            "match": [
                                {
                                    "host": [
                                        "test.hez2010.com"
                                    ]
                                }
                            ],
                            "handle": [
                                {
                                    "handler": "subroute",
                                    "routes": [
                                        {
                                            "handle": [
                                                {
                                                    "handler": "reverse_proxy",
                                                    "transport": {
                                                        "keep_alive": {
                                                            "idle_timeout": "64s"
                                                        },
                                                        "protocol": "http",
                                                        "tls": {
                                                            "insecure_skip_verify": true
                                                        }
                                                    },
                                                    "upstreams": [
                                                        {
                                                            "dial": "localhost:1080"
                                                        }
                                                    ]
                                                }
                                            ]
                                        }
                                    ]
                                }
                            ]
                        }
                    ]
                }
            }
        }
    }
}

Whitestrake · October 27, 2019, 10:33am

Probably because you’ve configured the upstream as HTTP and it’s HTTPS. Caddy’s expecting HTTP but the server is sending TLS negotiation back, causing the error. Configure the upstream as HTTPS and give it another go.

hez2010 · October 27, 2019, 12:34pm

I changed the line

"dial": "localhost:1080"

to

"dial": "https://localhost:1080"

but got a new error:

reverseproxy.(*Handler).ServeHTTP (reverseproxy.go:296): HTTP 502: dial https:: unknown network https:

Whitestrake · October 27, 2019, 12:35pm

~~I might be wrong, but I think you want to leave dial set to localhost:1080 and change the protocol from http to https.~~

(I was wrong, haha)

hez2010 · October 27, 2019, 3:03pm

Still not working

2019/10/27 15:02:40 run: starting caddy administration endpoint: loading initial config: running: loading app module 'http': provision http: setting up server routes: loading handler module in position 0: loading module 'subroute': provision http.handlers.subroute: setting up routes: loading handler module in position 0: loading module 'reverse_proxy': provision http.handlers.reverse_proxy: loading transport module: loading module 'https': unknown module: http.handlers.reverse_proxy.transport.https

matt · October 27, 2019, 3:17pm

Edit: Sorry, after actually waking up and re-reading this later, I see you already have tls specified.

Might be a bug in the http transport. Would you be willing to add some additional debug logs and recompile? I can give you some suggestions.

hez2010 · October 28, 2019, 8:48am

Definitely. How to add those debug logs?

matt · October 29, 2019, 5:36am

Great, thank you for being willing!

Actually, I was able to reproduce the error! @Whitestrake’s speculation was right, this happens when using HTTP to connect to an HTTPS server.

Weirdly, specifying "tls": {} with no parameters makes it work. I had this at first and was scratching my head because it was working and I couldn’t reproduce the problem. So it’s something about insecure_skip_verify…

Investigating…

matt · October 29, 2019, 6:11am

Ohh… we have to explicitly enable HTTP/2 still. Weird, I thought that worked by default in newer versions of Go. Sigh…

github.com

golang/net/blob/fe3aa8a4527195a6057b3fad46619d7d090e99b5/http2/transport.go#L136

    
      
          	}
          	return t.MaxHeaderListSize
          }
          
          
func (t *Transport) disableCompression() bool {
          	return t.DisableCompression || (t.t1 != nil && t.t1.DisableCompression)
          }
          
          
// ConfigureTransport configures a net/http HTTP/1 Transport to use HTTP/2.
          // It returns an error if t1 has already been HTTP/2-enabled.
          func ConfigureTransport(t1 *http.Transport) error {
          	_, err := configureTransport(t1)
          	return err
          }
          
          
func configureTransport(t1 *http.Transport) (*Transport, error) {
          	connPool := new(clientConnPool)
          	t2 := &Transport{
          		ConnPool: noDialClientConnPool{connPool},
          		t1:       t1,
          	}

The error makes sense though, since we explicitly set “h2” as one of the ALPN values for the TLS handshake, meaning the HTTPS server on the backend says, “Oh you support HTTP/2, cool, so do I. Now that the handshake is over, here’s my HTTP/2 response: …” but actually the client’s HTTP transport (for some reason!?!) hadn’t enabled HTTP/2, so it was expecting an ASCII HTTP/1.1 response.

Anyway, I’ve pushed a fix in proxy: Enable HTTP/2 on transport to backend · caddyserver/caddy@813fff0 · GitHub

Can you please build from source on the v2 branch and try again?

hez2010 · October 30, 2019, 2:29am

The authorization process now can successfully start, but 502 again after authorization. (caddy2 preview 8)

2019/10/30 02:26:03 [INFO][cache:0xc00009d090] Started certificate maintenance routine
1.5724023737244468e+09  e[31mERRORe[0m  http.log.error  stream error: stream ID 5; HTTP_1_1_REQUIRED    {"request": {"method": "GET", "uri": "/", "proto": "HTTP/1.1", "remote_addr": "120.236.174.144:37497", "host": "test.hez2010.com", "headers": {"Connection": ["keep-alive"], "Dnt": ["1"], "Upgrade-Insecure-Requests": ["1"], "User-Agent": ["Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/79.0.3945.8 Safari/537.36 Edg/79.0.309.5"], "Cache-Control": ["max-age=0"], "Authorization": ["Negotiate TlRMTVNTUAABAAAAl4II4gAAAAAAAAAAAAAAAAAAAAAKALpHAAAADw=="], "Accept": ["text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.9"], "Accept-Encoding": ["gzip, deflate"], "Accept-Language": ["en-US,en;q=0.9"]}}, "status": 502, "err_id": "fhr7zm7it", "err_trace": "reverseproxy.(*Handler).ServeHTTP (reverseproxy.go:296)"}

You can try accessing test.hez2010.com with username hez2010 and password hez2010!hez2010.
The original Windows Admin Center was served on test.hez2010.com:1080.

My caddyfile:

{
        "apps": {
                "http": {
                        "http_port": 80,
                        "servers": {
                                "srv0": {
                                        "listen": [
                                                ":80"
                                        ],
                                        "routes": [
                                                {
                                                        "match": [
                                                                {
                                                                              "host": [
                                                                                      "test.hez2010.com"
                                                                              ]
                                                                }
                                                        ],
                                                        "handle": [
                                                                {
                                                                              "handler": "subroute",
                                                                              "routes": [
                                                                                      {
                                                                                              "handle": [
                                                                                                      {
                                                                                                              "handler": "reverse_proxy",
                                                                                                              "transport": {
                                                                                                                      "keep_alive": {
                                                                                                                              "idle_timeout": "64s"
                                                                                                                      },
                                                                                                                      "protocol": "http",
                                                                                                                      "tls": {
                                                                                                                              "insecure_skip_verify": true
                                                                                                                      }
                                                                                                              },
                                                                                                              "upstreams": [
                                                                                                                      {
                                                                                                                              "dial": "localhost:1080"
                                                                                                                      }
                                                                                                              ]
                                                                                                      }
                                                                                              ]
                                                                                      }
                                                                              ]
                                                                }
                                                        ]
                                                }
                                        ]
                                }
                        }
                }
        }
}

matt · October 30, 2019, 5:09am

Thank you for trying it again!

And thanks for the live example, I can see the error you are talking about.

And boy, that’s a doozy. It’s a very obscure error – try googling for it, you’ll have to put “HTTP_1_1_REQUIRED” in quotes because it’s so uncommon.

The most relevant result I’ve found so far is here: HTTP_1_1_REQUIRED error returned on calls · Issue #428 · hashicorp/terraform-provider-azurerm · GitHub

This does appear to be a bug in WAC rather than in Caddy. (Here’s why I think so)

One possibility is to disable HTTP/2? Can you try running Caddy with this env variable set?

GODEBUG=http2client=0

(I got the idea from Intermittent errors when destroying resources and reapplying tf apply using azurerm · Issue #503 · hashicorp/terraform-provider-azurerm · GitHub, but it’s also in the docs for Go’s net/http package.)

hez2010 · October 30, 2019, 5:27am

I’ve set GODEBUG=http2client=0 but still not working.
I will upgrade WAC to the latest preview version and try again.