Doesn't work when reverse proxy Windows Admin Center

1. My Caddy version (caddy -version):

Caddy v1.0.3 (h1:i9gRhBgvc5ifchwWtSe7pDpsdS9+Q0Rw9oYQmYUTw1w=)

2. How I run Caddy:

a. System environment:

Windows Server 2019 Data Center Core

b. Command:

caddy -http-port 80 -https-port 443 -log stdout -agree=true -conf=Caddyfile.txt

c. Service/unit/compose file:

None

d. My complete Caddyfile:

(proxy) {
    websocket
    transparent
}

0.0.0.0:80 {
    proxy / https://localhost:1080 {
        import proxy
        insecure_skip_verify
        keepalive 64
    }
}

3. The problem I’m having:

If I try to access the caddy reverse proxy, it will return a ‘502 bad gateway’ after authorized.

4. Error messages and/or full log output:

Oct 16 02:10:16 mypc caddy[71188]: 16/Oct/2019:02:10:16 +0800 [ERROR 502 /] stream error: stream ID 13; HTTP_1_1_REQUIRED
Oct 16 02:10:16 mypc caddy[71188]: 16/Oct/2019:02:10:16 +0800 [ERROR 502 /favicon.ico] stream error: stream ID 17; HTTP_1_1_REQUIRED

5. Related resources for testing

I created a temporary test server and deployed Windows Admin Center on it. I will delete it after several days.

Access caddy reverse proxy to windows admin center: http://test.hez2010.com/
Access windows admin center directly: https://test.hez2010.com:1080/

username: hez2010, password: hez2010!hez2010
you can also use above credentials to access the server using RDP.

Thanks!

Hi @hez2010, welcome to the Caddy community.

Thanks especially for throwing up that temporary test server. I had a quick look, found pretty much exactly what you’re running into (502s from Caddy).

Just to check, you’re running Caddy on Windows on the same server that’s serving WAC, right? Can you throw up a PowerShell and run Invoke-WebRequest -SkipCertificateCheck -Uri https://localhost:1080 | Select-Object -ExpandProperty Headers just to see what we get back?

Yes, Caddy is on the same server that’s serving WAC.

I ran the following script:

# bypass TLS check
add-type @"
    using System.Net;
    using System.Security.Cryptography.X509Certificates;
    public class TrustAllCertsPolicy : ICertificatePolicy {
        public bool CheckValidationResult(
            ServicePoint srvPoint, X509Certificate certificate,
            WebRequest request, int certificateProblem) {
            return true;
        }
    }
"@
[System.Net.ServicePointManager]::CertificatePolicy = New-Object TrustAllCertsPolicy

[Net.ServicePointManager]::SecurityProtocol = [Net.SecurityProtocolType]::Ssl3, [Net.SecurityProtocolType]::Tls, [Net.SecurityProtocolType]::Tls11, [Net.SecurityProtocolType]::Tls12

# get credential
$cred = Get-Credential

# curl
Invoke-WebRequest -Uri https://localhost:1080 -Credential $cred -UseBasicParsing | Select-Object -ExpandProperty Headers

output:

Key             Value
---             -----
Pragma          no-cache
X-Frame-Options sameorigin
Content-Length  6559
Cache-Control   no-cache,no-store,must-revalidate
Content-Type    text/html
Date            Thu, 24 Oct 2019 00:48:25 GMT
Expires         0
ETag            "1d50c1410ead59f"
Last-Modified   Thu, 16 May 2019 18:20:40 GMT
Set-Cookie      XSRF-TOKEN=fb1d8489-4969-41a5-9d40-cfd4319db2b8; path=/; secure
Server          Microsoft-HTTPAPI/2.0

What do you get if you don’t provide a credential (i.e. the user has not authenticated yet)?

Will get 401 Unauthorized.

Alright. Well, this is the stage where I advise opening an issue on the Caddy repository.

That said, this is a v1 issue, and development is full steam ahead on the beta of v2. Would you be interested in giving Caddy v2 a try and see if it produces a similar result? Given that it’s being worked on right now, it might see updates much faster if this is an issue with the proxy module.

It seems that Caddyfile in v1 is not compatible with caddy v2.

That’s correct, you’d need to adapt to the v2 Caddyfile spec. That said, you have a relatively straightforward proxy, so it should be as simple as the v1 Caddyfile.

I’ve tried caddy 2, but it still not working.
This time I cannot even see the authorization process.

Logs:

2019/10/27 08:29:47 Caddy 2 admin endpoint listening on localhost:2019
2019/10/27 08:29:47 [INFO] Server srv0 is only listening on the HTTP port 80, so no automatic HTTPS will be applied to this server
2019/10/27 08:29:47 [INFO] tls: Cleaned up storage unit(s)
2019/10/27 08:29:47 Caddy 2 serving initial configuration
2019/10/27 08:29:47 [INFO][cache:0xc00009efa0] Started certificate maintenance routine
2019/10/27 08:30:05 [ERROR] [GET /] {id=k1qwti3jt} reverseproxy.(*Handler).ServeHTTP (reverseproxy.go:296): HTTP 502: net/http: HTTP/1.x transport connection broken: malformed HTTP response "\x00\x00\x12\x04\x00\x00\x00\x00\x00\x00\x03\x00\x00\x00d\x00\x10\x00\x00\x00\x02\x00\x04\x00\x10\x00\x00\x00\x00\x04\b\x00\x00\x00\x00\x00\x00\x0f\x00\x01"

My Caddyfile:

{
    "apps": {
        "http": {
            "servers": {
                "srv0": {
                    "listen": [
                        ":80"
                    ],
                    "routes": [
                        {
                            "match": [
                                {
                                    "host": [
                                        "test.hez2010.com"
                                    ]
                                }
                            ],
                            "handle": [
                                {
                                    "handler": "subroute",
                                    "routes": [
                                        {
                                            "handle": [
                                                {
                                                    "handler": "reverse_proxy",
                                                    "transport": {
                                                        "keep_alive": {
                                                            "idle_timeout": "64s"
                                                        },
                                                        "protocol": "http",
                                                        "tls": {
                                                            "insecure_skip_verify": true
                                                        }
                                                    },
                                                    "upstreams": [
                                                        {
                                                            "dial": "localhost:1080"
                                                        }
                                                    ]
                                                }
                                            ]
                                        }
                                    ]
                                }
                            ]
                        }
                    ]
                }
            }
        }
    }
}

Probably because you’ve configured the upstream as HTTP and it’s HTTPS. Caddy’s expecting HTTP but the server is sending TLS negotiation back, causing the error. Configure the upstream as HTTPS and give it another go.

I changed the line

"dial": "localhost:1080"

to

"dial": "https://localhost:1080"

but got a new error:

reverseproxy.(*Handler).ServeHTTP (reverseproxy.go:296): HTTP 502: dial https:: unknown network https:

I might be wrong, but I think you want to leave dial set to localhost:1080 and change the protocol from http to https.

(I was wrong, haha)

1 Like

Still not working

2019/10/27 15:02:40 run: starting caddy administration endpoint: loading initial config: running: loading app module 'http': provision http: setting up server routes: loading handler module in position 0: loading module 'subroute': provision http.handlers.subroute: setting up routes: loading handler module in position 0: loading module 'reverse_proxy': provision http.handlers.reverse_proxy: loading transport module: loading module 'https': unknown module: http.handlers.reverse_proxy.transport.https

Edit: Sorry, after actually waking up and re-reading this later, I see you already have tls specified.

Might be a bug in the http transport. Would you be willing to add some additional debug logs and recompile? I can give you some suggestions.

Definitely. How to add those debug logs?

1 Like

Great, thank you for being willing! :raised_hands:

Actually, I was able to reproduce the error! @Whitestrake’s speculation was right, this happens when using HTTP to connect to an HTTPS server.

Weirdly, specifying "tls": {} with no parameters makes it work. I had this at first and was scratching my head because it was working and I couldn’t reproduce the problem. So it’s something about insecure_skip_verify

Investigating…

1 Like

Ohh… we have to explicitly enable HTTP/2 still. Weird, I thought that worked by default in newer versions of Go. Sigh…

The error makes sense though, since we explicitly set “h2” as one of the ALPN values for the TLS handshake, meaning the HTTPS server on the backend says, “Oh you support HTTP/2, cool, so do I. Now that the handshake is over, here’s my HTTP/2 response: …” but actually the client’s HTTP transport (for some reason!?!) hadn’t enabled HTTP/2, so it was expecting an ASCII HTTP/1.1 response.

Anyway, I’ve pushed a fix in https://github.com/caddyserver/caddy/commit/813fff05844f4aee35bdca37eab76eddeb57d2dd

Can you please build from source on the v2 branch and try again?

1 Like

The authorization process now can successfully start, but 502 again after authorization. (caddy2 preview 8)

2019/10/30 02:26:03 [INFO][cache:0xc00009d090] Started certificate maintenance routine
1.5724023737244468e+09  e[31mERRORe[0m  http.log.error  stream error: stream ID 5; HTTP_1_1_REQUIRED    {"request": {"method": "GET", "uri": "/", "proto": "HTTP/1.1", "remote_addr": "120.236.174.144:37497", "host": "test.hez2010.com", "headers": {"Connection": ["keep-alive"], "Dnt": ["1"], "Upgrade-Insecure-Requests": ["1"], "User-Agent": ["Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/79.0.3945.8 Safari/537.36 Edg/79.0.309.5"], "Cache-Control": ["max-age=0"], "Authorization": ["Negotiate TlRMTVNTUAABAAAAl4II4gAAAAAAAAAAAAAAAAAAAAAKALpHAAAADw=="], "Accept": ["text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.9"], "Accept-Encoding": ["gzip, deflate"], "Accept-Language": ["en-US,en;q=0.9"]}}, "status": 502, "err_id": "fhr7zm7it", "err_trace": "reverseproxy.(*Handler).ServeHTTP (reverseproxy.go:296)"}

You can try accessing test.hez2010.com with username hez2010 and password hez2010!hez2010.
The original Windows Admin Center was served on test.hez2010.com:1080.

My caddyfile:

{
        "apps": {
                "http": {
                        "http_port": 80,
                        "servers": {
                                "srv0": {
                                        "listen": [
                                                ":80"
                                        ],
                                        "routes": [
                                                {
                                                        "match": [
                                                                {
                                                                              "host": [
                                                                                      "test.hez2010.com"
                                                                              ]
                                                                }
                                                        ],
                                                        "handle": [
                                                                {
                                                                              "handler": "subroute",
                                                                              "routes": [
                                                                                      {
                                                                                              "handle": [
                                                                                                      {
                                                                                                              "handler": "reverse_proxy",
                                                                                                              "transport": {
                                                                                                                      "keep_alive": {
                                                                                                                              "idle_timeout": "64s"
                                                                                                                      },
                                                                                                                      "protocol": "http",
                                                                                                                      "tls": {
                                                                                                                              "insecure_skip_verify": true
                                                                                                                      }
                                                                                                              },
                                                                                                              "upstreams": [
                                                                                                                      {
                                                                                                                              "dial": "localhost:1080"
                                                                                                                      }
                                                                                                              ]
                                                                                                      }
                                                                                              ]
                                                                                      }
                                                                              ]
                                                                }
                                                        ]
                                                }
                                        ]
                                }
                        }
                }
        }
}
1 Like

Thank you for trying it again!

And thanks for the live example, I can see the error you are talking about.

And boy, that’s a doozy. It’s a very obscure error – try googling for it, you’ll have to put “HTTP_1_1_REQUIRED” in quotes because it’s so uncommon.

The most relevant result I’ve found so far is here: https://github.com/terraform-providers/terraform-provider-azurerm/issues/428

This does appear to be a bug in WAC rather than in Caddy. (Here’s why I think so)

One possibility is to disable HTTP/2? Can you try running Caddy with this env variable set?

GODEBUG=http2client=0

(I got the idea from https://github.com/terraform-providers/terraform-provider-azurerm/issues/503#issuecomment-345919279, but it’s also in the docs for Go’s net/http package.)

I’ve set GODEBUG=http2client=0 but still not working.
I will upgrade WAC to the latest preview version and try again.