PHP and 502 errors (in Windows)

pwhodges · November 4, 2021, 12:57pm

1. Caddy version (`caddy version`):

Caddy 2.4.5 with replace_response and format plugins

2. How I run Caddy:

Windows service

a. System environment:

Windows Server 2019

b. Command:

nssm start caddy, with command line:

caddy run

c. Service/unit/compose file:

n/a

d. My complete Caddyfile or JSON config:

flextype.cassland.org {
	root * ..\Flextype

	php_fastcgi {
		to localhost:9001
		to localhost:9002
		to localhost:9003
		to localhost:9004
		to localhost:9005
		to localhost:9006
		to localhost:9007
		to localhost:9008
	}
	file_server
	log {
		output file .\Logs\TESTaccess.log
		format formatted
	}
}

3. The problem I’m having:

I am trying to test a CMS called Flextype. The installation instructions are as thin as “upload to your host which supports PHP7.4”. I therefore have to assume that some “standard” is followed which enables a PHP site to work on some default system. I further assume that Caddy’s php_fastcgi directive does what is normally expected, not least because I have some long-running sites where this works (and other test sites where sometimes it doesn’t!).

I can go to the root of the flextype site using “https://flextype.mydomain/” and get the “successfully installed” message. But when I try to go to a subpage such as “https://flextype.mydomain/admin” I simply get back 502. Also the favicon request “https://flextype.mydomain/favicon.ico” results in 502 rather than the 404 I’d expect.

4. Error messages and/or full log output:

502

I do not know where to find any information which might explain why I am getting a 502. The Caddy access log contains no other information. The php-errors log, for instance, has no record of this at all (it appears only to contain errors pertaining to exif data in images!)

5. What I already tried:

I’ve tried decomposing the php_fastcgi command and using a number of different try_files formats that I have seen recommended for other similar systems, but that seems too random an approach.

I have read around the forum messages about PHP and 502 errors, but none help (they are largely about Caddy v1 or v2 betas, anyway).

I’m pretty sure my problems would just go away if I simply changed to Apache, or maybe running Caddy on Linux, either of which I could do. But I’d like to persist in trying to find a solution to reliable PHP in Caddy on Windows, because I try to fight the general perception that Windows shouldn’t be taken seriously for web serving.

I’d be happy to do more diagnosis if I had some suggestions where to look.

6. Links to relevant resources:

Their certificate expired today (!), so not all browsers will let you view it… More useful today is:

francislavoie · November 4, 2021, 9:31pm

AFAIK, php-fpm doesn’t exist on Windows, so you need to run php-cgi.

To be clear, Caddy doesn’t come with PHP, it only comes with support for sending requests to a PHP FastCGI server (usually php-fpm, on Linux). So you need to install and run PHP separately.

HTTP status 502 means that Caddy wasn’t able to connect to the upstream, meaning it couldn’t reach a server on any of the ports you listed in your config. Which leads me to believe you don’t have php-cgi running separately.

Apache has a module called mod_php which allows Apache to directly execute PHP scripts without needing to send it to a separate server.

Nginx works similarly to Caddy, where it has a fastcgi_pass directive to send requests to a PHP FastCGI server.

Honestly, that “general perception” is for good reason. Windows is just not that well suited for the reliability, stability and performance that web servers require. It’s fine for development and toying around at home, but I can’t honestly recommend it for production workloads. But that’s my opinion.

pwhodges · November 4, 2021, 11:43pm

I wrote a long rant, and then deleted it .

As I said, I have several PHP sites running, using the same PHP set up. It’s fine - fast and reliable.

I have this one new site in which going to the root (i.e. index.php) works, but specifying an additional path, e.g. /admin (as specified) gives a 502.

I would like to know how I can debug what Caddy is doing with this path and how it is passing it to PHP to try to work out what’s wrong.

Whatever Caddy does works for some sites, but not here, even though the installation instructions imply nothing different is required from whatever is “usual” for the server to do. As PHP is there and listening, presumably the connection from Caddy gets started, but PHP turns it down - if I could find out why, then hopefully I could fix it.

Paul

francislavoie · November 5, 2021, 12:15am

Caddy outputs all its logs to stderr. How are you running Caddy exactly? Is the stderr output being written somewhere? That’s where you need to look.

pwhodges · November 5, 2021, 10:06am

OK, thanks. I had completely forgotten that I’d see that by running caddy in a command line window. What I get is:

In access log:

[2a02:8010:f01e::73]:62066 - - [05/Nov/2021:08:39:18 +0000] "GET /admin HTTP/2.0" 502 0

On command line:

2021/11/05 08:39:18.600 ERROR   http.log.error.log6     
malformed MIME header: missing colon: "HTTP/1.1 302 Found" {
    "request": {
        "remote_addr": "[2a02:8010:f01e::73]:62066", 
        "proto": "HTTP/2.0", 
        "method": "GET", 
        "host": "flextype.cassland.org", 
        "uri": "/admin", 
        "headers": {
            "User-Agent": ["Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:94.0) Gecko/20100101 Firefox/94.0"], 
            "Referer": ["https://flextype.cassland.org/"], 
            "Upgrade-Insecure-Requests": ["1"], 
            "Sec-Fetch-User": ["?1"], 
            "Accept": ["text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,*/*;q=0.8"], 
            "Accept-Language": ["en-GB,en;q=0.5"], 
            "Accept-Encoding": ["gzip, deflate, br"], 
            "Cookie": ["Flextype=a82b7df79fb858731f1c35c022bfe50b"], 
            "Sec-Fetch-Dest": ["document"], 
            "Sec-Fetch-Mode": ["navigate"], 
            "Sec-Fetch-Site": ["same-origin"], 
            "Te": ["trailers"]
        }, 
        "tls": {
            "resumed": true, 
            "version": 772, 
            "cipher_suite": 4865, 
            "proto": "h2", 
            "proto_mutual": true, 
            "server_name": "flextype.cassland.org"
        }
    }, 
    "duration": 0.4449831, 
    "status": 502, 
    "err_id": "m5agv6w88", 
    "err_trace": "reverseproxy.statusError (reverseproxy.go:858)"
}

I’m not certain how to interpret this; that’s to say, which bit of code is making this error, but I’m assuming that it is Flextype itself, and that Caddy is being more particular about it than some other servers (because the code is in use elsewhere). I also noticed the same error reported for a 404 response to a favicon request.

I guess from Caddy’s point of view there’s nothing to see here - thanks for the pointer.

Paul

pwhodges · November 5, 2021, 12:06pm

I mentioned Grav earlier; its recent failures being why I am looking at alternative CMSs. I now find that the failure I am getting with Grav is the exact same one as I’m getting with Flextype (i.e. missing colon in response header).

My guess was that these packages use a library in common, which is where the error arises - and indeed, they both use a package called GuzzleHTTP which I shall now investigate in more depth (it’s not the only package the products use in common, but it seems by far the most likely candidate).

Actually, looking further, I’m not sure from the error message where the colon is meant to be missing anyway. Presumably I need to use a network sniffer to see the part of the message which Caddy is finding the fault in.

OK, the response appears to be basically OK, but to have some extra bytes on the end which Caddy doesn’t like (output from WireShark):

E@y@#¨ÿ¿V°¶úÚP'ô7®þHTTP/1.1 302 Found
X-Powered-By: PHP/7.4.25
Expires: Thu, 19 Nov 1981 08:52:00 GMT
Cache-Control: no-store, no-cache, must-revalidate
Pragma: no-cache
Content-Type: text/html; charset=UTF-8
Location: /admin/accounts/login
Access-Control-Allow-Origin: *
Access-Control-Allow-Headers: X-Requested-With, Content-Type, Accept, Origin, Authorization
Access-Control-Allow-Methods: GET, POST, PUT, DELETE, PATCH, OPTIONS
Access-Control-Allow-Expose:
Access-Control-Allow-Credentials:
Content-Length: 0

doc

Paul

francislavoie · November 5, 2021, 1:22pm

This error is coming from the Go stdlib:

github.com

golang/go/blob/4d8db00641cc9ff4f44de7df9b8c4f4a4f9416ee/src/net/textproto/reader.go#L515


      
          
          	for {
          		kv, err := r.readContinuedLineSlice(mustHaveFieldNameColon)
          		if len(kv) == 0 {
          			return m, err
          		}
          
          		// Key ends at first colon.
          		k, v, ok := bytes.Cut(kv, colon)
          		if !ok {
          			return m, ProtocolError("malformed MIME header line: " + string(kv))
          		}
          		key := canonicalMIMEHeaderKey(k)
          
          		// As per RFC 7230 field-name is a token, tokens consist of one or more chars.
          		// We could return a ProtocolError here, but better to be liberal in what we
          		// accept, so if we get an empty key, skip it.
          		if key == "" {
          			continue
          		}

But that said, I don’t know why your upstreams would be responding with bad content. It might be a misconfiguration of your PHP runtime which is causing some stuff to be appended or prepended to the response sent back to Caddy, I dunno.

Edit: Caddy is directly calling that function triggering the error in the fastcgi client:

github.com

caddyserver/caddy/blob/24fda7514ddeebf19a1e872343b45ec523eefcba/modules/caddyhttp/reverseproxy/fastcgi/client.go#L420


      
          	r, err := c.Do(p, req)
          	if err != nil {
          		return
          	}
          
          	rb := bufio.NewReader(r)
          	tp := textproto.NewReader(rb)
          	resp = new(http.Response)
          
          	// Parse the response headers.
          	mimeHeader, err := tp.ReadMIMEHeader()
          	if err != nil && err != io.EOF {
          		return
          	}
          	resp.Header = http.Header(mimeHeader)
          
          	if resp.Header.Get("Status") != "" {
          		statusParts := strings.SplitN(resp.Header.Get("Status"), " ", 2)
          		resp.StatusCode, err = strconv.Atoi(statusParts[0])
          		if err != nil {
          			return

What I’m getting from this is that Caddy didn’t expect to see HTTP/1.1 302 Found on that line, which is the start of a regular HTTP response. For fastcgi it expects a Status: header which would have the status code number.

So it seems like you’re actually proxying to an HTTP server, not a fastcgi server. So you should use reverse_proxy, not php_fastcgi here. Maybe. Or your upstream isn’t configured to respond the right way.

pwhodges · November 5, 2021, 3:03pm

Well, something’s messed up. It seems that (on the problem sites at least) OK responses are handled correctly (without the “200” response, for instance) but erroneous ones are not.

Hang on - one of my error samples is the response to a request for “favicon.ico”. Surely that request should be handled by the fileserver in any case, not passed through fastcgi to PHP. That would, I suppose, explain the anomalous response… Shouldn’t the php_fastcgi directive sort that out?

Now I’m really confused.

Paul

francislavoie · November 5, 2021, 3:49pm

The way php_fastcgi works, is it first looks if a file exists on disk at the given request path. If it doesn’t, then it’ll rewrite the request path to index.php (if that exists) which will cause the request to get proxied to your PHP backend.

This is to give your backend the opportunity to handle routing for any path that Caddy knows it can’t handle successfully. Using index.php as a routing entrypoint is very common in modern PHP frameworks/applications, so that’s the default Caddy goes for.

pwhodges · November 5, 2021, 5:58pm

Right; I sort-of worked that out while walking the dogs just now. So I’m back to the application generating the back response - and since two of them do it, a library which they share. OTOH, these apps are widely used elsewhere without this problem, which implies a problem in my system. Time to dissect my PHP ini file, I think.

(later)

And BINGO!

; cgi.rfc2616_headers configuration option tells PHP what type of headers to
; use when sending HTTP response code. If it's set 0 PHP sends Status: header that
; is supported by Apache. When this option is set to 1 PHP will send
; RFC2616 compliant header.
; Default is zero.
; http://php.net/cgi.rfc2616-headers
;cgi.rfc2616_headers = 1
cgi.rfc2616_headers = 0

That was set to 1 (I guess I thought “RFC standard” - must be good). Setting it to 0 cured the problems I’ve been concentrating on.

Moral of the story - do not change any PHP default unless you truly understand it!

Thanks for your patient guidance which has led me along the right path.

Paul

francislavoie · November 5, 2021, 6:55pm

Fascinating. I didn’t know that option existed. Good to know

system · December 4, 2021, 12:58pm

This topic was automatically closed after 30 days. New replies are no longer allowed.

PHP and 502 errors (in Windows)

1. Caddy version (caddy version):

2. How I run Caddy:

a. System environment:

b. Command:

c. Service/unit/compose file:

d. My complete Caddyfile or JSON config:

3. The problem I’m having:

4. Error messages and/or full log output:

5. What I already tried:

6. Links to relevant resources:

1. Caddy version (`caddy version`):