Websocket traffic isn't reaching chatserver on frankenphp

Help
1

1. The problem I’m having:

I have a chatserver running as a service, which listens on localhost:5502

[Unit]
Description=Chat Server

[Service]
ExecStart=php /var/www/html/SportchLegacyApp/system.router.cli.php chat-server
WorkingDirectory=/var/www/html/SportchLegacyApp
StandardOutput=append:/var/log/chatserver/output.log
StandardError=append:/var/log/chatserver/error.log
Restart=on-failure
RestartSec=5
User=root

[Install]
WantedBy=multi-user.target

I have frankephp configured to provide websocket proxy to that service (see below).

I can telnet into port 5502, so I know the port is open:

curl -v http://82.180.154.41:5502

*   Trying 82.180.154.41:5502...
* Connected to 82.180.154.41 (82.180.154.41) port 5502
> GET / HTTP/1.1
> Host: 82.180.154.41:5502
> User-Agent: curl/8.6.0
> Accept: */*
>
* Empty reply from server
* Closing connection
curl: (52) Empty reply from server

There’s no output from Caddy to the logs for incoming 5502 requests and the chatserver logs show no activity either.

2. Error messages and/or full log output:

There is no log output. That's the problem.

3. Caddy version:

FrankenPHP v1.2.3 PHP 8.3.10 Caddy v2.8.4 h1:q3pe0wpBj1OcHFZ3n/1nl4V4bxBrYoSoab7rL9BMYNk=

4. How I installed and ran Caddy:

a. System environment:

Alma Linux 9, systemd
frankenphp latest static build

b. Command:

systemctl start frankenphp

c. Service/unit/compose file:

[Unit]
Description=FrankenPHP Server
After=network.target network-online.target
Requires=network-online.target

[Service]
Restart=on-failure
Type=notify
User=frankenphp
Group=frankenphp
ExecStartPre=/usr/bin/frankenphp validate --config /etc/frankenphp/Caddyfile
ExecStart=/usr/bin/frankenphp run --environ --config /etc/frankenphp/Caddyfile
ExecReload=/usr/bin/frankenphp reload --config /etc/frankenphp/Caddyfile --force
TimeoutStopSec=5s
LimitNOFILE=1048576
PrivateTmp=true
ProtectSystem=full
AmbientCapabilities=CAP_NET_ADMIN CAP_NET_BIND_SERVICE

Environment=PHP_INI_SCAN_DIR=/etc/php.d/

[Install]
WantedBy=multi-user.target

d. My complete Caddy config:

{
    # Enable FrankenPHP
    frankenphp

    # Logging
    log {
        output stderr
        level DEBUG
    }
}

dev.sportch.co.uk {
    # Enable compression (optional)
    encode zstd br gzip

    root * /var/www/html/SportchLegacyApp

    # Define a route for handling requests
    route {

        # Security: Hide specific paths
        @hide path /.env *.sql *.template /gtscripts/* *.txt *.md /random /support vapi/ var/ /support /system.* /lib
        respond @hide 404

        # Execute PHP files
        php_server

        # Rewrite subfolder requests for screenscrapers
        rewrite /ladder/* /ladder/index.php
        rewrite /manager/* /manager/index.php

         # Add WebSocket support
        @websocket {
            header Connection *Upgrade*
            header Upgrade websocket
        }
        reverse_proxy @websocket localhost:5502
    }
}

5. Links to relevant resources:

2

Looks like your config file is missing the port so it’s just listening on the default ports 80 and 443.

1 Like
3

Thanks for the reply! I’m not quite sure how to do that? Based on what you wrote, I tried this:

{
    # Enable FrankenPHP
    frankenphp

    # Logging
    log {
        output stderr
        level DEBUG
    }
}
dev.sportch.co.uk:5502 {
        # Add WebSocket support
        @websocket {
            header Connection *Upgrade*
            header Upgrade websocket
        }
        reverse_proxy @websocket localhost:5502
}
dev.sportch.co.uk {
    # Enable compression (optional)
    encode zstd br gzip

    root * /var/www/html/SportchLegacyApp

    # Define a route for handling requests
    route {

        # Security: Hide specific paths
        @hide path /.env *.sql *.template /gtscripts/* *.txt *.md /random /support vapi/ var/ /support /system.* /lib
        respond @hide 404

        # Execute PHP files
        php_server

        # Rewrite subfolder requests for screenscrapers
        rewrite /ladder/* /ladder/index.php
        rewrite /manager/* /manager/index.php

    }
}

But frankenphp won’t start. The startup logs have this error message:

Aug 22 16:24:21 dev.sportch.co.uk frankenphp[1157]: Error: loading initial config: loading new config: http app module: start: listening on :5502: listen tcp :5502: bind: address already
in use

I should also just point out that what I’m trying to do is to migrate from a Caddy/PHP-FPM setup, to a FrankenPHP setup.

With the former. the chatserver listens on port 5502 and the caddy server doesn’t actually need to get involved in routing that traffic. So the Caddyfile on that server has no route for 5502 and no @websocket named matcher. In case it helps, here’s that Caddyfile:

app.sportch.co.uk {

    log {
        output file /var/log/caddy/access.log
    }

        root * /var/www/html/SportchLegacyApp

        # handle other URLs
    php_fastcgi unix//run/php-fpm/www.sock

    # Serve files
    file_server

    root /phpmyadmin/* /var/www/html/myadmin/
    uri /phpmyadmin/* strip_prefix /phpmyadmin


    # Rewrite subfolder requests for screenscrapers
    rewrite /ladder/* /ladder/index.php
    rewrite /manager/* /manager/index.php

        # handle the /status URL
    reverse_proxy /status unix//run/php-fpm/www.sock {
        transport fastcgi {
            env SCRIPT_NAME /status
        }
    }


        @hide path /.env *.sql *.template /gtscripts/* *.txt *.md /random /support vapi/ var/ /support /system.* /lib
    respond @hide 404
}

However, on the frankenphp version, there seems to be some sort of disconnect that I haven’t been able to figure out, whereby the chatserver (which runs as a php app launched from the CLI via systemd but because frankenphp doesn’t provide a CLI it is therefor using a separate instance of PHP; I don’t know if that has an impact?) is not seeing traffic on that port. The traffic just seems to get swallowed up. I can see it coming in via tcpdump but it doesn’t seem to reach the listener.

4

What Matt was saying is that you were making requests to port 5502, but Caddy is listening on port 443. Make requests to port 443 (i.e. https://, you can omit the port) instead.

2 Likes
5

Ah, actually your problem is you put your reverse_proxy at the end of your route, so it never runs because php_server takes all requests and nothing goes past it. In route, the order you place directives matter. Move it above php_server. Your rewrites need to go earlier as well.

Though, you don’t need to be using route at all. Since v2.8.0, pretty sure FrankenPHP uses the new feature which allows plugins to set their own order for Caddyfile directives.

3 Likes
6

Ok… but I tried that and it made no appreciable difference. Everything seems as it was before. Same errors.

7

Which errors are you running into currently?

The 5502 bind errors you can revert back to the prior site address config to resolve, since as you’ve described you don’t want Caddy listening on 5502, you just want the chatserver there. Then implement Francis’ ordering advice.

1 Like
8

I’ve done ALL of that and it still doesn’t work.
I think this is an SSL issue. Using safari as the chat client I can see the following error messages:

WebSocket connection to 'wss://dev.sportch.co.uk:5502/' failed: An SSL error has occurred and a secure connection to the server cannot be made.

That would explain why the traffic doesn’t reach the chatserver. However, I don’t know how to fix that.

9

It’s trying to connect directly to the upstream bypassing Caddy, but I’m guessing your upstream doesn’t have a valid certificate like Caddy does.

It seems to me like you need to configure your client to connect to Caddy instead, on wss://dev.sportch.co.uk/ (note no specified port implies default WSS port which is 443), so Caddy can present a valid certificate and then proxy to your upstream.

2 Likes
10

Ok. Sounds good. But how? (And why isn’t this an issue with caddy / php-fpm?).

P.S. I can’t change the ws port as that’s baked into the client as 5502. I don’t want to have to completely redeploy all our client software just to get the new server to work.

Keep in mind – all of this works flawlessly with caddy / php-fpm. So the big question is – why isn’t it working with frankenphp? Like what’s the difference?

11

If you need your clients to connect to Caddy on port 5502, then you’ll need to put Caddy on 5502, and you’ll need to shift your upstream somewhere else. You obviously can’t have them both listening on the same port, so something’s gotta give.

The only difference I’m aware of between Caddy+PHP-FPM and FrankenPHP is that the former proxies PHP script requests to a separate FPM listener, and the latter instead handles the request in-process with a Go-based PHP processor, no proxy. So as far as I’m concerned, if you took your current config and just stripped out FrankenPHP and swapped in PHP-FPM, I’d have to assume it would fail in the same manner.

So I have to admit I’m not sure what the difference is. I’m not familiar with your upstream myself, and I haven’t seen your working Caddy+PHP-FPM config to compare and contrast with what you’re trying to achieve with FrankenPHP.

2 Likes
12

If that were true there would be no reason for this long thread.

13

omg… a colleague suggested restarting the chatserver service… and that worked. It had cached expired certs. I can’t believe that during all of that debugging, the one thing I obviously didn’t do, is restart the chatserver service.

14

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.