Reverse_proxy for Browser-based compiled Minecraft client using WebSocket not working

Help
1

1. Output of caddy version:

v2.3.0

2. How I run Caddy:

Running caddy as a reverse proxy in the system environment with a Caddyfile.

a. System environment:

Currently testing using a replit bash repl (which I don’t know too much about), but in production, I’ll use another hosting platform.

b. Command:

./caddy run

c. Service/unit/compose file:

N/A 
Don't think I have one

d. My complete Caddy config:

go.example.org:80, go.example.org:443 {
    reverse_proxy 172.96.140.17:25697
}

3. The problem I’m having:

A compiled version of Minecraft’s client that is run in a browser uses WebSocket protocols (specifically wss) and my reverse proxy from go.example.org to 172.96.140.17:25697 doesn’t seem to forward the connection. Also want to get wss working so that online clients can access the server.

4. Error messages and/or full log output:

curl -v go.example.org
*   Trying 34.149.204.188:80...
* Connected to go.example.org (34.149.204.188) port 80 (#0)
> GET / HTTP/1.1
> Host: go.example.org
> User-Agent: curl/7.80.0
> Accept: */*
> 
* Mark bundle as not supporting multiuse
< HTTP/1.1 308 Permanent Redirect
< Content-Type: text/html; charset=utf-8
< Location: https://go.example.org/
< Replit-Cluster: global
< Date: Tue, 10 Jan 2023 15:44:07 GMT
< Content-Length: 61
< Via: 1.1 google
< 
<a href="https://go.example.org/">Permanent Redirect</a>.

* Connection #0 to host go.example.org left intact

5. What I already tried:

  • Removing go.example.org in the config as the domain of the repl is already directed there with a CNAME record.
  • Making two different blocks for the different ports with the same text.
  • A lot of other things, including trying to serve a website on a file_server, which is not what I wanted.

6. Links to relevant resources:

Offline version of client (where screenshot above in section 3 was taken) https://eaglercraft.ru/dl/EaglerX_Offline.html
Browser-based Minecraft client (haven’t gotten connection to server working, linked below is a screenshot.) Minecraft 1.8.8
https://i.postimg.cc/Qt9SKD09/joinjeff-online.png

Update: Please look at Reverse_proxy for Browser-based compiled Minecraft client using WebSocket not working - #14 by devxan

2

try add http:// to instruct Caddy to only use http protocol when communicating with backend server.

go.jeffsnell.org:80, go.jeffsnell.org:443 {
    reverse_proxy http://172.96.140.17:25697
}
1 Like
3

Thank you so much! This totally worked and I’m going to put this onto a production server now.

1 Like
4

You made an HTTP request here, and the server responded with a Location header which instructs the client to try again (i.e. redirects the client) to https:// instead.

If you try curl -v https://go.jeffsnell.org it would avoid the redirect.

But here, we’re seeing that you’re not hitting Caddy at all, since we don’t see Server: Caddy.

My guess is you had incorrect DNS or port forwarding settings causing some other server to handle the request instead.

I guarantee this was not the fix. There’s absolutely no difference between having http:// in reverse_proxy upstreams and not having it. The default is HTTP for reverse_proxy.

You probably want to change this to simply go.jeffsnell.org.

Right now your config has two “servers” , one for port 80 and another for port 443. That’s unnecessary, and it means you’re allowing insecure connections to your server. Caddy will set up automatic HTTP->HTTPS redirects.

That’s a really old version. Please upgrade to v2.6.2, the latest. There’s a lot of bug fixes and new features since then.

I’m 99.99% sure adding http:// to the proxy was not the fix, and it only seemed like it. Something you did at the same time was probably the “fix”. For example DNS hadn’t propagated by the time you originally tested it, etc.

You didn’t actually post your Caddy logs at any point, so we’re in the dark on how Caddy was behaving.

2 Likes
5

Thanks for the reply, @francislavoie.

I changed the Caddyfile to

go.example.org {
  reverse_proxy http://172.96.140.17:25697
}

and this was the response from running curl -v https://go.example.org

*   Trying 34.149.204.188:443...
* Connected to go.example.org (34.149.204.188) port 443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
*  CAfile: /etc/ssl/certs/ca-certificates.crt
*  CApath: none
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* TLSv1.3 (IN), TLS handshake, Server hello (2):
* TLSv1.3 (IN), TLS handshake, Encrypted Extensions (8):
* TLSv1.3 (IN), TLS handshake, Certificate (11):
* TLSv1.3 (IN), TLS handshake, CERT verify (15):
* TLSv1.3 (IN), TLS handshake, Finished (20):
* TLSv1.3 (OUT), TLS change cipher, Change cipher spec (1):
* TLSv1.3 (OUT), TLS handshake, Finished (20):
* SSL connection using TLSv1.3 / TLS_AES_128_GCM_SHA256
* ALPN, server accepted to use h2
* Server certificate:
*  subject: CN=go.example.org
*  start date: Jan 10 04:24:33 2023 GMT
*  expire date: Apr 10 04:24:32 2023 GMT
*  subjectAltName: host "go.example.org" matched cert's "go.example.org"
*  issuer: C=US; O=Let's Encrypt; CN=R3
*  SSL certificate verify ok.
* Using HTTP2, server supports multiplexing
* Connection state changed (HTTP/2 confirmed)
* Copying HTTP/2 data in stream buffer to connection buffer after upgrade: len=0
* Using Stream ID: 1 (easy handle 0x101aa20)
> GET / HTTP/2
> Host: go.example.org
> user-agent: curl/7.80.0
> accept: */*
> 
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
* Connection state changed (MAX_CONCURRENT_STREAMS == 250)!
< HTTP/2 308 
< date: Tue, 10 Jan 2023 17:38:50 GMT
< expect-ct: max-age=2592000, report-uri="https://sentry.repl.it/api/10/security/?sentry_key=615192fd532445bfbbbe966cd7131791"
< location: https://go.example.org/
< replit-cluster: global
< server: Caddy
< content-length: 0
< 
* Connection #0 to host go.example.org left intact

Using the Caddyfile from the top of this reply, my client cannot access 172.96.140.17:25697.

Discourse disabled posting multiple images: https://i.postimg.cc/GmFqYD6k/joinjeff-online.png

Accessing wss://go.example.org worked with the Caddyfile configuration from @gyfer

This also didn’t work:

go.example.org {
  reverse_proxy 172.96.140.17:25697
}

A note on the version, I don’t know what replit uses for their platform like one listed in the dropdown on the binaries download page. (Actually, I forked a template with an old version of caddy)
When I tried Linux amd64, I got the following error when trying to run (./caddy_linux_amd64) it via the shell: bash: ./caddy_linux_amd64: Permission denied
Linux arm64 failed to upload to replit. These are the only binaries I’ve attempted to use right now.

Also, I’m using a regular CNAME record (with Cloudflare, not proxied) to replit: 415cb4c4-8584-4727-8b5f-223170d04e73.id.repl.co. Don’t know how that affects caddy, if at all.

6

That looks like the browser caching the 308 redirect more than the config change solving it.

8

I’m unable to clear my browser’s cache or use another browser. Does the Caddyfile config look right?

9

Why can’t you clear your browser cache? Or use another browser? Using incognito/private mode might also work. But I think what would be best would be to use curl -v https://... and make sure your browser is also making the request over HTTPS. Otherwise it’ll get a redirect to HTTPS, and if the client-side code can’t handle that, it’ll break the web app.

You can remove http:// from the reverse_proxy and it will still work.

Run caddy adapt --pretty with and without the https:// and let us know what differences you see. :pray:

1 Like
10

Using Caddyfile go.example.org { reverse_proxy 172.96.140.17:25697 }

New Caddyfile 
{
    "apps": {
        "http": {
            "servers": {
                "srv0": {
                    "listen": [
                        ":443"
                    ],
                    "routes": [
                        {
                            "match": [
                                {
                                    "host": [
                                        "go.example.org"
                                    ]
                                }
                            ],
                            "handle": [
                                {
                                    "handler": "subroute",
                                    "routes": [
                                        {
                                            "handle": [
                                                {
                                                    "handler": "reverse_proxy",
                                                    "upstreams": [
                                                        {
                                                            "dial": "172.96.140.17:25697"
                                                        }
                                                    ]
                                                }
                                            ]
                                        }
                                    ]
                                }
                            ],
                            "terminal": true
                        }
                    ]
                }
            }
        }
    }
}

Using Caddyfile go.example.org:80, go.example.org:443 { reverse_proxy http://172.96.140.17:25697 }

Working Caddyfile (with http:// behind reverse proxy) 
{
    "apps": {
        "http": {
            "servers": {
                "srv0": {
                    "listen": [
                        ":443"
                    ],
                    "routes": [
                        {
                            "match": [
                                {
                                    "host": [
                                        "go.example.org"
                                    ]
                                }
                            ],
                            "handle": [
                                {
                                    "handler": "subroute",
                                    "routes": [
                                        {
                                            "handle": [
                                                {
                                                    "handler": "reverse_proxy",
                                                    "upstreams": [
                                                        {
                                                            "dial": "172.96.140.17:25697"
                                                        }
                                                    ]
                                                }
                                            ]
                                        }
                                    ]
                                }
                            ],
                            "terminal": true
                        }
                    ]
                },
                "srv1": {
                    "listen": [
                        ":80"
                    ],
                    "routes": [
                        {
                            "match": [
                                {
                                    "host": [
                                        "go.example.org"
                                    ]
                                }
                            ],
                            "handle": [
                                {
                                    "handler": "subroute",
                                    "routes": [
                                        {
                                            "handle": [
                                                {
                                                    "handler": "reverse_proxy",
                                                    "upstreams": [
                                                        {
                                                            "dial": "172.96.140.17:25697"
                                                        }
                                                    ]
                                                }
                                            ]
                                        }
                                    ]
                                }
                            ],
                            "terminal": true
                        }
                    ]
                }
            }
        }

I ran curl -v https://go.example.org in replit’s shell. Here are those results:

New Caddyfile 
*   Trying 34.149.204.188:443...
* Connected to go.example.org (34.149.204.188) port 443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
*  CAfile: /etc/ssl/certs/ca-certificates.crt
*  CApath: none
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* TLSv1.3 (IN), TLS handshake, Server hello (2):
* TLSv1.3 (IN), TLS handshake, Encrypted Extensions (8):
* TLSv1.3 (IN), TLS handshake, Certificate (11):
* TLSv1.3 (IN), TLS handshake, CERT verify (15):
* TLSv1.3 (IN), TLS handshake, Finished (20):
* TLSv1.3 (OUT), TLS change cipher, Change cipher spec (1):
* TLSv1.3 (OUT), TLS handshake, Finished (20):
* SSL connection using TLSv1.3 / TLS_AES_128_GCM_SHA256
* ALPN, server accepted to use h2
* Server certificate:
*  subject: CN=go.example.org
*  start date: Jan 10 04:24:33 2023 GMT
*  expire date: Apr 10 04:24:32 2023 GMT
*  subjectAltName: host "go.example.org" matched cert's "go.example.org"
*  issuer: C=US; O=Let's Encrypt; CN=R3
*  SSL certificate verify ok.
* Using HTTP2, server supports multiplexing
* Connection state changed (HTTP/2 confirmed)
* Copying HTTP/2 data in stream buffer to connection buffer after upgrade: len=0
* Using Stream ID: 1 (easy handle 0x906a20)
> GET / HTTP/2
> Host: go.example.org
> user-agent: curl/7.80.0
> accept: */*
> 
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
* Connection state changed (MAX_CONCURRENT_STREAMS == 250)!
< HTTP/2 308 
< date: Tue, 10 Jan 2023 18:21:56 GMT
< expect-ct: max-age=2592000, report-uri="https://sentry.repl.it/api/10/security/?sentry_key=615192fd532445bfbbbe966cd7131791"
< location: https://go.example.org/
< replit-cluster: global
< server: Caddy
< content-length: 0
< 
* Connection #0 to host go.example.org left intact
Working Caddyfile (with http:// and specified ports) 
*   Trying 34.149.204.188:443...
* Connected to go.example.org (34.149.204.188) port 443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
*  CAfile: /etc/ssl/certs/ca-certificates.crt
*  CApath: none
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* TLSv1.3 (IN), TLS handshake, Server hello (2):
* TLSv1.3 (IN), TLS handshake, Encrypted Extensions (8):
* TLSv1.3 (IN), TLS handshake, Certificate (11):
* TLSv1.3 (IN), TLS handshake, CERT verify (15):
* TLSv1.3 (IN), TLS handshake, Finished (20):
* TLSv1.3 (OUT), TLS change cipher, Change cipher spec (1):
* TLSv1.3 (OUT), TLS handshake, Finished (20):
* SSL connection using TLSv1.3 / TLS_AES_128_GCM_SHA256
* ALPN, server accepted to use h2
* Server certificate:
*  subject: CN=go.example.org
*  start date: Jan 10 04:24:33 2023 GMT
*  expire date: Apr 10 04:24:32 2023 GMT
*  subjectAltName: host "go.example.org" matched cert's "go.example.org"
*  issuer: C=US; O=Let's Encrypt; CN=R3
*  SSL certificate verify ok.
* Using HTTP2, server supports multiplexing
* Connection state changed (HTTP/2 confirmed)
* Copying HTTP/2 data in stream buffer to connection buffer after upgrade: len=0
* Using Stream ID: 1 (easy handle 0xa1da20)
> GET / HTTP/2
> Host: go.example.org
> user-agent: curl/7.80.0
> accept: */*
> 
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
* Connection state changed (MAX_CONCURRENT_STREAMS == 250)!
< HTTP/2 200 
< cache-control: max-age=3600
< content-type: text/html; charset=utf-8
< date: 10 Jan 2023 18:17:21 GMT
< expect-ct: max-age=2592000, report-uri="https://sentry.repl.it/api/10/security/?sentry_key=615192fd532445bfbbbe966cd7131791"
< expires: 10 Jan 2023 19:17:21 GMT
< last-modified: 07 Jan 2023 20:14:47 GMT
< replit-cluster: global
< server: Caddy
< server: EaglerXBungee/1.0.1
< content-length: 289
< 
<!DOCTYPE html>
<html lang="en">
<head>
    <meta charset="UTF-8">
    <meta http-equiv="X-UA-Compatible" content="IE=edge">
    <meta name="viewport" content="width=device-width, initial-scale=1.0">
    <title>Document</title>
</head>
<body>
    <p>WHY DOESN'T IT WORK!?!?!?! 😭</p>
</body>
* Connection #0 to host go.example.org left intact
</html>

With the text above, I’m labeling “Working” as the client reached the Minecraft server.

11

This is telling me that the app you’re proxying to is triggering a redirect itself as well.

That sounds like the upstream app doesn’t understand that the request had already come in over HTTPS. Caddy passes the X-Forwarded-Proto: https header to the upstream which it should use to understand that.

Seems like a bug or a misconfiguration with your upstream app.

There might be a way to configure your upstream app to stop trying to serve HTTP->HTTPS redirects (i.e. expect connections over HTTP). Look into that.

You made two changes at once. Only do one at a time. Isolate the variables.

2 Likes
12

The IP address that go.example.org is going to changes the packets from a browser-based client to data that the Minecraft server can understand and vice versa (through a plugin on a bungeecord server).

13

Let me help a bit in narrowing down what happened.

First, a proper comparison of the change to the Caddyfile, where http:// is added, instead of comparing a change to the first line of the Caddyfile for some reason:

$ cat Caddyfile*
go.jeffsnell.org:80, go.jeffsnell.org:443 {
	reverse_proxy 172.96.140.17:25697
}
go.jeffsnell.org:80, go.jeffsnell.org:443 {
	reverse_proxy http://172.96.140.17:25697
}

$ diff <(caddy adapt --config Caddyfile --pretty) <(caddy adapt --config Caddyfile2 --pretty)

$

No difference. The two Caddyfiles are actually the exact same config.

So adding http:// was not the solution.

What is really happening is related to how the client/browser is making requests. I think Francis is onto something when he says:

A backend’s behavior can change if it is accessed by the frontend through HTTP versus HTTPS, even if the config is the same Browsers also do weird things related to caching and redirects, and JS code can do even weirder.

2 Likes
14

I’m using a different hosting platform now that runs on Ubuntu.
(Everything below is regarding that, posts above may no longer be a concern.)

This is the error I receive:

443 port already in use 
root@ifsmp-proxy:/home/ubuntu# caddy run
2023/01/10 22:55:59.734 INFO    using adjacent Caddyfile
2023/01/10 22:55:59.736 INFO    admin   admin endpoint started  {"address": "localhost:2019", "enforce_origin": false, "origins": ["//localhost:2019", "//[::1]:2019", "//127.0.0.1:2019"]}
2023/01/10 22:55:59.737 INFO    http    server is listening only on the HTTPS port but has no TLS connection policies; adding one to enable TLS {"server_name": "srv0", "https_port": 443}
2023/01/10 22:55:59.737 INFO    http    enabling automatic HTTP->HTTPS redirects    {"server_name": "srv0"}
2023/01/10 22:55:59.737 INFO    http.log    server running  {"name": "remaining_auto_https_redirects", "protocols": ["h1", "h2", "h3"]}
2023/01/10 22:55:59.737 INFO    http    enabling HTTP/3 listener    {"addr": ":443"}
2023/01/10 22:55:59.737 INFO    tls.cache.maintenance   started background certificate maintenance  {"cache": "0xc00047b500"}
2023/01/10 22:55:59.737 INFO    tls.cache.maintenance   stopped background certificate maintenance  {"cache": "0xc00047b500"}
Error: loading initial config: loading new config: http app module: start: listen udp :443: bind: address already in use
root@ifsmp-proxy:/home/ubuntu#

How do I make the TLS certificate work but not interrupt the reverse_proxy on the same port?
Here’s my Caddyfile right now:

:443 {
        reverse_proxy 172.96.140.17:25697
}
15

You’ll need to stop whatever other webserver is already running there so that Caddy can use port 443.

Or maybe you misunderstand how to run Caddy. If you installed it as a systemd service, Caddy is already running (and using port 443) so you should not run caddy run yourself.

16

It looks like only Caddy is using port 443:

~$ sudo lsof -i -P -n | grep LISTEN
systemd      1            root   48u  IPv4  15876      0t0  TCP *:111 (LISTEN)
systemd      1            root   50u  IPv6  15882      0t0  TCP *:111 (LISTEN)
rpcbind    643            _rpc    4u  IPv4  15876      0t0  TCP *:111 (LISTEN)
rpcbind    643            _rpc    6u  IPv6  15882      0t0  TCP *:111 (LISTEN)
systemd-r  735 systemd-resolve   14u  IPv4  21523      0t0  TCP 127.0.0.53:53 (LISTEN)
sshd      1131            root    3u  IPv4  22721      0t0  TCP *:22 (LISTEN)
sshd      1131            root    4u  IPv6  22723      0t0  TCP *:22 (LISTEN)
caddy     4211            root    3u  IPv4  48365      0t0  TCP 127.0.0.1:2019 (LISTEN)
caddy     4211            root    7u  IPv6  48369      0t0  TCP *:443 (LISTEN)
caddy     4211            root    9u  IPv6  47605      0t0  TCP *:80 (LISTEN)

~$ sudo lsof -i:443
COMMAND  PID USER   FD   TYPE DEVICE SIZE/OFF NODE NAME
caddy   4211 root    7u  IPv6  48369      0t0  TCP *:https (LISTEN)
caddy   4211 root    8u  IPv6  48370      0t0  UDP *:https

Oh, and about the system service. How would I view errors from Caddy when running caddy reload?

~$ systemctl status caddy
○ caddy.service - Caddy
     Loaded: loaded (/lib/systemd/system/caddy.service; enabled; vendor preset: enabled)
     Active: inactive (dead) since Tue 2023-01-10 21:16:41 UTC; 2h 2min ago
       Docs: https://caddyserver.com/docs/
    Process: 2864 ExecStart=/usr/bin/caddy run --environ --config /etc/caddy/Caddyfile (code=exited, status=0/SUCCESS)
   Main PID: 2864 (code=exited, status=0/SUCCESS)
        CPU: 1.265s
17

Same way as you see your logs. See the link I just sent, which explains how to look at your logs.

1 Like
18

This is implying that it’s not running as a systemd service right now.

Did you run caddy start at some point? If so, Caddy is running detached from anything else. You’ll need to kill that process, then start the systemd service again.

FYI, you didn’t say in your original post how you installed Caddy, so we had no way to know this might be the problem. So I had to make a blind guess that you did install via the apt repo. It’s important to give full and accurate information in the template, to save time and avoid lots of extra back-and-forth.

1 Like
19

I shutdown linux and then rebooted the instance. It’s running now :slight_smile:

Active: active (running) since Tue 2023-01-10 23:50:27 UTC; 34s ago

Oh sorry, in this new instance (not run with the old version of caddy running on replit, for that I had no idea how it was installed) I followed the documentation to install the stable ubuntu release.

1 Like
20

I think something’s up with Caddy’s SSL and(/or) how it interacts with http://172.96.140.17:25697/. How can I disable HTTP redirects using the Caddyfile? Also, what’s the best Caddyfile? Would it be this:

:443 {
        reverse_proxy 172.96.140.17:25697
}
21

You need a domain for TLS to work. Certificates need a valid domain name.

Simply using :443 as your site address without telling Caddy how to get valid certificates will not work.

Why do you want to turn off HTTP->HTTPS redirects? There’s rarely any good reason to do so.

2 Likes