Reverse_proxy with WebSockets (was Reverse_proxy with transparent upstream auth)

Hi,

I want to set up Caddy with reverse_proxy where Caddy authenticates against the upstream HTTPS server using NTLM. Clients should then be able to access Caddy over HTTP without authentication. In other words, I wan’t the Caddy/proxy handle the authentication so that the client does not have to.

Caddyfile pseudo code:

http://local-server {
   reverse_proxy https://upstream-server {
      transport http {
         tls_insecure_skip_verify
      }
      <NTLM upstream auth>
   }
}

Is this at all within the scope of Caddy to handle? I did not fill in the template because I am not at all sure if this is the right way to go at the moment. If you think it might be, I’ll be happy to put in all details.

Another option I have been looking at is the cntlm tool, but it is very old, and it is meant to work against upstream proxies and not directly with webservers (I think).

Thank you.

Is this what you need?

I am not sure. I had a look at the ntlm.go but cannot see how to provide username and password. It looks like it want to pass that on to the client, which is not what I need in this case.

I got the NTLM bits disabled, so not I am just doing a plain reverse_proxy setup, though it is not working. Static resources from the remote server loads fine, but websockets do not work.

1. The problem I’m having:

I am setting up a reverse proxy with Caddy to allow a legacy client access an web application on another server. The problem I am facing is an http status 500 for websocket requests.

2. Error messages and/or full log output:

Chrome shows this error in the webtools console:

WebSocket connection to 'ws://my-caddy-server.com:9991/system/pws/Andon_DX/253f4d88?token=Z6jj_b_9psYIt7lSJDsxtWN74e1Y4mAVO3RNOW1JuWc' failed: Error during WebSocket handshake: Unexpected response code: 500

Caddy logs show many rows like this:

{
	"level": "info",
	"ts": 1708949045.1448529,
	"logger": "http.log.access.log0",
	"msg": "handled request",
	"request": {
		"remote_ip": "10.12.143.250",
		"remote_port": "61285",
		"client_ip": "10.12.143.250",
		"proto": "HTTP/1.1",
		"method": "GET",
		"host": "my-caddy-server.com:9991",
		"uri": "/data/perspective/hello/Andon_DX/253f4d88",
		"headers": {
			"Accept-Language": [
				"en-US,en;q=0.9"
			],
			"Connection": [
				"keep-alive"
			],
			"Version-Code": [
				"0"
			],
			"Accept": [
				"application/json, text/plain, */*"
			],
			"Device-Id": [
				"86db965b-57c3-48a6-9805-c5a4fd89ec9b"
			],
			"User-Agent": [
				"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/116.0.0.0 Safari/537.36 Edg/116.0.1938.62"
			],
			"Referer": [
				"http://my-caddy-server.com:9991/data/perspective/client/Andon_DX/disign"
			],
			"Device-Type": [
				"browser"
			],
			"Client-Timezone": [
				"Europe/Stockholm"
			],
			"Accept-Encoding": [
				"gzip, deflate"
			]
		}
	},
	"bytes_read": 0,
	"user_id": "",
	"duration": 0.0037787,
	"size": 588,
	"status": 200,
	"resp_headers": {
		"Content-Length": [
			"588"
		],
		"Date": [
			"Mon, 26 Feb 2024 12:04:05 GMT"
		],
		"Referrer-Policy": [
			"strict-origin-when-cross-origin"
		],
		"X-Content-Type-Options": [
			"nosniff"
		],
		"Cache-Control": [
			"no-cache, no-store"
		],
		"Content-Type": [
			"application/json"
		],
		"Server": [
			"Caddy"
		],
		"Expires": [
			"Thu, 01 Jan 1970 00:00:00 GMT"
		],
		"X-Frame-Options": [
			"SAMEORIGIN"
		],
		"X-Xss-Protection": [
			"1; mode=block"
		],
		"Pragma": [
			"no-cache"
		],
		"Set-Cookie": [
			"JSESSIONID=node01vqzbicaejylo16dn80mynseu8616.node0; Path=/; Secure; HttpOnly"
		]
	}
}
{
	"level": "error",
	"ts": 1708949045.1563049,
	"logger": "http.log.access.log0",
	"msg": "handled request",
	"request": {
		"remote_ip": "10.12.143.250",
		"remote_port": "62960",
		"client_ip": "10.12.143.250",
		"proto": "HTTP/1.1",
		"method": "GET",
		"host": "my-caddy-server.com:9991",
		"uri": "/system/pws/Andon_DX/253f4d88?token=ONukfWtus3fmcTF_2ErxFNUlwKZrJbxM_dKTwDiy-E4",
		"headers": {
			"Connection": [
				"Upgrade"
			],
			"Cache-Control": [
				"no-cache"
			],
			"User-Agent": [
				"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/116.0.0.0 Safari/537.36 Edg/116.0.1938.62"
			],
			"Upgrade": [
				"websocket"
			],
			"Sec-Websocket-Version": [
				"13"
			],
			"Sec-Websocket-Extensions": [
				"permessage-deflate; client_max_window_bits"
			],
			"Origin": [
				"http://my-caddy-server.com:9991"
			],
			"Accept-Encoding": [
				"gzip, deflate"
			],
			"Accept-Language": [
				"en-US,en;q=0.9"
			],
			"Sec-Websocket-Key": [
				"gTko2CkUBXfW1BDQRZoKhw=="
			],
			"Pragma": [
				"no-cache"
			]
		}
	},
	"bytes_read": 0,
	"user_id": "",
	"duration": 0.0075772,
	"size": 357,
	"status": 500,
	"resp_headers": {
		"Content-Length": [
			"357"
		],
		"Referrer-Policy": [
			"strict-origin-when-cross-origin"
		],
		"X-Content-Type-Options": [
			"nosniff"
		],
		"X-Frame-Options": [
			"SAMEORIGIN"
		],
		"Server": [
			"Caddy"
		],
		"X-Xss-Protection": [
			"1; mode=block"
		],
		"Cache-Control": [
			"must-revalidate,no-cache,no-store"
		],
		"Content-Type": [
			"text/html;charset=iso-8859-1"
		]
	}
}

3. Caddy version:

2.7.6

4. How I installed and ran Caddy:

Downloaded the Windows binary from GitHub.

a. System environment:

Windows 10 x64

b. Command:

C:\Caddy\caddy.exe start -c c:\Caddy\Caddyfile

d. My complete Caddy config:

#main
{
	auto_https off
	auto_https disable_redirects
	log {
		output file C:\Caddy\caddy.log {
			roll_size 10M
		}
	}
	servers {
		log_credentials
	}
}

http://my-caddy-server.com:9991 {
	log {
		format json
		output file C:\Caddy\dxsrv11.log {
			roll_size 10M
		}
	}

	reverse_proxy https://upsteam-server:443 {
	}
}

I’ve also tried setting header_up Host {upstream_hostport} in the reverse_proxy section.

The 500 status error is coming from your upstream app. You’ll need to see what it’s logging. Doesn’t seem like a Caddy problem.

It is weird though. When the client connects directly against the server there is no error, but when I connect via Caddy, I get an error with WebSockets.

The server software is Ignition and it’s logging features is not the easiest to use. This is what I was able to export. Looks like there is some problem with Websocket upgrades. Looking at the Caddy logs I did not see Upgrade: websocket, so maybe it is something causing Caddy or the client/server to not send those headers?

java.lang.UnsupportedOperationException: Feature unsupported after Upgraded to WebSocket
	at org.eclipse.jetty.websocket.core.server.internal.UpgradeHttpServletRequest.getSession(UpgradeHttpServletRequest.java:293)
	at com.inductiveautomation.perspective.gateway.comm.Routes.getOrCreateSession(Routes.java:1728)
	at com.inductiveautomation.perspective.gateway.comm.PerspectiveWebSocketServlet$PerspectiveWebSocketCreator.createWebSocket(PerspectiveWebSocketServlet.java:185)
	at com.inductiveautomation.perspective.gateway.comm.PerspectiveWebSocketServlet$PerspectiveWebSocketCreator.createWebSocket(PerspectiveWebSocketServlet.java:123)
	at org.eclipse.jetty.websocket.server.JettyWebSocketServlet$WrappedJettyCreator.createWebSocket(JettyWebSocketServlet.java:276)
	at org.eclipse.jetty.websocket.core.server.internal.CreatorNegotiator.lambda$negotiate$0(CreatorNegotiator.java:64)
	at org.eclipse.jetty.server.handler.ContextHandler.handle(ContextHandler.java:1450)
	at org.eclipse.jetty.server.handler.ContextHandler.handle(ContextHandler.java:1487)
	at org.eclipse.jetty.websocket.core.server.internal.CreatorNegotiator.negotiate(CreatorNegotiator.java:63)
	at org.eclipse.jetty.websocket.core.server.internal.AbstractHandshaker.upgradeRequest(AbstractHandshaker.java:76)
	at org.eclipse.jetty.websocket.core.server.internal.HandshakerSelector.upgradeRequest(HandshakerSelector.java:39)
	at org.eclipse.jetty.websocket.core.server.WebSocketMappings.upgrade(WebSocketMappings.java:241)
	at org.eclipse.jetty.websocket.server.JettyWebSocketServlet.service(JettyWebSocketServlet.java:181)
	at javax.servlet.http.HttpServlet.service(HttpServlet.java:590)
	at com.inductiveautomation.ignition.gateway.bootstrap.MapServlet.service(MapServlet.java:86)
	at org.eclipse.jetty.servlet.ServletHolder$NotAsync.service(ServletHolder.java:1410)
	at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:764)
	at org.eclipse.jetty.servlet.ServletHandler$ChainEnd.doFilter(ServletHandler.java:1665)
	at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:527)
	at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:131)
	at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:578)
	at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:122)
	at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:223)
	at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1570)
	at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:221)
	at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1383)
	at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:176)
	at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:484)
	at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1543)
	at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:174)
	at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1305)
	at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:129)
	at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:122)
	at com.inductiveautomation.catapult.handlers.RemoteHostNameLookupHandler.handle(RemoteHostNameLookupHandler.java:121)
	at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:122)
	at org.eclipse.jetty.rewrite.handler.RewriteHandler.handle(RewriteHandler.java:301)
	at org.eclipse.jetty.server.handler.HandlerList.handle(HandlerList.java:51)
	at org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:141)
	at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:122)
	at org.eclipse.jetty.server.Server.handle(Server.java:563)
	at org.eclipse.jetty.server.HttpChannel.lambda$handle$0(HttpChannel.java:505)
	at org.eclipse.jetty.server.HttpChannel.dispatch(HttpChannel.java:762)
	at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:497)
	at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:282)
	at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:314)
	at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:100)
	at org.eclipse.jetty.io.ssl.SslConnection$DecryptedEndPoint.onFillable(SslConnection.java:558)
	at org.eclipse.jetty.io.ssl.SslConnection.onFillable(SslConnection.java:379)
	at org.eclipse.jetty.io.ssl.SslConnection$2.succeeded(SslConnection.java:146)
	at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:100)
	at org.eclipse.jetty.io.SelectableChannelEndPoint$1.run(SelectableChannelEndPoint.java:53)
	at org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.runTask(AdaptiveExecutionStrategy.java:416)
	at org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.consumeTask(AdaptiveExecutionStrategy.java:385)
	at org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.tryProduce(AdaptiveExecutionStrategy.java:272)
	at org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.lambda$new$0(AdaptiveExecutionStrategy.java:140)
	at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:411)
	at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:934)
	at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1078)
	at java.base/java.lang.Thread.run(Unknown Source)

I found an almost identical issue, but with the Apache server. The solution there was to enable mod_proxy_wstunnel:

From your logs in OP :point_down:

I don’t think this is a problem with Caddy. Or if it is, we don’t have any evidence that it is.

You’ll need to get help from Java communities or whatever for them to explain what that error means.

Yea, it’s annoying to not have a clear answer what is the actual problem. For example could it be that my client connects via HTTP while caddy connects to upstream via HTTPS? It could be other stuff like difference in hostnames between what the client connects to and what the server expects, and so on. I’ll see if I can get some more details on the error from Ignition.

Seems the issue is that the client is connecting to Caddy via HTTP while Caddy connects to upstream via HTTPS. I generated a self-signed certificate to use with Caddy and now it works. Unfortunately the clients cannot opt to support non-trusted certificates, so I am a little stuck at the moment.

If you use a real domain that resolves to your LAN IP, you could use the DNS challenge to issue a publicly trusted cert. You could get a DuckDNS domain for example, it’s free, and the DNS plugin works well.

The solution was in the end to get the upstream to accept http connections. Now everything works well.

Thanks!

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.