How to setup reverse proxy with path for GenAI Text Gen and SD gradio webui?

1. The problem I’m having:

I am trying to create a reverse proxy for famous GenAI tools text-generation-webui and stable diffusion. I believe both of them use Gradio for webui. I configured text gen with the port 8000 for web and 8001 for RestAPI, SD with port 8003 for both web and webui.

Attempt 1:
caddy reverse-proxy --from my.domain.demo --to :8000
Result:
Open my.domain.demo and it works fine although textgen has warning:

Unsupported upgrade request.
No supported WebSocket library detected. Please use "pip install 'uvicorn[standard]'", or install 'websockets' or 'wsproto' manually.

Attempt 2:

my.domain.demo {
	log {
		output file /var/log/caddy/mydomaincom.log {
			roll_size 10mb
			roll_keep 20
			roll_keep_for 720h
		}
	}

	# Reverse proxy from http://127.0.0.0:8000 to https://my.domain.demo/llm
	reverse_proxy /llm localhost:8000 {
		header_up Host {host}
		header_up Connection {>Connection}
		header_up Upgrade {>Upgrade}
		header_up X-Real-IP {remote_host}
		header_up X-Forwarded-For {remote_host}
		header_up X-Forwarded-Proto {scheme}
		flush_interval -1
	}

	# Reverse proxy from http://127.0.0.0:8001 to https://my.domain.demo/llmapi
	reverse_proxy /llmapi/* localhost:8001 {
		header_up Host {host}
        header_up Connection {>Connection}
        header_up Upgrade {>Upgrade}
        flush_interval -1
	}

	# Reverse proxy from http://127.0.0.0:8003 to https://my.domain.demo/sd
	reverse_proxy /sd/* localhost:8003 {
	header_up Host {host}
        header_up Connection {>Connection}
        header_up Upgrade {>Upgrade}
        flush_interval -1
	}

	# Automatic TLS
	tls {
		on_demand
	}
}

Upload Caddyfile and run commands:

service caddy start
caddy adapt

Save output into caddy.json then:
curl localhost:2019/load -H "Content-Type: application/json" -d @caddy.json

It returns 404/no content.
Also tried adding websocket and handle_path /llm/* but still got same results.

2. Error messages and/or full log output:

https://my.domain.demo/llm and https://my.domain.demo/llmapi

HTTP 404
{
    "detail": "Not Found"
}

https://my.domain.demo/sd

HTTP 200
No content

3. Caddy version:

v2.7.6 h1:w0NymbG2m9PcvKWsrXO6EEkY9Ru4FJK8uQbYcev1p3A=

4. How I installed and ran Caddy:

a. System environment:

Azure VM
Ubuntu 22.04.3 LTS

b. Command:

apt update
sudo apt install caddy

c. Service/unit/compose file:

none

d. My complete Caddy config:

{"apps":{"http":{"servers":{"srv0":{"listen":[":443"],"logs":{"logger_names":{"my.domain.demo":"log0"}},"routes":[{"handle":[{"handler":"subroute","routes":[{"handle":[{"flush_interval":-1,"handler":"reverse_proxy","headers":{"request":{"set":{"Connection":["{\u003eConnection}"],"Host":["{http.request.host}"],"Upgrade":["{\u003eUpgrade}"],"X-Forwarded-For":["{http.request.remote.host}"],"X-Forwarded-Proto":["{http.request.scheme}"],"X-Real-Ip":["{http.request.remote.host}"]}}},"upstreams":[{"dial":"localhost:8001"}]}],"match":[{"path":["/llmapi/*"]}]},{"handle":[{"flush_interval":-1,"handler":"reverse_proxy","headers":{"request":{"set":{"Connection":["{\u003eConnection}"],"Host":["{http.request.host}"],"Upgrade":["{\u003eUpgrade}"],"X-Forwarded-For":["{http.request.remote.host}"],"X-Forwarded-Proto":["{http.request.scheme}"],"X-Real-Ip":["{http.request.remote.host}"]}}},"upstreams":[{"dial":"localhost:8003"}]}],"match":[{"path":["/sd/*"]}]},{"handle":[{"flush_interval":-1,"handler":"reverse_proxy","headers":{"request":{"set":{"Connection":["{\u003eConnection}"],"Host":["{http.request.host}"],"Upgrade":["{\u003eUpgrade}"],"X-Forwarded-For":["{http.request.remote.host}"],"X-Forwarded-Proto":["{http.request.scheme}"],"X-Real-Ip":["{http.request.remote.host}"]}}},"upstreams":[{"dial":"localhost:8000"}]}],"match":[{"path":["/llm"]}]}]}],"match":[{"host":["my.domain.demo"]}],"terminal":true}]}}},"tls":{"automation":{"policies":[{"on_demand":true,"subjects":["my.domain.demo"]}]}}},"logging":{"logs":{"default":{"exclude":["http.log.access.log0"]},"log0":{"include":["http.log.access.log0"],"writer":{"filename":"/var/log/caddy/mydomaincom.log","output":"file","roll_keep":20,"roll_keep_days":30,"roll_size_mb":10}}}}}

5. Links to relevant resources:

Is the webpage using ws:// or wss:// to establish the connection?

Remove all this stuff. It’s not useful, and invalid syntax.

I’m gonna guess you used ChatGPT or whatever to generate config. Don’t do that, LLMs have no idea how to write Caddy v2 config, it mixes concepts with Caddy v1 and gets it completely wrong.

This comment is backwards, it’s the other way around.

You should not turn on on_demand unless you don’t know your domain names ahead of time, which is not the case for you here.

You configured a matcher for /sd/* but made a request to /sd. That doesn’t match. You’d need to match /sd* for it to match.

You don’t have any handling for anything other than those 3 paths, so non-matching requests get served with no content because Caddy wasn’t configured to do anything else.

Thank you for promptly reply.

Yes I used chatgpt to generate Caddy script according to googled posts for Nginx, which might be necessary for Gradio I explain later.

First of all, I removed ‘unnecessary code’ and only kept the following lines:
Version 1:

my.domain.demo {
    reverse_proxy /llm* localhost:8000
    reverse_proxy /llmapi* localhost:8001
    reverse_proxy /sd* localhost:8003
}

It corrects /sd so that all 3 paths return HTTP 404 { "detail": "Not Found"} instead of no content.

I guess I have to remove the prefix like this post:
Version 2:

my.domain.demo {
	handle_path /llm* {
		reverse_proxy localhost:8000
	}

	handle_path /llmapi* {
		reverse_proxy localhost:8001
	}

	handle_path /sd* {
		reverse_proxy localhost:8003
	}
}

Results:

  1. , /llm page can be loaded but cannot displayed because resources cannot be loaded. I can see the source
    <script type="module" crossorigin src="./assets/index-a959df42.js"></script>
    Dynamically generated incorrect path
    <link rel="stylesheet" href="https://my.domain.demo/theme.css">

  2. …/llmapi/models returns Invalid HTTP request received.

  3. …/sd returns http 200 OK with no content.

I guess there must be something missing. According to the nginx conf for Gradio and another similar post on github, I should write Caddy config like:

...
proxy_set_header Upgrade $http_upgrade;
proxy_set_header Connection $http_connection;
proxy_http_version 1.1;

  location / {
    proxy_set_header Host $host;
    proxy_set_header X-Forwarded-Scheme $scheme;
    proxy_set_header X-Forwarded-Proto  $scheme;
    proxy_set_header X-Forwarded-For    $remote_addr;
    proxy_set_header X-Real-IP		$remote_addr;
    proxy_pass       http://10.241.2.1:7860;

    # Force SSL
    include conf.d/include/force-ssl.conf;

    proxy_set_header Upgrade $http_upgrade;
    proxy_set_header Connection $http_connection;
    proxy_http_version 1.1;
  }

Or

    location / {
        # Serve GRADIO 7860
        proxy_pass http://localhost:7860;
        proxy_http_version 1.1;
        proxy_set_header Upgrade $http_upgrade;
        proxy_set_header Connection 'upgrade';
        proxy_set_header Host $host;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_cache_bypass $http_upgrade;
        proxy_read_timeout 86400;
        proxy_redirect off;
    }

I don’t know how to write them for Caddy so I asked ChatGPT to translate it to Caddyfile format then got the version 3 in my original post.

@matt I am not a developer using Gradio/Python/Javascript but I guess it uses websocket from the error message.

See this article:

If you use handle_path to strip the path prefix, then your upstream app needs to be aware of that path prefix to properly build paths. If the app doesn’t have somekind of “base path” config, then you shouldn’t use subpaths like this, and using subdomains for each part is a better option.

The nginx config is only proxying to the one port, shouldn’t you just do that as well? Just have one reverse_proxy with no matcher.