Trying to use reverse_proxy to answer tls on_demand ask requests (correctly)

thedaveCA · August 11, 2023, 12:49am

1. The problem I’m having:

I have a backend server that will respond with either a 200 for a configured domain, or a 302 for a domain that is not configured.

The backend has no trivial way to implement the tls on_demand ask request directly but since I can get a 200 or a 302 from requests to / I thought I might be able to do this with a reverse_proxy.

A nice simple Caddyfile:

{
        debug
        auto_https disable_redirects
        ocsp_stapling off
}
babysitter.razx.com {
        # Grab the /check?domain= requests
        handle_path /check* {
                reverse_proxy http://127.0.0.1:3001 {
                        header_down X-UP-Host {http.request.uri.query.domain}
                        rewrite /
                        header_up Host {http.request.uri.query.domain}
                }
        }
        # Default handler
        #        reverse_proxy http://127.0.0.1:3001
}

I think I have the backend doing what I want, but I don’t get the expected result proxied.

I am testing using a curl request that should be reasonably similar to what I expect to be proxied (the backend is ignoring the ?domain=status.razx.com, but for consistency it is included in this query – I get the same result without it)

dave@babysitter:~$ curl --silent --resolve status.razx.com:3001:127.0.0.1 --head -X GET http://status.razx.com:3001/?domain=status.razx.com
HTTP/1.1 200 OK
X-Frame-Options: SAMEORIGIN
Content-Type: text/html; charset=utf-8
Content-Length: 6477
ETag: W/"194d-8R5HOeOMCjLmTsPxQpmttgMUEZQ"
Date: Fri, 11 Aug 2023 00:11:48 GMT
Connection: keep-alive
Keep-Alive: timeout=5

And then the results via caddy:

dave@babysitter:~$ curl --silent --resolve babysitter.razx.com:443:127.0.0.1 --head -X GET https://babysitter.razx.com/check?domain=status.razx.com
HTTP/2 302
alt-svc: h3=":443"; ma=2592000
content-type: text/plain; charset=utf-8
date: Fri, 11 Aug 2023 00:11:50 GMT
location: /dashboard
server: Caddy
vary: Accept
x-frame-options: SAMEORIGIN
x-up-host: status.razx.com
content-length: 32

And of course a site that does not exist is doing what I expect (caddy example only)

dave@babysitter:~$ curl --silent --resolve babysitter.razx.com:443:127.0.0.1 --head -X GET https://babysitter.razx.com/check?domain=nope.razx.com
HTTP/2 302
alt-svc: h3=":443"; ma=2592000
content-type: text/plain; charset=utf-8
date: Fri, 11 Aug 2023 00:13:17 GMT
location: /dashboard
server: Caddy
vary: Accept
x-frame-options: SAMEORIGIN
x-up-host: nope.razx.com
content-length: 32

2. Debug log

2023/08/10 23:45:30.083 DEBUG   http.handlers.rewrite   rewrote request {"request": {"remote_ip": "127.0.0.1", "remote_port": "42994", "proto": "HTTP/2.0", "method": "GET", "host": "babysitter.razx.com", "uri": "/check?domain=status.razx.com", "headers": {"User-Agent": ["curl/7.81.0"], "Accept": ["*/*"]}, "tls": {"resumed": false, "version": 772, "cipher_suite": 4865, "proto": "h2", "server_name": "babysitter.razx.com"}}, "method": "GET", "uri": "/?domain=status.razx.com"}
2023/08/10 23:45:30.084 DEBUG   http.handlers.reverse_proxy     selected upstream       {"dial": "127.0.0.1:3001", "total_upstreams": 1}
2023/08/10 23:45:30.090 DEBUG   http.handlers.reverse_proxy     upstream roundtrip      {"upstream": "127.0.0.1:3001", "duration": 0.005441887, "request": {"remote_ip": "127.0.0.1", "remote_port": "42994", "proto": "HTTP/2.0", "method": "GET", "host": "status.razx.com", "uri": "/?domain=status.razx.com", "headers": {"User-Agent": ["curl/7.81.0"], "Accept": ["*/*"], "X-Forwarded-For": ["127.0.0.1"], "X-Forwarded-Proto": ["https"], "X-Forwarded-Host": ["babysitter.razx.com"]}, "tls": {"resumed": false, "version": 772, "cipher_suite": 4865, "proto": "h2", "server_name": "babysitter.razx.com"}}, "headers": {"X-Frame-Options": ["SAMEORIGIN"], "Location": ["/dashboard"], "Content-Length": ["32"], "Date": ["Thu, 10 Aug 2023 23:45:30 GMT"], "Connection": ["keep-alive"], "Keep-Alive": ["timeout=5"], "Vary": ["Accept"], "Content-Type": ["text/plain; charset=utf-8"]}, "status": 302}

I think this is (similar enough to) my curl request directly to the backend?


upstream	`127.0.0.1:3001`
method	`GET`
host	`status.razx.com`
uri	`/?domain=status.razx.com`

3. Caddy version:

~~latest~~v2.7.3 h1:eMCNjOyMgB5A1KgOzT2dXKR4I0Va+YHCJYC8HHu+DP0=

a. System environment:

ubuntu jammy
caddy installed via apt via https://dl.cloudsmith.io/public/caddy/stable/deb/debian

b. Command:

sudo /usr/bin/caddy run --environ --config /etc/caddy/Caddyfile

I have a feeling this is going to be one of those “duh, obviously” typos because I’ve been staring at this for too long. Either way, thanks for your time!

francislavoie · August 11, 2023, 3:37am

Why 302? You can return a 4xx status, by the way. The ask endpoint won’t follow redirects.

Why not?

I’m kinda confused, I don’t see on_demand in there. Do you have two Caddy instances?

You can shorten this to {query.domain} btw.

I don’t think this is needed, because handle_path will strip away /check already.

This looks fine, seems like Host was properly updated.

Is your upstream reading from X-Forwarded-Host or something maybe?

You’ll probably need to do some trace debugging through your upstream to figure out what causes it to redirect. I don’t think it’s something I can answer without knowing intimately what it’s doing.

thedaveCA · August 11, 2023, 5:44am

(Paraphrasing the quote slightly so that I can reply in context)

There doesn’t happen to be a call that will explicitly answer the question in the backend.

I would like to avoid forking the project as that creates a permanent maintenance task in a language I’m not familiar with. This change is simple enough that I could probably implement it, but…

Frankly a few lines in Caddy’s config is less maintenance hassle and once I found a way to get a different response code from the “wanted” domains vs everything else it seemed like this would be a quick one-shot-and-finished without any ongoing maintenance required.

This seemed like “good clever”.

It’s not there yet. My eventual goal is to use a wildcard, so I need a way to answer the ask before going further.

Normally there’s more to the production Caddyfile, but I reduced it to the minimum possible configuration. To be clear, this is the entirety of the config file while I generated the debugging lines for this thread, otherwise I’d be risking wasting your time.

Good point.

I was mucking around with other ways to handle the request in a more complex version of the config and it got left behind. I hope to just stick with handle_path at this point.

I totally appreciate your time, and as long as I’m doing things properly in Caddy I don’t want to waste any more of it.

This is my first Caddy server and the whole thing is barely above a lab server. Although I’ve got a few other Caddy servers running now, I’m still new enough to expect to make genuine n00b mistakes.

Capturing the raw request(s) to compare curl vs caddy as seen by the backend was going to be my next step, and I’ll start with anything that mentions any domain names.

I think I ruled out X-Forwarded-Host and other headers early on as when I started I was testing from an external location (using a valid target domain and injecting the “bad” domain into the Host header), so it should have been the reverse (false positives instead of false negatives). But I might have missed something then.

I’ve got a direction to go, and I appreciate your time!

system · September 10, 2023, 5:45am

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.