Reverse Proxy routing based on directory in URL

Help
1

1. Caddy version (caddy version): 2.4.6

2. How I run Caddy:

I’m running Caddy on Ubuntu 20.06 using a Caddyfile in /etc/caddy/Caddyfile.

a. System environment:

AWS Lightsail, Unbuntu 20.06, with latest apt update + apt upgrade.

b. Command:

To start Caddy during a restart I’m using (whilst in /etc/caddy/):
caddy stop
caddy run

Paste command here.

d. My complete Caddyfile or JSON config:

{
email myemail@mymaindomain.com
http_port 80
https_port 443
}


site.mymaindomain.com {
reverse_proxy 11.22.33.44:80
reverse_proxy /nr/* 55.66.77.88:1880
file_server
}

3. The problem I’m having:

What I’m trying to achieve is: to have all requests to site.mymaindomain.com routed to ip:port 11.22.33.44:80, whilst any request to site.mymaindomain.com/nr gets routed to a different server, 55.66.77.88:1880.

Neither the first nor the second server has SSL enabled. Caddy is to handle the SSL.

4. Error messages and/or full log output:

No errors are being returned in the logs when debug is turned on by placing ‘debug’ in the global section of the caddy file. I believe this indicates my caddyfile is working as I’m telling it to, and I’m not telling it to do what I’m afte r correctly.

5. What I already tried:

I’ve tried breaking out the domain into a new domain with a trailing directory.

site.mymaindomain.com/nr {
    55.66.77.88:1880
    file_server
}

site.mymaindomain.com {
    11.22.33.44 
    file_server
}

I have also tried varying where the slashes are placed in the config, per the below:

{
email myemail@mymaindomain.com
http_port 80
https_port 443
}


site.mymaindomain.com {
reverse_proxy /* 11.22.33.44:80
reverse_proxy /nr* 55.66.77.88:1880
file_server
}

Any help you might be able to supply in getting the Caddyfile set up correctly would be most appreciated. The goal is to route a “directory” to a different server.

2

What are you seeing happen instead of what you expect, though?

The debug logs would show you which upstream is being connected to for a given request.

Please also make requests with curl -v and show us the output, if it’s not doing what you want.

Configuring file_server doesn’t make sense for you here, since you’re wanting to proxy to 2 different upstreams.

Also, file_server should almost always be paired with the root directive to tell Caddy where to read files from, if you needed it.

Path matchers are exact in Caddy, so this would only match requests to /nr but not /nr/foo.

Also, I strongly recommend not using path matchers in the site address, I consider it an unfortunate legacy behaviour, and I hope to deprecate and remove it later.

You can remove these, it’s redundant. These are already the defaults.

3

Thank you for the response.

Currently requests to mymaindomain.com resolve to 11.22.33.44:80 correctly, and my applications pages are delivered ok.
Requests to mymaindomain.com/nr are met with a Http Error 502. Hitting 55.66.77.88:1880 as I’m trying to direct in the log files results in successful delivery of the application, however.

Are you able to provide an example of what I should be doing instead of the path matches? I’ve tried a number of things I can see from the documentation, but don’t seem to be getting anywhere. I do not control DNS, so cannot affect CNAME changes, like adding subdomains, etc. Controlling for path is, however, something I can use. The intention is to have a request to the domain, or any sub folder that doesn’t include /nr resolve to one application server that is serving a site on :80, and a request to the domain that includes /nr to point to another application server that is serving an application on :1880

When trying mymaindomain.com/nr I recieve a status message:

Cannot GET /nr

I have changed the Caddyfile to reflect the suggestions I understand.

    {
        email myemail@mymaindomain.com
        debug
    }
    
    
    site.mymaindomain.com {
        reverse_proxy /nr* http://55.66.77.88:1880
        reverse_proxy /* 11.22.33.44:80
    }

Using debug from this point on returns the following:
At the outset it looks like the remote server is refusing the connection - but the thing is if I hit 55.66.77.88:1880 then the server will provide the intended application just fine.

2022/03/08 10:16:08.885 INFO    using adjacent Caddyfile
2022/03/08 10:16:08.888 WARN    input is not formatted with 'caddy fmt' {"adapter": "caddyfile", "file": "Caddyfile", "line": 2}
2022/03/08 10:16:08.889 INFO    admin   admin endpoint started  {"address": "tcp/localhost:2019", "enforce_origin": false, "origins": ["localhost:2019", "[::1]:2019", "127.0.0.1:2019"]}
2022/03/08 10:16:08.889 INFO    http    server is listening only on the HTTPS port but has no TLS connection policies; adding one to enable TLS   {"server_name": "srv0", "https_port": 443}
2022/03/08 10:16:08.889 INFO    http    enabling automatic HTTP->HTTPS redirects        {"server_name": "srv0"}
2022/03/08 10:16:08.890 DEBUG   http    starting server loop    {"address": "[::]:443", "http3": false, "tls": true}
2022/03/08 10:16:08.890 DEBUG   http    starting server loop    {"address": "[::]:80", "http3": false, "tls": false}
2022/03/08 10:16:08.890 INFO    http    enabling automatic TLS certificate management   {"domains": ["mymaindomain.com"]}
2022/03/08 10:16:08.890 DEBUG   tls     loading managed certificate     {"domain": "mymaindomain.com", "expiration": "2022/06/06 05:59:02.000", "issuer_key": "acme-v02.api.letsencrypt.org-directory", "storage": "FileStorage:/home/ec2-user/.local/share/caddy"}
2022/03/08 10:16:08.891 DEBUG   tls.cache       added certificate to cache      {"subjects": ["mymaindomain.com"], "expiration": "2022/06/06 05:59:02.000", "managed": true, "issuer_key": "acme-v02.api.letsencrypt.org-directory", "hash": "c9450812e869aceaaf34d68a283fb7133aae411e7fa15985efe6cc5c137c0e72", "cache_size": 1, "cache_capacity": 10000}
2022/03/08 10:16:08.891 INFO    autosaved config (load with --resume flag)      {"file": "/home/ec2-user/.config/caddy/autosave.json"}
2022/03/08 10:16:08.891 INFO    serving initial configuration
2022/03/08 10:16:08.891 INFO    tls.cache.maintenance   started background certificate maintenance      {"cache": "0xc00053dea0"}
2022/03/08 10:16:08.891 INFO    tls     cleaning storage unit   {"description": "FileStorage:/home/ec2-user/.local/share/caddy"}
2022/03/08 10:16:08.892 INFO    tls     finished cleaning storage units
2022/03/08 10:16:11.591 DEBUG   tls.handshake   choosing certificate    {"identifier": "mymaindomain.com", "num_choices": 1}
2022/03/08 10:16:11.591 DEBUG   tls.handshake   default certificate selection results   {"identifier": "mymaindomain.com", "subjects": ["mymaindomain.com"], "managed": true, "issuer_key": "acme-v02.api.letsencrypt.org-directory", "hash": "c9450812e869aceaaf34d68a283fb7133aae411e7fa15985efe6cc5c137c0e72"}
2022/03/08 10:16:11.591 DEBUG   tls.handshake   matched certificate in cache    {"subjects": ["mymaindomain.com"], "managed": true, "expiration": "2022/06/06 05:59:02.000", "hash": "c9450812e869aceaaf34d68a283fb7133aae411e7fa15985efe6cc5c137c0e72"}
2022/03/08 10:16:11.732 DEBUG   http.handlers.reverse_proxy     upstream roundtrip      {"upstream": "55.66.77.88:1880", "duration": 0.004099742, "request": {"remote_addr": "43.245.160.163:50239", "proto": "HTTP/2.0", "method": "GET", "host": "mymaindomain.com", "uri": "/nr", "headers": {"Sec-Fetch-Dest": ["document"], "Cookie": ["_ga=GA1.4.108457131.1645305609; wf_loginalerted_49ffc0b821102931d068b7166f82c16af328fc58e14540c4a7fe41fc4ed9961e=6dc650d3040c72c96bbc12dc093fbf63eee41e51c13ba920a3a1d39806e63bbd; wp-settings-1=libraryContent%3Dbrowse%26advImgDetails%3Dshow%26urlbutton%3Dnone%26editor%3Dtinymce%26siteorigin_panels_setting_tab%3Dgeneral%26hidetb%3D1%26posts_list_mode%3Dlist; wp-settings-time-1=1645305631; visid_incap_724216=BNkOBZJNTqSTYZmToZ+uTLB4FGIAAAAAQUIPAAAAAAB6AVp340sE9I0ysx8Es8xB; wf_loginalerted_e0a8157f9e573d5cff9a1e1fe27babd01a5e168da0bbffb47e99a7da12e719a5=6146fbb2dca4718dc23fef454dfc2e257fbd3a762cf3af54dc280b9053d156bd; wordpress_test_cookie=WP%20Cookie%20check; wordpress_logged_in_377fc503a3d94ab6ba73dd7ea7e0da85=catalystadmin01%7C1646572960%7CogzOTLVesr0OQTA3ubjuu2W1zu4uWvY3YfODAzZcS0T%7C48080dc9bd86f9f4c4cdee13d6a2873a7e7ed6daaf92a93441ed0dddcbf73762; wp_lang=en_US; _gid=GA1.4.1565826196.1646733499"], "Sec-Fetch-Mode": ["navigate"], "Sec-Ch-Ua-Mobile": ["?0"], "User-Agent": ["Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/98.0.4758.102 Safari/537.36"], "Accept": ["text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.9"], "Sec-Fetch-Site": ["none"], "Accept-Language": ["en-AU,en-GB;q=0.9,en-US;q=0.8,en;q=0.7"], "Cache-Control": ["max-age=0"], "Upgrade-Insecure-Requests": ["1"], "Sec-Ch-Ua-Platform": ["\"Windows\""], "Sec-Fetch-User": ["?1"], "Accept-Encoding": ["gzip, deflate, br"], "X-Forwarded-For": ["43.245.160.163"], "X-Forwarded-Proto": ["https"], "Sec-Ch-Ua": ["\" Not A;Brand\";v=\"99\", \"Chromium\";v=\"98\", \"Google Chrome\";v=\"98\""]}, "tls": {"resumed": false, "version": 772, "cipher_suite": 4865, "proto": "h2", "proto_mutual": true, "server_name": "mymaindomain.com"}}, "headers": {"Content-Security-Policy": ["default-src 'none'"], "Content-Type": ["text/html; charset=utf-8"], "Connection": ["keep-alive"], "Keep-Alive": ["timeout=5"], "Access-Control-Allow-Origin": ["*"], "X-Powered-By": ["Express"], "X-Content-Type-Options": ["nosniff"], "Content-Length": ["141"], "Date": ["Tue, 08 Mar 2022 10:16:11 GMT"]}, "status": 404}

Additionally, curl -v shows:

> curl -v mymaindomain.com/nr
*   Trying 1.2.3.4:80...
* Connected to mymaindomain.com (1.2.3.4) port 80 (#0)
> GET /nr HTTP/1.1
> Host: mymaindomain.com
> User-Agent: curl/7.79.1
> Accept: */*
>
* Mark bundle as not supporting multiuse
< HTTP/1.1 308 Permanent Redirect
< Connection: close
< Location: https://mymaindomain.com/nr
< Server: Caddy
< Date: Tue, 08 Mar 2022 10:08:07 GMT
< Content-Length: 0
<
* Closing connection 0
4

Then that’s not a config problem, Caddy just can’t connect to that upstream. Make sure it’s actually running at that IP and port, and that there’s no firewall blocking the request.

You made an HTTP request here, not an HTTPS request – Caddy responded with an HTTP->HTTPS redirect. Specify https:// or use the -L flag to follow redirects (-L for the Location response header which is the redirect destination)

5

Thanks again for the reply.

The server has no firewalls / VPCs or anything else in the way that should be preventing the connection. It is most definitely connectable from anywhere, at the moment. All three of the machines concerned here (the two app servers and the Caddy server) are in the one VPC, and the firewall config is identical between the two application servers except for port 80 vs port 1880.

Here’s an updated version of the curl request

>curl -v -L https://mymaindomain.com/nr
*   Trying 1.2.3.4:443...
* Connected to mymaindomain.com (1.2.3.4) port 443 (#0)
* schannel: disabled automatic use of client certificate
* schannel: ALPN, offering http/1.1
* schannel: ALPN, server accepted to use http/1.1
> GET /nr HTTP/1.1
> Host: mymaindomain.com
> User-Agent: curl/7.79.1
> Accept: */*
>
* Mark bundle as not supporting multiuse
< HTTP/1.1 404 Not Found
< Access-Control-Allow-Origin: *
< Content-Length: 141
< Content-Security-Policy: default-src 'none'
< Content-Type: text/html; charset=utf-8
< Date: Tue, 08 Mar 2022 10:31:39 GMT
< Server: Caddy
< X-Content-Type-Options: nosniff
< X-Powered-By: Express
<
<!DOCTYPE html>
<html lang="en">
<head>
<meta charset="utf-8">
<title>Error</title>
</head>
<body>
<pre>Cannot GET /nr</pre>
</body>
</html>
* Connection #0 to host mymaindomain.com left intact
6

Alright, so your upstream is saying it doesn’t know how to route /nr.

Make sure your upstream app is configured to handle that path.

7

Is an alternative here to use a URL rewrite instead of configuring the upstream app to allow that path? In other words, would a url rewrite serve to change https://mymaindomain.com/nr to 55.66.77.88:1880 and back?

If so, that might be worth a shot - just got to work out how.

8

Yes, you can use handle_path to strip the path prefix before proxying:

site.mymaindomain.com {
	handle_path /nr* {
		reverse_proxy 55.66.77.88:1880
	}

	handle {
		reverse_proxy 11.22.33.44:80
	}
}

Keep in mind though, this isn’t bullet-proof because any URLs in HTML that the upstream app responds with may not work unless the app is configured to prefix them with /nr. This isn’t something you can solve easily with Caddy config. See this article for an explanation:

9

Thanks very much. It’s late here, and this is now a tomorrow problem. I suspect I’ll bump into that exact issue too.

10

I’ve modified the configuration of the upstream application so that now it is accessible via 55.66.77.88:1880/nr. I’ve also used the handle_path example you posted above. I’m back to receiving a 502 response.

At the bottom of a debug message I can see:

"error": "dial 55.66.77.88:1880: unknown network 55.66.77.88:1880"}

Hitting 55.66.77.88:1880/nr works in a browser
Hitting 55.66.77.88:1880 no longer works because the base url has been changed.
These changes are reflected in the Caddyfile

Caddyfile:

{
    email email@mymaindomain.com
    debug
}

mymaindomain.com {
    handle_path /nr* {
        reverse_proxy 55.66.77.88:1880/nr
    }

    handle {
        reverse_proxy 11.22.33.44:80
    }
}

I’m on the corporate network this morning, so curl -v -L looks a bit different.

curl -v -L https://mymaindomain.com/nr
* Uses proxy env variable https_proxy == 'http://corporateproxy.internal:8080'
*   Trying 10.2.106.100:8080...
* Connected to corporateproxy.internal (10.2.106.100) port 8080 (#0)
* allocate connect buffer!
* Establish HTTP proxy tunnel to mymaindomain.com:443
> CONNECT mymaindomain.com:443 HTTP/1.1
> Host: mymaindomain.com:443
> User-Agent: curl/7.79.1
> Proxy-Connection: Keep-Alive
>
< HTTP/1.1 200 Connection established
<
* Proxy replied 200 to CONNECT request
* CONNECT phase completed!
* schannel: disabled automatic use of client certificate
* schannel: ALPN, offering http/1.1
* ALPN, server did not agree to a protocol
> GET /nr HTTP/1.1
> Host: mymaindomain.com
> User-Agent: curl/7.79.1
> Accept: */*
>
* Mark bundle as not supporting multiuse
< HTTP/1.1 502 Bad Gateway
< Server: Caddy
< Date: Tue, 08 Mar 2022 22:04:15 GMT
< Content-Length: 0
< Set-Cookie: visid_incap_2249622=wxoJ2qjpRQSBTwUkyzJ1d9/SJ2IAAAAAQUIPAAAAAADJOvgd8M5aqo6cjit0EfhU; expires=Wed, 08 Mar 2023 14:53:16 GMT; HttpOnly; path=/; Domain=.qfes.qld.gov.au; Secure; SameSite=None
< Set-Cookie: incap_ses_341_2249622=ZFIsBju3llaYLN2WVHu7BN/SJ2IAAAAA1QCNfnf5Ndfr2N1yKNOT/g==; path=/; Domain=.qfes.qld.gov.au; Secure; SameSite=None
< Strict-Transport-Security: max-age=31536000
< X-CDN: Imperva
< X-Iinfo: 13-56170987-56170995 NNNN CT(14 14 0) RT(1646777054988 96) q(0 0 1 0) r(1 1) U11
<
* Connection #0 to host corporateproxy.internal left intact
11

It’s not valid to use a path in the proxy upstream. Remove the /nr here.

12

I had that in there because without it I noticed that the upstream application was receiving requests on / rather than on /nr.

So, now it’s removed my upstream is again showing that it’s receiving requests on /. When the base directory of this application is set to / then the assets don’t come through, like in the article you linked earlier. When the base dir is set to /nr then the assets all appear from /nr/stuff_goes_here.jpg as expected. Caddy doesn’t seem to be sending the /nr through, unfortunately.

13

Well, what you had originally, with reverse_proxy /nr* 55.66.77.88:1880 did pass through /nr. But then I explained you can use handle_path if you do need to strip the path before proxying.

It depends how your upstream app is configured. But it’s definitely not correct to use a path in the upstream address.

1 Like
14

Good pointer. Thanks.

What wound up working:

{
    email email@mymaindomain.com
    debug
}


mymaindomain.com {
    reverse_proxy /nr* 55.66.77.88:1880
    reverse_proxy 11.22.33.44:80
}

The upstream app, node-red, had to have two directives set to \ in settings.json

15

This topic was automatically closed after 30 days. New replies are no longer allowed.