Issues with header manipulation and Deluge

Help
1

1. My Caddy version (caddy version):

v2.0.0-rc.3 h1:z2H/QnaRscip6aZJxwTbghu3zhC88Vo8l/K57WUce4Q=

2. How I run Caddy:

I run caddy through systemd on a RHEL 8.2 server. It is a reverse proxy for local containers.

a. System environment:

RHEL 8.2 x64, systemd

b. Command:

/usr/local/bin/caddy run --config /etc/caddy/Caddyfile --environ

c. Service/unit/compose file:

[Unit]
Description=Caddy Web Server
Documentation=https://caddyserver.com/docs/
After=network.target

[Service]
User=caddy
Group=caddy
ExecStart=/usr/local/bin/caddy run --config /etc/caddy/Caddyfile --environ
ExecReload=/usr/local/bin/caddy reload --config /etc/caddy/Caddyfile
TimeoutStopSec=5s
LimitNOFILE=1048576
LimitNPROC=512
PrivateTmp=true
ProtectSystem=full
AmbientCapabilities=CAP_NET_BIND_SERVICE

[Install]
WantedBy=multi-user.target

d. My complete Caddyfile or JSON config:

mediabox.noodlesauce.com {
        reverse_proxy /deluge* 127.0.0.1:8112 {
                header_up X-Deluge-Base "/deluge/";
                header_down X-Frame-Options SAMEORIGIN;
        }
        log {
                level debug
        }
}

3. The problem I’m having:

I’m unable to access Deluge on a sub URI through a reverse proxy with Caddy. I receive a “No Such Resource” error from the twisted web server within my Deluge container.
This same configuration proxied through Nginx works fine. Relevant Nginx config:

location /deluge {
    proxy_pass http://localhost:8112/;
    proxy_set_header X-Deluge-Base "/deluge/";
    add_header X-Frame-Options SAMEORIGIN;
}

4. Error messages and/or full log output:

When visiting mediabox.noodlesauce.com/deluge, I’m greeted with:

No Such Resource

No such child resource.

Here’s the Caddy log:

May 02 03:37:31 exa.h.noodlesauce.com caddy[26052]: {“level”:“error”,“ts”:1588405051.5211046,“logger”:“http.log.access.log0”,“msg”:“handled request”,“request”:{“method”:“GET”,“uri”:"/deluge",“proto”:“HTTP/2.0”,“remote_addr”:“10.0.1.1:63033”,“host”:“mediabox.noodlesauce.com”,“headers”:{“Accept-Language”:[“en-us”],“Accept-Encoding”:[“gzip, deflate, br”],“User-Agent”:[“Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_4) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/13.1 Safari/605.1.15”],“Accept”:[“text/html,application/xhtml+xml,application/xml;q=0.9,/;q=0.8”]},“tls”:{“resumed”:false,“version”:772,“ciphersuite”:4865,“proto”:“h2”,“proto_mutual”:true,“server_name”:“mediabox.noodlesauce.com”}},“common_log”:“10.0.1.1 - - [02/May/2020:03:37:31 -0400] “GET /deluge HTTP/2.0” 404 153”,“latency”:0.000810056,“size”:153,“status”:404,“resp_headers”:{“Server”:[“Caddy”,“TwistedWeb”],“Date”:[“Sat, 02 May 2020 07:37:31 GMT”],“Content-Type”:[“text/html; charset=utf-8”],“Content-Length”:[“153”],“X-Frame-Options”:[“SAMEORIGIN;”]}}

5. What I already tried:

I’ve attempted numerous permutations of the “header” directive, and the “header_up” and “header_down” settings under reverse_proxy. Also many different permutations of the matcher (/deluge, /deluge/, /deluge*, etc.).

6. Links to relevant resources:

2

I think this has to do with your headers. The semicolon is likely causing the issue. Caddy doesn’t use exactly the same syntax as Nginx. Also note that double quotes are stripped unless escaped (in Caddyfile they just group multiple words with spaces as one argument instead of being parsed as separate arguments).

3

Good catch on the semicolons. I’ve changed it to:

        reverse_proxy /deluge* 127.0.0.1:8112 {
                header_up X-Deluge-Base /deluge/
                header_down X-Frame-Options SAMEORIGIN
        }

Same behavior.

4

Could you add this to the top of your Caddyfile and report back with the logs output?

{
    debug
}
5

With debug stanza:

May 02 12:18:39 exa.h.noodlesauce.com systemd[1]: Reloading Caddy Web Server.
May 02 12:18:39 exa.h.noodlesauce.com caddy[310445]: {"level":"info","ts":1588436319.3357277,"msg":"using provided configuration","config_file":"/etc/caddy/Caddyfile","config_adapter":""}
May 02 12:18:39 exa.h.noodlesauce.com caddy[26052]: {"level":"info","ts":1588436319.3376498,"logger":"admin.api","msg":"received request","method":"POST","host":"localhost:2019","uri":"/load","remote_addr":"127.0.0.1:39966","headers":{"Accept-Encoding":["gzip"],"Content-Length":["449"],"Content-Type":["application/json"],"Origin":["localhost:2019"],"User-Agent":["Go-http-client/1.1"]}}
May 02 12:18:39 exa.h.noodlesauce.com caddy[26052]: {"level":"info","ts":1588436319.3377695,"logger":"admin.api","msg":"config is unchanged"}
May 02 12:18:39 exa.h.noodlesauce.com caddy[26052]: {"level":"info","ts":1588436319.3377748,"logger":"admin.api","msg":"load complete"}
May 02 12:18:39 exa.h.noodlesauce.com systemd[1]: Reloaded Caddy Web Server.
May 02 12:18:41 exa.h.noodlesauce.com caddy[26052]: {"level":"debug","ts":1588436321.6408045,"logger":"http.handlers.reverse_proxy","msg":"upstream roundtrip","upstream":"127.0.0.1:8112","request":{"method":"GET","uri":"/deluge","proto":"HTTP/2.0","remote_addr":"10.0.1.1:52727","host":"mediabox.noodlesauce.com","headers":{"Accept-Language":["en-us"],"Accept-Encoding":["gzip, deflate, br"],"X-Forwarded-For":["10.0.1.1"],"Cookie":["PHPSESSID=fsht6sn8d3lura83fmfivc43ng"],"Accept":["text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8"],"X-Deluge-Base":["/deluge/"],"User-Agent":["Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_4) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/13.1 Safari/605.1.15"]},"tls":{"resumed":false,"version":772,"ciphersuite":4865,"proto":"h2","proto_mutual":true,"server_name":"mediabox.noodlesauce.com"}},"headers":{"Server":["TwistedWeb"],"Date":["Sat, 02 May 2020 16:18:41 GMT"],"Content-Type":["text/html; charset=utf-8"],"Content-Length":["153"]},"duration":0.000779933,"status":404}
6

Well, it does seem to be passing your header as expected to the upstream. I don’t know enough about Deluge to help much more.

What if you make a request with a slash appended at the end of the URL? Is Deluge not accepting it because it doesn’t end in a slash? If that’s the case you could add a redir directive to ensure there’s always a slash.

7

Thank you so much for your quick responses.

I’ve tried with the /, no luck.

I guess what intrigues me is that this isn’t a terribly complicated configuration but I think I’m missing something minor with the Caddyfile. Are you able to spot a difference functionality-wise between these three configurations?

NGINX (Works):

location /deluge {
    proxy_pass http://127.0.0.1:8112/;
    proxy_set_header X-Deluge-Base "/deluge/";
    add_header X-Frame-Options SAMEORIGIN;
}

Apache HTTPD (2.4) (Works):

ProxyPass /deluge http://127.0.0.1:8112/

<Location /deluge>
    ProxyPassReverse /
    ProxyPassReverseCookiePath / /deluge               
    RequestHeader set X-Deluge-Base "/deluge/"          
    Order allow,deny
    Allow from all
</Location>

Caddyfile (doesn’t work):

        reverse_proxy /deluge* 127.0.0.1:8112 {
                header_up X-Deluge-Base /deluge/
                header_down X-Frame-Options SAMEORIGIN
        }

Thank you!

8

A new data point. This Caddyfile works flawlessly with Caddy v1.0.4:

10.0.1.3:9999

proxy /deluge http://localhost:8112/ { 
	transparent 
	without /deluge 
	header_upstream X-Deluge-Base "/deluge" 
}

Any ideas on translating this to a v2 Caddyfile? I know that “transparent” and “without” were removed from v2.

I manually inserted the headers for which “transparent” is a shortcut. That didn’t fix it, so it must be the “without.”

From what I’m reading, “without” is a rewrite hack. I don’t understand what it’s doing though and how to translate it to v2. Please advise.

1 Like
9

Matching requests (i.e. using /deluge* as the request matcher for reverse_proxy) doesn’t modify the request. So if you want to change the request URI (i.e. strip a prefix from the path), you have to do that separately.

uri strip_prefix /deluge

should do the trick.

1 Like
10

Hey @matt , thanks for the response.

I did read a bit about the “uri” directive yesterday and tried the strip_prefix but that didn’t work. It looks like “uri” can’t live within a reverse_proxy block, so it has to live at the server level. In my case, adding the strip_prefix appears to do just that – stripping the prefix, making the request go to “/” and completely missing the reverse_proxy matcher. I just get a blank page. You can see in the debug output that the reverse_proxy is not matched.

mediabox.noodlesauce.com {
        reverse_proxy /deluge* 127.0.0.1:8112 {
                header_up X-Deluge-Base /deluge/
                header_down X-Frame-Options SAMEORIGIN
        }
        uri strip_prefix /deluge
}

May 03 09:26:59 exa.h.noodlesauce.com caddy[1034858]: {"level":"debug","ts":1588512419.034223,"logger":"http.handlers.rewrite","msg":"rewrote request","request":{"method":"GET","uri":"/deluge/","proto":"HTTP/2.0","remote_addr":"10.0.1.1:60259","host":"mediabox.noodlesauce.com","headers":{"Accept-Encoding":["gzip, deflate, br"],"Upgrade-Insecure-Requests":["1"],"Te":["trailers"],"User-Agent":["Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:75.0) Gecko/20100101 Firefox/75.0"],"Accept":["text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8"],"Accept-Language":["en-US,en;q=0.5"]},"tls":{"resumed":false,"version":772,"ciphersuite":4865,"proto":"h2","proto_mutual":true,"server_name":"mediabox.noodlesauce.com"}},"method":"GET","uri":"/"}
11 
route /deluge/* {
      uri strip_prefix deluge
      reverse_proxy  localhost:8112 {
          header_up X-Real-IP {remote}
          header_up X-Deluge-Base "/deluge"
      }
}
1 Like
12

@calisro thank you! This did it. I removed the trailing slash on the route matcher, which precludes the need for a trailing slash in the URL. Here’s my final WORKING config:

mediabox.noodlesauce.com {
        route /deluge* {
              uri strip_prefix /deluge
              reverse_proxy  localhost:8112 {
                  header_up X-Real-IP {remote}
                  header_up X-Deluge-Base "/deluge"
              }
        }
}

Thank you all. I love what this project is all about. I am a long-time IT guy and I’ve been evaluating Caddy in my homelab with the intent of using it at my business at some point. Caddy 2 seems to just have some very non-obvious configuration approaches. I’ll try and review all the docs again and provide feedback where I can.

Justin

1 Like
13

What’s non-obvious about it? Given the docs, it seems obvious to me :man_shrugging:

It’s a bit late in the cycle now but maybe we can make it more obvious? How would you like your config to look instead?

14

@matt thanks again for your responsiveness.

What’s non-obvious about it?

The fact that I would have to place my configuration block within a route, for example. Routes are great (now that I’ve been exposed to them) but there is no mention of them in the reverse proxy documentation (I understand that routes are a more general concept). From a noob perspective, I wouldn’t get to routes in the documentation until I read through the Caddyfile tutorial (which I did), and then Directives under further reading. I would then have to read through that doc (which I did) and understand that for this particular case that I need “A group of directives treated literally as single unit” (which I didn’t) and go on to read about the routes directive specifically (I didn’t because I didn’t understand what a route was). I may be an outlier though. I do see that there is a relevant example here at the bottom of the route directive doc page:

There’s a brief mention of the lack of “without” in the v2 upgrade documentation, but I’m not a v1 user, so I’ve never had exposure to Caddy prior to the v2 RCs.

Even with that mention, it talks about rewrite hacks, with a link to what they are, but there’s no indication of how to translate “without” to v2.

In this particular case, the part that I’m struggling with from a user perspective is that two other major HTTP servers only need to pass a header to the upstream server and it works. I don’t understand all of the internals, but my guess is that this is possibly due to assumptions made by these servers that Caddy does not make (which can be a good thing).

I am a huge fan of examples. They are incredibly helpful to me. It may be helpful to other users if you compiled together all of the examples in one page. Or had a repo of v2 examples like you do for v1:

tl;dr There feels like there’s a bit of glue missing in the docs. I would have never known to look up routes for this case. I didn’t understand what routes do by reading its description in the directives documentation.

Thank you for an awesome piece of software! LE has changed web certificates forever and Caddy is taking it to the next level! :metal:

2 Likes
15

Just to be clear, it’s not that the final config was a problem, I just didn’t understand how to get there. I think for this edge case, placing the reverse proxy directives inside of a route is just fine.

1 Like
16

Thanks @sixdrum - that’s great feedback. I’ll add a route example to the reverse proxy page.

1 Like
17

It may also be helpful to have a route example (or an explanation) on the uri page. I ended up on that page but didn’t know how to properly implement the strip_prefix for this case.

1 Like
18

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.