Caddy on seedbox slot as reverse proxy using Cloudflare DNS


(Cameron) #1

What am I trying to do?

I’m trying to setup Caddy as a reverse proxy to secure external access to several services, chiefly an Emby media server, running on a Whatbox slot with the DNS for mydomain.tld handled by Cloudflare.

As a background, I was following this guide on the Whatbox wiki but ran into issues when I tried it myself. I spoke to Whatbox support and was told that this user-created page is probably incomplete. The page is actually not among their curated list of officially supported software on their main wiki page, but it was the first result that appeared on Google when I searched reverse proxy whatbox.

So while the question of whether it can work at all is unconfirmed, I imagine the original author didn’t write that guide as a joke and was actually able to get it running at some point. I figured I should ask here before trying something else.

Current Setup

  • The host server has a static external address.
  • To my knowledge, I have no control over what ports are provided for the services I run.
  • My Emby server is directly accessible through HTTP from external-host-ip:7777. Emby also sets up port 8888 for HTTPS access but I cannot use it without first setting up an SSL cert.
  • I’m using Cloudflare for DNS since it’s compatible with LetsEncrypt. My SSL cert setting on Cloudflare is currently set to Full (strict) (I don’t know if this is the correct setting).

What have I tried?

I installed Caddy and setup up my Cloudflare credentials as environment variables according to the linked wiki guide.

Caddyfile
mydomain.tld {

 tls {
  dns cloudflare
 }

 gzip

 proxy /emby localhost:7777 {    # Should this be the HTTPS port 8888 instead?
  websocket
  transparent
 }
 
 log

}

I ran Caddy as a screen daemon. Although that wiki guide may not be officially supported, the ports provided were indeed opened properly.

screen -dmS caddy /path/caddy/caddy -conf /path/caddy/Caddyfile -http-port 20202 -https-port 40404

This is where I ran into confusion. When I try to access my service via https://mydomain.tld/emby, I get an Error 526 page (Invalid SSL certificate).

I checked the log using ./caddy -log -stderr and see this:

Activating privacy features... done.
2019/01/09 22:27:44 listen tcp :443: bind: permission denied

If I attach the screen with screen -r caddyID, I see this:

Activating privacy features... done.
https://mydomain.tld:40404
http://mydomain.tld:20202

…which is what I think I should be seeing. What’s going on?

The Caddy page on automatic-https has this paragraph…

Paragraph on ports 80 and 443

Ports 80 and 443 must be externally open

By default, Caddy will bind to ports 80 and 443 to serve HTTPS and redirect HTTP to HTTPS. This usually requires privilege escalation. On Linux systems, you can give Caddy permission to bind to port 80 and 443 without being root using setcap, like so: setcap cap_net_bind_service=+ep caddy . Don’t forget to configure all relevant firewalls to allow Caddy to use these ports for incoming and outgoing connections! Caddy must have claim on these ports to obtain certificates unless you enable the DNS challenge OR forward ports 80 and 443 to different ports internally (in which case you can change the HTTP and HTTPS ports using CLI flags.

…that makes it sound like not binding ports 80 and 443 was OK as long as I enabled the DNS challenge (which I think I am, through Cloudflare). This is what has me confused the most.

Is Caddy binding the new ports or is it trying to bind 80 and 443? If it is trying to bind the standard ports, why? Aren’t I explicitly setting them to be something else via CLI flags? Is this just some weirdness with running as screen daemon?

Finally, as this is a shared machine, I do not have root, but I did try running the setcap command suggested above…

setcap cap_net_bind_service=+ep /path/caddy/caddy

…but it was denied.

unable to set CAP_SETFCAP effective capability: Operation not permitted

Sorry for the wall of text. I hope I’ve given enough that someone will be able to help me understand what’s going on and maybe what I should do.


(Matthew Fay) #2

Hi @10days!

The wall of text is great; making it easier to understand where you’re coming from and what you’d like to achieve, along with all the information you’ve given, will hopefully make it very easy to help you get it all up and running.

First up, that guide isn’t bad! Looks like it’s got everything it needs to get you going. There’s just a couple of things it perhaps didn’t clarify 100%.

Cloudflare’s great, but Cloudflare’s complicated the moment you step out of a very traditional web server setup. They actually provide two services - the first is a nameserver (to host and serve your DNS zone). The second is a reverse proxy.

You might have noticed the “orange cloud” next to A or CNAME records in your Cloudflare dashboard. While that’s on, Cloudflare actually responds to DNS requests with the IP address of their own server, not yours. Then, when a client connects, they proxy to what they refer to as your origin server. They only do this over standard HTTP(S) ports, and they only proxy HTTP content. That means that this feature isn’t suitable if you want to host non-HTTP services or if you want to host on non-standard ports.

If Caddy were able to use 80 and 443 on your seedbox, we could configure Cloudflare properly, but in your case you’ll just need to “grey-cloud” those DNS records.

Firstly, Error 526 is a Cloudflare-specific error. Since you’ve got the record orange-clouded with SSL configured for Full (Strict), when Cloudflare connects to your origin, it does so on port 443 and requires a valid HTTPS certificate to be presented. When that fails, you get that error code.

Secondly, if you browse to https://mydomain.tld/emby, you won’t reach Caddy, even after you turn off the orange cloud. You’ve changed the HTTPS port Caddy uses to 40404, so you’ll need to specify this in your browser’s URL bar (or it’ll just try 443 by default). It should look like: https://example.com:40404/emby

This command won’t open any existing error logs. Instead, you’re running a new instance of Caddy (with logging configured to stdout in your shell). Note that -http-port and -https-port haven’t been specified like they were in the screen command - because of this, Caddy tries to use the defaults (80 and 443) and then fails, probably dumping you back to the prompt.

This is the command you should be using to see anything Caddy is logging to stdout, and is indeed the result you should be seeing. It means that Caddy successfully started, is serving your website on those scheme/port combinations, and is managing your certificate.

Yep, that’s exactly the case, on both counts.

Caddy is binding the new ports, as per the output in screen -r caddyID. It’s binding those standard ports for the reason stated above (you launched it fresh without overriding the default ports). Your explicit setting in the CLI flags of your initial screen command are correct. I don’t use screen to manage my Caddy, but I don’t think it’s acting out here.

Yeah, unfortunately, this is probably the entire reason the author of that guide you linked wrote it in the first place; to serve HTTPS without having to bind the (lower, privileged) default ports because you don’t have the power to do so.


In summary, go to Cloudflare, disable the orange-cloud option on your CNAME / A records, then browse to https://mydomain.tld:40404/emby (note the port has to be included here) once the DNS changes have propagated.


(Cameron) #3

Thank you, @Whitestrake! Your instructions were exactly what I needed to get it working.

I have a follow-up question. I suspect the answer again has to do with the caveat that I’m running on non-standard ports.

Given that the HTTPS link now works as intended, I also tried typing mydomain.tld:20202/emby and http://mydomain.tld:20202/emby with the expectation that they would automatically redirect to https://mydomain.tld:40404/emby. I instead land at a Your connection is not private/secure page. I don’t need those URLs to work, I just tried them out of curiosity.

Should I have expected this sort of behavior or is that strange?


(Matthew Fay) #4

It’s strange. Your expectation should have been correct. Can you run curl -kIL mydomain.tld:20202 and give us the output?


(Cameron) #5

Sure thing.

curl -kIL mydomain.tld:20202

HTTP/1.1 301 Moved Permanently
Connection: close
Content-Type: text/html; charset=utf-8
Location: https://mydomain.tld/
Server: Caddy
Date: Thu, 10 Jan 2019 03:47:45 GMT

HTTP/1.1 200 OK
Server: nginx
Date: Thu, 10 Jan 2019 03:47:45 GMT
Content-Type: text/html; charset=UTF-8
Connection: keep-alive
Vary: Accept-Encoding

Not sure if the nginx part is separate, but this the whole output.


(Matthew Fay) #6

Huh… Caddy’s HTTP->S upgrade redirect didn’t indicate the port.

Hey @matt, shouldn’t Automatic HTTPS set up a HTTP listener that respects the -https-port flag when redirecting? Is this a bug?

Anyway, @10days, you can override it by manually specifying the HTTP listener and it’s redirect behaviour in your Caddyfile. Do it like this:

http://example.com:20202 {
  redir https://example.com:40404/
}

https://example.com:40404 {
  [put your current site definition here]
}

This method explicitly specifies which schemes and ports should be running and how Caddy should handle that redirect. This also means that, for these sites, you don’t need -http-port and -https-port flags (but I’d leave them there anyway).


(Matt Holt) #7

Maybe – I don’t have time to look into it right now, but here is the relevant code: https://github.com/mholt/caddy/blob/f5aaa471deb35fad58a5a6394b05bbf7e42ac61a/caddyhttp/httpserver/https.go#L172


(Matthew Fay) #9

Yeah, it’s a logic error.

  1. The flag -https-port sets the HTTPSPort variable.
  2. redirPlaintextHost checks whether the site’s port is different than HTTPSPort.
    a. If it’s set differently to the default HTTPS port, it includes the port in the location header.
    b. If it’s not set, or set to the same as HTTPSPort, it does not include a port in the location header.

Since we’re on a nonstandard port via -https-port flag, but have not explicitly set the port in the site label, we have the scenario where the default HTTP listener does not correctly indicate the alternate HTTPS port.

It would be more correct to compare redirPort == 443, instead of redirPort == HTTPSPort, but the assumption made in the code was that these flags are for internal use only and Caddy should assume that the client will still be connecting on regular ports externally.

I have opened an issue addressing this behaviour:

https://github.com/mholt/caddy/issues/2422

I haven’t made any PR, because at present this is actually expected behaviour (as per the docs, “Using this flag implies you are forwarding port 443 internally”: https://caddyserver.com/docs/cli#https-port).


(Matthew Fay) #10

@10days:

Yep, official word down is that with this being an advanced setup, manually specifying ports in your site labels and explicitly defining the HTTP listener is the correct way to go about it, per my above post.


(Cameron) #11

Ok, I can confirm that doing so gives me the expected behavior. Thanks again for your help!