V2: Reverse Proxy setup : help required

inxsible · May 27, 2020, 12:04am

1. Caddy version (`caddy version`): v2.0.0 – commit id: e051e119d1dff75972ed9b07cf97bbb989ba8daa

2. How I run Caddy:

Using a systemd service as defined below.

a. System environment:

Archlinux LXC container on Proxmox VE 6.2-4

b. Command:

N/A if restarting the container

OR

# systemctl start caddy

c. Service/unit/compose file:

[Unit]
  2 Description=Caddy Web Server
  3 Documentation=https://caddyserver.com/docs/
  4 After=network.target
  5 
  6 [Service]
  7 User=http
  8 Group=http
  9 ExecStart=/usr/bin/caddy run --config /etc/caddy/Caddyfile --environ
 10 ExecReload=/usr/bin/caddy reload --config /etc/caddy/Caddyfile
 11 TimeoutStopSec=5s
 12 LimitNOFILE=1048576
 13 LimitNPROC=512
 14 
 15 # Hardening options
 16 PrivateTmp=true
 17 ProtectSystem=strict
 18 PrivateDevices=true
 19 ProtectHome=true
 20 ReadWritePaths=/var/lib/caddy /var/log/caddy
 21 AmbientCapabilities=CAP_NET_BIND_SERVICE
 22 CapabilityBoundingSet=CAP_NET_BIND_SERVICE
 23 NoNewPrivileges=true
 24 ProtectKernelTunables=true
 25 ProtectKernelModules=true
 26 ProtectControlGroups=true
 27 LockPersonality=true
 28 
 29 
 30 [Install]
 31 WantedBy=multi-user.target

d. My complete Caddyfile or JSON config:

{
    acme_ca https://acme-staging-v02.api.letsencrypt.org/directory
}

(cloudflare) {
    tls {
        dns cloudflare $CLOUDFLARE_AUTH_TOKEN
    }
}

bw.tabala.com {
    import cloudflare
    log {
        output file /var/log/caddy/caddy.log {
            roll_size 512mb
            roll_keep 5
            roll_keep_for 168h
        }
    }

    encode gzip
    header / {
        # Enable HTTP Strict Transport Security (HSTS)
        #Strict-Transport-Security "max-age=31536000;"
        # Enable cross-site filter (XSS) and tell browser to block detected attacks
        X-XSS-Protection "1; mode=block"
        # Disallow the site to be rendered within a frame (clickjacking protection)
        X-Frame-Options "DENY"
        # Prevent search engines from indexing (optional)
        X-Robots-Tag "none"
        # Server name removing
        -Server
    }
    
    # The negotiation endpoint is also proxied to Rocket
    reverse_proxy /notifications/hub/negotiate 192.168.1.25:80

    # Notifications redirected to the websockets server
    reverse_proxy /notifications/hub 192.168.1.25:3012

    # Proxy the Root directory to Rocket
    reverse_proxy 192.168.1.25:80
    

    #reverse_proxy * 192.168.1.25:8000
}

I got the header section and the proxy configuration setup from the Caddy 2 example from this link

3. The problem I’m having:

I cannot get to the bitwarden web-vault nor can i access the reverse proxy server which would then forward me to the bitwarden service using HTTPS.
Questions

What DNS records and their values do I need in my Cloudflare account to setup on my domain so that I can access the reverse proxy which would then forward me to the correct service based on what I type in the address bar?
Should I use wildcard SSL cert so that the same cert can be used for all the services on my network? Which DNS records in my Cloudflare account are required in this case?
Or Should I use separate certs for each service?

AIM : to simply type in the address bar (while on the LAN)

cloud - so that it would take me to my self-hosted Nextcloud instance
nas - so that it would take me to my nas login page etc.

4. Error messages and/or full log output:

I don’t see any logs generated under /var/log/caddy

Here’s what #systemctl status caddy shows:

service - Caddy Web Server
  3          Loaded: loaded (/usr/lib/systemd/system/caddy.service; enabled; vendor preset: disabled)
  4               Active: active (running) since Tue 2020-05-26 23:03:31 UTC; 2s ago
  5                      Docs: https://caddyserver.com/docs/
  6                         Main PID: 4237 (caddy)
  7                               Tasks: 7 (limit: 4915)
  8                                    Memory: 13.9M
  9                                         CGroup: /system.slice/caddy.service
 10                                                      `-4237 /usr/bin/caddy run --config /etc/caddy/Caddyfile --environ
 11 
 12 May 26 23:03:32 reverseproxy caddy[4237]: 2020/05/26 23:03:32 [INFO] [bw.tabala.com] acme: Obtaining bundled SAN certificate given a CSR
 13 May 26 23:03:32 reverseproxy caddy[4237]: 2020/05/26 23:03:32 [INFO] [bw.tabala.com] AuthURL: https://acme-staging-v02.api.letsencrypt.org/acme/authz-v3/59521055
 14 May 26 23:03:32 reverseproxy caddy[4237]: 2020/05/26 23:03:32 [INFO] [bw.tabala.com] acme: Could not find solver for: tls-alpn-01
 15 May 26 23:03:32 reverseproxy caddy[4237]: 2020/05/26 23:03:32 [INFO] [bw.tabala.com] acme: Could not find solver for: http-01
 16 May 26 23:03:32 reverseproxy caddy[4237]: 2020/05/26 23:03:32 [INFO] [bw.tabala.com] acme: use dns-01 solver
 17 May 26 23:03:32 reverseproxy caddy[4237]: 2020/05/26 23:03:32 [INFO] [bw.tabala.com] acme: Preparing to solve DNS-01
 18 May 26 23:03:33 reverseproxy caddy[4237]: 2020/05/26 23:03:33 [INFO] [bw.tabala.com] acme: Trying to solve DNS-01
 19 May 26 23:03:33 reverseproxy caddy[4237]: 2020/05/26 23:03:33 [INFO] [bw.tabala.com] acme: Checking DNS record propagation using [192.168.1.1:53]
 20 May 26 23:03:33 reverseproxy caddy[4237]: 2020/05/26 23:03:33 [INFO] Wait for propagation [timeout: 1m0s, interval : 2s]
 21 May 26 23:03:33 reverseproxy caddy[4237]: 2020/05/26 23:03:33 [INFO] [bw.tabala.com] acme: Waiting for DNS record propagation 
 22 May 26 23:03:33 reverseproxy caddy[4237]: 2020/05/26 23:03:33 [INFO] [bw.tabala.com] acme: Checking DNS record propagation using [192.168.1.1:53]
 23 May 26 23:03:33 reverseproxy caddy[4237]: 2020/05/26 23:03:33 [INFO] Wait for propagation [timeout: 1m0s, interval: 2s ]
 24 May 26 23:03:33 reverseproxy caddy[4237]: 2020/05/26 23:03:33 [INFO] [bw.tabala.com] acme: Waiting for DNS record propagation.
 25 May 26 23:03:35 reverseproxy caddy[4237]: 2020/05/26 23:03:35 [INFO] [bw.tabala.com] acme: Waiting for DNS record propagation.
 26 May 26 23:03:43 reverseproxy caddy[4237]: 2020/05/26 23:03:43 [INFO] [bw.tabala.com] The server validated our request
 27 May 26 23:03:43 reverseproxy caddy[4237]: 2020/05/26 23:03:43 [INFO] [bw.tabala.com] acme: Cleaning DNS-01 challenge
 28 May 26 23:03:43 reverseproxy caddy[4237]: 2020/05/26 23:03:43 [INFO] [bw.tabala.com] acme: Validations succeeded; requesting certificates
 29 May 26 23:03:44 reverseproxy caddy[4237]: 2020/05/26 23:03:44 [INFO] [bw.tabala.com] Server responded with a certificate.
 30 May 26 23:03:44 reverseproxy caddy[4237]: 2020/05/26 23:03:44 [INFO][bw.tabala.com] Certificate obtained successfully
 31 May 26 23:03:44 reverseproxy caddy[4237]: 2020/05/26 23:03:44 [INFO][bw.tabala.com] Obtain: Releasing lock

5. What I already tried:

Backstory: This all started with my Bitwarden web vault not being accessible after a Firefox update. I got errors saying:

Cannot read property 'importKey' of null

Apparently this post seemed to suggest it was a browser issue where the browser no longer supported HTTP access to the bitwarden service. So in enabling HTTPS for bitwarden, I thought I might as well also implement a reverse proxy so that I don’t have to remember the ports for all the various services that I have on my network. Enter caddy, since it supported easy configuration for Let’s Encrypt. Being new to this whole thing, it didn’t matter if I chose Apache, Nginx or caddy.

I went ahead and bought a domain for this purpose and since caddy didn’t have a dns provider for my domain registrar’s name servers, I switched over to Cloudflare. After installing caddy on my Archlinux container using this PKGBUILD from caddyserver/dist/archlinux , I built a new binary using the xcaddy command below and copied the newly built caddy binary to /usr/bin/caddy
xcaddy build --with github.com/caddy-dns/cloudflare

I don’t have anything hosted on my domain – if that matters. Maybe in the future I will buy a hosting package and put up a website there

I seemed to have picked an inopportune time to create a reverse proxy because caddy was in transition from v1 to v2. caddy was in the late betas at the time, so I went through the muck of various forum posts and incomplete (at that time) documentation to see how to create a Caddyfile for v2. After a lot of trial and error with various settings – some which happened to be from v1 and didn’t quite translate over to v2, I finally have that right and I can get the SSL certs from Let’s Encrypt by creating an ‘A’ record for bw and pointing it to my home’s WAN IP address.

Currently:
However, I use pfsense as my router and when I put in bw.tabala.com in the address bar (while being on the LAN), I get presented the NET::ERR_CERT_AUTHORITY_INVALID page. If I click Advanced and proceed I get the pfsense page with the following warning :

Potential DNS Rebind attack detected, see http://en.wikipedia.org/wiki/DNS_rebinding
Try accessing the router by IP address instead of by hostname.

instead of being presented the bitwarden login page.

What DNS records do I need to set in my DNS provider, in order for me to get to the reverse proxy – which would then forward me to the correct service (bitwarden or guacamole, or syncthing etc.)?

Currently I am trying this only with bitwarden, but eventually I want the same reverse proxy to be able to handle all of the services on my network (8 LXC containers, 3 VMs, Proxmox host itself, separate FreeNAS box and 3 plugins on the FreeNAS box, & pfsense router). Some of them currently use self signed certs and some just work on HTTP – I was hoping to use Let’s Encrypt certs for all the services. That way I won’t have to remember the port numbers etc. and would aim to simply put in the address bar (while on the LAN)
cloud - so that it would take me to my self-hosted Nextcloud instance
nas - so that it would take me to my nas login page etc.

6. Links to relevant resources:

I read a lot of forum posts and links which eventually helped me get to a point where I am able to get the SSL certs correctly, but I don’t think they are relevant to the issue that I have now.

francislavoie · May 27, 2020, 4:53am

FYI this will only apply those headers to requests to / only, and nothing else. Patch matching is exact in Caddy v2. Remove the / to make it apply to all paths. The example from the bitwarden wiki has been updated since you last looked to fix that.

I’ll let @Whitestrake answer the rest though, he has a better grasp on Cloudflare and home networking stuff than I do

inxsible · May 27, 2020, 3:31pm

Thanks for the reply @francislavoie.
It does look like the bitwarden wiki got updated. I will go ahead and remove the /.

Hopefully someone can explain this whole A/AAAA/CNAME records that need to be set up in Cloudflare in order to get to the correct internal LAN service using the DNS challenge.

inxsible · May 31, 2020, 5:36am

Any word on what I could be doing wrong here ?

Whitestrake · June 2, 2020, 2:33am

DNS rebind attack protection kicks in whenever you try to access pfSense’s HTTP server with a hostname that isn’t pfSense’s self-configured hostname.

The issue is that, internally (i.e. on LAN), a DNS A query for bw.tabala.com returns your external IP address.

Externally, this is fine. On the outside of your pfSense - i.e. the WAN interface - the firewall listens to all. A packet comes in for port 443. You’re probably forwarding it to Caddy (with NAT → Port Forward configuration). All is probably well from outside.

Internally, it’s a problem that can be a bit difficult to understand if you’re not familiar with it already. Think about the packet your browser sends: it looks up the website you requested and the DNS says (EXTERNAL IP). So, you’re on LAN, and our little packet has been told to go find this internet address. Your PC sends it in the direction of the gateway - your pfSense - and says “the internet is that way, go forth”. Our little packet reaches the pfSense and says, “I was sent for (EXTERNAL IP), do you know him?”, and the pfSense goes, “Well of course I know him, he’s me!”

So our packet has technically reached its destination and needs travel no further. Problem: what does the pfSense do with it now? YOU want pfSense to send our little packet onwards to Caddy; you’ve likely set up a port forward to enable this. But the packet didn’t come in from WAN, it actually came from the LAN interface. There’s no firewall and no NAT happening on the LAN side. Instead, pfSense is running its web server there!

So your browser sent a request for bw.tabala.com, it technically made it most of the way to where you want it, but it’s getting handled by pfSense instead of being sent to Caddy and now pfSense’s web server is upset because you forgot it’s name, how could you?

So that’s the problem. What’s the answer?

Well, there are two. The first is hairpinning, a.k.a. NAT loopback.

You flick a switch, and pfSense starts taking all the packets it receives on LAN and feeding them back through the WAN interface firewall - just in case there’s a NAT rule that applies! This approach pretty much instantly solves this problem (and a lot of problems like it). So why isn’t it on by default?

Well, it involves double-handling all those packets. It’s slightly less than ideal. Actually, though, it’s relatively common for consumer modem/routers to have hairpinning enabled by default. pfSense doesn’t, though, because it’s targeted at users with more technical expertise and they expect you’ll know to set it up if you want it.

The other option: split DNS.

Configure pfSense to return a different IP address for DNS requests that originate internally. Specifically, have pfSense return the local IP address of Caddy (rather than the external address of your pfSense) for a DNS request for your domain name. Then, those little packets don’t even need to even go to pfSense and loop around to Caddy; they can go straight to the Caddy host server.

This is what I do. There’s a few ways about it; you can either use a domain that pfSense has configured, with the subdomain CNAME’d to the server’s hostname, which is externally CNAME’d to your pfSense and internally resolves directly to the LAN IP of the server. The other way is to just set manual host overrides in the DNS resolver (unbound).

Check out Netgate’s own docs, which cover both methods in pfSense:

https://docs.netgate.com/pfsense/en/latest/nat/accessing-port-forwards-from-local-networks.html

inxsible · June 3, 2020, 1:51pm

Thanks for the detailed reply @Whitestrake.

Not having a networking background, I just didn’t know the terms to search for when trying to fix this. I think I had read something about Split DNS but just didn’t know enough to understand how it could apply. Thank you again, I’ll read through the pfSense docs, set it up and see how that works out.

system · June 26, 2020, 12:19am

This topic was automatically closed after 30 days. New replies are no longer allowed.