Use Caddy for local HTTPS (TLS) between front-end reverse proxy and LAN hosts

TL;DL

This Wiki explains how to enable https connections between hosts in a LAN with automatically renewed certificates. Just scroll down and go over the example Caddyfiles. They hopefully have enough comment to understand the idea :wink:

Introduction

If you have successfully followed the Wiki Using Caddy as a reverse proxy in a home network by @Matt, you have setup a reverse proxy that provides a TLS encrypted connection from the internet to that reverse proxy. This is mandatory and you shouldn’t want it any less. However, beyond the reverse proxy your connections are non-TLS.

This Wiki is about -also- adding TLS between that reverse proxy (frontend) and the hosts behind it (backends) with automatically renewed certificates. Whether this is important, is up to you. Normally, you should be able to trust your own LAN. But there may be situations you don’t want to take a chance. In my case I use the password manager Bitwarden and I didn’t want to send my passwords unencrypted over the LAN.

This Wiki assumes you have managed to setup a reverse proxy as explained in the Wiki Using Caddy as a reverse proxy in a home network. From here I have added TLS encryption to the backend(s).

Definitions and Requirements

The definitions and requirements from the Wiki mentioned above apply for this Wiki too. This Wiki has one extra requirement. You need to setup either a local DNS or split DNS.

With your own (split) DNS you can resolve Fully Qualified Domain Names (FQDN) in your LAN rather than using IP addresses. For example:

nextcloud.my.example.com 192.168.0.4 or
office.roadrunner192.168.0.4

Going into details about DNS or split DNS is out of the scope of this Wiki. Cloudflare has a pretty good explanation what is DNS and more about split DNS can be found here.

Two well-known DNS systems are:

I use unbound because it is by default available in OPNsense and comes with a handy GUI.

Alternatively, you could also fill up your hosts files but I have not tested this myself. It will be much more work to maintain though.

Topology

The below figure is somewhat similar to the version without TLS in the LAN. There is a frontend host and two backend hosts. Each host can be virtual or physical. The frontend hosts acts as reverse proxy to the outside WWWorld and provides trusted certificates from Let’s Encrypt. The backend hosts are hosting several services.

The difference is that the same frontend Caddy service is also setup as a certificate authority using the ACME protocol (powered by smallstep) to provide certificates for the backends.

Each backend also has a Caddy service running that requests the certificate and to either act as a web server or reverse proxy (or both) to the desired services on that backend.

Note that the hosts with docker containers is just presented as a possibility. This Wiki will focus on the 2nd host with Nextcloud on it.

DNS

To set up the DNS you can follow the same guidance as in Using Caddy as a reverse proxy in a home network.

Local (Split) DNS

Each host should be given a FQDN to resolve the IP of the corresponding host in the local network. In this example I created:

caddy.roadrunner192.168.0.2 to reach the frontend (downstream) Caddy service
services.roadrunner192.168.0.3 to reach the 1st backend (upstream) host with services
office.roadrunner192.168.2.4 to reach the 2nd backend (upstream) host with services

I created domain names for the hosts because pointing directly to the IP addresses didn’t work with the ACME server. I was not able to get certificates for the backends.

Services

Services can either be installed natively or you can use docker containers. The Caddy services should not be in a docker container because this will complicate the automated certificate renewal.

Port forwarding

To set up the port forwarding you can follow the same guidance as in Using Caddy as a reverse proxy in a home network.

Local HTTPS

To enable local HTTPS you need to enable the ACME server in the Frontend Caddy service in the Caddyfile. In the backend you have to point to this server in the Caddyfile to request a certificate.

Frontend
To enable the ACME server in the frontend, include the acme_server directive in the Caddyfile.

# ACME Server
caddy.roadrunner {
	acme_server
	tls internal
}

Note that the FQDN caddy.roadrunner is used here.

Secondly, you need define or update the FQDN where Caddy listens to and reverse proxies accordingly with TLS.

nextcloud.my.example.com {
	reverse_proxy https://office.roadrunner {
		header_up Host {upstream_hostport}
	}
}

In the above example, connections from nextcloud.my.example.com coming from the WAN are being forwarded to office.roadrunner with TLS enabled.

There is one extra line header_up Host {upstream_hostport} which is a host header with placeholder that will override the host header with the host name in the proxy upstream.

Read my question topic for more info.

Backends
There are two ways to configure the backends to request a certificate from the frontend.

  1. By setting the TLS directive for each domain that is configured in the Caddyfile, for example;
office.roadrunner {
	tls {
		ca https://caddy.roadrunner/acme/local/directory
		ca_root /etc/ssl/certs/root.crt
	}

Here the FQDN office.roadrunner is told to request a certificate from caddy.roadrunner.

  1. By setting one directive for the whole Caddyfile in the global section.
# Global Option Block

# TLS Options
acme_ca https://caddy.roadrunner/acme/local/directory
acme_ca_root /etc/ssl/certs/root.crt

Here a certificate is requested from caddy.roadrunner for each defined domain (sites) in the Caddyfile.

To request a certificate, a root certificate from the server is required. (Read this blog from Mike Malone to understand why). Once you have started the Caddy service on the frontend for the first time and there are no errors, a file named root.crt is generated in .local/share/caddy/pki/authorities/local

NOTE: The absolute path for the certificate depends on how you run Caddy. Typically when you use systemd the absolute path will be /var/lib/caddy/.local/share/caddy/pki/authorities/local but if you run caddy with caddy start the path .local/share/caddy/pki/authorities/local

You will have to copy this file to the backend and point to it in the Caddyfile. In the above example the file root.crt has been copied to /etc/ssl/certs/root.crt will be created in the in the directory you initiated the command (and where the Caddyfile is located).

If you are working with SSH terminals, you can just copy / paste the content of root.crt to the other terminal with your favourite txt editor, ie vim. The content will look something like this:

-----BEGIN CERTIFICATE-----
Mkd2PrifjSodwIBAgIQICdiSD4Yf9/2x0JL2MKkqzAKBggqhkjOPQQDAjAwMS4w
LAYDVQQDEyVDYWRkeSBMb2NhbCBBdXRob3JpdHkgLSAyMDIxIEVDQyBSb290MB4X
DTIxMDIwMzE3NDKEWRJ#fFJEklejejelWo3oFj3NTYyMlowMDEuMCwGA1UEAxMlQ2FkZHkg
TG9jYWwgQXV0aG9yaXR5IC0gMjAyMSBFQ0MgUm9vdDBZMBMGByqGSM49AgEGCCqG
SM49AwEHA0IABF6n6K32IongBmQmrFuTKCL0LlNWGY/aLUsWQtb6WazWMtEJxoH/
9FTO9zoUl5tPGyKU7yty6xFkejelfFDpoa[fpfWPOfkjewBDMA4GA1UdDwEB/wQEAwIBBjAS
BgNVHRMBAf8ECDAGAQH/AgEBMB0GA1UdDgQWBBShK2n24nbgtXrJTNjmfHIPfUTh
4jAKBggqhkjOPQQDAgNIADBFAiEAvAZzPc05PTyL2ilxcwIB5LEq9HDgl/eEtQok
Fu1VERUCIC4JxfRKk4PfhRcjBJ8Rz0cjYFJR9fOV90wFo9FEKw[pwq
-----END CERTIFICATE-----

Now the only thing left to do is to add a service to the defined domain office.roadrunner. You could directly point to a fileserver (Caddy is also a fileserver) or reverse proxy to another service on this backend. In this example I will use Nextcloud. The config is heavily inspired by this great Wiki by @basil

office.roadrunner {

	root * /var/www/html
	php_fastcgi 127.0.0.1:9000 {
		env front_controller_active true # Remove index.php form url
	}
	file_server

	header Strict-Transport-Security max-age=31536000; # enable HSTS

	# .htaccess / data / config / ... shouldn't be accessible from outside
	@forbidden {
		path /.htaccess
		path /data/*
		path /config/*
		path /db_structure
		path /.xml
		path /README
		path /3rdparty/*
		path /lib/*
		path /templates/*
		path /occ
		path /console.php
	}
	respond @forbidden 404
}

Test

After this you should be ready to caddy run or caddy reload if Caddy is already running.

When someone enters https://nextcloud.my.example.com in his browser a TLS encrypted connection is made to the frontend host with a certificate provided by Let’s Encrypt. The connection is then forwarded with TLS from the local ACME server to https://office.roadrunner on the backend host.

For debugging you can enable the debug option in the global section of each Caddy service.

# Global Option Block
{
	debug
}

Caddyfiles + comments

For completeness I have copied two full Caddyfiles for reference.

Frontend

# Global Option Block
{
	debug		# enable debugging. Comment out when everything is working
}

# ACME Server
acme.roadrunner {		# defining FQDN for ACME server
	acme_server		# defining the ACME server
	tls internal
}

# Nextcloud
nextcloud.my.example.com {		# defining incoming FQDN
	reverse_proxy https://office.roadrunner {
		header_up Host {upstream_hostport}
	}
}

Backend

# Global Option Block
{
	# General Option
	debug

	# TLS Options
	acme_ca https://acme.roadrunner/acme/local/directory # point to ACME server
	acme_ca_root /etc/ssl/certs/root.crt  # define root certificate
}

office.roadrunner {		# FQDN for the backend

        root * /var/www/html
        file_server

        php_fastcgi 127.0.0.1:9000 {
                env front_controller_active true # Remove index.php form url
        }

        header {
                Strict-Transport-Security max-age=31536000; # enable HSTS
        }


        # .htaccess / data / config / ... shouldn't be accessible from outside
        @forbidden {
                path /.htaccess
                path /data/*
                path /config/*
                path /db_structure
                path /.xml
                path /README
                path /3rdparty/*
                path /lib/*
                path /templates/*
                path /occ
                path /console.php
        }

        respond @forbidden 404
}

Disclaimer
Most of the information in this Wiki can be found in either the official documentation or in the many posts and Wiki’s on this forum. I tried to link as much as possible to the original source. Together with a lot of help from @francislavoie and @matt I was able to get the local https to work.

To come
In seperate Wiki’s I will post more Caddyfile examples how to host Nextcloud with Collabora, Bitwarden and Papermerge.

Caddy is really great in doing this because it’s got everything. Most of the times you can strip down the docker containers and remove Apache, Nginx Trafic and Cerbot and just let (one) Caddy service do the job.

7 Likes

I was having a look at the acme directive last night and wondering how I might use it? Lo and behold, I now have a concrete example of its application in the wiki. Writing a well-thought-out wiki article like this one takes a lot of effort. @Rob789 Well done!

@matt You and your small team have done an amazing job of bringing Caddy this far. I’m sure it’s an understatement to say that you’re all being stretched thin on the ground, on top of giving up your free time in the forum. This is by no means a criticism, but I do find much of the documentation terse and difficult to put in context. You do a sterling job of tackling the ‘what’ and ‘how’. This is really a call out to the Caddy community to assist with the ‘when’, ‘where’ and ‘why’; to continue to ‘put some meat on the bones’ and deliver some super useful wiki articles like this one to help unravel the mysteries of the documentation.

Food for thought: The help section of the forum is largely about tapping into the depth and breadth of knowledge the Caddy team has. We’re ever so grateful when the team assists us in solving a problem that has us stumped. The wiki section of the forum is our opportunity to give back to the Caddy team for the tremendous support we receive.

3 Likes

A post was split to a new topic: Reverse proxy parsing difficulties

I was going to try setting up my net like this but then I realized there’s no need to have Caddy be man-in-the-middle, pointlessly decrypting and then re-encrypting the data when it could* just forward the raw data according to SNI (as I don’t need the gateway to do anything else with it). *Well it can’t but I mean I’m thinking to use this instead GitHub - dlundquist/sniproxy: Proxies incoming HTTP and TLS connections based on the hostname contained in the initial request of the TCP session.

Would be cool and especially efficient to have something like this right on the gateway router too.

Of course using different TCP ports or better-yet different IPv6 addresses would be even more efficient/simple but the web has not evolved in such a way that makes that practical :pensive:

TCP ports could be fine if u dont mind the number in urls but getting https for it is a mess cause the ACME challenge needs port 80/443 for that :confused: hey dont convert that emoji i want raw ‘:/

@ledlamp you can use GitHub - mholt/caddy-l4: Layer 4 (TCP/UDP) app for Caddy for this.

1 Like

Oh cool, that looks like awesome functionality for caddy to have!

A post was split to a new topic: Setting up NextCloud behind Caddy

Does the ACME server require using domain names or were you just unable to get it to work with the IP addresses? (@matt for additional insight) For my use-case I would prefer to just use IP addresses and not have a DNS system at all, but it is not obvious to me if that is possible. (Great post btw!)

Public ACME servers currently don’t generally issue certificates for IP addresses. (ZeroSSL I think does issue IP certs, but I’m not sure if their ACME endpoint does.)

Caddy’s internal (but not publicly-trusted) ACME server does issue IP certificates, IIRC.

However if you trust your internal network and the connections are purely internal, you probably don’t need TLS at all.

3 posts were split to a new topic: Local domains with public certs