Help getting caddy with tailscale to work, cant seem to figure this out

Help
1

1. The problem I’m having:

Trying to get reverse_proxy to work with tailscale using tailscale magig-dns and domain hosted through Cloudflare.

I cannot get it to work the old-school way of port-forward and no tailscale as well. I thought this was a firewall-rules issue, but no-one have found an error with my firewall-rules.
I run proxmox as the host. with several CT/VM running, one of the VMs TrueNAS-Scale, and TrueNAS has an instance of Nextcloud and Immich running.

TrueNAS = 192.168.200.50:442 (nas.tsc.heime.pro)
Nextcloud = 192.168.200.50:9001 (cloud.tsc.heime.pro)
Immich = 192.168.200.50:30041 (immich.tsc.heime.pro)

I’m trying to use caddy and tailscale together to avoid port forwarding, I would like to be able to reach these services from outside of my LAN.
I use cloudflare with the acme-dns-cloudflare module, and I get emails from cloudflare saying it has issued certificates, to my subdomains.

I have even installed Cloudflare-rootCA and Origin certs on my Debian vm which caddy and tailscale lives on. 192.168.200.60.
But when I try to use nas.tsc.heime.pro I get an “error this page isn’t working HTTP ERROR 502”
I dont know how to start troubleshooting this any help would be greatly appreciated.

2. Error messages and/or full log output:

journalctl -u caddy --no-pager | less +G`
^^ running this command gives no output.

2024/05/26 03:33:28.008	INFO	using adjacent Caddyfile
2024/05/26 03:33:28.008	INFO	using provided configuration	{"config_file": "Caddyfile", "config_adapter": ""}
2024/05/26 03:33:28.013	INFO	tls.cache.maintenance	started background certificate maintenance	{"cache": "0xc0000b3f00"}
2024/05/26 03:33:28.013	INFO	http.auto_https	server is listening only on the HTTPS port but has no TLS connection policies; adding one to enable TLS	{"server_name": "srv0", "https_port": 443}
2024-05-25 22:33:28.013812449 -0500 CDT m=+0.068846840 write error: can't make directories for new logfile: mkdir <: permission denied
2024-05-25 22:33:28.013610946 -0500 CDT m=+0.068645270 write error: can't make directories for new logfile: mkdir <: permission denied
2024/05/26 03:33:28.014	INFO	http.auto_https	enabling automatic HTTP->HTTPS redirects	{"server_name": "srv0"}
2024-05-25 22:33:28.01425491 -0500 CDT m=+0.069289208 write error: can't make directories for new logfile: mkdir <: permission denied
2024/05/26 03:33:28.014	DEBUG	http.auto_https	adjusted config	{"tls": {"automation":{"policies":[{"subjects":["immich.tsc.heime.pro","cloud.tsc.heime.pro","nas.tsc.heime.pro"]},{"subjects":["nas.heime.pro"]},{}]}}, "http": {"servers":{"remaining_auto_https_redirects":{"listen":[":80"],"routes":[{},{}]},"srv0":{"listen":[":443"],"routes":[{"handle":[{"handler":"subroute","routes":[{"handle":[{"handler":"reverse_proxy","upstreams":[{"dial":"192.168.200.50:30041"}]}]}]}],"terminal":true},{"handle":[{"handler":"subroute","routes":[{"handle":[{"handler":"reverse_proxy","upstreams":[{"dial":"192.168.200.50:9001"}]}]}]}],"terminal":true},{"handle":[{"handler":"subroute","routes":[{"handle":[{"handler":"reverse_proxy","upstreams":[{"dial":"192.168.200.50:442"}]}]}]}],"terminal":true},{"handle":[{"handler":"subroute","routes":[{"handle":[{"handler":"reverse_proxy","upstreams":[{"dial":"192.168.200.50:442"}]}]}]}],"terminal":true}],"tls_connection_policies":[{}],"automatic_https":{}}}}}
2024-05-25 22:33:28.014554302 -0500 CDT m=+0.069588650 write error: can't make directories for new logfile: mkdir <: permission denied
2024/05/26 03:33:28.015	INFO	tls.cache.maintenance	stopped background certificate maintenance	{"cache": "0xc0000b3f00"}
2024-05-25 22:33:28.015511378 -0500 CDT m=+0.070545733 write error: can't make directories for new logfile: mkdir <: permission denied
Valid configuration

3. Caddy version:

v2.7.6 h1:w0NymbG2m9PcvKWsrXO6EEkY9Ru4FJK8bYcev1p3A=

4. How I installed and ran Caddy:

xcaddy build \
 --with github.com/caddy-dns/cloudflare \
 --with github.com/WeidiDeng/caddy-cloudflare-ip \
 --with github.com/tailscale/caddy-tailscale

Move the caddy dir to /usr/bin and export the path:

  • sudo mv caddy /usr/bin/
  • sudo groupadd --system caddy

Create a user named “caddy” with a writable home directory:

sudo useradd --system \
 --gid caddy \
 --create-home \
 --home-dir /var/lib/caddy \
 --shell /usr/sbin/nologin \
 --comment "Caddy web server" \
caddy

Creating the /etc/caddy and Caddy file premissions:

  • mkdir /etc/caddy
  • touch /etc/caddy/Caddyfile
  • chown caddy:caddy /etc/caddy/Caddyfile
  • nano /etc/caddy/Caddyfile

create a systemd config:

  • sudo nano /etc/systemd/system/caddy.service

^ pasted this service text into the caddy.service file.##

####################################################################

# caddy.service
#
# For using Caddy with a config file.
#
# Make sure the ExecStart and ExecReload commands are correct
# for your installation.
#
# See https://caddyserver.com/docs/install for instructions.
#
# WARNING: This service does not use the --resume flag, so if you
# use the API to make changes, they will be overwritten by the
# Caddyfile next time the service is restarted. If you intend to
# use Caddy's API to configure it, add the --resume flag to the
# `caddy run` command or use the caddy-api.service file instead.
[Unit]
Description=Caddy
Documentation=https://caddyserver.com/docs/
After=network.target network-online.target
Requires=network-online.target
[Service]
Type=notify
User=caddy
Group=caddy
ExecStart=/usr/bin/caddy run --environ --config /etc/caddy/Caddyfile
ExecReload=/usr/bin/caddy reload --config /etc/caddy/Caddyfile --force
TimeoutStopSec=5s
LimitNOFILE=1048576
PrivateTmp=true
ProtectSystem=full
AmbientCapabilities=CAP_NET_ADMIN CAP_NET_BIND_SERVICE
[Install]
WantedBy=multi-user.target

after saving ^ the file run:

  • sudo systemctl daemon-reload
  • sudo systemctl enable --now caddy

verify that the service is running:

  • systemctl status caddy

a. System environment:

  • systemd
  • PRETTY_NAME=“Debian GNU/Linux 12 (bookworm)”
  • NAME=“Debian GNU/Linux”
  • VERSION_ID=“12”
  • VERSION=“12 (bookworm)”
  • VERSION_CODENAME=bookworm
  • ID=debian

b. Command:

  • sudo systemctl start caddy
  • caddy reload
  • caddy validate

c. Service/unit/compose file:

d. My complete Caddy config:

{
	# Debugging mode
	debug
	# Email address for cloudflare ssl
	email cloxxxxxre.xxxxxxx@silomails.com
	#
	# Cloudflare API key for Heime.pro acme_dns-module
	acme_dns cloudflare xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
	#
	log [debugging] {
		output file </var/log/caddylog.log>
		#format  <encoder_module>
		#level   <level>
		#include <namespaces...>
		#exclude <namespaces...>
	}
	#
	tailscale {
		# Tailscale auth key used to register nodes.
		auth_key <xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx>
		#
		# Alternate control server URL. Leave empty to use the default server.
		#control_url <control_url>
		#
		# If true, register ephemeral nodes that are removed after disconnect.
		# Default: false (true|false)
		ephemeral false
		#
		# Directory to store Tailscale state in. A subdirectory will be created for each node.
		# The default is to store state in the user's config dir (see os.UserConfDir).
		state_dir /home/kds/.tailscale-state-dir/
		# If true, run the Tailscale web UI for remotely managing the node. (https://tailscale.com/kb/1325)
		# Default: false (true|false)
		webui true
		#
		# Any number of named node configs can be specified to override global options.
		caddy-tsc {
			# Tailscale auth key used to register this node.
			auth_key <xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx>
			#  
			# Alternate control server URL.
			#control_url <control_url>
			#  
			# If true, remove this node after disconnect. true|fals
			ephemeral false
			#  
			# Hostname to request when registering this node.
			# Default: <node_name> used for this node configuration
			hostname caddy-tsc
			#  
			# Directory to store Tailscale state in for this node. No subdirectory is created.
			state_dir /home/kds/.tailscale-state-dir/
			# If true, run the Tailscale web UI for remotely managing this node. true|false
			webui false
		}
	}
}
nas.tsc.heime.pro {
	reverse_proxy 192.168.200.50:442
}
cloud.tsc.heime.pro {
	reverse_proxy 192.168.200.50:9001
}
immich.tsc.heime.pro {
	reverse_proxy 192.168.200.50:30041
}

nas.heime.pro {
	reverse_proxy 192.168.200.50:442
	tls {
		dns cloudflare {xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx}
	}
}

5. Links to relevant resources:

2

If you’ve got debug enabled, and you’re getting a 502 error, and there’s no log entry for it, I suspect that you are not reaching Caddy like you expect you are.

Also, all of that Tailscale config doesn’t have any impact on your heime.pro domain. Tailscale HTTPS only functions over the tailnet domain.

If you want to serve Caddy sites directly into your Tailnet, with the caddy-tailscale plugin, you need to bind Caddy to the Tailscale interface using network listeners.

If you just want to get a Tailscale cert for the node Caddy is running on, then you don’t need to use the plugin. You just define a site with the FQDN of that machine on your Tailnet and Caddy. (/var/run/tailscale/tailscaled.sock needs to be accessible if you need to mount that into containers, etc.)

1 Like
3

Update:

I tried both of these options and I still cant get it to work. I cannot believe that this should be that difficult.

I’ll start over fresh and try one of each of the ways you are pointing out, I think I was in over my head right now. so you are probably correct. I’m just starting out learning this networking stuff.

4

I actually tried this out just to give it a shot serving into my tailnet. Here was the minimum working configuration:

{
	tailscale {
		auth_key tskey-auth-[snip]
	}
}
:80 {
	bind tailscale/hello
	reverse_proxy 172.22.0.3
}

Then I navigated to http://hello.[my-tailnet].ts.net over Tailscale and got a response from the server at 172.22.0.3.

To build out to HTTPS, you currently need to turn auto_https off, which means you’d then need to replicate the HTTP->S redirect behaviour. It would look something like this:

{
	auto_https off
	tailscale {
		auth_key tskey-auth-[snip]
	}
}
:80 {
	bind tailscale/hello
	redir https://{host}
}
:443 {
	bind tailscale+tls/hello
	reverse_proxy 172.22.0.3
}

To change the node name Tailscale creates, you just replace /hello in the bind directive, e.g. bind tailscale/myapp to become accessible at myapp.[my-tailnet].ts.net.

Incidentally, there’s a PR in place as of this past week, looks like, to improve the HTTPS ergonomics: https://github.com/tailscale/caddy-tailscale/pull/53

As for the auth_key, you’ll need to generate one in your Tailscale account. These generated auth keys last for a maximum of 90 days. This obviously looks pretty suboptimal if you want to be able to set-and-forget this server - but it’s not quite as bad as it seems. You only need the auth_key to be present when the node is first provisioned. The tailscale/caddy-tailscale plugin can keep getting certs for the node for as long as that node exists, which it would indefinitely as long as key expiry is disabled. So the auth_key becomes something that you only need to worry about when you wanna add new configuration.

Also, there’s an open feature request for the plugin to support OAuth client support. Using OAuth keys would allow Caddy itself to generate the auth_key as required to authenticate a node. https://github.com/tailscale/caddy-tailscale/issues/48

I honestly think that tailscale/caddy-tailscale is already an incredible work, and when these two additions make their way in, it’s going to be an easy recommendation to literally anyone who’s not opposed to making use of Tailscale.

1 Like
5

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.