Caddy reverse proxy to https upstream, only works with tls_insecure_skip_verify

1. The problem I’m having:

Setup reverse proxy to https server (nextcloud) upstream.

Caddy is working great as a reverse proxy for other services. When I try to add a nextcloud vm, I can only get it to work using tls_insecure_skip_verify, which as I understand is a bad idea.
I have tried configuring the Caddyfile with and without forwarding the header, but I get the error in the logs shown below. Configuring Caddy to skip TLS verification works, but again as I understand, that is not something I want to do permanently?

-edit I’ve added caddy’s ip address as a trusted proxy in nextcloud’s config file.

2. Error messages and/or full log output:

Oct 24 22:01:13 caddy caddy[147]: {"level":"error","ts":1729832473.3701081,"logger":"http.log.error","msg":"tls: failed to verify certificate: x509: cannot validate certificate for 192.168.2.217 because it doesn't contain any IP SANs","request":{"remote_ip":"192.168.2.1","remote_port":"53462","client_ip":"192.168.2.1","proto":"HTTP/3.0","method":"GET","host":"cloud.rulytafzil.com","uri":"/","headers":{"User-Agent":["Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:131.0) Gecko/20100101 Firefox/131.0"],"Accept-Encoding":["gzip, deflate, br, zstd"],"Priority":["u=0, i"],"Alt-Used":["cloud.rulytafzil.com"],"Sec-Fetch-Site":["none"],"Accept":["text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/png,image/svg+xml,*/*;q=0.8"],"Cookie":["REDACTED"],"Upgrade-Insecure-Requests":["1"],"Accept-Language":["en-US,en;q=0.5"],"Sec-Fetch-Dest":["document"],"Sec-Fetch-Mode":["navigate"],"Sec-Fetch-User":["?1"]},"tls":{"resumed":true,"version":772,"cipher_suite":4865,"proto":"h3","server_name":"cloud.rulytafzil.com"}},"duration":0.001556261,"status":502,"err_id":"3d4rupffk","err_trace":"reverseproxy.statusError (reverseproxy.go:1269)"}

3. Caddy version:

2.8.4

4. How I installed and ran Caddy:

a. System environment:

Debian LXC in proxmox. Installed via the instruction in the docs.

curl -1sLf 'https://dl.cloudsmith.io/public/caddy/stable/gpg.key' | sudo gpg --dearmor -o /usr/share/keyrings/caddy-stable-archive-keyring.gpg
curl -1sLf 'https://dl.cloudsmith.io/public/caddy/stable/debian.deb.txt' | sudo tee /etc/apt/sources.list.d/caddy-stable.list
sudo apt update
sudo apt install caddy

b. Command:

Runs as a systemd service. I’m using the Caddyfile.

c. Service/unit/compose file:

not running it in docker.

d. My complete Caddy config:

What I’m currently using, that “works”:

wiki.rulytafzil.com {
	reverse_proxy 192.168.2.231:8080
}

cloud.rulytafzil.com {
	reverse_proxy https://192.168.2.217:443 {
		transport http {
			tls
			tls_insecure_skip_verify
		}
	}
}

The config I thought should work, but gives the error shown above:

wiki.rulytafzil.com {
	reverse_proxy 192.168.2.231:8080
}

cloud.rulytafzil.com {
	reverse_proxy https://192.168.2.217:443 {
		header_up Host {upstream_hostport}
	}
}

5. Links to relevant resources:

Howdy @Rukiri, welcome to the Caddy community.

HTTPS essentially sets out to achieve two goals:

  1. Prevent snooping by providing a platform to negotiate encryption of communication between server and client.
  2. Assure you, the client, that the server you’re talking to is legitimate and not some possibly-malicious impostor.

Both of these are way more important over the open internet than they are over your local area network. By configuring tls_insecure_skip_verify, you’re telling Caddy to go ahead with #1 but don’t worry at all about #2. Obviously this is very much throwing out one of the major benefits of HTTPS.

The reason we’re opinionated against skipping verification is because that with HTTPS still in place, users may be misled into thinking their setup is secure and trustworthy, when in fact a bad actor could try to impersonate your server and the skipped verification will not prevent this. We usually advise people to simply proxy over HTTP - within the context of a local area network where you have control over a relatively limited number of devices, insecure HTTP is usually plenty safe.

If you’d prefer not to use HTTP and want to get fully validated HTTPS working, though…

I’m guessing this error is because you’re proxying to an IP address and the certificate the upstream is presenting probably has a domain name instead.

The examples in the reverse_proxy documentation do a little more explaining, including the use of tls_server_name in order to tell Caddy what domain name it should be validating against on the upstream certificate.

Reverse proxy to an HTTPS upstream, but :warning: disable TLS verification. This is NOT RECOMMENDED, since it disables all security checks that HTTPS offers; proxying over HTTP in private networks is preferred if possible, because it avoids the false sense of security:

example.com {
	reverse_proxy 10.0.0.1:443 {
		transport http {
			tls_insecure_skip_verify
		}
	}
}

Instead you may establish trust with the upstream by explicitly trusting the upstream’s certificate, and (optionally) setting TLS-SNI to match the hostname in the upstream’s certificate:

example.com {
	reverse_proxy 10.0.0.1:443 {
		transport http {
			tls_trusted_ca_certs /path/to/cert.pem
			tls_server_name app.example.com
		}
	}
}

https://caddyserver.com/docs/caddyfile/directives/reverse_proxy#examples

You may not be required to use tls_trusted_ca_certs if your upstream’s certificate is publicly trusted. (It’s also been deprecated and should be tls_trust_pool instead.)

1 Like

Thank you for the detailed response. That really helps me understand what’s going on better. I confess I saw that part of the documentation but didn’t understand what it was saying, but seeing it after your explanation makes it clear.

You’re exactly right, the existing certificate for the nextcloud vm was created with letsencrypt and was for the domain.

Adding tls_sever_name fixed my issue! thank you!

I am curious though, when behind a reverse proxy, instead of configuring Caddy to trust the upstream certificate, is it better practice to turn off the upstream server’s ssl and leave that to Caddy? It seems like the upstream’s certificate doesn’t serve a purpose anymore?

Similarly, it seems like I’ll need to edit certbot on the vm to authenticate over webroot instead of tls in the future? to be able to authenticate behind a proxy?

No worries!

I wouldn’t say better practice - either way is pretty valid. Much like Caddy’s certificate now provides security to clients connecting from elsewhere, Nextcloud’s certificate is providing security to its own, likely singular, client - Caddy. It’s ensuring the last leg of the journey is safe. If you have this set up and automated, there’s almost no reason not to continue on this way.

Like you say, though, you’ll probably need to configure certbot on the Nextcloud VM to use validation other than TLS-ALPN, as that won’t work with Caddy in front. Your options are HTTP validation or DNS validation.

There’s a minor possible pitfall with proceeding with HTTP validation, which is that if your Nextcloud certificate runs out without getting renewed, Caddy will stop talking to it (because of the required validation) and won’t pass on the HTTP validation request, locking it out of renewing the certificate again until you manually intervene. That might not ever happen, though, as long as you maintain enough uptime on Caddy and the Nextcloud VM to keep renewing those certificates ahead of expiry. You could also go to DNS validation to solve that.

Or, like some people do - in a LAN context especially, if you’re happy with your relative level of internal network security, and most people have no reason not to be - just reverse proxy HTTP over the LAN instead.

1 Like