Struggling with first real reverse proxy

Help
1

1. The problem I’m having:

I’m trying to setup a simple reverse proxy using two servers. I have setup the two web servers using Ubuntu 23.10 x64. My web app, soulpickleball.com, is on one with nginx php, mysql, and php. It is a very simple web site with two pages.

My Caddy server is setup with only Ubuntu and Caddy. I have Caddy installed as a service on the Ubuntu server and it seems to be running correctly.

soulpickleball.com now points at the IP for the Ubuntu server (it was initially pointed at my web app). My reverse proxy points to the IP for the web app server.

When I go to https://soulpickleball.com it just sits there with no feedback. I see nothing in the logs. It seems I have a fundamental misunderstanding of how to setup a reverse proxy. (Sorry I couldn’t provide a ‘curl -vL’ for this, but I’m not sure how to do it. If you can help with this I’m happy to post any results).

2. Error messages and/or full log output:


No error messages and nothing in the logs according to 'journalctl -u caddy --no-pager | less +G' It just leaves a dead page.

3. Caddy version:

v2.7.5 h1:HoysvZkLcN2xJExEepaFHK92Qgs7xAiCFydN5x5Hs6Q=

4. How I installed and ran Caddy:

I installed Caddy on our Ubuntu server using the instructions for Debian, Ubuntu, Raspbian found on your site. Everything went quite smoothly and Caddy seems to be running as a service.

a. System environment:

1 GB Memory / 25 GB Disk / SFO3 - Ubuntu 23.10 x64

Very plain Ubuntu server with nothing else installed. systemd yes, docker no.

b. Command:

Not currently using any commands, running Caddy as a service. Using a Caddyfile for config.

c. Service/unit/compose file:

Using systemd but nothing else.

d. My complete Caddy config:

Very simple at this point:

https://soulpickleball.com {
	# Another common task is to set up a reverse proxy:
	reverse_proxy https://146.190.125.203 {
		header_up Host soulpickleball.com
	}
}

5. Links to relevant resources:

None currently.

6. Additional comments

My goal is to use the Caddy server to sit between the user and my web app server, handling all the ssl/tls issues. Currently the Caddy server has no certificates setup (hoping that Caddy will handle this). The web app server had a certificate for soulpickleball.com, but since I switched the DNS to point to the Caddy server, I’m not sure it’s valid. But it is important that all communication from the user to → Caddy server → web app server gets encrypted.

Again, I seem to have a fundamental misunderstanding of how all this works, so I’m hopeful someone can straighten me out. With a little direction I’ll understand pretty quickly.

2

Running a curl command just means opening your Terminal or Command Prompt and typing curl -v "https://soulpickleball.com", basically.

When I do, I see:

$ curl -vL "https://soulpickleball.com"
*   Trying 159.223.195.50:443...

and it just sits there.

The good news is, it’s not Caddy! No connection is even being made, or attempting to be made, because it looks like packets aren’t even getting to the machine.

Ensure your network and firewall configurations are correct.

3

You were right, the firewall wasn’t configured to allow http/https. Here are my current ufw settings:

Status: active

To                         Action      From
--                         ------      ----
OpenSSH                    ALLOW       Anywhere
80/tcp                     ALLOW       Anywhere
443                        ALLOW       Anywhere
80,443/tcp                 ALLOW       Anywhere
OpenSSH (v6)               ALLOW       Anywhere (v6)
80/tcp (v6)                ALLOW       Anywhere (v6)
443 (v6)                   ALLOW       Anywhere (v6)
80,443/tcp (v6)            ALLOW       Anywhere (v6)

With this fixed, curl now works, but fails. Here is the output from curl:

» curl -vL https://soulpickleball.com
*   Trying 159.223.195.50:443...
* Connected to soulpickleball.com (159.223.195.50) port 443 (#0)
* ALPN: offers h2,http/1.1
* (304) (OUT), TLS handshake, Client hello (1):
*  CAfile: /etc/ssl/cert.pem
*  CApath: none
* LibreSSL/3.3.6: error:1404B438:SSL routines:ST_CONNECT:tlsv1 alert internal error
* Closing connection 0
curl: (35) LibreSSL/3.3.6: error:1404B438:SSL routines:ST_CONNECT:tlsv1 alert internal error

Note the CAfile does not exist and the CApath is set to none. Do I need to provide for these in the Caddyfile somewhere?

If there are other flaws in the Caddyfile that jump out at you, can you point them out?

4

Now you should have some log output from Caddy. Use journalctl to view the service logs and see what Caddy is saying about failed TLS handshakes.

5

It seems I must have been too quick after chaning the firewall in checking with curl. When I did it again, I got positive results with good TLS handshakes.

So then I gave my reverse proxy another try, but it failed with:

This site can’t provide a secure connection 
www.soulpickleball.com sent an invalid response.
ERR_SSL_PROTOCOL_ERROR

So I checked the logs and this appears to be the culprit:

Nov 14 21:18:19 Dig-Ocean04-Caddy caddy[22109]: {"level":"error","ts":1699996699.89812,"logger":"http.log.error","msg":"tls: failed to verify certificate: x509: cannot validate certificate for 146.190.125.203 beca]

The IP is the correct one that I’m trying to reach. I had created a cert for soulpickleball.com on it when the DNS pointed there. If type in the IP in the browser, I can view the (invalid) cert in http: mode and it looks like this:

Issued to:
 Common Name (CN)	soulpickleball.com
 Organization (O)	<Not Part Of Certificate>

Issued by:
 Common Name (CN)	R3
 Organization (O)	Let's Encrypt
 Organizational Unit (OU)	<Not Part Of Certificate>

Validity Period
 Issued On	Sunday, November 5, 2023 at 6:36:30 AM
 Expires On	Saturday, February 3, 2024 at 6:36:29 AM

SSA-256
Fingerprints
 Certificate	 
 1c56373205caea543bc4a3b3d81aee4859abb5b42ec7d2548cc2e5617d6e38af
 Public Key	 
 3e50448b3e7c48310bdf7923a0cf07f1b9ab94743c5b7408ef1408c69d12caca

So it seems I need to do something on my backend server with regards to the certificate, and maybe something in my Caddyfile as well. This, I think, is where my fundamental misunderstanding of reverse proxy lies. I do need all transmissions between my caddy server and app server to be encrypted.

6

The error message is cut off:

… beca]

What is the rest of it?

Anyway, is your domain www.soulpickleball.com or soulpickleball.com? I don’t see the www. form in your config at all…

7

Sorry, I didn’t see the verbose option at first. Here’s the complete message:

    MESSAGE={"level":"error","ts":1700001941.5305698,"logger":"http.log.error","msg":"tls: failed to verify certificate: x509: cannot validate certificate for 146.190.125.203 because it doesn't contain any IP SANs>

I’m sure this must have something to do with the cert on my app server not being associated with soulpickleball via dns anymore or the IP I’m using for the reverse proxy. Just not sure how to correct it.

The intention is both bare and www domains. I have the dns records set (A and CNAME), but you’re right it’s not in my Caddyfile yet. However, I haven’t tried anything with www yet, so is it important for now?

I really appreciate your help!

8

That’s also cut off, i.e. >

Please read Keep Caddy Running — Caddy Documentation for the recommended command to read your logs.

Is your upstream app in the same network as Caddy? If so, don’t proxy over HTTPS, there’s no benefit to doing so. Proxy over HTTP instead. It’s only beneficial to proxy over HTTPS if there’s risk of attackers getting between your server and upstream, i.e. the traffic is going over the public internet.

Since you configured Caddy to proxy to https://146.190.125.203, it’s expecting to see a certificate which has that IP address in it. But it doesn’t, so it doesn’t validate.

1 Like
9

Thanks. So that error means you are proxying to an IP address, 146.190.125.203 (which is in your config) but the server doesn’t have a certificate with that IP in it.

A simple solution should be to use the hostname instead of the IP address for proxying. However, it seems weird in your case because the hostname you previously pointed at that machine is now pointed to Caddy instead.

That’s fine, but you’ll probably want to either get a new hostname to point to that machine (and replace the certificate there), or generate your own IP cert for that machine. If going over a public network, though, you should use TLS one way or another.

10

That’s exactly my point of confusion!. I want the user to always be connected to soulpickleball.com, but have the files served from 159.223.195.50. And that conversation should be encrypted.

Maybe I can create a subdomain named app.soulpickleball.com on my app server and create a cert pointed to it. I guess it might be preferable to include the IP in the cert, so that it could pass things along to virtual hosts or server blocks. Would that make sense?

1 Like
11

Thank you, I’ll take a look. I want to make you helping me easier for you!

Yes, that seems to be the problem, now how to fix it in the best way.

Thanks!

1 Like
12

Here is the response I got using the recommended “sudo journalctl -u caddy --no-pager | less +G”

Nov 15 01:46:48 Dig-Ocean04-Caddy caddy[22109]: {"level":"error","ts":1700012808.4052055,"logger":"http.log.error","msg":"tls: failed to verify certificate: x509: cannot validate certificate for 146.190.125.203 because it doesn't contain any IP SANs","request":{"remote_ip":"47.242.224.70","remote_port":"47240","client_ip":"47.242.224.70","proto":"HTTP/1.1","method":"GET","host":"soulpickleball.com","uri":"/","headers":{"User-Agent":["Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.9.0.1) Gecko/2008072820 Ubuntu/8.04 (hardy) (Linux Mint)"],"Accept-Encoding":["gzip"]},"tls":{"resumed":false,"version":772,"cipher_suite":4865,"proto":"","server_name":"soulpickleball.com"}},"duration":0.004398345,"status":502,"err_id":"gg52igrvi","err_trace":"reverseproxy.statusError (reverseproxy.go:1265)"}

Doesn’t look like the message was cut off. I used this command at first but the logs were kind of a mess and hard to read, so I looked for a better way of formatting the output. Maybe there isn’t one. I’ll use your command in the future!

Thanks!

2 Likes
13

That’s not possible. ACME issuers don’t issue certificates for IP addresses.

Yeah, you could do that.

14

I’m still struggling with this. I added an SSL cert on my backend server using the subdomain apps,soulpickleball.com. It provides a secure connection when you go to htttps://apps.soulpickleball.com (it’s basically an empty site for now).

Then I update my complete Caddyfile as shown below:

https://soulpickleball.com {

        # Another common task is to set up a reverse proxy:
        reverse_proxy https://apps.soulpickleball.com {
                #header_up Host apps.soulpickleball.com
        }
}

I tried it with and without the header_up directive. Either way, when I go to soulpickleball.com I get a secure, blank page, with the cert that was issued through Caddy. The log also shows the following entry:

Nov 16 19:13:48 Dig-Ocean04-Caddy caddy[22109]: 
{
    "level":"error",
    "ts":1700162028.3892753,
    "logger":"http.log.error",
    "msg":"tls: failed to verify certificate: x509: cannot validate certificate for 146.190.125.203 because it doesn't contain any IP SANs",
    "request":{"remote_ip":"68.8.166.206",
    "remote_port":"62622",
    "client_ip":"68.8.166.206",
    "proto":"HTTP/2.0",
    "method":"GET",
    "host":"soulpickleball.com",
    "uri":"/",
    "headers":{"Accept-Language":["en-US,en;q=0.9"],
    "Sec-Fetch-Dest":["document"],
    "Accept":["text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8"],
    "Sec-Fetch-Site":["none"],"Accept-Encoding":["gzip, deflate, br"],
    "Sec-Fetch-Mode":["navigate"],
    "User-Agent":["Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/17.0 Safari/605.1.15"]},
    "tls":{"resumed":false,
    "version":772,
    "cipher_suite":4865,
    "proto":"h2",
    "server_name":"soulpickleball.com"}},
    "duration":0.004686397,"status":502,
    "err_id":"79uxyzkm2",
    "err_trace":"reverseproxy.statusError (reverseproxy.go:1265)"
}

I can successfully curl -v into both soulpickleball.com and apps.soulpickleball.com. I can provide the results if helpful.

I guess I’m still stumped and struggling…

15

Are you sure your reloaded or restarted Caddy after making your config change? That error doesn’t make sense given your config.

1 Like
16

Thank you, that was the problem. I feel pretty dumb right now! :wink:

2 Likes
17

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.