Struggling with first real reverse proxy

1. The problem I’m having:

I’m trying to setup a simple reverse proxy using two servers. I have setup the two web servers using Ubuntu 23.10 x64. My web app,, is on one with nginx php, mysql, and php. It is a very simple web site with two pages.

My Caddy server is setup with only Ubuntu and Caddy. I have Caddy installed as a service on the Ubuntu server and it seems to be running correctly. now points at the IP for the Ubuntu server (it was initially pointed at my web app). My reverse proxy points to the IP for the web app server.

When I go to it just sits there with no feedback. I see nothing in the logs. It seems I have a fundamental misunderstanding of how to setup a reverse proxy. (Sorry I couldn’t provide a ‘curl -vL’ for this, but I’m not sure how to do it. If you can help with this I’m happy to post any results).

2. Error messages and/or full log output:

No error messages and nothing in the logs according to 'journalctl -u caddy --no-pager | less +G' It just leaves a dead page.

3. Caddy version:

v2.7.5 h1:HoysvZkLcN2xJExEepaFHK92Qgs7xAiCFydN5x5Hs6Q=

4. How I installed and ran Caddy:

I installed Caddy on our Ubuntu server using the instructions for Debian, Ubuntu, Raspbian found on your site. Everything went quite smoothly and Caddy seems to be running as a service.

a. System environment:

1 GB Memory / 25 GB Disk / SFO3 - Ubuntu 23.10 x64

Very plain Ubuntu server with nothing else installed. systemd yes, docker no.

b. Command:

Not currently using any commands, running Caddy as a service. Using a Caddyfile for config.

c. Service/unit/compose file:

Using systemd but nothing else.

d. My complete Caddy config:

Very simple at this point: {
	# Another common task is to set up a reverse proxy:
	reverse_proxy {
		header_up Host

5. Links to relevant resources:

None currently.

6. Additional comments

My goal is to use the Caddy server to sit between the user and my web app server, handling all the ssl/tls issues. Currently the Caddy server has no certificates setup (hoping that Caddy will handle this). The web app server had a certificate for, but since I switched the DNS to point to the Caddy server, I’m not sure it’s valid. But it is important that all communication from the user to → Caddy server → web app server gets encrypted.

Again, I seem to have a fundamental misunderstanding of how all this works, so I’m hopeful someone can straighten me out. With a little direction I’ll understand pretty quickly.

Running a curl command just means opening your Terminal or Command Prompt and typing curl -v "", basically.

When I do, I see:

$ curl -vL ""
*   Trying

and it just sits there.

The good news is, it’s not Caddy! No connection is even being made, or attempting to be made, because it looks like packets aren’t even getting to the machine.

Ensure your network and firewall configurations are correct.

You were right, the firewall wasn’t configured to allow http/https. Here are my current ufw settings:

Status: active

To                         Action      From
--                         ------      ----
OpenSSH                    ALLOW       Anywhere
80/tcp                     ALLOW       Anywhere
443                        ALLOW       Anywhere
80,443/tcp                 ALLOW       Anywhere
OpenSSH (v6)               ALLOW       Anywhere (v6)
80/tcp (v6)                ALLOW       Anywhere (v6)
443 (v6)                   ALLOW       Anywhere (v6)
80,443/tcp (v6)            ALLOW       Anywhere (v6)

With this fixed, curl now works, but fails. Here is the output from curl:

» curl -vL
*   Trying
* Connected to ( port 443 (#0)
* ALPN: offers h2,http/1.1
* (304) (OUT), TLS handshake, Client hello (1):
*  CAfile: /etc/ssl/cert.pem
*  CApath: none
* LibreSSL/3.3.6: error:1404B438:SSL routines:ST_CONNECT:tlsv1 alert internal error
* Closing connection 0
curl: (35) LibreSSL/3.3.6: error:1404B438:SSL routines:ST_CONNECT:tlsv1 alert internal error

Note the CAfile does not exist and the CApath is set to none. Do I need to provide for these in the Caddyfile somewhere?

If there are other flaws in the Caddyfile that jump out at you, can you point them out?

Now you should have some log output from Caddy. Use journalctl to view the service logs and see what Caddy is saying about failed TLS handshakes.

It seems I must have been too quick after chaning the firewall in checking with curl. When I did it again, I got positive results with good TLS handshakes.

So then I gave my reverse proxy another try, but it failed with:

This site can’t provide a secure connection sent an invalid response.

So I checked the logs and this appears to be the culprit:

Nov 14 21:18:19 Dig-Ocean04-Caddy caddy[22109]: {"level":"error","ts":1699996699.89812,"logger":"http.log.error","msg":"tls: failed to verify certificate: x509: cannot validate certificate for beca]

The IP is the correct one that I’m trying to reach. I had created a cert for on it when the DNS pointed there. If type in the IP in the browser, I can view the (invalid) cert in http: mode and it looks like this:

Issued to:
 Common Name (CN)
 Organization (O)	<Not Part Of Certificate>

Issued by:
 Common Name (CN)	R3
 Organization (O)	Let's Encrypt
 Organizational Unit (OU)	<Not Part Of Certificate>

Validity Period
 Issued On	Sunday, November 5, 2023 at 6:36:30 AM
 Expires On	Saturday, February 3, 2024 at 6:36:29 AM

 Public Key	 

So it seems I need to do something on my backend server with regards to the certificate, and maybe something in my Caddyfile as well. This, I think, is where my fundamental misunderstanding of reverse proxy lies. I do need all transmissions between my caddy server and app server to be encrypted.

The error message is cut off:

… beca]

What is the rest of it?

Anyway, is your domain or I don’t see the www. form in your config at all…

Sorry, I didn’t see the verbose option at first. Here’s the complete message:

    MESSAGE={"level":"error","ts":1700001941.5305698,"logger":"http.log.error","msg":"tls: failed to verify certificate: x509: cannot validate certificate for because it doesn't contain any IP SANs>

I’m sure this must have something to do with the cert on my app server not being associated with soulpickleball via dns anymore or the IP I’m using for the reverse proxy. Just not sure how to correct it.

The intention is both bare and www domains. I have the dns records set (A and CNAME), but you’re right it’s not in my Caddyfile yet. However, I haven’t tried anything with www yet, so is it important for now?

I really appreciate your help!

That’s also cut off, i.e. >

Please read Keep Caddy Running — Caddy Documentation for the recommended command to read your logs.

Is your upstream app in the same network as Caddy? If so, don’t proxy over HTTPS, there’s no benefit to doing so. Proxy over HTTP instead. It’s only beneficial to proxy over HTTPS if there’s risk of attackers getting between your server and upstream, i.e. the traffic is going over the public internet.

Since you configured Caddy to proxy to, it’s expecting to see a certificate which has that IP address in it. But it doesn’t, so it doesn’t validate.

1 Like

Thanks. So that error means you are proxying to an IP address, (which is in your config) but the server doesn’t have a certificate with that IP in it.

A simple solution should be to use the hostname instead of the IP address for proxying. However, it seems weird in your case because the hostname you previously pointed at that machine is now pointed to Caddy instead.

That’s fine, but you’ll probably want to either get a new hostname to point to that machine (and replace the certificate there), or generate your own IP cert for that machine. If going over a public network, though, you should use TLS one way or another.

That’s exactly my point of confusion!. I want the user to always be connected to, but have the files served from And that conversation should be encrypted.

Maybe I can create a subdomain named on my app server and create a cert pointed to it. I guess it might be preferable to include the IP in the cert, so that it could pass things along to virtual hosts or server blocks. Would that make sense?

1 Like

Thank you, I’ll take a look. I want to make you helping me easier for you!

Yes, that seems to be the problem, now how to fix it in the best way.


1 Like

Here is the response I got using the recommended “sudo journalctl -u caddy --no-pager | less +G”

Nov 15 01:46:48 Dig-Ocean04-Caddy caddy[22109]: {"level":"error","ts":1700012808.4052055,"logger":"http.log.error","msg":"tls: failed to verify certificate: x509: cannot validate certificate for because it doesn't contain any IP SANs","request":{"remote_ip":"","remote_port":"47240","client_ip":"","proto":"HTTP/1.1","method":"GET","host":"","uri":"/","headers":{"User-Agent":["Mozilla/5.0 (X11; U; Linux i686; en-US; rv: Gecko/2008072820 Ubuntu/8.04 (hardy) (Linux Mint)"],"Accept-Encoding":["gzip"]},"tls":{"resumed":false,"version":772,"cipher_suite":4865,"proto":"","server_name":""}},"duration":0.004398345,"status":502,"err_id":"gg52igrvi","err_trace":"reverseproxy.statusError (reverseproxy.go:1265)"}

Doesn’t look like the message was cut off. I used this command at first but the logs were kind of a mess and hard to read, so I looked for a better way of formatting the output. Maybe there isn’t one. I’ll use your command in the future!



That’s not possible. ACME issuers don’t issue certificates for IP addresses.

Yeah, you could do that.

I’m still struggling with this. I added an SSL cert on my backend server using the subdomain apps, It provides a secure connection when you go to htttps:// (it’s basically an empty site for now).

Then I update my complete Caddyfile as shown below: {

        # Another common task is to set up a reverse proxy:
        reverse_proxy {
                #header_up Host

I tried it with and without the header_up directive. Either way, when I go to I get a secure, blank page, with the cert that was issued through Caddy. The log also shows the following entry:

Nov 16 19:13:48 Dig-Ocean04-Caddy caddy[22109]: 
    "msg":"tls: failed to verify certificate: x509: cannot validate certificate for because it doesn't contain any IP SANs",
    "Sec-Fetch-Site":["none"],"Accept-Encoding":["gzip, deflate, br"],
    "User-Agent":["Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/17.0 Safari/605.1.15"]},
    "err_trace":"reverseproxy.statusError (reverseproxy.go:1265)"

I can successfully curl -v into both and I can provide the results if helpful.

I guess I’m still stumped and struggling…

Are you sure your reloaded or restarted Caddy after making your config change? That error doesn’t make sense given your config.

1 Like

Thank you, that was the problem. I feel pretty dumb right now! :wink: