HOW TO: Reverse Proxy with wildcard ssl's with private key password

Help
1

1. The problem I’m having:

Where to place, and how to configure, reverse proxy ssl certs with private key password?

2. Error messages and/or full log output:

Not getting any errors yet, trying to work out how to do it

3. Caddy version:

2.7.4 on MacOS

4. How I installed and ran Caddy:

https://ports.macports.org/port/caddy/

a. System environment:

MacOS Catalina on 2012 i7 Mac Mini

b. Command:

None Yet

d. My complete Caddy config:

None yet, still at planning stage

5. Links to relevant resources:

I need to set-up a reverse proxy to a number of individual hardware (MacOS) database servers that are all on subdomains to example.com
Google tells me Caddy might be the best solution and the easiest to configure.
My environment is all MacOS, I have downloaded and installed Caddy 2.7.4 via MacPorts, on a Mac Mini running Catalina.

I am now planning the configuration of the reverse proxy and trying to understand how to manage the SSL certificates.

DNS is managed by my ISP / IT Contractor. I have a commercial wildcard ssl cert issued by GoDaddy for: *.example.com
It is in 3 parts
123456.cert
123456.pem
bundle.cert
Research suggests I need to concatenate the two certs and place this and the pem in etc/ssl/ and add a line to the .caddy file
tls /etc/ssl/concatenate-bundle.cert /etc/ssl/123456.pem
With the correct DNS configuration this should give me https://proxy.example.com for the caddy server.

HOWEVER I don’t see any documention for how to place or manage the Reverse Proxy ssl certs for the 3x database servers, data1.example.com; data2.example.com; data3.example.com; each have the identical *.example.com ssl certs installed.

The database servers ssl pems are created via the certificate signing request/private key file/private key password process.

The database servers use the 3x certs: 123456.cert; bundle.cert; and key.pem (generated from 123456.pem using the private key password when installed). NOTE once I have generated key.pem I can copy it to the other database servers, because they are all using the identical set of wildcard certs for *.example.com

MY QUESTION IS: Where to place and how to configure the reverse proxy ssl certs for the database servers?

MAYBE THIS FOR EACH DATABASE SERVER? tls /etc/ssl/123456.cert /etc/ssl/bundle.cert /etc/ssl/key.pem

DO I NEED TO SCRIPT PROVIDING the private key password for key.pem, IF SO, HOW?

Thanks in advance for any help!

2

Hi there.

Password-protected private keys don’t really make sense on servers, unless you want to type it in manually every time the server reboots or something.

The way system permission models work, if you stored the password there, you wouldn’t really have any protection because an attacker wouldn’t have to cross permission boundaries to access the password if they already have access to the key.

I would use OpenSSL to remove the password once the key is safely stored with proper permissions:

3

Hi Matt, Are we talking about the same password? On my Db server the encryption password on the private key file is compulsory. It is created when creating the certificate signing request, and is only entered once, when installing the certs received back from the CA. It is not required after every server reboot.

  1. Anyway, suppose I remove the password from key.pem as described above, on caddy am I then going to follow the same 3 part cert scheme as my Db server SSL store?

tls /etc/ssl/123456.cert /etc/ssl/bundle.cert /etc/ssl/key.pem

Or do I still concatenate the certs?

  1. Given that when requesting the wildcard certs from GoDaddy, I had to provide an encryption password, does that mean I will also have to remove the password from 123456.pem so that it will work for the caddy server?

tls /etc/ssl/concatenate-bundle.cert /etc/ssl/123456.pem

4

To clarify, you’ll want to give Caddy the certificate bundle (that’s the leaf cert and intermediate – usually two total, SOMETIMES three but that’s rare I think) as a single file, and the private key as a separate file. Both should be PEM-encoded. If you don’t do the bundle, browsers/clients will reject the certificate because they only trust roots, not intermediates.

I don’t know what GoDaddy does, but you’ll want to make sure that you have those requirements met ^.

Then yes, use the tls directive. Make sure key.pem is not encrypted.

If you’re not sure you can use openssl commands to inspect; or just try loading it with Caddy, and if it gives you an error, you will know :slight_smile:

5

thanks Matt, I removed the encryption password from key.pem and concatenated the certs and converted that to .pem and used the tls directive in a very basic caddyfile

I have caddy serving a secure https page however it is giving me a number of permission errors and warnings about depreciated methods so I will start a new help request to resolve those.

1 Like
6

Sounds good! Congrats on getting it working anyway.

7

Hi Matt, now following on re my initial question above about reverse_proxy from a caddyserver on a subdomain, to a number of Db servers on subdomains. I cannot find an example in the documentation or other sources that covers this exact scenario.

MY QUESTION: Do I need to declare the (identical set of) ssl certificates for each upstream server?

My current caddyfile:

serve.example.com
tls /etc/ssl/certificate.pem /etc/ssl/key.pem
respond “Hiya World!”

Proposed reverse_proxy caddyfile: (with identical certs declared for one upstream DbServer)

serve.example.com
 tls /etc/ssl/certificate.pem /etc/ssl/key.pem{ 
  reverse_proxy data1.example.com
  tls /etc/ssl/certificate.pem /etc/ssl/key.pem
}

NOTE: the DbServers each have their own identical set of ssl certs installed for *.example.com

8

Are you getting trying to proxy to your upstream over HTTPS? If so, see reverse_proxy (Caddyfile directive) — Caddy Documentation

If your upstream is serving a publicly trusted cert (signed by a public CA) then there should be nothing else to do because Caddy will use your system’s trust store to verify the cert’s signature. If not, then you can configure tls_trusted_ca_certs to give Caddy the root cert to trust (or just add that root cert to your system’s trust store yourself).

Please remember to use code blocks in your posts, it’s important for it to be readable.

9

Hey @francislavoie, yes as I say several times above all my servers including CaddyServer (serve.example.com) and the upstream DbServers (data1.example.com, data2.example.com, etc) have the same commercial wildcard ssl (*.example.com).

This is provisioned to serve.example.com in the caddyfile directly above (Line2 as follows)

tls /etc/ssl/certificate.pem /etc/ssl/key.pem

The DbServers (separate individual MacOS machines) each have their own trust stores with the same wildcard ssl *.example.com certs installed.

So if I understand you correctly, I do not need to reference the tls directive an additional time in the caddyfile for each reverse_proxy, and to reverse_proxy to just one DbServer, in the simplest form, the caddyfile could look like this?

serve.example.com
 tls /etc/ssl/certificate.pem /etc/ssl/key.pem{ 
  reverse_proxy https://data1.example.com
 }
10

No, your { } are in the wrong spot. The tls directive doesn’t take other directives as subdirectives. Both reverse_proxy and tls are separate directives, and directives go within a site block.

It should look like this:

serve.example.com {
	tls /etc/ssl/certificate.pem /etc/ssl/key.pem
	reverse_proxy https://data1.example.com
}

See Caddyfile Concepts — Caddy Documentation

The tls directive configures TLS for the server. The reverse_proxy is an HTTP client and you may configure how it performs TLS connections via the HTTP transport config. You said your root cert is in your system’s trust store, so no config should be necessary if that’s the case

But you may still need to override the Host header as per my first link in my previous comment.

Did you try it? What’s in Caddy’s logs? What behaviour are you seeing?

1 Like
11

Thanks for your assistance, it is the middle of the night in my timezone and I will not be able to test this for 12 hours or so as I need my IT contractor to provision the test environment. I will report back as soon as I have a result. Thanks again.

12

Hi @francislavoie OK we tried it using your caddyfile example above and got connection refused as follows:

2023/10/03 20:51:45.430	ERROR	http.log.error	dial tcp 999.53.148.67:443: connect: connection refused	{"request": {"remote_ip": "999.53.148.1", "remote_port": "49643", "client_ip": "999.53.148.1", "proto": "HTTP/2.0", "method": "GET", "host": "serve.example.com", "uri": "/", "headers": {"Sec-Ch-Ua-Mobile": ["?0"], "Upgrade-Insecure-Requests": ["1"], "User-Agent": ["Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/117.0.0.0 Safari/537.36 Edg/117.0.2045.47"], "Accept": ["text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.7"], "Sec-Fetch-Mode": ["navigate"], "Sec-Fetch-User": ["?1"], "Sec-Fetch-Dest": ["document"], "Accept-Language": ["en-US,en;q=0.9,en-NZ;q=0.8"], "Sec-Ch-Ua": ["\"Microsoft Edge\";v=\"117\", \"Not;A=Brand\";v=\"8\", \"Chromium\";v=\"117\""], "Sec-Ch-Ua-Platform": ["\"Windows\""], "Sec-Fetch-Site": ["none"], "Accept-Encoding": ["gzip, deflate, br"]}, "tls": {"resumed": false, "version": 772, "cipher_suite": 4865, "proto": "h2", "server_name": "data1.example.com"}}, "duration": 0.05199789, "status": 502, "err_id": "e5zpyqtjy", "err_trace": "reverseproxy.statusError (reverseproxy.go:1248)"}

It looks to me like the caddyserver “serve.example.com” is refusing to accept the initial connection?

Per your suggestion to overide the host header, when I try that I get:

Error: parsing caddyfile tokens for 'reverse_proxy': Caddyfile:3 - Error during parsing: due to parsing difficulties, placeholders are not allowed when an upstream address contains a scheme, import chain: ['']
13

No, this is saying that the connection from Caddy to your upstream app was refused. Are you sure that’s the right IP address? (it’s obvious you obfuscated it, 999 is not a valid number in an IPv4 address), are you sure your firewall isn’t blocking the connection, etc?

What exactly did you write in your Caddyfile? That’s a specific config error.

1 Like
14

thanks, I will look to see why the upstream refused the connection, there is no firewall or authentication in play, the incoming url requests a static document, which is served by the upstream.

Could it relate to the host header issue? the Caddyfile that gave the Error: parsing caddyfile tokens etc follows:

serve.example.com {
	tls /etc/ssl/certificate.pem /etc/ssl/key.pem
	reverse_proxy https://data1.example.com{
	header_up Host {upstream_hostport}
}
}
15

You must use a space before {, otherwise Caddy thinks it’s part of your upstream address. That’s why you get that error.

1 Like
16

Thanks for all your help guys, the syntax correction re space before { has corrected the caddyfile, however the upstream is still refusing connection.

We can now reverse proxy using apache in the same environment, so it is not some inherent problem with the upstream box. It was Matt’s advice about reconfiguring the SSL certs that got apache going, so a big thank you for that.

I’m impressed with Caddy’s apparent simplicity and brevity compared to the alternatives and I would love to make this work, however I’ve now run out of time, and will have to park this project for a while. When I can come back to it I will raise any issues as new topics, thanks again.

1 Like
17

Same IP, same port, from the same machine? That doesn’t make sense if so. There’s nothing different about how Caddy makes requests upstream vs any other HTTP client on the same machine.

:blush:

1 Like
18

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.