How about that "This Connection Is Not Secure"

1. The problem I’m having:

Hello! I am back looking for some more help understanding things.

For reference, this post has the background:

To recap, I have been trying to setup self-hosted apps locally and remove the need to use IP address in the browser and from configuration files.

The concept in my head was pretty simple, I have a DHCP server (router) that knows all of the devices on the network. I should be able to ask the router for the IP based on hostname.domain (and probably just by hostname) and get to the app without IPs.

Whitesnake was extremely helpful in getting that issue resolved. Full transparency, I did have some continued struggles, but Whitesnake armed me with much needed context and information which allowed me to better understand what I still had wrong.

The challenge of late has been that darn “This connection is not secure” warnings. I can’t find the post, but I did find a comment on a thread that indicated that I needed to get the root and intermediate certificates from the PKI folder of Caddy onto what I understood as my computer. (The one I am using to submit this on). I put them into my Apply Keychain and trusted them. Made no difference.

I attempted to follow this conversation Local valid SSL cert setup - #5 by Rocker8607 but I wasn’t.

Knowing that I do tend to comprehend better from videos, I turned YT and tried to search but nothing caddy specific seemed to touch the issue, at least not in a way that I could translate into my issue.

What I did find was a video by The_Sudo which was not about Caddy but was about the issue: https://www.youtube.com/watch?v=PQh3sDaPJyg

This video made me question whether or not I was doing the right thing with the certificates from Caddy. In the video, he is using Vault and after generating all the certs and keys, at the end he is taking the root and intermediate certs and putting them into his Heimdall site. I started wondering if that was what I was supposed to be doing as well with Caddy’s.

The other thing, I don’t understand the whole 12hr expiration thing and I can’t imagine that the expectation for Caddy users is to go copy certificates every 12 hours and install everywhere needed.

My relevant Caddyfile section looks like this. Note, I am generalizing the hostnames, but again, the issue is about what I am supposed to be doing with the certs and I can’t imagine anyone needs the hostnames to help. Or maybe these are the actual hostnames if that helps! :wink:

# Wildcard certificate for all internal services
*.internal {
    tls internal
    log {
        output file /var/log/caddy/internal-services.log
        format json
        level DEBUG
    }

    @a host a.internal
    handle @a {
        reverse_proxy a.internal:1234
    }

    @b host b.internal
    handle @b {
        reverse_proxy b.internal:2345
    }

    @c host c.internal
    handle @c {
        reverse_proxy c.internal:3456
    }
}

I appreciate any guidance from the community to help me fill in the gaps of my understanding. Visual, as in videos, content is something I will absorb and understand more fully. I am not looking for do this and then that and would prefer an approach similar to Whitesnake in the original thread. I learned from Whitesnake and was able to use that knowledge to get here.

Thanks!

2. Error messages and/or full log output:

[
	{
		"level": "info",
		"ts": 1724540845.3384328,
		"logger": "http.log.access.log0",
		"msg": "handled request",
		"request": {
			"remote_ip": "192.168.1.5",
			"remote_port": "60014",
			"client_ip": "192.168.1.5",
			"proto": "HTTP/1.1",
			"method": "GET",
			"host": "a.internal",
			"uri": "/signalr/messages?access_token=a4d3455865204d6786bd75ac83632436&id=FX0I3ZtMq05d-UevqfAO3Q",
			"headers": {
				"Connection": ["Upgrade"],
				"Sec-Fetch-Dest": ["websocket"],
				"User-Agent": [
					"Mozilla/5.0 (iPhone; CPU iPhone OS 17_6_1 like Mac OS X) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/17.6 Mobile/15E148 Safari/604.1"
				],
				"Sec-Websocket-Key": ["SGum0PMRLFX85OahJEicug=="],
				"Sec-Websocket-Extensions": ["permessage-deflate"],
				"Accept": ["*/*"],
				"Origin": ["https://a.internal"],
				"Upgrade": ["websocket"],
				"Sec-Fetch-Site": ["same-origin"],
				"Pragma": ["no-cache"],
				"Sec-Websocket-Version": ["13"],
				"Cache-Control": ["no-cache"],
				"Sec-Fetch-Mode": ["websocket"],
				"Accept-Language": ["en-US,en;q=0.9"],
				"Accept-Encoding": ["gzip, deflate"]
			},
			"tls": {
				"resumed": false,
				"version": 772,
				"cipher_suite": 4865,
				"proto": "http/1.1",
				"server_name": "a.internal"
			}
		},
		"bytes_read": 72,
		"user_id": "",
		"duration": 39.267173594,
		"size": 581,
		"status": 101,
		"resp_headers": {
			"Alt-Svc": ["h3=\":443\"; ma=2592000"],
			"Upgrade": ["websocket"],
			"Connection": ["Upgrade"],
			"Date": ["Sat, 24 Aug 2024 23:06:45 GMT"],
			"Cache-Control": ["no-cache, no-store"],
			"Server": ["Caddy", "Kestrel"],
			"Expires": ["-1"],
			"Pragma": ["no-cache"],
			"Sec-Websocket-Accept": ["MwgsQ/AeFXeWEkUDx02rbVbML4U="]
		}
	},
	{
		"level": "info",
		"ts": 1724545179.18059,
		"logger": "http.log.access.log0",
		"msg": "handled request",
		"request": {
			"remote_ip": "192.168.1.2",
			"remote_port": "55978",
			"client_ip": "192.168.1.2",
			"proto": "HTTP/2.0",
			"method": "GET",
			"host": "a.internal",
			"uri": "/",
			"headers": {
				"Accept": [
					"text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8"
				],
				"Sec-Fetch-Site": ["none"],
				"Accept-Encoding": ["gzip, deflate, br"],
				"Sec-Fetch-Mode": ["navigate"],
				"User-Agent": [
					"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/17.6 Safari/605.1.15"
				],
				"Accept-Language": ["en-US,en;q=0.9"],
				"Sec-Fetch-Dest": ["document"]
			},
			"tls": {
				"resumed": false,
				"version": 772,
				"cipher_suite": 4865,
				"proto": "h2",
				"server_name": "a.internal"
			}
		},
		"bytes_read": 0,
		"user_id": "",
		"duration": 0.002500461,
		"size": 1952,
		"status": 200,
		"resp_headers": {
			"Date": ["Sun, 25 Aug 2024 00:19:38 GMT"],
			"Server": ["Caddy", "Kestrel"],
			"Alt-Svc": ["h3=\":443\"; ma=2592000"],
			"Cache-Control": ["no-cache, no-store"],
			"Expires": ["-1"],
			"Pragma": ["no-cache"],
			"Content-Length": ["1952"],
			"Content-Type": ["text/html"]
		}
	}
]

3. Caddy version:

caddy --version
v2.8.4 h1:q3pe0wpBj1OcHFZ3n/1nl4V4bxBrYoSoab7rL9BMYNk=

4. How I installed and ran Caddy:

xbuild caddy

a. System environment:

Caddy is in a Proxmox LXC

b. Command:

systemctl start caddy

c. Service/unit/compose file:

d. My complete Caddy config:

Relevant section, again, this is working for the most part. I am addressing the SSL portion.

# Wildcard certificate for all internal services
*.internal {
    tls internal
    log {
        output file /var/log/caddy/internal-services.log
        format json
        level DEBUG
    }

    @a host a.internal
    handle @a {
        reverse_proxy a.internal:1234
    }

    @b host b.internal
    handle @b {
        reverse_proxy b.internal:2345
    }

    @c host c.internal
    handle @c {
        reverse_proxy c.internal:3456
    }
}

5. Links to relevant resources:

Just continued thanks for the assistance.

Below I’ve provided some content that might help you troubleshoot.

While you can leverage Caddy to manage the certs for you, since you’re having trouble with the trust perhaps take a moment to walk through the process a bit more manually?

Here is a small shell script, it’s just two commands with step-cli, you can just copy/paste to run those individually or as a generate-certs.sh file and run it with sh.

Make sure you change the --san lines to suit your needs.

#!/usr/bin/env bash

# Provisions root CA and leaf certs
# Both are with ECDSA-256 keys + a decade for the expiry date (not advisable).
#
# NOTE: Caddy uses smallstep for this same functionality, where the Caddy data
# directory stores the root CA it creates at `./pki/authorities/local/root.crt`.
# Caddy will offer to install this into the OS trust store and manage renewals
# for your leaf certs.
# Other clients will still need to add root CA to trust stores to avoid warnings.

# Docs for `step-cli certificate create ...`:
# https://smallstep.com/docs/step-cli/reference/certificate/create/


# The Root CA. Your OS has many root CA certs already trusted, like LetsEncrypt.
# NOTE: Any connecting client must have `ca-cert.pem` added to their trust stores
step-cli certificate create 'Smallstep Root CA' ca-cert.pem ca-key.pem \
  --no-password --insecure \
  --profile root-ca \
  --not-before '2024-01-01T00:00:00+00:00' \
  --not-after '2034-01-01T00:00:00+00:00' \
  --kty EC --crv P-256


# Create a leaf cert (what you typically have LetsEncrypt provisions you with).
# Creates a cert for `example.com` that supports any subdomain (via wildcard SAN):
# NOTE: Add or remove `--san` lines to control what FQDN are valid for the cert.
step-cli certificate create 'Smallstep Leaf' cert.pem key.pem \
  --no-password --insecure \
  --profile leaf \
  --ca ca-cert.pem \
  --ca-key ca-key.pem \
  --not-before '2024-01-01T00:00:00+00:00' \
  --not-after '2034-01-01T00:00:00+00:00' \
  --san 'example.com' \
  --san '*.example.com' \
  --kty EC --crv P-256

After you’ve provisioned your root CA and leaf certs, you’ll need each client to trust the root CA cert by adding to their trust store(s):

# You can now install this to the trust store(s)
step-cli certificate install ca-cert.pem
# https://smallstep.com/docs/step-cli/reference/certificate/install
#
# Caddy has equivalent functionality if not using `step-cli`:
# https://caddyserver.com/docs/command-line#caddy-trust
# https://caddyserver.com/docs/command-line#caddy-untrust

# Alternatively the manual way on Linux:
cp ca-cert.pem /usr/local/share/ca-certificates/smallstep-ca.crt
# This command appends the contents to `/etc/ssl/certs/ca-certificates.crt`
# Requires the common `ca-certificates` package if the command is missing.
update-ca-certificates

# For other devices like Android, you'll need to look up how to add the root CA cert to the trust store.
# This is why some prefer to use public DNS-01 challenge (such as with Cloudflare)
# with a public domain name they have, since using LetsEncrypt for certs will
# avoids this requirement for clients.

Next for the leaf certificate, any service that needs this such as Caddy should be configured to use it. So in your Caddyfile you’d swap your tls internal directive for tls /path/to/cert.pem /path/to/key.pem.

Now you should have success with the HTTPS connection and no warning shown if the OS recognizes the root CA from the trust store. If it doesn’t, the software you’re using may have it’s own trust store and need to be configured to be aware of it (some software like curl provides options in the CLI to point to these as another example).

For a web browser you should be able to click a padlock symbol by the URL to inspect the certificate, it should say it’s using the one we named earlier as “Smallstep Leaf”, and you should be able to also get information about related “Smallstep Root CA” :+1:

If that’s not what you see when encountering that security warning something else is misconfigured / misbehaving.

Good day polarathene (note I try to not @ very much).

First, thank you for taking the time to offer some thoughts. I took the opportunity to see what I could learn about smallstep as I recalled reading a post about when it was being integrated with Caddy. I had also come across it online as well but didn’t understand at the time what I was learning about.

I went back to YT and found some more videos where people were using it:

Apalrd’s Adventure: https://www.youtube.com/watch?v=BKCj6A4CHV4&t=729s&pp=ygURc21hbGxzdGVwIGhvbWVsYWI%3D

Sass Drew: https://www.youtube.com/watch?v=aAirdY2fdmM&pp=ygURc21hbGxzdGVwIGhvbWVsYWI%3D

and

Tech Tutorials: https://www.youtube.com/watch?v=fhqnj4J7gpU&pp=ygUMc21hbGxzdGVwIGNh

Blog: Automated Local TLS Certificates With Step-CA | Tech Tutorials

(I also checked out a number of the videos on the smallstep channel)

Tech Tutorials seemed to be most similar to my setup / goal, but I am still a little unclear. This is what I think I learned:

  • Install smallstep ca into it’s own Proxmox LXC so I can give it a hostname and make the service reachable via a domain name.
  • run step-cli command on individual hosts using the fingerprint that is created when setting up ca.

Honestly, there is little confidence in that understanding as it seems vastly different than your steps. I am also not sure if I need to be running the step cli (or manually placing the root cert) in the container of the services or if I am just adding it to caddy (lxc) and my Mac.

Obviously, I need to continue to educate myself as I need to understand what is happening before just running any code. The other thing that was unclear was how to account for *.internal. If I am saying that I want any services on the internal domain to be covered, what I exactly I am doing. Not even sure I need to be doing that.

Anyway, like I said, thank you very much for the suggestion and I will continue to make an effort to learn more.

@polarathene recommending to use Smallstep directly is only going to be more confusing. That’s inherently more complex.

@wdpronovost when you use tls internal, Caddy sets up its own local CA (using Smallstep under the hood, it’s built-in).

The detail you probably missed is you need to install Caddy’s root CA cert to any of the systems that will be connecting to your server, so that they trust the connection. You can find it at /var/lib/caddy/.local/share/caddy/pki/authorities/local/root.crt

For non-internal certs (e.g. Let’s Encrypt), those are automatically trusted without that step because OSes and browsers ship a default trust store, i.e. a list of root certificates from CAs that are widely trusted around the world. But if you have Caddy generate its own CA automatically, obviously that doesn’t exist in the public world so you need to manually trust it.

For install it, google instructions for your various devices OSes and browsers - modern browsers have their own trust stores separate from the OS these days.

2 Likes

It’s really not? I wasn’t suggesting the Smallstep CA service, I was suggesting their step CLI program. Single binary to run, easy to get via that link or via a container.

I actually thought this would simplify the whole troubleshooting process (explained in a latter section). I was not expecting OP to sink time into YT videos on Smallstep CA :man_shrugging:

While my response was a tad verbose, the intent was to provide ample commentary and doc links for additional context, and there was no expectation/requirement to understand the advice too deeply beyond the instructions given.


Simple - Only 3 straight-forward commands to run

Using Smallstep the way I showed above is only 3 commands.

  • step-cli certificate create (create a root cert)
  • step-cli certificate create (create a leaf cert)
  • step-cli certificate install (install the root cert to the trust store, aka caddy trust)
step-cli certificate create 'Smallstep Root CA' ca-cert.pem ca-key.pem \
  --profile root-ca --no-password --insecure

step certificate create 'Smallstep Leaf' cert.pem key.pem \
  --ca ca-cert.pem --ca-key ca-key.pem \
  --profile leaf --no-password --insecure --san 'example.com'

step-cli certificate install ca-cert.pem

Followed by replacing the tls internal directive in their Caddyfile to use tls /path/to/cert.pem /path/to/key.pem instead.


Misunderstandings due to similar named product from Smallstep

The assumption was that this would be a far more transparent approach for troubleshooting by provisioning and configuring the cert files manually (which as can be seen above is very simple).

Once they have that working, they could go back to their implicit Caddy managed setup for certs, and if there was any remaining issues with that, troubleshooting should at least be easier via comparing a working explicit config.

Clearly I made a mistake here with my advice?

I don’t believe I directed anyone towards Smallstep CA, but only the step-cli command (with a download link and doc links), yet both of you got the impression I was referring to the full CA service.

Docker Compose

For those comfortable with Docker Compose, here is a single compose.yaml file that you can copy/paste to run via a container to create the certificates.

# Run this with `docker compose run --rm get-certs`
services:
  get-certs:
    image: smallstep/step-ca
    # Your new cert files will be provisioned into this local `certs/` folder:
    volumes:
      - "./certs/:/tmp/certs/"
    # non-root (1000:1000) by default, change to the UID/GID of your `certs/` dir:
    user: root
    # Everything below is container specific:
    working_dir: /tmp/certs
    entrypoint: /tmp/generate-certs.sh
    configs:
      - source: generate-certs
        target: /tmp/generate-certs.sh
        # Make script executable:
        mode: 0500

# NOTE: The `smallstep/step-ca` container provides `step-cli` as `step` instead
configs:
  generate-certs:
    content: |
      #!/usr/bin/env bash

      step certificate create 'Smallstep Root CA' ca-cert.pem ca-key.pem \
        --profile root-ca --no-password --insecure

      step certificate create 'Smallstep Leaf' cert.pem key.pem \
        --ca ca-cert.pem --ca-key ca-key.pem \
        --profile leaf --no-password --insecure --san 'example.com'

NOTE:

  • The example uses the compose feature configs (instead of volumes) to embed the shell script into the same compose.yaml file. Thus the Docker Compose version must be a release from at least 2024.
  • The example only covers the first two step-cli commands to provision certs, since it’s containerized it’s not really ideal for updating a trust store? :man_shrugging:

I didn’t share this originally as I thought it would only add to any potential confusion (given the image name), even though it’s a simple copy/paste + single command to run.

I did point this out, although it might have been less visible as it was in script comments. I didn’t provide an absolute path since the actual location does vary depending on how Caddy is run (systemd / docker are documented with different paths for Caddy’s data dir IIRC)

I also addressed that with caddy trust and step certificate install, but perhaps it got lost from the verbosity of my response. I’ll need to learn to communicate better like your answers tend to do :sweat_smile:

Sorry for sending you down such a rabbit hole.

  • Smallstep CA is an entirely different beast, that’s a full self-host CA service.
  • step-cli is just a command (single file) you can use like Caddy but offers functionality like creating and inspecting certificates.

I meant to show you roughly the equivalent process Caddy was handling for you to create a root CA + leaf cert and add the root CA into the trust store. I’m not sure how many actually read through my extra commentary like in the scripts shared (perhaps you chose to search more on the topic before seeing that, or I didn’t explain something well enough).

That was the intent of the added commentary and links to relevant docs for step-cli not Smallstep CA. There’s a lot on the topic you could learn about, but don’t really need to know about for the same reason you’d choose to delegate such responsibilities to Caddy. I tried to keep a balance with that.

You can see a compose.yaml example above. You could run that if it’s more comfortable due to the isolation. Likewise you can run the commands in a Dockerfile that extends that image, and inspect it with a tool like dive to see the actual changes made within a container when you do something like adding the root CA to a trust store.

It’s good that you want to understand before blindly running commands though :slight_smile:

Your question isn’t quite clear here.

When you want to use a wildcard certificate in Caddyfile and rely on local_certs / tls internal or the external ACME automation services, Caddy presently only supports the approach you’ve shown with the handle directive in a wildcard site block (as shown in the Caddy docs). There is a PR to provide an opt-in improvement where wildcard certs would be preferred, but it’s unclear when that’ll land.

Without the handle approach, if you have separate site blocks that will currently provision new certs for those site-addresses instead of using the wildcard.

If you use the tls /path/to/cert /path/to/key support instead for external cert management, then each site block using that can reference the wildcard cert and you can avoid the handle workaround.

However, for internal usage it shouldn’t matter too much. Your site blocks can just have Caddy provision their leaf certs via local_certs / tls internal and you’ll be fine.

  • You only need the root CA cert in the trust store (for which all the leaf certs are derived from). The only downside here is you need each client / container to have awareness of this root CA cert, and ensure it’s updated before it would expire. This is usually why some self-host setups prefer to have a public domain for LetsEncrypt with DNS challenge, since the root CA trust maintenance is less hassle.
  • The value of a wildcard leaf cert is often in privacy, when you use LetsEncrypt or similar services there are online sites that will keep a historic log that can be searched, which can reveal some insights of what services you have running if your subdomains reflect that. Other benefits are fairly minor usually.

If you had other concerns in mind with *.internal you’d need to clarify.

Man. @polarathene, you gotta write shorter posts. I don’t have time to read all that. And I don’t think OP has time to either. It really feels like you’re derailing this. Please stay on topic. Not the place for tangential discussion. Recognize who your audience is.

Same thing. Step CLI sets up and manages a CA.

Which also happens to be what Caddy does, built-in, with tls internal. I don’t like that you recommend using other software when the functionality already exists in Caddy itself.

I disagree that there’s a misunderstanding. I know the difference, but the difference between the Smallstep CLI and the service is not relevant to this conversation.

3 Likes

I’ll try, but I’m not sure the reduced context helps.

  • Caddy lacks an equivalent to step certificate create? (to manually reference the leafs via tls /path/to/cert /path/to/key), an explicit command + output files is easier to advise.
  • While Caddy does that implicitly with tls internal, I provided explicit config advice for OP to better grok what Caddy does implicitly, which helps troubleshoot.
  • You stated the smallstep advice was more complex and would confuse OP. The advice was 3 simple commands, basically copy/paste.
  • OP clearly had a misunderstanding with their mentioning of the CA service (which would be much more complex). Thus clarifying the difference after is relevant.

Responding to their questions is not derailing and off-topic?

If there’s something specific that went wildly off-topic let me know. I’ll refrain from expanding on this concern further to keep my response terse.

Good morning!

Regarding the rabbit hole that’s totally a me thing. I try to understand what I am doing and I am not a person that will take code and just run it without knowing what I am doing. I was aware that @polarathene was talking about step cli and the reason step ca was being discussed by me was because when I went to go learn about step cli what I was finding was step ca.

Hopefully this clears up how I took us down the step ca path.

I think it is also possible that since I tend to write long posts some of the facts of what I had done already got missed. I tend to feel a need to provide as much context as possible and that has a downside I am very familiar with but I thought it was fine here because the template we are asked to fill out is verbose on it’s own.

To be clear, the very first thing I did was go to /var/lib/caddy/.local/share/caddy/pki/authorities/local/root.crt and copy it to my Mac which is the device I am physically using. When I took the cert and dropped it into the KeyChain App, I got an error but the cert did stick as well. I was then able to mark it as trusted. I also was under the impression that I needed to install the intermediate and the wildcard *.internal cert. I couldn’t tell from the docs what exactly I was supposed to be doing and why I was doing it. Each time I dropped a cert into KeyChain, I got the same error which I can’t recall the number now and I didn’t get the error now as I deleted the certificates from KeyChain and re-added the root cert.

Because I had installed and trusted the certs and was still getting the warning on my browser (Safari, Firefox, and Chrome) I thought maybe I misunderstood where I was supposed to be installing the certs or that I was doing something wrong altogether which is why I came back for help.

I am still uncertain if I need to be installing the cert in other places (to be understood as on the actual services) or if we are only talking about the device I am actually using … my Mac. I feel like we are just talking about the Mac and if that is the case, then that is what I did.

If the above is correct, and I installed the root on my Mac and that is the only place I needed to install it, then I’ve missed something else it appears. While I did run caddy trust at one time, maybe I need to run it again. The other thing because the error I get is indicating a mismatch of the certs name *.internal and the service being hostname.internal I am thinking the use of the of the wildcard to create a block was incorrect and that I just need to remove that and have the tls internal on each of the individual code block and not use the grouping at all.

I own the way this has progressed. It’s on me and my desire to understand what it is I am doing. I hate the fact that despite how pretty and clean the documentation is for Caddy that for some reason it doesn’t make sense to me. Not that I expect Caddy to teach me about webservers and http requests.

I am going to rework my Caddyfile and see what happens.

1 Like

Whelp, I feel like such a moron now. I just commented out all of the original block of code where I had everything nested under *.internal and just tried 1 of the services and it worked without error.

:cry:

My sincere apologies for wasting anyones time.

I am going to go back to the documentation to see if I can figure out how I went astray.

1 Like

This was due to provisioning a wildcard for an entire TLD (reference).

Some software like curl appears to error about no matching SAN, whereas it would be valid if the wildcard belonged to a domain under the TLD like *.example.internal.

You need to install only the root cert (nothing else, don’t touch the intermediate/leaf, Caddy manages those) on any device that acts as a client to your server. If you only ever connect to your Caddy server using your Mac, then that’s the only device that matters. But if you also connect with your phone or another family member’s laptop/desktop, then those also need the root cert.

And yeah, need to install in the browser’s trust store as well if you’re using that browser to connect since they don’t always check the OS’s trust store.

2 Likes

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.