V2: Local development self signed certs

Hello, I am using caddy 2 beta 10.

From having a look through the source code I don’t think the self signed certs have been ported to v2.

I think my ideal state is that we use self signed certs for local development and lets encrypt for production.

I wanted to check that I am wanting the right thing, and have I missed that it is already available?

I would be curious about a mechanism that could take a caddyfile and environmentally switch between self signed and lets encrypt modes.

Cheers

Mark

Great question.

v2 doesn’t yet have a mechanism for generating and using self-signed certificates. For now, you can generate one yourself and then load it up manually like any other certificate.

My longer-term plan is to embed Smallstep CA, which is also an ACME server, which I think is a much more desirable alternative to “self-signed” certs and will scale a lot better.

Right I think I see, we could flick between ACME servers in dev and prod? Possibly just a url or a switch to indicate which ACME configuration?

Have you got any thoughts on the configuration for making that work, I would be interested in helping out.

1 Like

Yes, something like that!

This would probably be the same component which is used for automatic mTLS in clusters – basically an internal PKI that is fully-managed for you – which we’re planning.

In the case where the ACME server is internal-only, we may not even need to open a socket, and might be able to use function calls directly to provide self-signed certs.

So, I’m not sure yet what the solution looks like in great detail. But the basic idea is you should be able to “flip a switch” and get self-signed certs from a locally-trusted internal CA that auto-renew and are just for the specific (internal) hostnames you need.

And I would really love some help here, because it’s kind of a big task to take on alone! How involved can you be? I would really like to work with someone else to develop this feature.

I have some capacity during my day job and have some summer holidays coming up which I like to have a project to work on (some coding zen).

Perhaps some idea about how you think Caddy and Small Step CA would interact would be interesting. If Small Step would be completely embedded into Caddy (licensing) or as a plugin / go module.

Curious about how you want to approach the design process, I haven’t done much of this in an open source manner so happy to take your lead on this.

Great! Letsee…

FWIW, almost every feature of Caddy is a “plugin” – the question is mainly whether we should include it as a standard plugin that ships with mainline builds of Caddy, or whether it is hosted in another repo and is its own project. Frankly, I think this is (eventually) a big-enough endeavor that it might be its own project. Still, where the code is hosted and stuff is actually pretty inconsequential as far as its development is concerned. Caddy doesn’t care whether a plugin is in the same repo or not, it’s all the same.

Sure. I need to first gain some more familiarity with Smallstep, before I know what the design/config would look like. It’d take some dedicated effort which I can’t spare right now as I have some work to do for a customer and I’ve got to finish up the new documentation system, as well as finish prepping a
stable 2.0 release in Q1 2020… I can definitely use help for this, but I am not sure how quickly it will happen right now, is that okay?

It’s definitely on my roadmap. :slight_smile: However, we can prioritize this right away if a customer needs this sooner!

Hi Matt,

I have had a bit of a play with small-step, I found your suggestion on their issues list. Basically stalking you at this point.:sweat_smile:

My discoveries sit like this at the moment.

Publicly accessible site options:

  • lets encrypt (ACME)
    • supporting HTTP-01 or DNS-01 challenges

Intranet site options:

  • you could setup a small-step ca server (ACME)
    • already supported using HTTP-01 challenges

Local development site options:

My thoughts are that ACME is not appropriate for local development. We could add support for another challenge type targeted at the local development use case. The implications for that I am not sure of.

We could instead integrate Small Step more directly, and create a specific integration that will leverage some of the existing Small Step apis to mint a cert and install it directly.

Thoughts

Mark

Great analysis!

+ TLS-ALPN challenge

I don’t view this as (very) different from local dev, see below.

This is more along the lines of what I was thinking.

There are two different issues here, and the challenge types is only the solution to one of them IMO:

  1. DNS resolution of hostnames. This is something that may be more likely in an intranet with actual internal DNS configured, etc. Here, in theory, any challenge type should be fine, but it has to be with an internal CA, not Let’s Encrypt.

  2. Binding to low ports. This can be annoying on local dev but not impossible. Agreed, that for local development, we should be able to bypass the ACME challenge entirely and simply call a function in the Smallstep library to get a new cert, without needing validations per-se. Setting up DNS for local resolution can also be a bit tedious/annoying.

I think both options need to be available: set up and use an internal ACME CA, or just call a function to get a cert directly.

Using an internal CA will pretty much just work from what I have seen. There are a few nice to haves like specifying the root_ca_pem cert in the Caddyfile which I will push fix for.

I would be interested in a steer on the configuration side of things.

tls {

  small-step {
    ca-host [internal | url]
  }
}

Might be enough? Perhaps a password file?

@matt Ok a little update.

Small Step as you know probably isn’t a big deal to get running.

Provisioning certs from it would probably take one of two things.
a) loosen up the Small Step API, they have done a good job of hiding the implementations
b) start the small step service and call the same API endpoints. While this is clean on one hand it will still require some clear configuration about the ports being used. So it will give you one more possible port collision on start up. You would be running the small step client to connect to the small step ca via http over some local port.

An alternative would be to have 3 different approaches

  • Publicly accessible sites - lets encrypt via ACME
  • Internal sites - Small Step via ACME
  • Development sites - something like minica

I wonder if the model of having self-signed like in caddy v1 was a better approach. It is unclear what process we would need to do to self provision Small Step even if we have have it hosted inside Caddy. Small Step have done a great job of smoothing out the onboarding flow, and makes sense if you are planning on running an actual ca. But in the dev case of running something locally I think it will be more effort for the developer than self-signed currently is.

I hope that makes sense.

@sarge Just so you know, I am following your updates, I just have been very busy with some other things right now.

Allow me to paste in some tips from the Smallstep team on our Slack a while back, it may be helpful!

Mariano:

The pki package is used to create the configuration for the ca, in your case you will only need to use GenerateRootCertificate and GenerateIntermediateCertificate .
These methods, are just helpers over x509util package - github.com/smallstep/cli/crypto/x509util - pkg.go.dev, you can also use pemutil package - github.com/smallstep/cli/crypto/pemutil - pkg.go.dev (pemutil.Serialize()) to store the pem files if your disk, encrypted or not.

If you look into the code you will see two different x509 packages, one that we usually call stepx509 that is embedded in cli/pkg/x509 . This one is likely to go away soon as Go 1.13 already supports Ed25519, and we want to enforce this version. Note that the pki package is going to change and the constructor for the new version won’t accept any parameters.Then for the ACME part, if you’re using step-ca for this, all the handlers are in api package - github.com/smallstep/certificates/acme/api - pkg.go.dev. They use a special authority acme package - github.com/smallstep/certificates/acme - pkg.go.dev on top of the main one certificates/authority that is used to sign certificates.

Currently, the only way to build the authority authority package - github.com/smallstep/certificates/authority - pkg.go.dev is using a Config struct, and this should have at least the path for the root certificate, intermediate certificate and key, at least one acme provisioner and a DNSName to create the actual server certificate, other properties might also be required but they are not that important. Right now the main authority enforces encryption in the intermediate key, but you can set the password in the config.

At some point, sooner than later, we want to change the authority constructor not require all that, and you can set up your own signers just passing variadic parameters, the configuration will be an extra one, that sets up everything from a Config struct or file. Imagine something like authority.New("localhost", authority. WithTLSSigner(mySigner))

One thing more, the acme requires a db, by default step-ca will use a local badger db, but you can always build an interface if you want to store it in memory: database package - github.com/smallstep/nosql/database - pkg.go.dev

Trusting the development CA could be done with this, I think: GitHub - smallstep/truststore: Package to locally install development certificates

Then, a few of my own thoughts in response to you @sarge:

I am hoping that we do not need to expose the “smallstep” brand/name in the config, as that seems like an implementation detail. Rather, I think what the user really would want to configure is whether a site/domain gets TLS. For local/internal names, it should get them from smallstep under the hood; for public sites, it should use a public CA. I think it’s that straightforward: it’s just a matter of deciding this automatically and exposing the right config to the user for choosing whether to use an internal or public CA. If they want to use an internal or local one, it should be smallstep by default, but of course they could configure this.

I think the point is that we do want to run an actual, self-contained, internal CA for non-public names. So I still think smallstep is the right way to go here.

We don’t have to make the user configure every detail – in the simplest form, they should only choose whether to use a local, internal, or public CA for certs, and only if we can’t determine it automatically. Does that make sense?

Hi @matt,

Thanks for tips. I have pulled back another layer and have generated some certs based on a prior install of small step. I will explore inital configuration of small step later.

In terms of plumbing it into the certificate renewal process my plan was to create a another AutomationPolicy Manager
https://github.com/caddyserver/caddy/blob/v2/modules/caddyhttp/caddyhttp.go#L378

This seems to be a reasonable place to create some policy about which AutomationPolicy Manager to use.

I have made caddytls.SmallStepManager similar to the caddytls.ACMEManagerMaker. It implements ManagerMaker

// ManagerMaker makes a certificate manager.
type ManagerMaker interface {
	NewManager(interactive bool) (certmagic.Manager, error)
}

And certmagic.Manager

type Manager interface {
	Obtain(name string) error
	Renew(name string) error
	Revoke(name string) error
}

When I implement the Obtain method I have everything I need to generate a certificate, it gets called, but there is a challenge with where to put the generated cert?

I get an error from certmagic, which references a path, presumably I could write it there but an odd bit of coupling?
/home/fedora/.local/share/caddy/acme/acme-v02.api.letsencrypt.org/sites/test.sarge.com/test.sarge.com.crt

I had a play trying to access the the certmagic.CertCache via the tls app, thinking I could write it there, but that supports reading and not writing to it.

It is a complex area which I certainly am only getting a sense of at this stage, but I would appreciate a moment to vet the approach taken thus far and any suggestions about alternative paths.

1 Like

This is awesome progress! Thanks for the update.

I think this is a great start. The SmallStepManager type might even embed ACMEManagerMaker (or certmagic.Config directly?) if it needs to do any ACME operations. For local dev, I could see how it wouldn’t need ACME at all – just call functions in smallstep directly. But for internal intranets, you will probably need ACME to facilitate certs over the wire. This is an implementation detail that might even be an unnecessary optimization! But for sure, the user will almost never care either way, as long as they can configure it if needed.

Notice how ACMEManagerMaker takes a Storage module, which it sets up here – and if it’s not set by the user, it uses the global/default storage: acmemanager.go - caddyserver/caddy - Sourcegraph

Once it is written to storage, you don’t need to worry about loading it into the cert cache.

I have managed to get something working with a fairly small MR add support for a ca.local to provide self signed certs by sarge · Pull Request #2948 · caddyserver/caddy · GitHub

It doesn’t run a full SmallStep server, but probably could. The cut down version doesn’t need to make any http calls so that keeps the complexity down.

Anyway I would be interested in your thoughts

1 Like

This is awesome, @sarge! I can’t wait to check it out – will get around to it as soon as I have the chance.

This topic was automatically closed 90 days after the last reply. New replies are no longer allowed.