Caddy 2: wildcard + on demand certificates

[jmstfv (Jama)]

1. My Caddy version (caddy version):

v2.0.0-beta.14 h1:QX1hRMfTA5sel53o5SuON1ys50at6yuSAnPr56sLeK8=

2. How I run Caddy:

a. System environment:

• Ubuntu 18.04.4
• systemd 237

c. Service/unit/compose file:

[Unit]
Description=Caddy Web Server
Documentation=https://caddyserver.com/docs/
After=network.target

[Service]
User=caddy
Group=caddy
ExecStart=/usr/bin/caddy run --config /etc/caddy/caddy.json --resume --environ
ExecReload=/usr/bin/caddy reload --config /etc/caddy/caddy.json
TimeoutStopSec=5s
LimitNOFILE=1048576
LimitNPROC=512
PrivateTmp=true
ProtectSystem=full
AmbientCapabilities=CAP_NET_BIND_SERVICE

[Install]
WantedBy=multi-user.target

d. My complete Caddyfile or JSON config:

Caddyfile:

*.tryhexadecimal.com {
  encode gzip zstd

  reverse_proxy 0.0.0.0:3000 {
    header_up X-Forwarded-For {remote_host}
    header_up X-Forwarded-Port {server_port}
    header_up X-Forwarded-Proto {scheme}
  }
}

And the equivalent JSON config:

{
  "apps": {
    "http": {
      "servers": {
        "srv0": {
          "listen": [
            ":443"
          ],
          "routes": [
            {
              "match": [
                {
                  "host": [
                    "*.tryhexadecimal.com"
                  ]
                }
              ],
              "handle": [
                {
                  "handler": "subroute",
                  "routes": [
                    {
                      "handle": [
                        {
                          "encodings": {
                            "gzip": {},
                            "zstd": {}
                          },
                          "handler": "encode"
                        },
                        {
                          "handler": "reverse_proxy",
                          "headers": {
                            "request": {
                              "set": {
                                "X-Forwarded-For": [
                                  "{http.request.remote.host}"
                                ],
                                "X-Forwarded-Port": [
                                  "{server_port}"
                                ],
                                "X-Forwarded-Proto": [
                                  "{http.request.scheme}"
                                ]
                              }
                            }
                          },
                          "upstreams": [
                            {
                              "dial": "0.0.0.0:3000"
                            }
                          ]
                        }
                      ]
                    }
                  ]
                }
              ],
              "terminal": true
            }
          ]
        }
      }
    }
  }
}

3. What I am trying to accomplish:

1. Obtain a wildcard certificate. But first, I need to enable a DNS challenge.
2. Issue certificates on demand.

Below is my hand-written JSON configuration (for tls only, http part of the config can be found above). I’d be grateful if you could give it a look to see if it needs improvement.

{
  "apps": {
    "tls": {
      "automation": {
        "on_demand": {
          "ask": "MY WEBHOOK URL"
        },
        "policies": [{
          "management": {
            "module": "acme",
            "email": "MY EMAIL",
            "challenges": {
              "dns": {
                "provider": "cloudflare",
                "api_token": "MY API TOKEN"
              }
            }
          }
        }]
      }
    }
  }
}

Also, is there a way to store certificates on a remote file system (e.g. AWS EFS)?

4. Error messages and/or full log output:

{"error": "loading config: loading new config: loading app modules: module name 'tls': provision tls: loading TLS automation management module: loading module 'acme': provision tls.management.acme: loading DNS provider module: loading module 'cloudflare': unknown module: tls.dns.cloudflare", "status_code": 400}

[francislavoie]

The issue is that Caddy doesn’t come with the cloudflare DNS module by default. It need to be compiled in to work.

@matt will need to give you the instructions for that, I don’t know the correct way to do this right now.

[matt (Matt Holt)]

Correct; instructions are here: https://github.com/caddyserver/tls.dns

[jmstfv (Jama)]

Thanks @francislavoie & @matt!

Post for future lurkers who might have the similar problem.

Provision a new server, and install Go from scratch:

$ sudo snap install go --classic
$ go version
go version go1.13.8 linux/amd64

Create a new directory and download the main.go file:

$ mkdir -p caddy && cd caddy
$ wget https://raw.githubusercontent.com/caddyserver/caddy/v2/cmd/caddy/main.go

Add the tls.dns module to the main.go file:

// Copyright 2015 Matthew Holt and The Caddy Authors
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
//     http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.

// Package main is the entry point of the Caddy application.
// Most of Caddy's functionality is provided through modules,
// which can be plugged in by adding their import below.
//
// There is no need to modify the Caddy source code to customize your
// builds. You can easily build a custom Caddy with these simple steps:
//
//   1. Copy this file (main.go) into a new folder
//   2. Edit the imports below to include the modules you want plugged in
//   3. Run `go mod init caddy`
//   4. Run `go install` or `go build` - you now have a custom binary!
//
package main

import (
	caddycmd "github.com/caddyserver/caddy/v2/cmd"

	// plug in Caddy modules here
	_ "github.com/caddyserver/caddy/v2/modules/standard"
	_ "github.com/caddyserver/tls.dns/providers/cloudflare"
)

func main() {
	caddycmd.Main()
}

Finish the build:

$ go mod init caddy
$ go get github.com/caddyserver/caddy/v2@v2.0.0-beta.14
$ go build

Here’s what output from ./caddy list-modules should look like:

caddy.logging.encoders.console
caddy.logging.encoders.filter
caddy.logging.encoders.filter.delete
caddy.logging.encoders.filter.ip_mask
caddy.logging.encoders.json
caddy.logging.encoders.logfmt
caddy.logging.encoders.string
caddy.logging.writers.discard
caddy.logging.writers.file
caddy.logging.writers.net
caddy.logging.writers.stderr
caddy.logging.writers.stdout
caddy.storage.file_system
http
http.authentication.hashes.bcrypt
http.authentication.hashes.scrypt
http.authentication.providers.http_basic
http.encoders.brotli
http.encoders.gzip
http.encoders.zstd
http.handlers.authentication
http.handlers.cache
http.handlers.encode
http.handlers.error
http.handlers.file_server
http.handlers.headers
http.handlers.request_body
http.handlers.reverse_proxy
http.handlers.rewrite
http.handlers.static_response
http.handlers.subroute
http.handlers.templates
http.handlers.vars
http.matchers.file
http.matchers.header
http.matchers.header_regexp
http.matchers.host
http.matchers.method
http.matchers.not
http.matchers.path
http.matchers.path_regexp
http.matchers.protocol
http.matchers.query
http.matchers.remote_ip
http.matchers.vars
http.matchers.vars_regexp
http.reverse_proxy.circuit_breakers.local
http.reverse_proxy.selection_policies.first
http.reverse_proxy.selection_policies.header
http.reverse_proxy.selection_policies.ip_hash
http.reverse_proxy.selection_policies.least_conn
http.reverse_proxy.selection_policies.random
http.reverse_proxy.selection_policies.random_choose
http.reverse_proxy.selection_policies.round_robin
http.reverse_proxy.selection_policies.uri_hash
http.reverse_proxy.transport.fastcgi
http.reverse_proxy.transport.http
http.reverse_proxy.transport.http_ntlm
tls
tls.certificate_selection.custom
tls.certificates.automate
tls.certificates.load_files
tls.certificates.load_folders
tls.certificates.load_pem
tls.dns.cloudflare
tls.handshake_match.sni
tls.management.acme
tls.stek.distributed
tls.stek.standard

[jmstfv (Jama)]

For future visitors: API permissions you need to give (from your Cloudflare dashboard) so that Caddy could request a wildcard certificate