Attempting to use pki module in json caddyfile

1. Caddy version (caddy version):

eeuser@elite-pi:~ $ caddy version
v2.4.0 h1:yHnnbawH2G3ZBP2mAJF4XBLnJanqhULLP/wu01Qi9Io=

2. How I run Caddy:

a. System environment:

$ cat /etc/*release*
PRETTY_NAME="Raspbian GNU/Linux 10 (buster)"
NAME="Raspbian GNU/Linux"
VERSION_ID="10"
VERSION="10 (buster)"
VERSION_CODENAME=buster
ID=raspbian
ID_LIKE=debian
...

b. Command:

systemctl start caddy

c. Service/unit/compose file:

# caddy.service
#
# For using Caddy with a config file.
#
# Make sure the ExecStart and ExecReload commands are correct
# for your installation.
#
# See https://caddyserver.com/docs/install for instructions.
[Unit]
Description=Caddy
Documentation=https://caddyserver.com/docs/
After=network.target network-online.target
Requires=network-online.target

[Service]
User=root
Group=root
ExecStart=/usr/bin/caddy run --environ --config /etc/caddy/caddy.json
ExecReload=/usr/bin/caddy reload --config /etc/caddy/caddy.json
TimeoutStopSec=5s
LimitNOFILE=1048576
LimitNPROC=512
PrivateTmp=true
ProtectSystem=full
AmbientCapabilities=CAP_NET_BIND_SERVICE

[Install]
WantedBy=multi-user.target

d. My complete Caddyfile or JSON config:

{
	"logging": {
		"logs": {
			"default": {
				"exclude": [
					"http.log.access.log0"
					]
			}                       ,
			"log0": {
				"writer": {
					"filename": "/var/log/caddy/caddy.log",
					"output": "file"
				}                               ,
				"encoder": {
					"format": "console"
				}                               ,
				"level": "INFO",
				"include": [
				"http.log.access.log0"
				]
			}
		}
	}       ,
	"apps": {
		"http": {
			"http_port": 80,
			"https_port": 443,
			"servers": {
				"demo": {
					"listen": [
					":443"
					],
					"routes": [ {
						"match": [ {
							"host": [
							"http://192.168.1.10"
							]
						}
						],
						"handle": [ {
							"handler": "subroute",
							"routes": [ {
								"handle": [ {
									"handler": "subroute",
									"routes": [ {
										"handle": [ {
											"handler": "authentication",
											"providers": {
												"jwt": {
													"authorizer": {
														"access_list": [ {
															"action": "allow",
															"claim": "roles",
															"values": [
															"superadmin",
															"manager"
															]
														}
														],
														"context": "default",
														"pass_claims_with_headers": true
													}
												}
											}
										}
										]
										},
                                                                          {
                                                                                "handle": [ {
                                                                                        "handler": "reverse_proxy",
                                                                                        "upstreams": [ {
                                                                                                "dial": "localhost:5555"
                                                                                        }
                                                                                        ]
                                                                                }
                                                                                ]
                                                                        }
                                                                        ]
                                                                }
                                                                ],
                                                                "match": [ {
                                                                        "path": [
                                                                        "/user-mgr/*"
                                                                        ]
                                                                }
                                                                ]
                                                        }                                                       , {
                                                                "handle": [ {
                                                                        "handler": "subroute",
                                                                        "routes": [ {
                                                                                "handle": [ {
                                                                                        "body": "2.0.0-a",
                                                                                        "handler": "static_response",
                                                                                        "status_code": 200
                                                                                }
                                                                                ]
                                                                        }
                                                                        ]
                                                                }
                                                                ],
                                                                "match": [ {
                                                                        "path": [
                                                                        "/version*"
                                                                        ]
                                                                }
                                                                ]
                                                        }                                                       , {
                                                                "handle": [ {
                                                                        "handler": "subroute",
                                                                        "routes": [ {
                                                                                "handle": [ {
                                                                                        "handler": "subroute",
                                                                                        "routes": [ {
                                                                                                "handle": [ {
                                                                                                        "handler": "auth_portal",
                                                                                                        "portal": {
                                                                                                                "auth_url_path": "/auth",
                                                                                                                "backends": [ {
                                                                                                                        "method": "local",
                                                                                                                        "name": "local_backend",
                                                                                                                        "path": "/opt/caddy/auth/user_db.json",
                                                                                                                        "realm": "local"
                                                                                                                }
                                                                                                                ],
                                                                                                                "context": "default",
                                                                                                                "cookies": {},
                                                                                                                "jwt": {
                                                                                                                        "token_lifetime": 43200,
                                                                                                                        "token_name": "access_token",
                                                                                                                        "token_secret": "AnExampleSecretString123"
                                                                                                                }                                               ,
                                                                                                                "primary": true,
                                                                                                                "registration": {
                                                                                                                        "dropbox": "/opt/caddy/auth/registrations_db.json",
                                                                                                                        "title": "User Registration"
                                                                                                                }                                               ,
                                                                                                                "ui": {
                                                                                                                        "auto_redirect_url": "",
                                                                                                                        "custom_css_path": "/opt/caddy/css/custom.css",
                                                                                                                        "logo_url": "assets/logo.png",
                                                                                                                        "password_recovery_enabled": false,
                                                                                                                        "private_links": [ {
                                                                                                                                "link": "/",
                                                                                                                                "title": "Example Manager"
                                                                                                                        }                                       , {
                                                                                                                                "link": "/auth/settings",
                                                                                                                                "title": "Auth Portal Settings"
                                                                                                                        }                                       , {
                                                                                                                                "link": "/auth/whoami",
                                                                                                                                "title": "Who Am I Check"
                                                                                                                        }                                       , {
                                                                                                                                "link": "/user-mgr/",
                                                                                                                                "title": "User Management"
                                                                                                                        }
                                                                                                                        ],
                                                                                                                        "realms": null,
                                                                                                                        "templates": {
                                                                                                                                "generic": "/opt/caddy/templates/em20-tmpl/generic.template",
                                                                                                                                "login": "/opt/caddy/templates/em20-tmpl/login.template",
                                                                                                                                "portal": "/opt/caddy/templates/em20-tmpl/portal.template",
                                                                                                                                "register": "/opt/caddy/templates/em20-tmpl/register.template",
                                                                                                                                "settings": "/opt/caddy/templates/em20-tmpl/settings.template",
                                                                                                                                "whoami": "/opt/caddy/templates/em20-tmpl/whoami.template"
                                                                                                                        }                                       ,
                                                                                                                        "theme": "basic"
                                                                                                                }
                                                                                                        }
                                                                                                }
                                                                                                ]
                                                                                        }
                                                                                        ]
                                                                                }
                                                                                ],
                                                                                "match": [ {
                                                                                        "path": [
                                                                                        "/auth*"
                                                                                        ]
                                                                                }
                                                                                ]
                                                                        }
                                                                        ]
                                                                }
                                                                ],
                                                                "match": [ {
                                                                        "path": [
                                                                        "/auth*"
                                                                        ]
                                                                }
                                                                ]
                                                        }                                                       , {
                                                                "handle": [ {
                                                                        "handler": "subroute",
                                                                        "routes": [ {
                                                                                "handle": [ {
                                                                                        "handler": "authentication",
                                                                                        "providers": {
                                                                                                "jwt": {
                                                                                                        "authorizer": {
                                                                                                                "context": "default",
                                                                                                                "pass_claims_with_headers": true
                                                                                                        }
                                                                                                }
                                                                                        }
                                                                                }
                                                                                ]
                                                                        }                                                                       , {
                                                                                "handle": [ {
                                                                                        "handler": "reverse_proxy",
                                                                                        "upstreams": [ {
                                                                                                "dial": "localhost:3000"
                                                                                        }
                                                                                        ]
                                                                                }
                                                                                ]
                                                                        }
                                                                        ]
                                                                }
                                                                ],
                                                                "match": [ {
                                                                        "path": [
                                                                        "/gr/*"
                                                                        ]
                                                                }
                                                                ]
                                                        }                                                       , {
                                                                "handle": [ {
                                                                        "handler": "subroute",
                                                                        "routes": [ {
                                                                                "handle": [ {
                                                                                        "handler": "authentication",
                                                                                        "providers": {
                                                                                                "jwt": {
                                                                                                        "authorizer": {
                                                                                                                "context": "default",
                                                                                                                "pass_claims_with_headers": true
                                                                                                        }
                                                                                                }
                                                                                        }
                                                                                }
                                                                                ]
                                                                        }                                                                       , {
                                                                                "handle": [ {
                                                                                        "handler": "reverse_proxy",
                                                                                        "upstreams": [ {
                                                                                                "dial": "localhost:1880"
                                                                                        }
                                                                                        ]
                                                                                }
                                                                                ]
                                                                        }
                                                                        ]
                                                                }
                                                                ],
                                                                "match": [ {
                                                                        "path": [
                                                                        "/ui/*"
                                                                        ]
                                                                }
                                                                ]
                                                        }                                                       , {
                                                                "handle": [ {
                                                                        "handler": "subroute",
                                                                        "routes": [ {
                                                                                "handle": [ {
                                                                                        "handler": "authentication",
                                                                                        "providers": {
                                                                                                "jwt": {
                                                                                                        "authorizer": {
                                                                                                                "context": "default",
                                                                                                                "pass_claims_with_headers": true,
                                                                                                                "primary": true,
                                                                                                                "trusted_tokens": [ {
                                                                                                                        "token_name": "access_token",
                                                                                                                        "token_secret": "AnExampleSecretString123"
                                                                                                                }
                                                                                                                ]
                                                                                                        }
                                                                                                }
                                                                                        }
                                                                                }
                                                                                ]
                                                                        }                                                                       , {
                                                                                "handle": [ {
                                                                                        "handler": "reverse_proxy",
                                                                                        "upstreams": [ {
                                                                                                "dial": "localhost:1081"
                                                                                        }
                                                                                        ]
                                                                                }
                                                                                ]
                                                                        }
                                                                        ]
                                                                }
                                                                ],
                                                                "match": [ {
                                                                        "path": [
                                                                        "/*"
                                                                        ]
								}
								]
							}
							]
						}
						],
						"terminal": true
					}
					],
					"logs": {
							"logger_names": {
							"192.168.1.10": "log0"
						}
					}
				}
			}
		 }
	}
}

3. The problem I’m having:

The following configuration effort is based on attempting to control the configuration of the auto-tls’s local certificate authority. In this first step I’m attempting to get our caddy deployment approach converted to using JSON based configuration files, including configuring the pki module.

One of the primary issues for switching to the json based configuration is to be able to manage the configuration around the local CA (specifically control the CN value of its subject DN strings to make them unique per install). The goal is to have caddy on startup with the proper configuration, generate and install trust for a local CA with the properties set by the PKI module configuration.

The “configuration example” from the caddyserver documentation presents the following:

{
	"certificate_authorities": {
		"": {
			"name": "",
			"root_common_name": "",
			"intermediate_common_name": "",
			"install_trust": false,
			"root": {
				"certificate": "",
				"private_key": "",
				"format": ""
			},
			"intermediate": {
				"certificate": "",
				"private_key": "",
				"format": ""
			},
			"storage": {•••}
		}
	}
}

I’m running into a couple of issues:

  1. I’m not finding a clear example of how to properly position the PKI module into our existing json file that was generated from our original Caddyfile with the caddy adapt command. I’m assuming it needs to be presented as a “peer” to the “http”: module declaration in the file but have not been successful in formatting it in a way that starts up properly. Does anyone have an example of using this configuration that can be shared?

  2. In getting into actually using the configuration, where should CA ID be presented? Is it in the first value within the “certificate_authorities”: object? For example if we wanted CA ID to be local-test would we use the following

"certificate_authorities": {
		"local-test": {
...
  1. what is “required” values in the pki module? In our use case we are specifically focused on programmatically setting "root_common_name": "", to a unique value per deployment. Do we have to set the other values beyond this or will defaults apply across them?
  2. If configuring the root and intermediate values are required; what is required around the “root”: and “intermediate”: child object entries with regard to the certificate, private key and format values. Will they fall back to defaults being generated by caddy on startup, or is this implying I need to provide already generated private key and generated CA certificate file (which is confusing at this point because I’m not sure if I’m pursuing the right configuration approach now).

4. Error messages and/or full log output:

5. What I already tried:

I’ve iterated with positioning the pki assertion outside of the main block, within it, and I seem to break the parsing regardless of how I introduce it to the file.

I’d rather focus on what is proper form for the configuration file and its required parameters than the iterations of configuration attempted to date; this should be a predictable specific configuration pattern, I’m not seeing in the docs how to include it in a json configuration file.

6. Links to relevant resources:

(looking to pointers for other contextual references…)

pki is an app, so it goes inside apps; so yes, “besides” http, which is also an app.

Better seen here (hover over the ... next to apps):

That’s right. See the documentation here, which mentions that the keys are the ID:

Generally, defaults will apply everywhere, unless otherwise mentioned. If something’s required, then there would be an error during provisioning of the config.

You can run the caddy validate command to check whether Caddy thinks your config makes sense (without actually running it).

1 Like

This is where I’m ending up every time. The documentation is not clear here as to whats expected to be in the file. Really at this point I’m looking for input from someone using the config (or that tested it) that might be able to share a parseable example. Our configuration ends up in a horrible json file that wastes 10’s of thousands of spaces to indent the thing (I had to manually convert to tabs to post our file).

eeuser@elite-pi:~/elite-pi/elite-caddy $ sudo caddy validate --config ./fmtcaddy.json
2021/06/02 23:52:38.088 INFO using provided configuration {“config_file”: “./fmtcaddy.json”, “config_adapter”: “”}
validate: decoding config: unexpected end of JSON input

So I think I have a working config, but from the logging it looks like its NOT attempting to generate local CA, and instead is attempting to get to online certificate issuance.

In the Caddyfile configuration approach there was a way to set TLS to local CA only… how would I go about doing that in the json configuration approach?

{
  "logging": {
    "logs": {
      "default": {
        "exclude": [
          "http.log.access.log0"
        ]
      },
      "log0": {
        "writer": {
          "filename": "/var/log/caddy/caddy.log",
          "output": "file"
        },
        "encoder": {
          "format": "console"
        },
        "level": "INFO",
        "include": [
          "http.log.access.log0"
        ]
      }
    }
  },
  "apps": {
      "pki":{
	"certificate_authorities": {
		"local": {
			"name": "EM Local",
			"root_common_name": "Elite Local Facility Root CA - 00000000",
			"intermediate_common_name": "Elite Intermediary CA",
			"install_trust": true
		}
	}
      },
      "http": {
      "http_port": 80,
      "https_port": 443,
      "servers": {
        "elite": {
          "listen": [
            ":443"
          ],
          "routes": [
            {
              "match": [
                {
                  "host": [
                    "https://10.252.252.21",
		    "https://epi.cust.elite-env.com"
                  ]
                }
              ],
              "handle": [
                {
                  "handler": "subroute",
                  "routes": [
                    {
                      "handle": [
                        {
                          "handler": "subroute",
                          "routes": [
                            {
                              "handle": [
                                {
                                  "handler": "authentication",
                                  "providers": {
                                    "jwt": {
                                      "authorizer": {
                                        "access_list": [
                                          {
                                            "action": "allow",
                                            "claim": "roles",
                                            "values": [
                                              "superadmin",
                                              "manager"
                                            ]
                                          }
                                        ],
                                        "context": "default",
                                        "pass_claims_with_headers": true
                                      }
                                    }
                                  }
                                }
                              ]
                            },
                            {
                              "handle": [
                                {
                                  "handler": "reverse_proxy",
                                  "upstreams": [
                                    {
                                      "dial": "localhost:5555"
                                    }
                                  ]
                                }
                              ]
                            }
                          ]
                        }
                      ],
                      "match": [
                        {
                          "path": [
                            "/user-mgr/*"
                          ]
                        }
                      ]
                    },
                    {
                      "handle": [
                        {
                          "handler": "subroute",
                          "routes": [
                            {
                              "handle": [
                                {
                                  "body": "2.0.0-a",
                                  "handler": "static_response",
                                  "status_code": 200
                                }
                              ]
                            }
                          ]
                        }
                      ],
                      "match": [
                        {
                          "path": [
                            "/version*"
                          ]
                        }
                      ]
                    },
                    {
                      "handle": [
                        {
                          "handler": "subroute",
                          "routes": [
                            {
                              "handle": [
                                {
                                  "handler": "subroute",
                                  "routes": [
                                    {
                                      "handle": [
                                        {
                                          "handler": "auth_portal",
                                          "portal": {
                                            "auth_url_path": "/auth",
                                            "backends": [
                                              {
                                                "method": "local",
                                                "name": "local_backend",
                                                "path": "/opt/caddy/auth/user_db.json",
                                                "realm": "local"
                                              }
                                            ],
                                            "context": "default",
                                            "cookies": {},
                                            "jwt": {
                                              "token_lifetime": 43200,
                                              "token_name": "access_token",
                                              "token_secret": "AnExampleSecretString123"
                                            },
                                            "primary": true,
                                            "registration": {
                                              "dropbox": "/opt/caddy/auth/registrations_db.json",
                                              "title": "User Registration"
                                            },
                                            "ui": {
                                              "auto_redirect_url": "",
                                              "custom_css_path": "/opt/caddy/css/custom.css",
                                              "logo_url": "assets/logo.png",
                                              "password_recovery_enabled": false,
                                              "private_links": [
                                                {
                                                  "link": "/",
                                                  "title": "Elite Manager"
                                                },
                                                {
                                                  "link": "/auth/settings",
                                                  "title": "Auth Portal Settings"
                                                },
                                                {
                                                  "link": "/auth/whoami",
                                                  "title": "Who Am I Check"
                                                },
                                                {
                                                  "link": "/user-mgr/",
                                                  "title": "User Management"
                                                }
                                              ],
                                              "realms": null,
                                              "templates": {
                                                "generic": "/opt/caddy/templates/em20-tmpl/generic.template",
                                                "login": "/opt/caddy/templates/em20-tmpl/login.template",
                                                "portal": "/opt/caddy/templates/em20-tmpl/portal.template",
                                                "register": "/opt/caddy/templates/em20-tmpl/register.template",
                                                "settings": "/opt/caddy/templates/em20-tmpl/settings.template",
                                                "whoami": "/opt/caddy/templates/em20-tmpl/whoami.template"
                                              },
                                              "theme": "basic"
                                            }
                                          }
                                        }
                                      ]
                                    }
                                  ]
                                }
                              ],
                              "match": [
                                {
                                  "path": [
                                    "/auth*"
                                  ]
                                }
                              ]
                            }
                          ]
                        }
                      ],
                      "match": [
                        {
                          "path": [
                            "/auth*"
                          ]
                        }
                      ]
                    },
                    {
                      "handle": [
                        {
                          "handler": "subroute",
                          "routes": [
                            {
                              "handle": [
                                {
                                  "handler": "authentication",
                                  "providers": {
                                    "jwt": {
                                      "authorizer": {
                                        "context": "default",
                                        "pass_claims_with_headers": true
                                      }
                                    }
                                  }
                                }
                              ]
                            },
                            {
                              "handle": [
                                {
                                  "handler": "reverse_proxy",
                                  "upstreams": [
                                    {
                                      "dial": "localhost:3000"
                                    }
                                  ]
                                }
                              ]
                            }
                          ]
                        }
                      ],
                      "match": [
                        {
                          "path": [
                            "/gr/*"
                          ]
                        }
                      ]
                    },
                    {
                      "handle": [
                        {
                          "handler": "subroute",
                          "routes": [
                            {
                              "handle": [
                                {
                                  "handler": "authentication",
                                  "providers": {
                                    "jwt": {
                                      "authorizer": {
                                        "context": "default",
                                        "pass_claims_with_headers": true
                                      }
                                    }
                                  }
                                }
                              ]
                            },
                            {
                              "handle": [
                                {
                                  "handler": "reverse_proxy",
                                  "upstreams": [
                                    {
                                      "dial": "localhost:1880"
                                    }
                                  ]
                                }
                              ]
                            }
                          ]
                        }
                      ],
                      "match": [
                        {
                          "path": [
                            "/ui/*"
                          ]
                        }
                      ]
                    },
                    {
                      "handle": [
                        {
                          "handler": "subroute",
                          "routes": [
                            {
                              "handle": [
                                {
                                  "handler": "authentication",
                                  "providers": {
                                    "jwt": {
                                      "authorizer": {
                                        "context": "default",
                                        "pass_claims_with_headers": true,
                                        "primary": true,
                                        "trusted_tokens": [
                                          {
                                            "token_name": "access_token",
                                            "token_secret": "AnExampleSecretString123"
                                          }
                                        ]
                                      }
                                    }
                                  }
                                }
                              ]
                            },
                            {
                              "handle": [
                                {
                                  "handler": "reverse_proxy",
                                  "upstreams": [
                                    {
                                      "dial": "localhost:1081"
                                    }
                                  ]
                                }
                              ]
                            }
                          ]
                        }
                      ],
                      "match": [
                        {
                          "path": [
                            "/*"
                          ]
                        }
                      ]
                    }
                  ]
                }
              ],
              "terminal": true
            }
          ],
          "logs": {
            "logger_names": {
              "10.252.252.21": "log0"
            }
          }
        }
      }
    }
  }
}

Here is what I get from journalctl -xe after startup. I’m noticing a couple of things, it appears to fail to handle setup properly (sudo error, the thing is running as root? I updated the systemd to run as root not caddy at this point.) And then it starts to attempt online certificate issuance which is not what I’m expecting, I’m expecting local CA to issue the certs.

s 1846-1901/1901 (END)
Jun 02 20:12:22 elite-pi caddy[8124]: {"level":"info","ts":1622686342.2923243,"logger":"http.handlers.auth_portal","msg":"local backend configuration","db_path":"/opt/caddy/auth/user_db.json","require_mfa":false}
Jun 02 20:12:22 elite-pi caddy[8124]: {"level":"info","ts":1622686342.293469,"logger":"http.handlers.auth_portal","msg":"validating local backend","db_path":"/opt/caddy/auth/user_db.json"}
Jun 02 20:12:22 elite-pi caddy[8124]: {"level":"info","ts":1622686342.3070905,"logger":"http.handlers.auth_portal","msg":"JWT token validator provisioned","instance_name":"portal-1","access_list":[{"action":"allow","values":["anonymous","guest","*"],"claim":"
Jun 02 20:12:22 elite-pi caddy[8124]: {"level":"info","ts":1622686342.3071828,"logger":"http.handlers.auth_portal","msg":"provisioned plugin instance","instance_name":"portal-1","started_at":1622686342.2922094}
Jun 02 20:12:22 elite-pi caddy[8124]: {"level":"info","ts":1622686342.3083808,"logger":"http.authentication.providers.jwt","msg":"provisioned plugin instance","instance_name":"jwt-4","started_at":1622686342.3083124}
Jun 02 20:12:22 elite-pi caddy[8124]: {"level":"info","ts":1622686342.3118165,"logger":"tls","msg":"cleaning storage unit","description":"FileStorage:/root/.local/share/caddy"}
Jun 02 20:12:22 elite-pi caddy[8124]: {"level":"info","ts":1622686342.3119755,"logger":"http","msg":"enabling automatic TLS certificate management","domains":["https://10.252.252.21","https://epi.cust.elite-env.com"]}
Jun 02 20:12:22 elite-pi caddy[8124]: {"level":"info","ts":1622686342.3127623,"logger":"tls","msg":"finished cleaning storage units"}
Jun 02 20:12:22 elite-pi caddy[8124]: {"level":"info","ts":1622686342.3136792,"logger":"tls.obtain","msg":"acquiring lock","identifier":"https://10.252.252.21"}
Jun 02 20:12:22 elite-pi caddy[8124]: {"level":"info","ts":1622686342.313837,"logger":"tls.obtain","msg":"acquiring lock","identifier":"https://epi.cust.elite-env.com"}
Jun 02 20:12:22 elite-pi caddy[8124]: {"level":"info","ts":1622686342.3165538,"logger":"tls.obtain","msg":"lock acquired","identifier":"https://epi.cust.elite-env.com"}
Jun 02 20:12:22 elite-pi caddy[8124]: {"level":"info","ts":1622686342.3165538,"logger":"tls.obtain","msg":"lock acquired","identifier":"https://10.252.252.21"}
Jun 02 20:12:22 elite-pi caddy[8124]: {"level":"info","ts":1622686342.3226008,"logger":"tls.issuance.acme","msg":"waiting on internal rate limiter","identifiers":["https://epi.cust.elite-env.com"]}
Jun 02 20:12:22 elite-pi caddy[8124]: {"level":"info","ts":1622686342.3226624,"logger":"tls.issuance.acme","msg":"done waiting on internal rate limiter","identifiers":["https://epi.cust.elite-env.com"]}
Jun 02 20:12:22 elite-pi caddy[8124]: {"level":"info","ts":1622686342.323995,"logger":"tls.issuance.acme","msg":"waiting on internal rate limiter","identifiers":["https://10.252.252.21"]}
Jun 02 20:12:22 elite-pi caddy[8124]: {"level":"info","ts":1622686342.3244846,"logger":"tls.issuance.acme","msg":"done waiting on internal rate limiter","identifiers":["https://10.252.252.21"]}
Jun 02 20:12:22 elite-pi caddy[8124]: {"level":"warn","ts":1622686342.325318,"logger":"tls.issuance.zerossl","msg":"missing email address for ZeroSSL; it is strongly recommended to set one for next time"}
Jun 02 20:12:22 elite-pi caddy[8124]: {"level":"warn","ts":1622686342.3264968,"logger":"tls.issuance.zerossl","msg":"missing email address for ZeroSSL; it is strongly recommended to set one for next time"}
Jun 02 20:12:22 elite-pi caddy[8124]: {"level":"warn","ts":1622686342.4120698,"logger":"pki.ca.local","msg":"installing root certificate (you might be prompted for password)","path":"storage:pki/authorities/local/root.crt"}
Jun 02 20:12:22 elite-pi caddy[8124]: 2021/06/02 20:12:22 not NSS security databases found
Jun 02 20:12:22 elite-pi caddy[8124]: 2021/06/02 20:12:22 define JAVA_HOME environment variable to use the Java trust
Jun 02 20:12:22 elite-pi sudo[8134]:     root : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/usr/bin/tee /usr/local/share/ca-certificates/Elite_Local_Facility_Root_CA_-_00000000_135570175785920494935986578965199803499.crt
Jun 02 20:12:22 elite-pi sudo[8134]: pam_unix(sudo:session): session opened for user root by (uid=0)
Jun 02 20:12:22 elite-pi sudo[8134]: pam_unix(sudo:session): session closed for user root
Jun 02 20:12:22 elite-pi caddy[8124]: {"level":"error","ts":1622686342.4348652,"logger":"pki.ca.local","msg":"failed to install root certificate","error":"failed to execute sudo: exit status 1","certificate_file":"storage:pki/authorities/local/root.crt"}
Jun 02 20:12:22 elite-pi caddy[8124]: {"level":"info","ts":1622686342.435452,"msg":"autosaved config (load with --resume flag)","file":"/root/.config/caddy/autosave.json"}
Jun 02 20:12:22 elite-pi caddy[8124]: {"level":"info","ts":1622686342.4355195,"msg":"serving initial configuration"}
Jun 02 20:12:22 elite-pi caddy[8124]: {"level":"info","ts":1622686342.9392276,"logger":"tls.issuance.zerossl","msg":"generated EAB credentials","key_id":"rY5tLH-2aQkR9aoEWbZPVA"}
Jun 02 20:12:22 elite-pi caddy[8124]: {"level":"info","ts":1622686342.9429705,"logger":"tls.issuance.zerossl","msg":"generated EAB credentials","key_id":"9zlmkfjubitu41ejDPn4-w"}
Jun 02 20:12:23 elite-pi caddy[8124]: {"level":"info","ts":1622686343.6460388,"logger":"tls.issuance.acme","msg":"waiting on internal rate limiter","identifiers":["https://epi.cust.elite-env.com"]}
Jun 02 20:12:23 elite-pi caddy[8124]: {"level":"info","ts":1622686343.6484163,"logger":"tls.issuance.acme","msg":"done waiting on internal rate limiter","identifiers":["https://epi.cust.elite-env.com"]}
Jun 02 20:12:23 elite-pi caddy[8124]: {"level":"error","ts":1622686343.649245,"logger":"tls.obtain","msg":"will retry","error":"[https://epi.cust.elite-env.com] Obtain: [https://epi.cust.elite-env.com] no identifiers found (ca=https://acme.zerossl.com/v2/DV90
Jun 02 20:12:24 elite-pi caddy[8124]: {"level":"info","ts":1622686344.167156,"logger":"tls.issuance.acme","msg":"waiting on internal rate limiter","identifiers":["https://10.252.252.21"]}
Jun 02 20:12:24 elite-pi caddy[8124]: {"level":"info","ts":1622686344.1673996,"logger":"tls.issuance.acme","msg":"done waiting on internal rate limiter","identifiers":["https://10.252.252.21"]}
Jun 02 20:12:24 elite-pi caddy[8124]: {"level":"error","ts":1622686344.1675544,"logger":"tls.obtain","msg":"will retry","error":"[https://10.252.252.21] Obtain: [https://10.252.252.21] no identifiers found (ca=https://acme.zerossl.com/v2/DV90)","attempt":1,"r
Jun 02 20:13:23 elite-pi caddy[8124]: {"level":"warn","ts":1622686403.6646206,"logger":"tls.issuance.zerossl","msg":"missing email address for ZeroSSL; it is strongly recommended to set one for next time"}
Jun 02 20:13:24 elite-pi caddy[8124]: {"level":"warn","ts":1622686404.1819944,"logger":"tls.issuance.zerossl","msg":"missing email address for ZeroSSL; it is strongly recommended to set one for next time"}
Jun 02 20:13:24 elite-pi caddy[8124]: {"level":"info","ts":1622686404.5033426,"logger":"tls.issuance.zerossl","msg":"generated EAB credentials","key_id":"opzSiuJT9elWE0QOpdNkhg"}
Jun 02 20:13:24 elite-pi caddy[8124]: {"level":"info","ts":1622686404.9533105,"logger":"tls.issuance.zerossl","msg":"generated EAB credentials","key_id":"vp51-hIAFSjQBnTgId_2-g"}
Jun 02 20:13:25 elite-pi caddy[8124]: {"level":"error","ts":1622686405.8877976,"logger":"tls.obtain","msg":"will retry","error":"[https://epi.cust.elite-env.com] Obtain: [https://epi.cust.elite-env.com] no identifiers found (ca=https://acme.zerossl.com/v2/DV9
Jun 02 20:13:26 elite-pi caddy[8124]: {"level":"error","ts":1622686406.12015,"logger":"tls.obtain","msg":"will retry","error":"[https://10.252.252.21] Obtain: [https://10.252.252.21] no identifiers found (ca=https://acme.zerossl.com/v2/DV90)","attempt":2,"ret
Jun 02 20:15:25 elite-pi caddy[8124]: {"level":"warn","ts":1622686525.9016778,"logger":"tls.issuance.zerossl","msg":"missing email address for ZeroSSL; it is strongly recommended to set one for next time"}
Jun 02 20:15:26 elite-pi caddy[8124]: {"level":"warn","ts":1622686526.1345332,"logger":"tls.issuance.zerossl","msg":"missing email address for ZeroSSL; it is strongly recommended to set one for next time"}
Jun 02 20:15:26 elite-pi caddy[8124]: {"level":"info","ts":1622686526.5825858,"logger":"tls.issuance.zerossl","msg":"generated EAB credentials","key_id":"fZlu1fuBVcUXBfc7MHmfUQ"}
Jun 02 20:15:26 elite-pi caddy[8124]: {"level":"info","ts":1622686526.6728904,"logger":"tls.issuance.zerossl","msg":"generated EAB credentials","key_id":"W27O5VII6-V6gPfNa6fEtg"}
Jun 02 20:15:27 elite-pi caddy[8124]: {"level":"error","ts":1622686527.346271,"logger":"tls.obtain","msg":"will retry","error":"[https://epi.cust.elite-env.com] Obtain: [https://epi.cust.elite-env.com] no identifiers found (ca=https://acme.zerossl.com/v2/DV90
Jun 02 20:15:27 elite-pi caddy[8124]: {"level":"error","ts":1622686527.5161915,"logger":"tls.obtain","msg":"will retry","error":"[https://10.252.252.21] Obtain: [https://10.252.252.21] no identifiers found (ca=https://acme.zerossl.com/v2/DV90)","attempt":3,"r
Jun 02 20:17:01 elite-pi CRON[8144]: pam_unix(cron:session): session opened for user root by (uid=0)
Jun 02 20:17:01 elite-pi CRON[8145]: (root) CMD (   cd / && run-parts --report /etc/cron.hourly)
Jun 02 20:17:02 elite-pi CRON[8144]: pam_unix(cron:session): session closed for user root
Jun 02 20:17:27 elite-pi caddy[8124]: {"level":"warn","ts":1622686647.3640358,"logger":"tls.issuance.zerossl","msg":"missing email address for ZeroSSL; it is strongly recommended to set one for next time"}
Jun 02 20:17:27 elite-pi caddy[8124]: {"level":"warn","ts":1622686647.5271525,"logger":"tls.issuance.zerossl","msg":"missing email address for ZeroSSL; it is strongly recommended to set one for next time"}
Jun 02 20:17:27 elite-pi caddy[8124]: {"level":"info","ts":1622686647.8993118,"logger":"tls.issuance.zerossl","msg":"generated EAB credentials","key_id":"ugGVZhBiw3ad1G1tHm3bVA"}
Jun 02 20:17:27 elite-pi caddy[8124]: {"level":"info","ts":1622686647.9426863,"logger":"tls.issuance.zerossl","msg":"generated EAB credentials","key_id":"aAmWtT3dR2gUkJ075Oeq1A"}
Jun 02 20:17:28 elite-pi caddy[8124]: {"level":"error","ts":1622686648.826648,"logger":"tls.obtain","msg":"will retry","error":"[https://10.252.252.21] Obtain: [https://10.252.252.21] no identifiers found (ca=https://acme.zerossl.com/v2/DV90)","attempt":4,"re
Jun 02 20:17:28 elite-pi caddy[8124]: {"level":"error","ts":1622686648.929608,"logger":"tls.obtain","msg":"will retry","error":"[https://epi.cust.elite-env.com] Obtain: [https://epi.cust.elite-env.com] no identifiers found (ca=https://acme.zerossl.com/v2/DV90

You need to configure TLS automation policies to use the internal issuer:

Configuring PKI just sets up the CA, but it doesn’t tell Caddy to use it. Automation policies are what tells Caddy what to do for each domain.

In what file, the config file? The config goes in the config file. What’s not clear exactly?

For that specifically, understanding the form necessary under apps as the object… e.g. what I eventually got to being the “pki”: { } wrapping the “certificate_authorities”:{ } configuration block.

ah thank you for that pointer, i’ll review and test.

Ok so with this appended for defining PKI and tls certificate

        "apps": {
                "pki": {
                        "certificate_authorities": {
                                "local-test": {
                                        "name": "EM Local",
                                        "root_common_name": "Elite Local Facility Root CA - 00000000",
                                        "intermediate_common_name": "Elite Intermediary CA",
                                        "install_trust": true
                                }
                        }
                }               ,
                "tls": {
                        "certificates": {
                                "automate": ""
                        }                       ,
                        "automation": {
                                "policies": [ {
                                        "subjects": ["192.168.37.145", "epi.int.elite-env.com"],
                                        "issuers": [
                                        {"internal": {
                                                "module": "internal",
                                                "ca": "local-test",
                                                "lifetime": "365d"
                                        }
                                }                               ]
                        }                       ]
                }
        }       ,

(I swear I ran the above through caddy fmt…)

I’m getting this from the validate

eeuser@elite-pi:~ $ sudo caddy validate --config /etc/caddy/caddy.json
2021/06/03 18:12:36.219 INFO    using provided configuration    {"config_file": "/etc/caddy/caddy.json", "config_adapter": ""}
2021/06/03 18:12:36.232 INFO    tls.cache.maintenance   started background certificate maintenance      {"cache": "0x2a5cc30"}
2021/06/03 18:12:36.232 INFO    tls.cache.maintenance   stopped background certificate maintenance      {"cache": "0x2a5cc30"}
validate: loading tls app module: provision tls: loading certificate loader modules: module name 'automate': decoding module config: tls.certificates.automate: json: cannot unmarshal string into Go value of type caddytls.AutomateLoader

I think I’m messing up something on the tls automate, but after iterating back through the docs a few times now I’m lost to what its requiring at this point?

So I guess a fundementals of json configuration question would be: When using a json config object for things like tls automation policy, do I need to fully “use” all the key: value fields, even if I’m just leaving them as default? Will the json file configuration parser assume if its not present its default, or does using partial entry values like I’m doing in my config above break things being read properly?

You don’t need to fill in all the fields. Most of them have the omitempty tag (in the Go code) which leaves it as the default of that type (e.g. nil or 0 or false depending on the type).

The documentation will specify if something is required.

In this case I don’t think you need certificates -> automate at all.

Also, please consider using an IDE (e.g. VS Code or something) and have it format your JSON for you. It’ll save you plenty of headaches with syntax. And set an indentation level of 2 spaces, since it can get pretty deep.

1 Like

Ok so with this as the pki and tls configuration (note this is being edited in jsonedit, I’ve removed the linefeeds for closing brackets to shorten the output as well).

 "apps": {
  "pki":{
    "certificate_authorities": {
     "local-test": {
     "name": "EM Local",
     "root_common_name": "Elite Local Facility Root CA - 00000000",
     "intermediate_common_name": "Elite Intermediary CA",
     "install_trust": true
     }}
   },
  "tls": {
   "automation": {
   "policies": [{
   "subjects": ["192.168.37.145", "epi.int.elite-env.com"],
   "issuers": [
    { "internal": {
     "module": "internal",
     "ca": "local-test",
     "lifetime": "360d"
    }}]}]}
   },

The validate from that seems to be indicating module is not being provided, when it is?

$ sudo caddy validate --config /etc/caddy/caddy.json                                                                                                                                                                                             2021/06/03 20:01:57.621 INFO    using provided configuration    {"config_file": "/etc/caddy/caddy.json", "config_adapter": ""}
2021/06/03 20:01:57.629 INFO    tls.cache.maintenance   started background certificate maintenance      {"cache": "0x4514d70"}
2021/06/03 20:01:57.630 INFO    tls.cache.maintenance   stopped background certificate maintenance      {"cache": "0x4514d70"}
validate: loading tls app module: provision tls: provisioning automation policy 0: loading TLS automation management module: position 0: module name not specified with key 'module' in map[internal:map[ca:local-test lifetime:360d module:internal]]

Or is this a “non error” message at this point and its parsing things correctly?

That’s a legitimate error. Issuers don’t have names, they’re just an array of objects. You nested it one level too deep.

The JSON docs will tell you the structure: https://caddyserver.com/docs/json - as will Abiola’s editor plugin. Just follow the structure.

I’m trying to follow the docs, this is what I was following to get to that last iteration of configuration I posted.

I’m not sure what Abiola’s editor is.

The issuers entry is nested in that documentation in a way I’m not following how to translate into the configuration statement. The issuers is represented at the top level as `“issuers”: [{…}] and then when you step down into it it lays out (as I’m reading it) like below.
So when you say (Francis) that its one level to deep, am I misunderstanding the issuers block? I’m not sure how to structure this then as I read through the levels of the JSON layout in the docs.

"issuers": [
    { "internal": {
{
	"module": "internal",
	"ca": "",
	"lifetime": 0,
	"sign_with_root": false
}
}
}
]
1 Like

Ok so Abiola’s editor appears to be abiosoft/caddy-json-schema coupled with a schema aware json editor.

1 Like

It should be like this:

"issuers": [
	{
		"module": "internal",
		"ca": "",
		"lifetime": 0,
		"sign_with_root": false
	}
]

You have an extra {"internal": { bit in there that doesn’t make sense.

1 Like

Thanks, I’ll adjust how I’m interpreting it when the docs presents the [{…}] and the child pages of detail. That worked, it appears we have a working configuration at this point.

$ sudo caddy validate --config /etc/caddy/caddy.json
2021/06/04 14:44:18.925 INFO    using provided configuration    {"config_file": "/etc/caddy/caddy.json", "config_adapter": ""}
2021/06/04 14:44:18.934 INFO    tls.cache.maintenance   started background certificate maintenance      {"cache": "0x2c08ff0"}
2021/06/04 14:44:18.939 INFO    http    server is listening only on the HTTPS port but has no TLS connection policies; adding one to enable TLS {"server_name": "elite", "https_port": 443}
2021/06/04 14:44:18.939 INFO    http    enabling automatic HTTP->HTTPS redirects        {"server_name": "elite"}
2021/06/04 14:44:18.945 INFO    http.handlers.auth_portal       JWT token name found    {"instance_name": "portal-1", "token_name": "access_token"}
2021/06/04 14:44:18.945 WARN    http.handlers.auth_portal       JWT token origin not found, using default       {"instance_name": "portal-1"}
2021/06/04 14:44:18.945 INFO    http.handlers.auth_portal       local backend configuration     {"db_path": "/opt/caddy/auth/user_db.json", "require_mfa": false}
2021/06/04 14:44:18.946 INFO    http.handlers.auth_portal       validating local backend        {"db_path": "/opt/caddy/auth/user_db.json"}
2021/06/04 14:44:18.966 INFO    http.handlers.auth_portal       JWT token validator provisioned {"instance_name": "portal-1", "access_list": [{"action":"allow","values":["anonymous","guest","*"],"claim":"roles"}]}
2021/06/04 14:44:18.966 INFO    http.handlers.auth_portal       provisioned plugin instance     {"instance_name": "portal-1", "started_at": "2021/06/04 14:44:18.945"}
2021/06/04 14:44:18.967 INFO    http.authentication.providers.jwt       provisioned plugin instance     {"instance_name": "jwt-4", "started_at": "2021/06/04 14:44:18.967"}
2021/06/04 14:44:18.968 INFO    tls.cache.maintenance   stopped background certificate maintenance      {"cache": "0x2c08ff0"}
Valid configuration

I’ll test further and post the working configuration example.
Thanks for the configuration details & clarification.

1 Like

OK so for the TLDR Crew wanting to get JSON based caddyfile configuration in place and start working on TLS and PKI for controlling / customizing the local CA implementation; here is the path to success for us (thanks @matt and @francislavoie for the pointers)

  1. It is folly to attempt managing the caddy.json without a schema re-enforced editor/ide. Best path I found is to set yourself up with Visual Studio Code ( Download Visual Studio Code - Mac, Linux, Windows ) on your favorite desktop. When you install VSC, select the options to add context menu for directory and files (this will make loading the caddy.json with custom schema easier).

  2. Go to Download Caddy and pick your download for your target platform and select your normal plugins you use, but make sure to add the “abiosoft/caddy-json-schema” plugin as well.

  3. Generate the proper vscode directory structure with schema for use with Visual Studio Code in a shell on your target platform with your custom downloaded caddy instance (this example is on Raspbian, but is generally applicable, adjust your paths accordingly)

$ caddy json-schema --vscode
json-schema .vscode/caddy_schema.json written.
json-schema .vscode/settings.json written.

Note if you are not using VSC and just need the schema in a json file you can use the caddy command line to create just that file. This assumes caddy is in your path and will create the file in your current working path.

$ caddy json-schema --output ./caddy_schema.json
  1. zip up the new .vscode path that is generated by the command and copy it to your system you will be running vscode on.

  2. create a directory (path) that you will have your copy of your caddy.json file in. See the note below on creating a caddy.json file (*).

  3. unzip your .vscode.zip file that you created into the working directory you created, copy your caddy.json to that path

  4. (on windows) you can right-click on the directory and select “open in vcode” and it will bring your caddy.json into the schema re-enforced IDE, using your generated caddy.json. On a new line you can prompt for available configuration values by opening double quotes, and a contextual menu will come up with your available commands at that level of your caddyfile (within structural reason, based on the caddy documentation on the json config file layout)

  5. To customize the local CA, this is the “condensed version” of the caddyfile we ended up with, note I’ve collapsed the http: down to only the relevant stuff to this discussion. Some of the finer points to note is the correlation between specific tls and pki block values.

{
  "apps": {
    "http": {
      "https_port": 443,
      "servers": {
        "srv0-Label": {
          "listen": [
            ":443"
          ],
          "routes": [
            {
              "match": [
                {
                  "host": [
                    "198.51.100.10",
                    "host.example.com"
                  ]
                }
**... this is all site specific config here** 
** below is the end of the http:{ block **
    },
    "tls":{
      "automation": {
        "policies": [
          {"issuers": [{
            "module":"internal",
            "ca": "local-test",
            "lifetime": "365d"
            }]
          }
        ]
      }
    },
    "pki": {
      "certificate_authorities": {
        "local-test":{
          "install_trust": true,
          "root_common_name": "Custom Named Local Root CA",
          "intermediate_common_name": "Custom Named Intermediary CA",
          "name": "Elite CA"
        }
      }            
    }
  }
}

(*) You can convert your Caddyfile to json with caddy adapt --config ./Caddyfile -validate and then once you have checked if its valid, generate a file with caddy adapt -pretty >> config.json to generate your config.json to bring into your editor.

I ended up shuttling caddy.json files to the server a few times in testing, but it was more effective to be in a reasonable IDE to get the configuration built properly.

Once we had a working configuration, it results in our local root CA having unique naming from the default caddy CA (we’re running as root so the store is in its path, not running caddy as root creates system trust setup issues). Note the … in the content below indicates clipped content for readability.

# openssl x509 -noout -text -in .local/share/caddy/pki/authorities/local-test/root.crt
Certificate:
   Data:
       Version: 3 (0x2)
       Serial Number:
           15:49:a2:5d:dc:e5:27:25:d9:15:78:cf:71:69:75:e9
       Signature Algorithm: ecdsa-with-SHA256
       Issuer: CN = Custom Named Local Root CA
       Validity
           Not Before: Jun  3 18:08:15 2021 GMT
           Not After : Apr 12 18:08:15 2031 GMT
       Subject: CN = Custom Named Local CA
...
# openssl x509 -noout -text -in .local/share/caddy/pki/authorities/local-test/intermediate.crt
Certificate:
   Data:
       Version: 3 (0x2)
       Serial Number:
           05:50:e1:1d:57:fc:7d:91:d2:d6:d1:68:f3:0a:7d:c7
       Signature Algorithm: ecdsa-with-SHA256
       Issuer: CN = Custom Named Local CA
       Validity
           Not Before: Jun  3 18:08:15 2021 GMT
           Not After : Jun 10 18:08:15 2021 GMT
       Subject: CN = Custom Named Intermediary CA
...

Keep your custom named unique value for the certificate subject DN’s CN value < 64 characters, otherwise you can create scenarios where the open source libraries wont be able to resolve the hash of the subject CN uniquely when attempting to map CA’s to a trusted root CA. This is a ultra edge case, but be aware.

When looking at the significant portion of the generated server instance certificates, we see:

# openssl x509 -noout -text -in .local/share/caddy/certificates/local/198.51.100.10/198.51.100.10.crt
Certificate:
    Data:
       ...
        Signature Algorithm: ecdsa-with-SHA256
        Issuer: CN = Custom Named Intermediary CA
        Validity
            Not Before: Jun  9 22:02:27 2021 GMT
            Not After : Jun 10 10:02:27 2021 GMT
        Subject:
       ...
            X509v3 Subject Alternative Name: critical
                IP Address:198.51.100.10
# openssl x509 -noout -text -in .local/share/caddy/certificates/local/host.example.com/host.example.com.crt
Certificate:
    Data:
     ...
        Issuer: CN = Custom Named Intermediary CA
        Validity
            Not Before: Jun  8 23:22:27 2021 GMT
            Not After : Jun 15 14:42:27 2021 GMT
        Subject:
      ...
            X509v3 Subject Alternative Name: critical
                DNS:host.example.com

Note that a CN value representing the host that the certificate is being issued to is no longer required in the certificate Subject DN value (its deprecated). Best practice moving forward is to use the Subject Alternative Name, which is what Caddy server is doing.

At this point you are ready to go.

If you run into issues to troubleshoot make sure to remember to use the caddy validate --config /etc/caddy/caddy.json against the file for sanity test. Your initial http: block must have a configuration representing your actual IP/FQDN that you are expecting caddy to serve TLS connections for. I also found a good reference to get through understand sane JSON layout in the Caddy Auth Portal source code’s configuration example for a config.json that includes the TLS and PKI module, here (note I had to adapt for the local CA, but it helped understand further).

2 Likes