ACME self-signed certificate setup with caddy server and client instance

I have a situation, where i have 2 caddy instances (acme_server and acme_client) serving mainly as reverse proxies for their respective web application. The acme_server instance should have a CA and provide self-signed certificates internally. Additionally it should have an ACME server, so the acme_client instance can get certificates signed by the acme_server. This way, the user only needs to install the CA of acme_server to trust both caddy instances.

1. Caddy version:

v2.6.2

2. How I installed, and run Caddy:

I have the binary installed on the device

a. System environment:

Debain 10 without systemd, arm64 architecture

b. Command:

/usr/bin/caddy run --pidfile /run/caddy-standard.pid --resume --environ --envfile /etc/caddy/caddy-standard.env

d. My complete Caddy config:

acme_server:

{
  "admin": {
    "listen": "unix//run/caddy-standard.sock",
    "origins": [
      "localhost"
    ]
  },
  "apps": {
    "http": {
      "servers": {
        "external-interfaces": {
          "automatic_https": {
            "disable_redirects": true
          },
          "listen": [
            "0.0.0.0:80",
            "0.0.0.0:443"
          ],
          "routes": [
            {
              "@id": "acme_server",
              "handle": [
                {
                  "handler": "subroute",
                  "routes": [
                    {
                      "handle": [
                        {
                          "ca": "root_ca",
                          "handler": "acme_server"
                        }
                      ]
                    }
                  ]
                }
              ],
              "match": [
                {
                  "remote_ip": {
                    "ranges": [
                      "10.33.41.89"
                    ]
                  }
                },
                {
                  "host": [
                    "acme_client"
                  ]
                }
              ]
            },
            {
              "@id": "device-certificates",
              "handle": [
                {
                  "handler": "subroute",
                  "routes": [
                    {
                      "handle": [
                        {
                          "handler": "static_response",
                          "headers": {
                            "Location": [
                              "/device-certificates/root-ca/"
                            ]
                          },
                          "status_code": "302"
                        }
                      ],
                      "match": [
                        {
                          "path": [
                            "/device-certificates/root-ca"
                          ]
                        }
                      ]
                    },
                    {
                      "handle": [
                        {
                          "handler": "subroute",
                          "routes": [
                            {
                              "handle": [
                                {
                                  "handler": "rewrite",
                                  "strip_path_prefix": "/device-certificates/root-ca"
                                }
                              ]
                            },
                            {
                              "handle": [
                                {
                                  "handler": "headers",
                                  "response": {
                                    "add": {
                                      "Content-Disposition": [
                                        "attachment; filename=\"acme_root.crt\""
                                      ]
                                    }
                                  }
                                },
                                {
                                  "handler": "file_server",
                                  "hide": [
                                    "*.key"
                                  ],
                                  "index_names": [
                                    "root.crt"
                                  ],
                                  "root": "/home/sick/apps/standard/config/caddy/pki/authorities/root_ca"
                                }
                              ]
                            }
                          ]
                        }
                      ],
                      "match": [
                        {
                          "path": [
                            "/device-certificates/root-ca/*"
                          ]
                        }
                      ]
                    }
                  ]
                }
              ]
            },
            {
              "handle": [
                {
                  "handler": "reverse_proxy",
                  "upstreams": [
                    {
                      "dial": "localhost:7000"
                    }
                  ]
                }
              ]
            }
          ],
          "tls_connection_policies": [
            {
              "cipher_suites": [
                "TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256",
                "TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256",
                "TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384",
                "TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384",
                "TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305_SHA256",
                "TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305_SHA256"
              ],
              "protocol_max": "tls1.3",
              "protocol_min": "tls1.2"
            }
          ]
        }
      }
    },
    "pki": {
      "certificate_authorities": {
        "root_ca": {
          "intermediate_common_name": "Standard Intermediate CA",
          "root_common_name": "Standard Root CA"
        }
      }
    },
    "tls": {
      "automation": {
        "on_demand": {
          "rate_limit": {
            "burst": 5,
            "interval": "1m"
          }
        },
        "policies": [
          {
            "issuers": [
              {
                "ca": "root_ca",
                "module": "internal"
              }
            ],
            "key_type": "rsa2048",
            "on_demand": true
          }
        ]
      }
    }
  },
  "logging": {
    "logs": {
      "logging": {
        "level": "DEBUG",
        "writer": {
          "filename": "/home/sick/var/caddy/caddy-standard.log",
          "output": "file",
          "roll_size_mb": 5
        }
      }
    },
    "sink": {
      "writer": {
        "filename": "/home/sick/var/caddy/caddy-standard.sink",
        "output": "file",
        "roll_size_mb": 5
      }
    }
  }
}

acme_client:

{
  "admin": {
    "listen": "unix//run/caddy-standard.sock",
    "origins": [
      "localhost"
    ]
  },
  "apps": {
    "http": {
      "servers": {
        "external-interfaces": {
          "automatic_https": {
            "disable_redirects": true
          },
          "listen": [
            "0.0.0.0:80",
            "0.0.0.0:443"
          ],
          "routes": [
            {
              "@id": "sopasair-standard-engine",
              "handle": [
                {
                  "handler": "reverse_proxy",
                  "upstreams": [
                    {
                      "dial": "localhost:7000"
                    }
                  ]
                }
              ]
            }
          ],
          "tls_connection_policies": [
            {
              "cipher_suites": [
                "TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256",
                "TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256",
                "TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384",
                "TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384",
                "TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305_SHA256",
                "TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305_SHA256"
              ],
              "protocol_max": "tls1.3",
              "protocol_min": "tls1.2"
            }
          ]
        }
      }
    },
    "tls": {
      "automation": {
        "on_demand": {
          "rate_limit": {
            "burst": 5,
            "interval": "1m"
          }
        },
        "policies": [
          {
            "issuers": [
              {
                "ca": "https://acme_server/acme/root_ca/directory",
                "module": "acme",
                "trusted_roots_pem_files": [
                  "/datafs/certs/acme_root.crt"
                ]
              }
            ],
            "key_type": "rsa2048",
            "on_demand": true,
            "subjects": [
              "10.33.41.89",
              "acme_client"
            ]
          }
        ]
      },
      "certificates": {
        "automate": [
          "10.33.41.89",
          "acme_client"
        ]
      }
    }
  },
  "logging": {
    "logs": {
      "logging": {
        "level": "DEBUG",
        "writer": {
          "filename": "/home/sick/var/caddy/caddy-standard.log",
          "output": "file",
          "roll_size_mb": 5
        }
      }
    },
    "sink": {
      "writer": {
        "filename": "/home/sick/var/caddy/caddy-standard.sink",
        "output": "file",
        "roll_size_mb": 5
      }
    }
  }
}

For debugging I manually imported the acme_ca from the acme_server to /datafs/certs/acme_root.crt on the acme_client.

3. The problem I’m having:

The acme_client is unable to get a certificate signed by the acme_server

4. Error messages and/or full log output:

logs I get repeatedly on acme_client

{"level":"debug","ts":1675946209.1758292,"logger":"events","msg":"event","name":"cert_failed","id":"0282b3a3-c277-4661-821a-562e08712e0b","origin":"tls","data":{"error":{},"identifier":"acme_client","issuers":["acme_server-acme-root_ca-directory"],"renewal":false}}
{"level":"error","ts":1675946209.1759293,"logger":"tls.obtain","msg":"will retry","error":"[acme_client] Obtain: [acme_client] creating new order: attempt 1: https://acme_server/acme/root_ca/new-order: HTTP 0 urn:ietf:params:acme:error:malformed - The request message was malformed (ca=https://acme_server/acme/root_ca/directory)","attempt":2,"retrying_in":120,"elapsed":66.725021625,"max_duration":2592000}

logs on acme_server

{"level":"info","ts":1675785103.356883,"logger":"http","msg":"enabling automatic TLS certificate management","domains":["acme_client"]}
{"level":"info","ts":1675945947.7689118,"logger":"http","msg":"enabling automatic TLS certificate management","domains":["acme_client"]}

5. What I already tried:

6. Links to relevant resources:

apps/tls/…/issuers/acme - JSON Config Structure - Caddy Documentation (caddyserver.com)
issuers/acme - JSON Config Structure - Caddy Documentation (caddyserver.com)

Can you enable access logs on your ACME server? You need to configure the HTTP app, i.e. apps/http/servers/logs – you can just set it to "logs": {} to enable it.

I’m not sure what’s going on here. It would probably help to see more detail of what the ACME server side is doing with the request.

After enabling logs on HTTP I get these logs on the acme server:

{"level":"info","ts":1676296498.4102032,"logger":"http.log.access","msg":"handled request","request":{"remote_ip":"10.33.41.89","remote_port":"53526","proto":"HTTP/2.0","method":"GET","host":"acme_server","uri":"/acme/root_ca/directory","headers":{"User-Agent":["Caddy/2.6.2 CertMagic acmez (linux; arm64)"],"Accept-Encoding":["gzip"]},"tls":{"resumed":false,"version":772,"cipher_suite":4865,"proto":"h2","server_name":"acme_server"}},"user_id":"","duration":0.008448,"size":382,"status":200,"resp_headers":{"Content-Type":["application/json"],"Server":["Caddy"],"Alt-Svc":["h3=\":443\"; ma=2592000"]}}
{"level":"info","ts":1676296498.4184043,"logger":"http.log.access","msg":"handled request","request":{"remote_ip":"10.33.41.89","remote_port":"53526","proto":"HTTP/2.0","method":"HEAD","host":"acme_server","uri":"/acme/root_ca/new-nonce","headers":{"User-Agent":["Caddy/2.6.2 CertMagic acmez (linux; arm64)"]},"tls":{"resumed":false,"version":772,"cipher_suite":4865,"proto":"h2","server_name":"acme_server"}},"user_id":"","duration":0.004717125,"size":0,"status":200,"resp_headers":{"Link":["<https://acme_server/acme/root_ca/directory>;rel=\"index\""],"Server":["Caddy"],"Alt-Svc":["h3=\":443\"; ma=2592000"],"Replay-Nonce":["VzZ2Ym1RTnh4cngyZGNJUFhaSWRUak5ad0VYMU12MTc"],"Cache-Control":["no-store"]}}
{"level":"error","ts":1676296498.4470663,"logger":"http.log.access","msg":"handled request","request":{"remote_ip":"10.33.41.89","remote_port":"53526","proto":"HTTP/2.0","method":"POST","host":"acme_server","uri":"/acme/root_ca/new-order","headers":{"Content-Type":["application/jose+json"],"User-Agent":["Caddy/2.6.2 CertMagic acmez (linux; arm64)"],"Content-Length":["658"],"Accept-Encoding":["gzip"]},"tls":{"resumed":false,"version":772,"cipher_suite":4865,"proto":"h2","server_name":"acme_server"}},"user_id":"","duration":0.022881375,"size":93,"status":400,"resp_headers":{"Server":["Caddy"],"Alt-Svc":["h3=\":443\"; ma=2592000"],"Replay-Nonce":["U0hyQWRFd09idllZM0duRE1ydExxR2hSQ3NiWlcwZXE"],"Cache-Control":["no-store"],"Link":["<https://acme_server/acme/root_ca/directory>;rel=\"index\""],"Content-Type":["application/problem+json"]}}

10.33.41.89 is the IP address of the acme client

Hmmm… Could you try some earlier versions of Caddy to see if it works with those? I’m thinking this might have been a regression that wasn’t caught because the ACME server stuff is within an external dependency.

Oh, maybe this is meant to be acme_server?

I get the same error. Omitting the whole “match” module also produces the same error.

Did you remove all other handlers from the ACME server? Try doing that too, just to be absolutely sure requests aren’t being mishandled.

Did you try that? What versions did you try?

This issue seemed to be caused by an incorrect installation.

In what way was it incorrect? What did you do to fix it exactly?

I was working with marcsal on this.

We first played with two existing machines that had Caddy running and ran into the problem. Then we didn’t run into them anymore with another machine. Unfortunately because we tried so many things we now don’t have this installation anymore so we don’t know what was different :see_no_evil:

1 Like

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.