Where are the tls certs stored?

reubano · July 20, 2023, 6:33am

1. The problem I’m having:

What are you trying to do? Use vouch-proxy to authenticate with azure active directory. Vouch requires a path to the tls fullchain.pem and privkey.pem files

What isn’t working? I can’t figure out the filepath for the tls certs after caddy aquires them. From reading the forum, I’ve gather it should be in the data/caddy/certificates/acme-v02.api.letsencrypt.org/sites/{domain}/ folder. But I don’t know what the file names are and if they change after each cert renewal.

2. Error messages and/or full log output:

{"level":"info","ts":1689833783.4240146,"msg":"using provided configuration","config_file":"/etc/caddy/Caddyfile","config_adapter":"caddyfile"}
{"level":"warn","ts":1689833783.4342937,"msg":"Caddyfile input is not formatted; run the 'caddy fmt' command to fix inconsistencies","adapter":"caddyfile","file":"/etc/caddy/Caddyfile","line":2}
{"level":"info","ts":1689833783.4421346,"logger":"admin","msg":"admin endpoint started","address":"localhost:2019","enforce_origin":false,"origins":["//localhost:2019","//[::1]:2019","//127.0.0.1:2019"]}
{"level":"info","ts":1689833783.4451523,"logger":"tls.cache.maintenance","msg":"started background certificate maintenance","cache":"0xc00035f880"}
{"level":"info","ts":1689833783.4240146,"msg":"using provided configuration","config_file":"/etc/caddy/Caddyfile","config_adapter":"caddyfile"}
{"level":"warn","ts":1689833783.4342937,"msg":"Caddyfile input is not formatted; run the 'caddy fmt' command to fix inconsistencies","adapter":"caddyfile","file":"/etc/caddy/Caddyfile","line":2}
{"level":"info","ts":1689833783.4421346,"logger":"admin","msg":"admin endpoint started","address":"localhost:2019","enforce_origin":false,"origins":["//localhost:2019","//[::1]:2019","//127.0.0.1:2019"]}
{"level":"info","ts":1689833783.4451523,"logger":"tls.cache.maintenance","msg":"started background certificate maintenance","cache":"0xc00035f880"}
{"level":"info","ts":1689833783.4240146,"msg":"using provided configuration","config_file":"/etc/caddy/Caddyfile","config_adapter":"caddyfile"}
{"level":"warn","ts":1689833783.4342937,"msg":"Caddyfile input is not formatted; run the 'caddy fmt' command to fix inconsistencies","adapter":"caddyfile","file":"/etc/caddy/Caddyfile","line":2}
{"level":"info","ts":1689833783.4421346,"logger":"admin","msg":"admin endpoint started","address":"localhost:2019","enforce_origin":false,"origins":["//localhost:2019","//[::1]:2019","//127.0.0.1:2019"]}
{"level":"info","ts":1689833783.4451523,"logger":"tls.cache.maintenance","msg":"started background certificate maintenance","cache":"0xc00035f880"}
{"level":"info","ts":1689833783.4240146,"msg":"using provided configuration","config_file":"/etc/caddy/Caddyfile","config_adapter":"caddyfile"}
{"level":"warn","ts":1689833783.4342937,"msg":"Caddyfile input is not formatted; run the 'caddy fmt' command to fix inconsistencies","adapter":"caddyfile","file":"/etc/caddy/Caddyfile","line":2}
{"level":"info","ts":1689833783.4421346,"logger":"admin","msg":"admin endpoint started","address":"localhost:2019","enforce_origin":false,"origins":["//localhost:2019","//[::1]:2019","//127.0.0.1:2019"]}
{"level":"info","ts":1689833783.4451523,"logger":"tls.cache.maintenance","msg":"started background certificate maintenance","cache":"0xc00035f880"}
{"level":"info","ts":1689833783.4240146,"msg":"using provided configuration","config_file":"/etc/caddy/Caddyfile","config_adapter":"caddyfile"}
{"level":"warn","ts":1689833783.4342937,"msg":"Caddyfile input is not formatted; run the 'caddy fmt' command to fix inconsistencies","adapter":"caddyfile","file":"/etc/caddy/Caddyfile","line":2}
{"level":"info","ts":1689833783.4421346,"logger":"admin","msg":"admin endpoint started","address":"localhost:2019","enforce_origin":false,"origins":["//localhost:2019","//[::1]:2019","//127.0.0.1:2019"]}
{"level":"info","ts":1689833783.4451523,"logger":"tls.cache.maintenance","msg":"started background certificate maintenance","cache":"0xc00035f880"}
{"level":"info","ts":1689833783.4240146,"msg":"using provided configuration","config_file":"/etc/caddy/Caddyfile","config_adapter":"caddyfile"}
{"level":"warn","ts":1689833783.4342937,"msg":"Caddyfile input is not formatted; run the 'caddy fmt' command to fix inconsistencies","adapter":"caddyfile","file":"/etc/caddy/Caddyfile","line":2}
{"level":"info","ts":1689833783.4421346,"logger":"admin","msg":"admin endpoint started","address":"localhost:2019","enforce_origin":false,"origins":["//localhost:2019","//[::1]:2019","//127.0.0.1:2019"]}
{"level":"info","ts":1689833783.4451523,"logger":"tls.cache.maintenance","msg":"started background certificate maintenance","cache":"0xc00035f880"}
{"level":"info","ts":1689833783.4240146,"msg":"using provided configuration","config_file":"/etc/caddy/Caddyfile","config_adapter":"caddyfile"}
{"level":"warn","ts":1689833783.4342937,"msg":"Caddyfile input is not formatted; run the 'caddy fmt' command to fix inconsistencies","adapter":"caddyfile","file":"/etc/caddy/Caddyfile","line":2}
{"level":"info","ts":1689833783.4421346,"logger":"admin","msg":"admin endpoint started","address":"localhost:2019","enforce_origin":false,"origins":["//localhost:2019","//[::1]:2019","//127.0.0.1:2019"]}
{"level":"info","ts":1689833783.4451523,"logger":"tls.cache.maintenance","msg":"started background certificate maintenance","cache":"0xc00035f880"}
{"level":"info","ts":1689833783.4240146,"msg":"using provided configuration","config_file":"/etc/caddy/Caddyfile","config_adapter":"caddyfile"}
{"level":"warn","ts":1689833783.4342937,"msg":"Caddyfile input is not formatted; run the 'caddy fmt' command to fix inconsistencies","adapter":"caddyfile","file":"/etc/caddy/Caddyfile","line":2}
{"level":"info","ts":1689833783.4421346,"logger":"admin","msg":"admin endpoint started","address":"localhost:2019","enforce_origin":false,"origins":["//localhost:2019","//[::1]:2019","//127.0.0.1:2019"]}
{"level":"info","ts":1689833783.4451523,"logger":"tls.cache.maintenance","msg":"started background certificate maintenance","cache":"0xc00035f880"}
{"level":"info","ts":1689833783.4240146,"msg":"using provided configuration","config_file":"/etc/caddy/Caddyfile","config_adapter":"caddyfile"}
{"level":"warn","ts":1689833783.4342937,"msg":"Caddyfile input is not formatted; run the 'caddy fmt' command to fix inconsistencies","adapter":"caddyfile","file":"/etc/caddy/Caddyfile","line":2}
{"level":"info","ts":1689833783.4421346,"logger":"admin","msg":"admin endpoint started","address":"localhost:2019","enforce_origin":false,"origins":["//localhost:2019","//[::1]:2019","//127.0.0.1:2019"]}
{"level":"info","ts":1689833783.4451523,"logger":"tls.cache.maintenance","msg":"started background certificate maintenance","cache":"0xc00035f880"}
{"level":"info","ts":1689833783.4240146,"msg":"using provided configuration","config_file":"/etc/caddy/Caddyfile","config_adapter":"caddyfile"}
{"level":"warn","ts":1689833783.4342937,"msg":"Caddyfile input is not formatted; run the 'caddy fmt' command to fix inconsistencies","adapter":"caddyfile","file":"/etc/caddy/Caddyfile","line":2}
{"level":"info","ts":1689833783.4421346,"logger":"admin","msg":"admin endpoint started","address":"localhost:2019","enforce_origin":false,"origins":["//localhost:2019","//[::1]:2019","//127.0.0.1:2019"]}
{"level":"info","ts":1689833783.4451523,"logger":"tls.cache.maintenance","msg":"started background certificate maintenance","cache":"0xc00035f880"}
{"level":"info","ts":1689833783.4240146,"msg":"using provided configuration","config_file":"/etc/caddy/Caddyfile","config_adapter":"caddyfile"}
{"level":"warn","ts":1689833783.4342937,"msg":"Caddyfile input is not formatted; run the 'caddy fmt' command to fix inconsistencies","adapter":"caddyfile","file":"/etc/caddy/Caddyfile","line":2}
{"level":"info","ts":1689833783.4421346,"logger":"admin","msg":"admin endpoint started","address":"localhost:2019","enforce_origin":false,"origins":["//localhost:2019","//[::1]:2019","//127.0.0.1:2019"]}
{"level":"info","ts":1689833783.4451523,"logger":"tls.cache.maintenance","msg":"started background certificate maintenance","cache":"0xc00035f880"}
{"level":"info","ts":1689833783.4240146,"msg":"using provided configuration","config_file":"/etc/caddy/Caddyfile","config_adapter":"caddyfile"}
{"level":"warn","ts":1689833783.4342937,"msg":"Caddyfile input is not formatted; run the 'caddy fmt' command to fix inconsistencies","adapter":"caddyfile","file":"/etc/caddy/Caddyfile","line":2}
{"level":"info","ts":1689833783.4421346,"logger":"admin","msg":"admin endpoint started","address":"localhost:2019","enforce_origin":false,"origins":["//localhost:2019","//[::1]:2019","//127.0.0.1:2019"]}
{"level":"info","ts":1689833783.4451523,"logger":"tls.cache.maintenance","msg":"started background certificate maintenance","cache":"0xc00035f880"}
{"level":"info","ts":1689833783.4240146,"msg":"using provided configuration","config_file":"/etc/caddy/Caddyfile","config_adapter":"caddyfile"}
{"level":"warn","ts":1689833783.4342937,"msg":"Caddyfile input is not formatted; run the 'caddy fmt' command to fix inconsistencies","adapter":"caddyfile","file":"/etc/caddy/Caddyfile","line":2}
{"level":"info","ts":1689833783.4421346,"logger":"admin","msg":"admin endpoint started","address":"localhost:2019","enforce_origin":false,"origins":["//localhost:2019","//[::1]:2019","//127.0.0.1:2019"]}
{"level":"info","ts":1689833783.4451523,"logger":"tls.cache.maintenance","msg":"started background certificate maintenance","cache":"0xc00035f880"}
{"level":"info","ts":1689833783.4240146,"msg":"using provided configuration","config_file":"/etc/caddy/Caddyfile","config_adapter":"caddyfile"}
{"level":"warn","ts":1689833783.4342937,"msg":"Caddyfile input is not formatted; run the 'caddy fmt' command to fix inconsistencies","adapter":"caddyfile","file":"/etc/caddy/Caddyfile","line":2}
{"level":"info","ts":1689833783.4421346,"logger":"admin","msg":"admin endpoint started","address":"localhost:2019","enforce_origin":false,"origins":["//localhost:2019","//[::1]:2019","//127.0.0.1:2019"]}
{"level":"info","ts":1689833783.4451523,"logger":"tls.cache.maintenance","msg":"started background certificate maintenance","cache":"0xc00035f880"}
{"level":"info","ts":1689833783.4240146,"msg":"using provided configuration","config_file":"/etc/caddy/Caddyfile","config_adapter":"caddyfile"}
{"level":"warn","ts":1689833783.4342937,"msg":"Caddyfile input is not formatted; run the 'caddy fmt' command to fix inconsistencies","adapter":"caddyfile","file":"/etc/caddy/Caddyfile","line":2}
{"level":"info","ts":1689833783.4421346,"logger":"admin","msg":"admin endpoint started","address":"localhost:2019","enforce_origin":false,"origins":["//localhost:2019","//[::1]:2019","//127.0.0.1:2019"]}
{"level":"info","ts":1689833783.4451523,"logger":"tls.cache.maintenance","msg":"started background certificate maintenance","cache":"0xc00035f880"}
{"level":"info","ts":1689833783.4240146,"msg":"using provided configuration","config_file":"/etc/caddy/Caddyfile","config_adapter":"caddyfile"}
{"level":"warn","ts":1689833783.4342937,"msg":"Caddyfile input is not formatted; run the 'caddy fmt' command to fix inconsistencies","adapter":"caddyfile","file":"/etc/caddy/Caddyfile","line":2}
{"level":"info","ts":1689833783.4421346,"logger":"admin","msg":"admin endpoint started","address":"localhost:2019","enforce_origin":false,"origins":["//localhost:2019","//[::1]:2019","//127.0.0.1:2019"]}
{"level":"info","ts":1689833783.4451523,"logger":"tls.cache.maintenance","msg":"started background certificate maintenance","cache":"0xc00035f880"}
{"level":"info","ts":1689833783.4240146,"msg":"using provided configuration","config_file":"/etc/caddy/Caddyfile","config_adapter":"caddyfile"}
{"level":"warn","ts":1689833783.4342937,"msg":"Caddyfile input is not formatted; run the 'caddy fmt' command to fix inconsistencies","adapter":"caddyfile","file":"/etc/caddy/Caddyfile","line":2}
{"level":"info","ts":1689833783.4421346,"logger":"admin","msg":"admin endpoint started","address":"localhost:2019","enforce_origin":false,"origins":["//localhost:2019","//[::1]:2019","//127.0.0.1:2019"]}
{"level":"info","ts":1689833783.4451523,"logger":"tls.cache.maintenance","msg":"started background certificate maintenance","cache":"0xc00035f880"}
{"level":"info","ts":1689833783.4240146,"msg":"using provided configuration","config_file":"/etc/caddy/Caddyfile","config_adapter":"caddyfile"}
{"level":"warn","ts":1689833783.4342937,"msg":"Caddyfile input is not formatted; run the 'caddy fmt' command to fix inconsistencies","adapter":"caddyfile","file":"/etc/caddy/Caddyfile","line":2}
{"level":"info","ts":1689833783.4421346,"logger":"admin","msg":"admin endpoint started","address":"localhost:2019","enforce_origin":false,"origins":["//localhost:2019","//[::1]:2019","//127.0.0.1:2019"]}
{"level":"info","ts":1689833783.4451523,"logger":"tls.cache.maintenance","msg":"started background certificate maintenance","cache":"0xc00035f880"}
{"level":"info","ts":1689833783.4240146,"msg":"using provided configuration","config_file":"/etc/caddy/Caddyfile","config_adapter":"caddyfile"}
{"level":"warn","ts":1689833783.4342937,"msg":"Caddyfile input is not formatted; run the 'caddy fmt' command to fix inconsistencies","adapter":"caddyfile","file":"/etc/caddy/Caddyfile","line":2}
{"level":"info","ts":1689833783.4421346,"logger":"admin","msg":"admin endpoint started","address":"localhost:2019","enforce_origin":false,"origins":["//localhost:2019","//[::1]:2019","//127.0.0.1:2019"]}
{"level":"info","ts":1689833783.4451523,"logger":"tls.cache.maintenance","msg":"started background certificate maintenance","cache":"0xc00035f880"}
{"level":"info","ts":1689833783.4240146,"msg":"using provided configuration","config_file":"/etc/caddy/Caddyfile","config_adapter":"caddyfile"}
{"level":"warn","ts":1689833783.4342937,"msg":"Caddyfile input is not formatted; run the 'caddy fmt' command to fix inconsistencies","adapter":"caddyfile","file":"/etc/caddy/Caddyfile","line":2}
{"level":"info","ts":1689833783.4421346,"logger":"admin","msg":"admin endpoint started","address":"localhost:2019","enforce_origin":false,"origins":["//localhost:2019","//[::1]:2019","//127.0.0.1:2019"]}
{"level":"info","ts":1689833783.4451523,"logger":"tls.cache.maintenance","msg":"started background certificate maintenance","cache":"0xc00035f880"}
{"level":"info","ts":1689833783.4240146,"msg":"using provided configuration","config_file":"/etc/caddy/Caddyfile","config_adapter":"caddyfile"}
{"level":"warn","ts":1689833783.4342937,"msg":"Caddyfile input is not formatted; run the 'caddy fmt' command to fix inconsistencies","adapter":"caddyfile","file":"/etc/caddy/Caddyfile","line":2}
{"level":"info","ts":1689833783.4421346,"logger":"admin","msg":"admin endpoint started","address":"localhost:2019","enforce_origin":false,"origins":["//localhost:2019","//[::1]:2019","//127.0.0.1:2019"]}
{"level":"info","ts":1689833783.4451523,"logger":"tls.cache.maintenance","msg":"started background certificate maintenance","cache":"0xc00035f880"}
{"level":"info","ts":1689833783.4240146,"msg":"using provided configuration","config_file":"/etc/caddy/Caddyfile","config_adapter":"caddyfile"}
{"level":"warn","ts":1689833783.4342937,"msg":"Caddyfile input is not formatted; run the 'caddy fmt' command to fix inconsistencies","adapter":"caddyfile","file":"/etc/caddy/Caddyfile","line":2}
{"level":"info","ts":1689833783.4421346,"logger":"admin","msg":"admin endpoint started","address":"localhost:2019","enforce_origin":false,"origins":["//localhost:2019","//[::1]:2019","//127.0.0.1:2019"]}
{"level":"info","ts":1689833783.4451523,"logger":"tls.cache.maintenance","msg":"started background certificate maintenance","cache":"0xc00035f880"}
{"level":"info","ts":1689833783.4240146,"msg":"using provided configuration","config_file":"/etc/caddy/Caddyfile","config_adapter":"caddyfile"}
{"level":"warn","ts":1689833783.4342937,"msg":"Caddyfile input is not formatted; run the 'caddy fmt' command to fix inconsistencies","adapter":"caddyfile","file":"/etc/caddy/Caddyfile","line":2}
{"level":"info","ts":1689833783.4421346,"logger":"admin","msg":"admin endpoint started","address":"localhost:2019","enforce_origin":false,"origins":["//localhost:2019","//[::1]:2019","//127.0.0.1:2019"]}
{"level":"info","ts":1689833783.4451523,"logger":"tls.cache.maintenance","msg":"started background certificate maintenance","cache":"0xc00035f880"}
{"level":"info","ts":1689833783.4240146,"msg":"using provided configuration","config_file":"/etc/caddy/Caddyfile","config_adapter":"caddyfile"}
{"level":"warn","ts":1689833783.4342937,"msg":"Caddyfile input is not formatted; run the 'caddy fmt' command to fix inconsistencies","adapter":"caddyfile","file":"/etc/caddy/Caddyfile","line":2}
{"level":"info","ts":1689833783.4421346,"logger":"admin","msg":"admin endpoint started","address":"localhost:2019","enforce_origin":false,"origins":["//localhost:2019","//[::1]:2019","//127.0.0.1:2019"]}
{"level":"info","ts":1689833783.4451523,"logger":"tls.cache.maintenance","msg":"started background certificate maintenance","cache":"0xc00035f880"}
{"level":"info","ts":1689833783.4240146,"msg":"using provided configuration","config_file":"/etc/caddy/Caddyfile","config_adapter":"caddyfile"}
{"level":"warn","ts":1689833783.4342937,"msg":"Caddyfile input is not formatted; run the 'caddy fmt' command to fix inconsistencies","adapter":"caddyfile","file":"/etc/caddy/Caddyfile","line":2}
{"level":"info","ts":1689833783.4421346,"logger":"admin","msg":"admin endpoint started","address":"localhost:2019","enforce_origin":false,"origins":["//localhost:2019","//[::1]:2019","//127.0.0.1:2019"]}
{"level":"info","ts":1689833783.4451523,"logger":"tls.cache.maintenance","msg":"started background certificate maintenance","cache":"0xc00035f880"}
{"level":"info","ts":1689833783.4240146,"msg":"using provided configuration","config_file":"/etc/caddy/Caddyfile","config_adapter":"caddyfile"}
{"level":"warn","ts":1689833783.4342937,"msg":"Caddyfile input is not formatted; run the 'caddy fmt' command to fix inconsistencies","adapter":"caddyfile","file":"/etc/caddy/Caddyfile","line":2}
{"level":"info","ts":1689833783.4421346,"logger":"admin","msg":"admin endpoint started","address":"localhost:2019","enforce_origin":false,"origins":["//localhost:2019","//[::1]:2019","//127.0.0.1:2019"]}
{"level":"info","ts":1689833783.4451523,"logger":"tls.cache.maintenance","msg":"started background certificate maintenance","cache":"0xc00035f880"}
{"level":"info","ts":1689833783.447647,"logger":"http","msg":"server is listening only on the HTTPS port but has no TLS connection policies; adding one to enable TLS","server_name":"srv0","https_port":443}
{"level":"info","ts":1689833783.4476907,"logger":"http","msg":"enabling automatic HTTP->HTTPS redirects","server_name":"srv0"}
{"level":"info","ts":1689833783.452004,"logger":"tls","msg":"cleaning storage unit","description":"FileStorage:/data/caddy"}
{"level":"info","ts":1689833783.4524798,"logger":"http","msg":"enabling HTTP/3 listener","addr":":443"}
{"level":"info","ts":1689833783.452803,"msg":"failed to sufficiently increase receive buffer size (was: 208 kiB, wanted: 2048 kiB, got: 416 kiB). See https://github.com/quic-go/quic-go/wiki/UDP-Receive-Buffer-Size for details."}
{"level":"info","ts":1689833783.4925442,"logger":"http.log","msg":"server running","name":"srv0","protocols":["h1","h2","h3"]}
{"level":"info","ts":1689833783.4926584,"logger":"http.log","msg":"server running","name":"remaining_auto_https_redirects","protocols":["h1","h2","h3"]}
{"level":"info","ts":1689833783.4926705,"logger":"http","msg":"enabling automatic TLS certificate management","domains":["prometheusweb-dev.centralus.azurecontainer.io"]}
{"level":"info","ts":1689833783.7282891,"logger":"tls","msg":"finished cleaning storage units"}
{"level":"info","ts":1689833783.755422,"msg":"autosaved config (load with --resume flag)","file":"/config/caddy/autosave.json"}
{"level":"info","ts":1689833783.7554777,"msg":"serving initial configuration"}

3. Caddy version:

2.6.4

4. How I installed and ran Caddy:

a. System environment:

Docker via Azure Container Instance via Pulumi
https://hub.docker.com/_/caddy
image:caddy

b. Command:

pulumi up  # this deploys and runs the caddy docker container

c. Service/unit/compose file:

pulumi

"""An Azure RM Python Pulumi program"""
from pathlib import Path
from base64 import b64encode
from secrets import token_urlsafe

from pulumi import Output, Config, ResourceOptions

from pulumi_azure_native import storage
from pulumi_azure_native.storage import StorageAccount, FileShare
from pulumi_azure_native.containerinstance import ContainerGroup, VolumeMountArgs, ContainerArgs
from pulumi_azure_native.resources import ResourceGroup

config = Config()
location = config.require("location")
env = config.require("env")
lowered_env = env.lower()
resource_group_config = config.require_object("resourceGroup")
vouch_config = config.require_object("vouch")

subdomain = f"web-{lowered_env}"
container_base = "centralus.azurecontainer.io"
domain = f"{subdomain}.{container_base}"

resource_group = ResourceGroup(
    f"app{env}",
    opts=ResourceOptions(protect=True),
    location=location,
    resource_group_name=resource_group_config["name"],
    tags=resource_group_config["tags"],
)

storage_account = StorageAccount(
    f"pulumiStorage{env}",
    minimum_tls_version="TLS1_2",
    account_name=f"pulumistorageacct{lowered_env}",
    allow_blob_public_access=False,
    resource_group_name=resource_group.name,
    sku=storage.SkuArgs(name=storage.SkuName.STANDARD_LRS),
    kind=storage.Kind.STORAGE_V2,
)


def create_file_share(name, quota=1, protect=False):
    return FileShare(
        f"{name}-fileshare-{lowered_env}",
        opts=ResourceOptions(protect=protect),
        account_name=storage_account.name,
        resource_group_name=resource_group.name,
        share_quota=quota
    )

caddy_config_fileshare = create_file_share("caddy-config")
caddy_data_fileshare = create_file_share("caddy-data", protect=True)
vouch_secret_fileshare = create_file_share("vouch-secret")

primary_storage_account_key = Output.secret(
    Output.all(resource_group.name, storage_account.name).apply(
        lambda args: storage.list_storage_account_keys(
            resource_group_name=args[0], account_name=args[1]
        )
    ).apply(lambda keys: keys.keys[0].value)
)

def get_file_share_config(name, read_only=False):
    return {
        "share_name": name,
        "storage_account_name": storage_account.name,
        "read_only": read_only,
        "storage_account_key": primary_storage_account_key
    }

# https://hub.docker.com/_/caddy
# https://learn.microsoft.com/en-us/azure/container-instances/container-instances-container-group-automatic-ssl
caddy_container = ContainerArgs(
    name=f"caddy-{lowered_env}",
    image="caddy",
    resources={"requests": {"memory_in_gb": .5, "cpu": .5}},
    ports=[{"port": 80}, {"port": 443}],
    volume_mounts=[
        VolumeMountArgs(mount_path="/config", name="caddy-config", read_only=False),
        VolumeMountArgs(mount_path="/data", name="caddy-data", read_only=False),
        VolumeMountArgs(mount_path="/etc/caddy", name="caddyfile", read_only=False),
    ],
    # command=["/bin/bash", "-c", f"caddy reverse-proxy --from {domain} --to :8080"]
)

# https://hub.docker.com/_/nginx
nginx_container = ContainerArgs(
    name=f"nginx-{lowered_env}",
    image="nginx",
    resources={"requests": {"memory_in_gb": 1, "cpu": 1}},
    ports=[{"port": 8080}],
    volume_mounts=[
        VolumeMountArgs(mount_path="/etc/nginx/templates", name="nginx-templates", read_only=False),
    ],

    environment_variables=[
        {"name": "NGINX_HOST", "value": domain},
    ]
)

# https://github.com/vouch/vouch-proxy#running-from-docker
vouch_container = ContainerArgs(
    name=f"vouch-{lowered_env}",
    image="quay.io/vouch/vouch-proxy:latest",
    resources={"requests": {"memory_in_gb": 1, "cpu": 1}},
    ports=[{"port": 9091}],
    volume_mounts=[
        VolumeMountArgs(mount_path="/config/secret", name="vouch-secret", read_only=False),
        VolumeMountArgs(mount_path="/data", name="caddy-data", read_only=False),
    ],

    environment_variables=[
        {"name": "OAUTH_PROVIDER", "value": "azure"},
        {"name": "OAUTH_CLIENT_ID", "value": vouch_config["clientID"]},
        {"name": "OAUTH_CLIENT_SECRET", "value": vouch_config["clientSecret"]},
        {"name": "OAUTH_CALLBACK_URL", "value": f"https://{domain}/oauth2/auth"},
        {"name": "OAUTH_AUTH_URL", "value": "https://login.microsoftonline.com/{tenantID}/oauth2/v2.0/authorize".format(**vouch_config)},
        {"name": "OAUTH_TOKEN_URL", "value": "https://login.microsoftonline.com/{tenantID}/oauth2/v2.0/token".format(**vouch_config)},
        {"name": "OAUTH_USER_INFO_URL", "value": "https://graph.microsoft.com/oidc/userinfo"},
        {"name": "OAUTH_SCOPES", "value": "openid profile email"},
        {"name": "VOUCH_SESSION_KEY", "value": token_urlsafe(128)},
        {"name": "VOUCH_JWT_SECRET", "value": token_urlsafe(128)},
        {"name": "VOUCH_LOGLEVEL", "value": "debug"},
        {"name": "VOUCH_DOCUMENT_ROOT", "value": "/oauth2"},
        {"name": "VOUCH_TLS_PROFILE", "value": "intermediate"},
        {"name": "VOUCH_PORT", "value": "9091"},
        {"name": "VOUCH_ALLOWALLUSERS", "value": True},
        {"name": "VOUCH_COOKIE_DOMAIN", "value": domain},
        {"name": "VOUCH_TLS_CERT", "value": f"/data/caddy/certificates/acme-v02.api.letsencrypt.org/sites/{domain}/fullchain.pem"},
        {"name": "VOUCH_TLS_KEY", "value": f"/data/caddy/certificates/acme-v02.api.letsencrypt.org/sites/{domain}/privkey.pem"},
    ]
)

def get_container_volume(name, fileshare=None, secret=None):
    volume = {"name": name}

    if secret:
        volume["secret"] = {}
        path = Path(secret)

        with path.open("rb") as f:
            volume["secret"][path.name] = b64encode(f.read()).decode("ascii")
    elif fileshare:
        volume["azure_file"] = get_file_share_config(fileshare.name)
    else:
        raise ValueError("Must provide either fileshare or secret")

    return volume

caddyfile_text = f"""{domain} {{
    reverse_proxy http://localhost:8080
    header Strict-Transport-Security max-age=31536000;
}}"""

container_volumes = [
    get_container_volume("caddy-config", caddy_config_fileshare),
    get_container_volume("caddy-data", caddy_data_fileshare),
    get_container_volume("caddyfile", secret_text=caddyfile_text, file_name="Caddyfile"),
    get_container_volume("nginx-templates", secret="../configs/nginx/templates/default.conf.template"),
    get_container_volume("vouch-secret", vouch_secret_fileshare),
]

container_group = ContainerGroup(
    f"containerGroup{env}",
    containers=[
        caddy_container,
        nginx_container,
        vouch_container,
    ],
    ip_address={
        "ports": [{"port": 80}, {"port": 443}],
        "type": "Public",
        "dns_name_label": subdomain,
        "auto_generated_domain_name_label_scope": "TenantReuse"
    },
    os_type="Linux",
    resource_group_name=resource_group.name,
    container_group_name=f"container-group-{lowered_env}",
    location=resource_group.location,
    restart_policy="OnFailure",
    volumes=container_volumes,
)

d. My complete Caddy config:

prometheusweb-dev.centralus.azurecontainer.io {
    reverse_proxy http://localhost:8080
    header Strict-Transport-Security max-age=31536000;
}

5. Links to relevant resources:

github.com

vouch/vouch-proxy/blob/master/config/config.yml_example

# Vouch Proxy configuration

# you should probably start with one of the other example configs in this directory
# Vouch Proxy does a fairly good job of setting its config to sane defaults

# be aware of the yaml indentation, the only top level elements are `vouch` and `oauth`. 

# Vouch Proxy can also be configured using Environmental Variables.  The associated env var for
# each configuration is shown such as VOUCH_LOGLEVEL.

vouch:
  # logLevel: debug # VOUCH_LOGLEVEL
  logLevel: info

  # testing: false - VOUCH_TESTING
  # force all 302 redirects to be rendered as a webpage with a link
  # if you're having problems, turn on testing
  testing: true

  listen: 0.0.0.0  # VOUCH_LISTEN

This file has been truncated. show original

VOUCH_TLS_CERT and VOUCH_TLS_KEY

francislavoie · July 20, 2023, 6:50am

Correct.

The filenames will not change, but Caddy may choose to use ZeroSSL instead of Let’s Encrypt, which means the directory will change (i.e. it won’t be in acme-v02.api/letsencrypt.org). A renewal may switch to a different issuer if it has problems when trying the same one first.

I suggest not taking this approach. This really complicates things. You should let Caddy be the thing in front, don’t put something else in front of Caddy.

You can have Caddy proxy to vouch or something instead.

Caddy has forward_auth (Caddyfile directive) — Caddy Documentation which you can use have Caddy send requests to another upstream to authenticate before continuing.

I’ve never heard of Vouch before, but it might be possible to use forward_auth with Vouch. Or consider using Authelia instead which we know works well with Caddy.

reubano · July 20, 2023, 6:33pm

@francislavoie can you link to documentation on using Authelia with Azure AD? I’ve found Microsoft | Caddy Security but am not sure if this is what I should use or not.

francislavoie · July 20, 2023, 6:46pm

I don’t use any of these authentication tools/libs so I can’t really help much further. But you can probably use Authelia’s LDAP integration to make it work with Azure AD: LDAP - Configuration - Authelia

reubano · July 20, 2023, 7:17pm

Ok, thanks. I’m using OAuth/OIDC instead of LDAP though. I know Vouch works and I’d rather stick with what I know than try to figure out Authelia which has no documented way of integrating Azure AD. Could you tell me what the equivalent TLS_CERT and TLS_KEY files are and how I could go about restricting Caddy to only use LetsEncrypt? My idea is to use Caddy strictly for obtaining and updating certs and then proxying all requests to Nginx to handle the auth.

francislavoie · July 21, 2023, 2:17am

Okay, then use OpenID Connect - Configuration - Authelia, I guess.

I can’t help you there. You should reach out to Vouch devs to see if/how it could integrate with Caddy’s forward_auth (which works essentially the same as Traefik’s forward auth or Nginx’s auth_request).

The default is to use Let’s Encrypt and/or ZeroSSL. See Automatic HTTPS — Caddy Documentation

The tls directive allows you to specify a cert and key file if you don’t want to automate.

reubano · July 21, 2023, 5:47pm

Okay, I think I’ve figured it out. The tls certs are at the following locations:

/data/caddy/certificates/acme-v02.api.letsencrypt.org-directory/{domain}/{domain}.crt
/data/caddy/certificates/acme-v02.api.letsencrypt.org-directory/{domain}/{domain}.key

And to force Caddy to only use LetsEncrypt it seems like I need this global option:

{
    cert_issuer letsencrypt
}

Is this correct? Of course, I’d prefer to use Authelia’s OpenID Connect, but Azure AD integration is undocumented. And since I am using Caddy for automatic certs, I can’t use the tls directive.

matt · July 21, 2023, 6:03pm

This is not correct. I’m not sure why that would be necessary. In fact you should be getting an error…

Error: parsing caddyfile tokens for 'cert_issuer': Caddyfile:2 - Error during parsing: getting module named 'tls.issuance.letsencrypt': module not registered: tls.issuance.letsencrypt, import chain: ['']

I would make sure you are running the config you think you are

reubano · July 21, 2023, 9:07pm

@matt I hadn’t yet deployed the config with the new global option so I was just assuming that it was correct. What is the proper way to force certs to only be from letsencrypt? I need to do this so that I can have a persistent location of the tls certs which I feed to the vouch config.

matt · July 21, 2023, 9:09pm

The easiest way, with the Caddyfile, is to use the acme_ca global option and set it to Let’s Encrypt’s directory endpoint:

reubano · July 21, 2023, 9:14pm

So this?

{
    acme_ca https://acme-v02.api.letsencrypt.org/directory
}

francislavoie · July 21, 2023, 11:56pm

I think cert_issuer acme would also work.

See tls (Caddyfile directive) — Caddy Documentation, the issuers that exist are acme (which is Let’s Encrypt by default, but is generic so it can support any ACME issuer), zerossl (which is the same implementation as acme but with different defaults to connect to ZeroSSL) and internal (which sets up a local CA and uses that to sign certs instead of using a public ACME issuer).

Either way, I recommend not doing that. You’re throwing away a key piece of what makes Caddy so reliable. Issuer fallback is important to ensure your sites don’t go down if Let’s Encrypt has a major outage etc.

Like I said earlier, you’re overcomplicating things, copying certs and keys adds a lot of complexity. Just have Caddy be the thing in front, and have it proxy to whatever services.

Also like I said, I suggest you reach out to the Vouch devs to see if it can integrate with Caddy’s forward_auth. My hunch is that yes it would work, because it was designed to pair with Nginx’s auth_request, which is very similar.

reubano · July 22, 2023, 1:30am

Cool, Thanks!

Yes, I understand I’m removing issuer reliability. Right now I’m just trying to get something that works since my new environment (Azure Container Instance) makes it difficult to incorporate Let’s Encrypt with nginx as I was doing before.

Caddy is still in the front on 80/433, and it proxies to nginx on 8080. ngnix then routes auth via vouch proxy on 9091 and then proxies to prometheus on 9090. Believe me, if I could do everything in Caddy and remove the nginx dependency I would. However, (other than Caddy Security) I haven’t seen any documented way of getting Azure AD OAuth with Caddy.

P.S. the certs aren’t being copied. Vouch just needs the path to their locations.
P.P.S. I’m not opposed to Caddy Security either if you think that would be better. Installation just didn’t look straightforward.

Yes, that would be ideal. Caddy/etc. integration is on vouch’s radar but not yet implemented/documented.

francislavoie · July 22, 2023, 1:34am

Okay, then you don’t need to extract the cert from Caddy. You’re proxying over HTTP to nginx, and nginx proxies over HTTP to your apps. Caddy terminates TLS.

reubano · August 4, 2023, 3:54pm

In case in anyone else is in the same boat, here’s what I came up with. It uses Caddy for SSL certs only and then proxies to Nginx for auth (via Vouch Proxy) and routing to Prometheus. It also restricts Caddy to only use lets encrypt so that we can pass the tls certs to vouch.

Unfortunately, the vouch config requires it. Also, I had to change the nginx proxy_pass to vouch from http to https in order to make it all work.

system · September 3, 2023, 3:55pm

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.