Two Caddy instances sharing ACME certs, deactivate provisioning on one instance

1. Caddy version (caddy version):

Latest docker version (while testing/experimenting).

2. How I run Caddy:

a. System environment:

Fedora 35, podman 3.4.4

b. Command:

letsencrypt-instance:

podman run -d -p [eth0-ip]:80:80 -p [eth0-ip]:443:443 --name caddy_outside -v $PWD/caddy_outside/Caddyfile:/etc/caddy/Caddyfile:U,Z -v $PWD/caddy_data:/data:U,z caddy

wireguard-instance:

podman run -d -p [wg-ip]:80:80 -p [wg-ip]:443:443 --name caddy_inside -v $PWD/caddy_inside/Caddyfile:/etc/caddy/Caddyfile:U,Z -v $PWD/caddy_data:/data:U,z caddy

c. Service/unit/compose file:

None

d. My complete Caddyfile or JSON config:

letsencrypt-instance:

{
    admin off
    email my-email@example.com
    acme_ca https://acme-staging-v02.api.letsencrypt.org/directory
    log
    debug
}

example.com {
    respond 403
}

www.example.com {
    respond 403
}

cloud.example.com {
    respond 403
}

www.example.com {
    respond 403
}

wireguard-instance:

{
    admin off
    email my-email@example.com 
    acme_ca https://acme-staging-v02.api.letsencrypt.org/directory
    auto_https off
    log
    debug
}

(httpredirect) {
        @http {
                protocol http
        }
        redir @http https://{host}{uri}
}


example.com  {
    import httpredirect
    respond "Inside"
}

www.example.com  {
    import httpredirect
    respond "www.Inside"
}

cloud.example.com  {
    import httpredirect
    respond "Inside.cloud"
}

www.cloud.example.com  {
    import httpredirect
    respond "www.Inside.cloud"
}

3. The problem I’m having:

I want to use Caddy as a reverse proxy with automatic certificate provisioning with Let’s Encrypt. I’m using two instances of Caddy (containers), one for provisioning and managing certificates, one to do the actual reverse proxying on the wireguard interface. I’m using the wireguard interface so I don’t have to expose any service directly to the internet.
With auto-https enabled on both instances, it seems to work but can run into issues with the wireguard-instance trying to provision certificates (which obviously doesn’t work). My idea was to disable auto-https on the wireguard-instance but I can’t figure out how to use the certificates provisioned by the letsencrypt-instance with the wireguard-instance.

My ideal solution would be to just deactivate the auto-provisioning on the wireguard-instance and leave the rest of the auto-https functionality active. I couldn’t find a way to do that with the Caddyfile config. A skip-provisioning option would be great as a parameter for auto_https.

Any help or hints how to get the this working are greatly appreciated.

4. Error messages and/or full log output:

{"level":"info","ts":1644911332.881876,"msg":"using provided configuration","config_file":"/etc/caddy/Caddyfile","config_adapter":"caddyfile"}
{"level":"warn","ts":1644911332.8861623,"msg":"input is not formatted with 'caddy fmt'","adapter":"caddyfile","file":"/etc/caddy/Caddyfile","line":2}
{"level":"warn","ts":1644911332.8870971,"logger":"admin","msg":"admin endpoint disabled"}
{"level":"info","ts":1644911332.8879244,"logger":"tls.cache.maintenance","msg":"started background certificate maintenance","cache":"0xc00045e000"}
{"level":"info","ts":1644911332.8884325,"logger":"tls","msg":"cleaning storage unit","description":"FileStorage:/data/caddy"}
{"level":"debug","ts":1644911332.8887951,"logger":"http","msg":"starting server loop","address":"[::]:443","http3":false,"tls":true}
{"level":"info","ts":1644911332.8891673,"msg":"autosaved config (load with --resume flag)","file":"/config/caddy/autosave.json"}
{"level":"info","ts":1644911332.8893547,"msg":"serving initial configuration"}
{"level":"info","ts":1644911332.890386,"logger":"tls","msg":"finished cleaning storage units"}
{"level":"debug","ts":1644911351.941252,"logger":"tls.handshake","msg":"no matching certificates and no custom selection logic","identifier":"www.example.com"}
{"level":"debug","ts":1644911351.9413471,"logger":"tls.handshake","msg":"no matching certificates and no custom selection logic","identifier":"*.example.com"}
{"level":"debug","ts":1644911351.9413526,"logger":"tls.handshake","msg":"no matching certificates and no custom selection logic","identifier":"*.*.com"}
{"level":"debug","ts":1644911351.9413567,"logger":"tls.handshake","msg":"no matching certificates and no custom selection logic","identifier":"*.*.*"}
{"level":"debug","ts":1644911351.941366,"logger":"tls.handshake","msg":"no certificate matching TLS ClientHello","server_name":"www.example.com","remote":"[IP]:52130","identifier":"www.example.com","cipher_suites":[4865,4867,4866,49195,49199,52393,52392,49196,49200,49162,49161,49171,49172,156,157,47,53],"cert_cache_fill":0,"load_if_necessary":true,"obtain_if_necessary":true,"on_demand":false}
{"level":"debug","ts":1644911351.9414945,"logger":"http.stdlib","msg":"http: TLS handshake error from [ip]]:52130: no certificate available for 'www.example.com'"}

5. What I already tried:

auto_https on both instances lead to issues with the wireguard-instance trying to provision certificates.

load /data/caddy in the wiregaurd-instance didn’t find any certificates.

I think I found a possible solution/work-around:

wireguard-instance Caddyfile:

{
    admin off
    email my-email@example.com
    acme_ca https://acme-staging-v02.api.letsencrypt.org/directory
    auto_https off
    log
    debug
}

(http_redirect) {
        @http {
                protocol http
        }
        redir @http https://{host}{uri}
}

(tls_config) {
    tls /data/caddy/certificates/acme-staging-v02.api.letsencrypt.org-directory/{args.0}/{args.0}.crt /data/caddy/certificates/acme-staging-v02.api.letsencrypt.org-directory/{args.0}/{args.0}.key
}

example.com {
    import http_redirect
    import tls_config example.com
    respond "Inside"
}

www.example.com {
    import http_redirect
    import tls_config www.example.com
    respond "www.Inside"
}

I’d prefer a skip_provisioning option. Being able to specify skip_certificates in the Caddyfile could also help, but I’m not sure whether this works as I think it does.

Does this keep working when the letsencrypt-instance provisions new certificates or does the config need to be reloaded? I think a nightly restart of both containers could still make it work.

Config will need a reload. Caddy doesn’t watch for file changes to certificates (but maybe it should, like daily or something)

Also, keep in mind if Caddy issues a cert for ZeroSSL, this path will be incorrect. Caddy will fallback to ZeroSSL if there’s a problem with Let’s Encrypt for whatever reason, and that cert will have a different path on disk.

But generally, I don’t have a great answer for you, because this is an unusual requirement. Could you open an issue on Github for your suggestion? I think it would probably also require changes in Certmagic (the underlying lib that performs cert maintenance).

1 Like

Thanks for the heads up about ZeroSSL, I didn’t think of that.

I think I found a more ergonomic solution only using one container. Is there anything I’m missing with this config or can I run this without issues? I think Caddy should not respond on the alt_http_port and the alt_tlsalpn_port if it isn’t currently provisioning a new certificate, is that correct?

If this is a somewhat reasonable solution, I don’t think any changes/suggestions would be needed.

Single-instance Caddyfile:

{
    admin off
    log
    debug
}

(tls_config) {
        tls {
            issuer acme  {
                dir https://acme-staging-v02.api.letsencrypt.org/directory
                email my-email@example.com
                alt_http_port 81
                alt_tlsalpn_port 444
            }
        }
}

example.com {
    import tls_config
    respond "Inside"
}

www.example.com {
    import tls_config
    redir https://example.com{uri}
    respond "www.Inside"
}

Podman command:

podman run -d -p [wg-ip]:80:80 -p [wg-ip]:443:443 -p [eth0-ip]:80:81 -p [eth0-ip]:443:444 --name caddy_single -v $PWD/caddy_single/Caddyfile:/etc/caddy/Caddyfile:U,Z -v $PWD/caddy_single_data:/data:U,z caddy
1 Like

Yeah, that does seem much simpler :+1:

Great, thanks for your help!

1 Like

This topic was automatically closed after 30 days. New replies are no longer allowed.