Caddy v2 - Reusable snippets

smebberson · January 8, 2020, 12:31am

1. My Caddy version (`caddy -version`):

v2.0.0-beta12 h1:LZnXOGDr1SbeJNyln8Xc/hXjWCa/a9qFpCbWt2iwJPw=

2. How I run Caddy:

a. System environment:

MacOS 10.15.1

b. Command:

KUEUI_BASIC_AUTH_USERNAME=$(echo "user" | base64) KUEUI_BASIC_AUTH_PASSWORD=$(echo "pass" | base64) TARGET_ENV=local ./caddy adapt --config ./static/root/etc/Caddyfile --pretty

c. Service/unit/compose file:

N/A

d. My complete Caddyfile:

I’ve just started converting to Caddy v2 format so it’s more v1 than v2 at the moment.

# TLS for local.
(tls-local) {
    tls {env.ACME_CA_EMAIL} {
        # ca https://acme-staging-v02.api.letsencrypt.org/directory
        dns route53
    }
}

# TLS for everything else.
(tls-majority) {

    tls {env.ACME_CA_EMAIL} {
        dns route53
    }

    header / Strict-Transport-Security max-age=31536000

}

(tls-beta) {
    import tls-majority
}

(tls-production) {
    import tls-majority
}

{env.APP_DOMAIN} {

    log stdout {
        except /_status/ping
    }

    log stderr {
        except /_status/ping
    }

    gzip

    import tls-{env.TARGET_ENV}

    root /www

    header / {
        Content-Security-Policy "frame-ancestors 'none'"
        X-Content-Type-Options "nosniff"
        X-Frame-Options "DENY"
        X-XSS-Protection "1; mode=block"
    }

    proxy / app:4000 {
        except /css /fonts /images /js /favicon.ico /robots.txt /default.html
        transparent
    }

}

import /etc/caddy/*.conf

3. The problem I’m having:

It would seem as though Caddy v2 doesn’t yet support imports?

4. Error messages and/or full log output:

adapt: Caddyfile:40 - Error during parsing: File to import not found: tls-{env.TARGET_ENV}

5. What I already tried:

I’ve confirmed that the environment variable is available to Caddy via KUEUI_BASIC_AUTH_USERNAME=$(echo "user" | base64) KUEUI_BASIC_AUTH_PASSWORD=$(echo "pass" | base64) TARGET_ENV=local ./caddy environ | grep TARGET_ENV and the output is TARGET_ENV=local as expected.

6. Links to relevant resources:

https://caddyserver.com/v1/docs/caddyfile#snippets

I can’t find any documentation about the import directive at Home · caddyserver/caddy Wiki · GitHub or JSON Config Structure - Caddy Documentation.

I did note that reusable snippets was mentioned here… V2 - adapt `unrecognized directive: proxy` - moving from Caddy v1

matt · January 8, 2020, 12:39am

Hmm, I don’t know that you will be able to use environment variables for importing in v2. We’re trying to take env variable logic out of the Caddyfile parser, because they are evaluated when modules are loaded, deeper in the system.

Does the import work if you hard-code it?

smebberson · January 8, 2020, 12:56am

So this fails with adapt: Caddyfile:22 - Error during parsing: File to import not found: tls-{env.TARGET_ENV}.

(tls-local) {

    tls {
        protocols tls1.3 tls1.3
    }

}

example.com {

    matcher all {
        path /
    }

    matcher all-excluding-local-files {
        path /
        not {
            path /css /fonts /images /js /favicon.ico /robots.txt /default.html
        }
    }

    import tls-{env.TARGET_ENV}

    encode match:all {
        gzip
    }

    headers match:all {
        Content-Security-Policy "frame-ancestors 'none'"
        Strict-Transport-Security "max-age=31536000"
        X-Content-Type-Options "nosniff"
        X-Frame-Options "DENY"
        X-XSS-Protection "1; mode=block"
    }

    reverse_proxy match:all-excluding-local-files app:4000 {
        header_up Host {host}
        header_up X-Real-IP {remote}
        header_up X-Forwarded-For {remote}
        header_up X-Forwarded-Port {server-port}
        header_up X-Forwarded-Proto {scheme}
    }

    file_server * {
        root /www
    }

}

But this works…

(tls-local) {

    tls {
        protocols tls1.3 tls1.3
    }

}

example.com {

    matcher all {
        path /
    }

    matcher all-excluding-local-files {
        path /
        not {
            path /css /fonts /images /js /favicon.ico /robots.txt /default.html
        }
    }

    import tls-local

    encode match:all {
        gzip
    }

    headers match:all {
        Content-Security-Policy "frame-ancestors 'none'"
        Strict-Transport-Security "max-age=31536000"
        X-Content-Type-Options "nosniff"
        X-Frame-Options "DENY"
        X-XSS-Protection "1; mode=block"
    }

    reverse_proxy match:all-excluding-local-files app:4000 {
        header_up Host {host}
        header_up X-Real-IP {remote}
        header_up X-Forwarded-For {remote}
        header_up X-Forwarded-Port {server-port}
        header_up X-Forwarded-Proto {scheme}
    }

    file_server * {
        root /www
    }

}

So I guess this answer is yes, it works when hard-coded. Sounds like it’s going to stay this way too?

smebberson · January 8, 2020, 1:01am

Also, I just wanted to note here (I assume you’re already aware though) that:

(tls-local) {

    tls {
        protocols tls1.3 tls1.3
    }

}

example.com {

    import tls-local

}

works, but fails when you update the tls directive to:

tls {
    dns route53
    protocols tls1.3 tls1.3
}

matt · January 8, 2020, 1:03am

Not necessarily, since we’re still in beta it’s easier to take feedback like this. We might be able to make it work, for imports specifically: https://github.com/caddyserver/caddy/pull/2963#issuecomment-571843648 – but not making any promises right now.

I think that’s expected, yes; the dns subdirective isn’t implemented, so I wouldn’t expect it to work (yet).

I think you can remove the extra tls1.3 too, unless I’m forgetting something…

smebberson · January 8, 2020, 1:10am

Okay, no worries.

While we’re on the topic of environment variables logic in the Caddfile parser, I thought I should bring this to your attention.

This works:

example.com {

    file_server * {
        root /www
    }

}

But the following fails with adapt: server block 0, key 0 ({env.APP_DOMAIN}): determining listener address: parsing key: parse //{env.APP_DOMAIN}: invalid character "{" in host name when being executed with APP_DOMAIN=example.com ./caddy adapt --config ./static/root/etc/Caddyfile --pretty:

{env.APP_DOMAIN} {

    file_server * {
        root /www
    }

}

matt · January 8, 2020, 5:12am

Ah, fun… guess we’ll need to find a way to hopefully benefit from the Go standard library’s URL parser while allowing placeholders. Okay, added to my list…

matt · January 9, 2020, 3:35am

If you’ve been following along, you’ll notice that we’re making it so that you can use {$ENV_VAR} syntax in the Caddyfile to be replaced before parsing even begins. That should probably meet your needs, I think.

This implies that {env.VAR}-style placeholders are still deferred until runtime to be evaluated.

So, this:

I am not sure we can get this to work. The Caddyfile needs to know something about the site: the port, the domain name, a path, something… but since {env.APP_DOMAIN} would only be evaluated at runtime (after the Caddyfile is adapted into JSON), the Caddyfile adapter won’t know how to structure the JSON, because it’s not sure what to make of the site address (it remains a placeholder!).

Now, in your case, you can just use {$APP_DOMAIN} and that will be replaced by the Caddyfile before tokenizing and parsing even begins, and so that should work, because in theory Caddy would have a value to work with. But using {env.APP_DOMAIN} would leave it clueless, so… I’m not sure we can get that to work.

Is that OK with you? (If not, do you have an alternative idea/solution?)

smebberson · January 9, 2020, 3:58am

@matt, yeah, I’ve been following along!

I can see how the following will never work:

{env.APP_DOMAIN} {
    file_server ...
}

It will always generate an error similar to what I demonstrated as you mention, it needs to know something about site at adapt time.

I think there are two solutions. The first being as you mentioned, making sure APP_DOMAIN exists at adapt-time, in this scenario, the following should work:

{$APP_DOMAIN} {
    file_server ...
}

The other is the following:

https://{env.APP_DOMAIN} {
    file_server ...
}

I think that would work based on what I’ve seen in the PR on GitHub? It will give the Caddyfile adapter enough information to proceed:

port: 443 (implied by https).
domain: {env.APP_DOMAIN} that will be passed through into the JSON and then evaluated at runtime.

Is that correct?

matt · January 9, 2020, 4:02am

To be clear, there already will be a solution, if you don’t mind the substitution happening at the time the Caddyfile is adapted: use {$APP_DOMAIN} instead of {env.APP_DOMAIN}.

However, if the runtime environment will be different and thus you have to use {env.APP_DOMAIN}, then yeah that won’t work.

However, if you can at least give it a hint:

https://{env.APP_DOMAIN} {
 ...
}

That might be enough. Or {env.APP_DOMAIN}:1234 or example.com:{env.PORT} – I think these should all be able to work.

Given the alternative is landing probably tomorrow, do you really need this to work? I can add it to my list but the parsing logic to support this probably wouldn’t be simple, might take some time to figure out. (I’d rather not do it right now, unless there’s an actual and urgent need for it, if that’s alright.) Will the {$APP_DOMAIN} solution work for you?

smebberson · January 9, 2020, 4:05am

@matt I can use {$APP_DOMAIN} for now, for sure That will at least allow me to use v2 with Caddyfile.

Ultimately, at some point before v2, I think it would be awesome for Caddy if you did happen to include support for:

https://{env.APP_DOMAIN} {
 ...
}

matt · January 9, 2020, 4:10am

Okay. Good to know.

Does that mean you will eventually be adapting the Caddyfile in a different environment from the server?

smebberson · January 9, 2020, 4:19am

Yes. Please refer to my use case as outlined here.

At start time, a Caddyfile in the static container will configure Caddy. At runtime (sometime later), a dynamically generated Caddyfile compiled in the supervisor container will update Caddy’s configuration in static via the Configuration API and the POST /load endpoint.

I should say, this will all be running in Kubernetes…

matt · January 9, 2020, 7:36pm

Okay, implemented in caddyfile: Less strict URL parsing; allows placeholders · caddyserver/caddy@8aef859 · GitHub.

The placeholder can only represent a single component of the possible address, however: scheme, host, port, or path.

system · April 8, 2020, 7:36pm

This topic was automatically closed 90 days after the last reply. New replies are no longer allowed.