What's a equivalent for nginx map module

[al-caddyserver (Aleks)]

we use the Module ngx_http_map_module for some dynamic directory decisions.

# configuration file /home/nginx/server/conf/global_map_data_storage.conf:
map $camid_date $storageid {
  default 3;

  include /home/nginx/server/conf/global_map_data_storage_data.conf;
}

# configuration file /home/nginx/server/conf/global_map_data_storage_data.conf:
'7/2010/08/10' 10;
'7/2010/08/11' 10;
'7/2010/08/12' 10;
'7/2010/08/13' 10;
'7/2010/08/14' 10;
...
536491 more lines
11M file size

# configuration file /home/nginx/nginx.conf

    location ~ /cams/(\d+)/\w+\.(jpg|xml)$ {
      set  $camid $1;
      root /web/data/data$storageid;
   }

What’s the best equivalent solution in caddy v2 for that setup?

[matt (Matt Holt)]

I’m hoping to work on this today or tomorrow.

Question: Do you like how your current mapping configuration works, or would you like anything changed about it? i.e. if you could have it ANY WAY YOU WANT with your web server, how would it be? Just like it is now?

One thing to consider: do you pre-generate this config and then reload it when any of those half-million-lines change, as a stopgap for a static config? (In other words, are you using this huge static config in place of a dynamic config?) Would you rather this mapping be dynamic? If so, how would you like it to work?

But if all things are to stay the same, this will be very straightforward and I can have this done by the end of the week.

[al-caddyserver (Aleks)]

@matt

I’m hoping to work on this today or tomorrow.

Oh wow I haven’t expect that.

@matt

Question: Do you like how your current mapping configuration works, or would you like anything changed about it? i.e. if you could have it ANY WAY YOU WANT with your web server, how would it be? Just like it is now?

One thing to consider: do you pre-generate this config and then reload it when any of those half-million-lines change, as a stopgap for a static config? (In other words, are you using this huge static config in place of a dynamic config?) Would you rather this mapping be dynamic? If so, how would you like it to work?

But if all things are to stay the same, this will be very straightforward and I can have this done by the end of the week.

Well as the map implementation looks quite fast it would be okay for me to have it as it is in nginx. We pre-generate the map file and run a nginx reload as there is no service interruption at reload time.

The code for the map module is here

When I take a look into the v2 concept it would be nice to have a key value “database” which can handle the map dynamically. There are several possible options for such a DB GitHub - avelino/awesome-go: A curated list of awesome Go frameworks, libraries and software

[matt (Matt Holt)]

Cool, I’m working on this today. I’m going to first implement the map module with similar functionality so that you can keep your existing processes and so other users can migrate more easily. But I’m still interested in doing the mapping “better” in v2, since making your web server config a read-only database seems like a bit of a hack around lack of dynamic logic.

@al-caddyserver

When I take a look into the v2 concept it would be nice to have a key value “database” which can handle the map dynamically.

Yes, I agree – but what specifically would work best with your setup?

[matt (Matt Holt)]

@al-caddyserver What do you think of this structure?

{
	"handler": "switch",
	"on": "{http.vars.foobar}",
	"set": ["foo", "bar"],
	"map_vars": {
		"if this value": ["then foo is this", "and bar is this"]
	},
	"map_routes": {
		"if this value": [
			... route ...
		]
	}
}

It allows you to set variables or execute routes based on the value of a certain input.

Still, I’d like to know if there’s a particular dynamic solution to this that would work really nicely with your setup.

[al-caddyserver (Aleks)]

@matt

@al-caddyserver What do you think of this structure?

{
	"handler": "switch",
	"on": "{http.vars.foobar}",
	"set": ["foo", "bar"],
	"map_vars": {
		"if this value": ["then foo is this", "and bar is this"]
	},
	"map_routes": {
		"if this value": [
			... route ...
		]
	}
}

It allows you to set variables or execute routes based on the value of a certain input.

Looks good but the map is more generic as it sets some variables which makes something like below possible

map $request_method $purge_method {
  PURGE   $purge_allowed;
  default 0;
}
...
fastcgi_cache_purge $purge_method;

or set expire based on content type. This can be handled with https://caddyserver.com/docs/http.expires AFAIK

map $sent_http_content_type $expires {
  default         off;
  text/html       +1h;
  image/jpeg       5m;
}
expires $expires

In summary the map directive can be used to check some http-header, methodes and a lot more and set the variable to the defined value.

@matt

Still, I’d like to know if there’s a particular dynamic solution to this that would work really nicely with your setup.

In the original use case we change the root line dynamically like this root /web/data/data$storageid; . Is this possible in Caddyserver v1 or v2 to change the root dynamically as I haven’t seen it in the doc if it’s possible or not https://caddyserver.com/docs/root ?
The nice part with the map and nginx is that the variables and directives are evaluated at request time is this also possible in Caddyserver v1 or v2?

[matt (Matt Holt)]

@al-caddyserver

Looks good but the map is more generic as it sets some variables

I’m not sure I follow as to why the map is more generic, what do you mean? The structure I proposed also sets variables which can be used in other places. (That’s why I intended with "map_vars" - did you see that?)

@al-caddyserver

In summary the map directive can be used to check some http-header, methodes and a lot more and set the variable to the defined value.

Yeah, so can this.

@al-caddyserver

In the original use case we change the root line dynamically like this root /web/data/data$storageid; . Is this possible in Caddyserver v1 or v2 to change the root dynamically

Yes, in Caddy 2 the root is dynamic: Home · caddyserver/caddy Wiki · GitHub - it doesn’t say so explicitly right there, but you can customize it per-request with the matcher, and use placeholders (variables) in the root path itself.

@al-caddyserver

The nice part with the map and nginx is that the variables and directives are evaluated at request time is this also possible in Caddyserver v1 or v2?

Yes, that is the point.

[al-caddyserver (Aleks)]

@matt

@al-caddyserver

Looks good but the map is more generic as it sets some variables

I’m not sure I follow as to why the map is more generic, what do you mean? The structure I proposed also sets variables which can be used in other places. (That’s why I intended with "map_vars" - did you see that?)

Well yes but it looks like I haven’t understood it. Let’s try a example with my original post.

The variable camid_date is set via the regex on the location line. I don’t know how to do this in caddy for now.

Nginx:

location ~ \/cams\/(\d+\/\d+\/\d+\/\d+)\/\d+-\d+-\d+_\w+\.jpg$ {
  set $camid_date $1;
  root /web/data/data$storageid;
}

{
	"handler": "switch",
	"on": "{http.vars.camid_date}",
	"set": ["storageid"],
	"map_vars": {
		'7/2010/08/10': ["10"]
        '7/2010/08/11': ["10"]
        '7/2010/08/12': ["10"]
        ...
	},
    root [<matcher>]? /web/data/data{storageid}
}

Is this what you have in mind?
What’s the matcher in this case?

[matt (Matt Holt)]

Ah, perhaps it would help to read up on how Caddy 2 is configured: spend a few minutes on the readme here, then dive into the docs where you want to learn more: https://github.com/caddyserver/caddy/tree/v2#caddy-2-development-branch

But, YES, that example you posted is pretty close to what I had in mind.

As for a Caddyfile syntax, it could be similar to nginx’s, which isn’t too bad really.

[al-caddyserver (Aleks)]

Okay thanks for your patience. I have now created the config below. Is this something it can work with caddy v2? I have a json parser error but can’t see where the problem is.

{"apps":{"http":{"servers":{"myserver":{"listen":[":443"],
  "routes":[{
    "match":[
      {"host":["example.com"]},
      { "path_regexp": {
	  "name": "camid_date",
	  "pattern": "^/(?P<CaDa>lala)>"
        }
      }
],
"handle":[
{
    "handler": "switch",
	"on": "{http.matchers.camid_date.CaDa}",
	"set": ["storageid"],
	"map_vars": {
          "7/2010/08/10": ["10"],
          "7/2010/08/11": ["10"],
          "7/2010/08/12": ["10"]
	},
  {
  "handler":"file_server",
  "root": "/web/data/data{storageid}"
  }
}]}]}}}}}

I tried to build caddy v2 today and get the following error, sadly therfore I can’t test the config. It would be nice to have a working example config somewhere

build github.com/caddyserver/caddy/v2/cmd/caddy: cannot load github.com/caddyserver/caddy/caddyconfig: cannot find module providing package github.com/caddyserver/caddy/caddyconfig

[matt (Matt Holt)]

Awesome, thanks for giving it a go.

We’re still in the very early stages of documentation (obviously, as Caddy 2 is still pre-beta!) but there are multiple pages of docs for v2 on the wiki: Home · caddyserver/caddy Wiki · GitHub

@al-caddyserver

It would be nice to have a working example config somewhere

One of the docs pages shows you how to build a HTTP server config from scratch: Home · caddyserver/caddy Wiki · GitHub - you can use that as a starting point.

You can then validate your JSON with a tool like https://jsonlint.com/.

I fixed a couple errors (extra closing ‘}’ and using a ‘]’ where ‘}’ was needed) for you:

{
	"apps": {
		"http": {
			"servers": {
				"myserver": {
					"listen": [
						":443"
					],
					"routes": [
						{
							"match": [
								{
									"host": [
										"example.com"
									]
								},
								{
									"path_regexp": {
										"name": "camid_date",
										"pattern": "^/(?P<CaDa>lala)>"
									}
								}
							],
							"handle": [
								{
									"handler": "switch",
									"on": "{http.matchers.camid_date.CaDa}",
									"set": [
										"storageid"
									],
									"map_vars": {
										"7/2010/08/10": [
											"10"
										],
										"7/2010/08/11": [
											"10"
										],
										"7/2010/08/12": [
											"10"
										]
									}
								},
								{
									"handler": "file_server",
									"root": "/web/data/data{storageid}"
								}
							]
						}
					]
				}
			}
		}
	}
}

That’s at least valid JSON.

An alternative way to do this is to use the Caddyfile – you don’t necessarily have to use the native JSON config directly.

[al-caddyserver (Aleks)]

Cool, thanks for the valid json
Are you now going to implement this?

@matt

You can then validate your JSON with a tool like https://jsonlint.com/.

I use normally jq but when the json is not valid then jq strikes.

[matt (Matt Holt)]

@al-caddyserver

Are you now going to implement this?

Soon, yeah! It’s going to take a backseat to fastcgi at this point (which is my next big project starting monday), but I’ll get to work on it sooner or later, or when I have a few minutes here and there.

[al-caddyserver (Aleks)]

@matt

Soon, yeah! It’s going to take a backseat to fastcgi at this point (which is my next big project starting monday), but I’ll get to work on it sooner or later, or when I have a few minutes here and there.

Sounds great

As we talk about features is/will this use case be possible?
I mean the cascading the matches. Maybe you can show us how to translate the conf snipplets below to caddy config

The most commands are quite straight forward, IMHO.

The directive try_files is documented there
http://nginx.org/en/docs/http/ngx_http_core_module.html#try_files

The line include /home/nginx/server/conf/global_valid-referer.conf; uses the following directive.
http://nginx.org/en/docs/http/ngx_http_referer_module.html#valid_referers

add_header can be used with this Home · caddyserver/caddy Wiki · GitHub , right?

How about the include directive, is this possible/planned into v2?

        location /interfaces {
            include add_header-Access-Control-Allow-Origin.conf;
            include add_header-Access-Control-Allow-Headers-X-Requested-With.conf;

            try_files $uri /app.php$is_args$args;

            location ~ /interfaces/json/(\S+) {
                include add_header-Access-Control-Allow-Origin.conf;
                include add_header-Content-Type_application_json;
                try_files $uri /app.php$is_args$args;
            }
        }

  location ~ /cams/(126|714|715|720|721|722|1229)/ {
    location ~ recent {
      add_header Cache-Control "no-cache";
      location ~ /cams/(\d+)/\w+\.(jpg|xml)$ {
        set  $camid $1;
        include /home/nginx/server/conf/global_valid-referer.conf;
        root /web/data/data$recentstorageid;
      }
    }
    location ~ /cams/(\d+)/\w+\.(jpg|xml)$ {
      set  $camid $1;
      include /home/nginx/server/conf/global_valid-referer.conf;
      root /web/data/data$storageid;
      if ($request_method = OPTIONS ) {
        add_header Access-Control-Allow-Origin "https://admin.DOMAIN.COM";
        add_header Access-Control-Allow-Methods "GET, OPTIONS";
        add_header Access-Control-Allow-Headers 'Authorization,Keep-Alive,User-Agent,X-Requested-With,If-Modified-Since,Cache-Control,Content-Type,X-CSRF-Token';
        add_header Access-Control-Allow-Credentials "true";
        add_header Content-Length 0;
        add_header Content-Type text/plain;
        return 200;
      }
    }
  }

[matt (Matt Holt)]

@al-caddyserver

The directive try_files is documented there
Module ngx_http_core_module

Caddy 2 has this, but even better. Its basic equivalent has a Caddyfile counterpart: Home · caddyserver/caddy Wiki · GitHub

But the file matcher can match based on other properties as well (not just existence), using the JSON.

@al-caddyserver

The line include /home/nginx/server/conf/global_valid-referer.conf; uses the following directive.
Module ngx_http_referer_module

Can probably be done with regex in Caddy 2. If not, or if a more convenient config is needed, this is easy to add.

@al-caddyserver

add_header can be used with this Home · caddyserver/caddy Wiki · GitHub , right?

Yep!

@al-caddyserver

How about the include directive, is this possible/planned into v2?

See import: The Caddyfile — Caddy Documentation

[al-caddyserver (Aleks)]

@matt

Caddy 2 has this, but even better. Its basic equivalent has a Caddyfile counterpart: Home · caddyserver/caddy Wiki · GitHub

Thanks. Can I redirect to another section like in nginx?

  try_files $uri @php_location

@matt

Can probably be done with regex in Caddy 2. If not, or if a more convenient config is needed, this is easy to add.

Please can you show us a example.

Can I suggest to create a similar explaining page like the below for nginx also for caddy v2?

http://nginx.org/en/docs/http/request_processing.html

[matt (Matt Holt)]

@al-caddyserver

Thanks. Can I redirect to another section like in nginx?

Yes, with even more flexibility.

@al-caddyserver

Please can you show us a example.

I don’t have time right now, but maybe somebody in the community will whip one up for you!

@al-caddyserver

Can I suggest to create a similar explaining page like the below for nginx also for caddy v2?

Yes, when Caddy 2 is released we’ll have thorough docs and videos.

[al-caddyserver (Aleks)]

Any update on this topic?

[matt (Matt Holt)]

I’ve been working on the reverse proxy lately, but will revisit this soon I hope!

[al-caddyserver (Aleks)]

Have you found some times to work on this?