What's a equivalent for nginx map module

al-caddyserver · August 19, 2019, 12:29am

we use the http://nginx.org/en/docs/http/ngx_http_map_module.html for some dynamic directory decisions.

# configuration file /home/nginx/server/conf/global_map_data_storage.conf:
map $camid_date $storageid {
  default 3;

  include /home/nginx/server/conf/global_map_data_storage_data.conf;
}

# configuration file /home/nginx/server/conf/global_map_data_storage_data.conf:
'7/2010/08/10' 10;
'7/2010/08/11' 10;
'7/2010/08/12' 10;
'7/2010/08/13' 10;
'7/2010/08/14' 10;
...
536491 more lines
11M file size

# configuration file /home/nginx/nginx.conf

    location ~ /cams/(\d+)/\w+\.(jpg|xml)$ {
      set  $camid $1;
      root /web/data/data$storageid;
   }

What’s the best equivalent solution in caddy v2 for that setup?

matt · August 19, 2019, 8:38pm

I’m hoping to work on this today or tomorrow.

Question: Do you like how your current mapping configuration works, or would you like anything changed about it? i.e. if you could have it ANY WAY YOU WANT with your web server, how would it be? Just like it is now?

One thing to consider: do you pre-generate this config and then reload it when any of those half-million-lines change, as a stopgap for a static config? (In other words, are you using this huge static config in place of a dynamic config?) Would you rather this mapping be dynamic? If so, how would you like it to work?

But if all things are to stay the same, this will be very straightforward and I can have this done by the end of the week.

al-caddyserver · August 19, 2019, 10:34pm

Oh wow I haven’t expect that.

Well as the map implementation looks quite fast it would be okay for me to have it as it is in nginx. We pre-generate the map file and run a nginx reload as there is no service interruption at reload time.

The code for the map module is here

When I take a look into the v2 concept it would be nice to have a key value “database” which can handle the map dynamically. There are several possible options for such a DB https://github.com/avelino/awesome-go#database

matt · August 22, 2019, 10:00pm

Cool, I’m working on this today. I’m going to first implement the map module with similar functionality so that you can keep your existing processes and so other users can migrate more easily. But I’m still interested in doing the mapping “better” in v2, since making your web server config a read-only database seems like a bit of a hack around lack of dynamic logic.

Yes, I agree – but what specifically would work best with your setup?

matt · August 22, 2019, 11:20pm

@al-caddyserver What do you think of this structure?

{
	"handler": "switch",
	"on": "{http.vars.foobar}",
	"set": ["foo", "bar"],
	"map_vars": {
		"if this value": ["then foo is this", "and bar is this"]
	},
	"map_routes": {
		"if this value": [
			... route ...
		]
	}
}

It allows you to set variables or execute routes based on the value of a certain input.

Still, I’d like to know if there’s a particular dynamic solution to this that would work really nicely with your setup.

al-caddyserver · August 23, 2019, 12:37am

matt:

@al-caddyserver What do you think of this structure?
{
	"handler": "switch",
	"on": "{http.vars.foobar}",
	"set": ["foo", "bar"],
	"map_vars": {
		"if this value": ["then foo is this", "and bar is this"]
	},
	"map_routes": {
		"if this value": [
			... route ...
		]
	}
}
It allows you to set variables or execute routes based on the value of a certain input.

Looks good but the map is more generic as it sets some variables which makes something like below possible

map $request_method $purge_method {
  PURGE   $purge_allowed;
  default 0;
}
...
fastcgi_cache_purge $purge_method;

or set expire based on content type. This can be handled with https://caddyserver.com/docs/http.expires AFAIK

map $sent_http_content_type $expires {
  default         off;
  text/html       +1h;
  image/jpeg       5m;
}
expires $expires

In summary the map directive can be used to check some http-header, methodes and a lot more and set the variable to the defined value.

In the original use case we change the root line dynamically like this root /web/data/data$storageid; . Is this possible in Caddyserver v1 or v2 to change the root dynamically as I haven’t seen it in the doc if it’s possible or not https://caddyserver.com/docs/root ?
The nice part with the map and nginx is that the variables and directives are evaluated at request time is this also possible in Caddyserver v1 or v2?

matt · August 23, 2019, 12:54am

I’m not sure I follow as to why the map is more generic, what do you mean? The structure I proposed also sets variables which can be used in other places. (That’s why I intended with "map_vars" - did you see that?)

Yeah, so can this.

Yes, in Caddy 2 the root is dynamic: https://github.com/caddyserver/caddy/wiki/v2:-Documentation#root - it doesn’t say so explicitly right there, but you can customize it per-request with the matcher, and use placeholders (variables) in the root path itself.

Yes, that is the point.

al-caddyserver · August 23, 2019, 1:11am

Well yes but it looks like I haven’t understood it. Let’s try a example with my original post.

The variable camid_date is set via the regex on the location line. I don’t know how to do this in caddy for now.

Nginx:

location ~ \/cams\/(\d+\/\d+\/\d+\/\d+)\/\d+-\d+-\d+_\w+\.jpg$ {
  set $camid_date $1;
  root /web/data/data$storageid;
}

{
	"handler": "switch",
	"on": "{http.vars.camid_date}",
	"set": ["storageid"],
	"map_vars": {
		'7/2010/08/10': ["10"]
        '7/2010/08/11': ["10"]
        '7/2010/08/12': ["10"]
        ...
	},
    root [<matcher>]? /web/data/data{storageid}
}

Is this what you have in mind?
What’s the matcher in this case?

matt · August 23, 2019, 1:35am

Ah, perhaps it would help to read up on how Caddy 2 is configured: spend a few minutes on the readme here, then dive into the docs where you want to learn more: https://github.com/caddyserver/caddy/tree/v2#caddy-2-development-branch

But, YES, that example you posted is pretty close to what I had in mind.

As for a Caddyfile syntax, it could be similar to nginx’s, which isn’t too bad really.

al-caddyserver · August 23, 2019, 11:03am

Okay thanks for your patience. I have now created the config below. Is this something it can work with caddy v2? I have a json parser error but can’t see where the problem is.

{"apps":{"http":{"servers":{"myserver":{"listen":[":443"],
  "routes":[{
    "match":[
      {"host":["example.com"]},
      { "path_regexp": {
	  "name": "camid_date",
	  "pattern": "^/(?P<CaDa>lala)>"
        }
      }
],
"handle":[
{
    "handler": "switch",
	"on": "{http.matchers.camid_date.CaDa}",
	"set": ["storageid"],
	"map_vars": {
          "7/2010/08/10": ["10"],
          "7/2010/08/11": ["10"],
          "7/2010/08/12": ["10"]
	},
  {
  "handler":"file_server",
  "root": "/web/data/data{storageid}"
  }
}]}]}}}}}

I tried to build caddy v2 today and get the following error, sadly therfore I can’t test the config. It would be nice to have a working example config somewhere

build github.com/caddyserver/caddy/v2/cmd/caddy: cannot load github.com/caddyserver/caddy/caddyconfig: cannot find module providing package github.com/caddyserver/caddy/caddyconfig

matt · August 23, 2019, 5:24pm

Awesome, thanks for giving it a go.

We’re still in the very early stages of documentation (obviously, as Caddy 2 is still pre-beta!) but there are multiple pages of docs for v2 on the wiki: https://github.com/caddyserver/caddy/wiki#version-2

One of the docs pages shows you how to build a HTTP server config from scratch: https://github.com/caddyserver/caddy/wiki/v2:-Config-from-Scratch - you can use that as a starting point.

You can then validate your JSON with a tool like https://jsonlint.com/.

I fixed a couple errors (extra closing ‘}’ and using a ‘]’ where ‘}’ was needed) for you:

{
	"apps": {
		"http": {
			"servers": {
				"myserver": {
					"listen": [
						":443"
					],
					"routes": [
						{
							"match": [
								{
									"host": [
										"example.com"
									]
								},
								{
									"path_regexp": {
										"name": "camid_date",
										"pattern": "^/(?P<CaDa>lala)>"
									}
								}
							],
							"handle": [
								{
									"handler": "switch",
									"on": "{http.matchers.camid_date.CaDa}",
									"set": [
										"storageid"
									],
									"map_vars": {
										"7/2010/08/10": [
											"10"
										],
										"7/2010/08/11": [
											"10"
										],
										"7/2010/08/12": [
											"10"
										]
									}
								},
								{
									"handler": "file_server",
									"root": "/web/data/data{storageid}"
								}
							]
						}
					]
				}
			}
		}
	}
}

That’s at least valid JSON.

An alternative way to do this is to use the Caddyfile – you don’t necessarily have to use the native JSON config directly.

al-caddyserver · August 23, 2019, 5:42pm

Cool, thanks for the valid json
Are you now going to implement this?

I use normally jq but when the json is not valid then jq strikes.

matt · August 24, 2019, 3:18am

Soon, yeah! It’s going to take a backseat to fastcgi at this point (which is my next big project starting monday), but I’ll get to work on it sooner or later, or when I have a few minutes here and there.

al-caddyserver · August 24, 2019, 11:25am

Sounds great

As we talk about features is/will this use case be possible?
I mean the cascading the matches. Maybe you can show us how to translate the conf snipplets below to caddy config

The most commands are quite straight forward, IMHO.

The directive try_files is documented there
http://nginx.org/en/docs/http/ngx_http_core_module.html#try_files

The line include /home/nginx/server/conf/global_valid-referer.conf; uses the following directive.
http://nginx.org/en/docs/http/ngx_http_referer_module.html#valid_referers

add_header can be used with this https://github.com/caddyserver/caddy/wiki/v2:-Documentation#httphandlersheaders , right?

How about the include directive, is this possible/planned into v2?

        location /interfaces {
            include add_header-Access-Control-Allow-Origin.conf;
            include add_header-Access-Control-Allow-Headers-X-Requested-With.conf;

            try_files $uri /app.php$is_args$args;

            location ~ /interfaces/json/(\S+) {
                include add_header-Access-Control-Allow-Origin.conf;
                include add_header-Content-Type_application_json;
                try_files $uri /app.php$is_args$args;
            }
        }

  location ~ /cams/(126|714|715|720|721|722|1229)/ {
    location ~ recent {
      add_header Cache-Control "no-cache";
      location ~ /cams/(\d+)/\w+\.(jpg|xml)$ {
        set  $camid $1;
        include /home/nginx/server/conf/global_valid-referer.conf;
        root /web/data/data$recentstorageid;
      }
    }
    location ~ /cams/(\d+)/\w+\.(jpg|xml)$ {
      set  $camid $1;
      include /home/nginx/server/conf/global_valid-referer.conf;
      root /web/data/data$storageid;
      if ($request_method = OPTIONS ) {
        add_header Access-Control-Allow-Origin "https://admin.DOMAIN.COM";
        add_header Access-Control-Allow-Methods "GET, OPTIONS";
        add_header Access-Control-Allow-Headers 'Authorization,Keep-Alive,User-Agent,X-Requested-With,If-Modified-Since,Cache-Control,Content-Type,X-CSRF-Token';
        add_header Access-Control-Allow-Credentials "true";
        add_header Content-Length 0;
        add_header Content-Type text/plain;
        return 200;
      }
    }
  }

matt · August 24, 2019, 3:26pm

Caddy 2 has this, but even better. Its basic equivalent has a Caddyfile counterpart: https://github.com/caddyserver/caddy/wiki/v2:-Documentation#try_files

But the file matcher can match based on other properties as well (not just existence), using the JSON.

Can probably be done with regex in Caddy 2. If not, or if a more convenient config is needed, this is easy to add.

Yep!

See import: https://caddyserver.com/docs/caddyfile#import

al-caddyserver · August 24, 2019, 8:54pm

Thanks. Can I redirect to another section like in nginx?

  try_files $uri @php_location

Please can you show us a example.

Can I suggest to create a similar explaining page like the below for nginx also for caddy v2?

http://nginx.org/en/docs/http/request_processing.html

matt · August 25, 2019, 6:25am

Yes, with even more flexibility.

I don’t have time right now, but maybe somebody in the community will whip one up for you!

Yes, when Caddy 2 is released we’ll have thorough docs and videos.

al-caddyserver · September 7, 2019, 2:32pm

Any update on this topic?

matt · September 7, 2019, 2:58pm

I’ve been working on the reverse proxy lately, but will revisit this soon I hope!

al-caddyserver · October 16, 2019, 6:21pm

Have you found some times to work on this?