we use the Module ngx_http_map_module for some dynamic directory decisions.
# configuration file /home/nginx/server/conf/global_map_data_storage.conf:
map $camid_date $storageid {
default 3;
include /home/nginx/server/conf/global_map_data_storage_data.conf;
}
# configuration file /home/nginx/server/conf/global_map_data_storage_data.conf:
'7/2010/08/10' 10;
'7/2010/08/11' 10;
'7/2010/08/12' 10;
'7/2010/08/13' 10;
'7/2010/08/14' 10;
...
536491 more lines
11M file size
# configuration file /home/nginx/nginx.conf
location ~ /cams/(\d+)/\w+\.(jpg|xml)$ {
set $camid $1;
root /web/data/data$storageid;
}
Whatās the best equivalent solution in caddy v2 for that setup?
matt
August 19, 2019, 8:38pm
2
Iām hoping to work on this today or tomorrow.
Question: Do you like how your current mapping configuration works, or would you like anything changed about it? i.e. if you could have it ANY WAY YOU WANT with your web server, how would it be? Just like it is now?
One thing to consider: do you pre-generate this config and then reload it when any of those half-million-lines change, as a stopgap for a static config? (In other words, are you using this huge static config in place of a dynamic config?) Would you rather this mapping be dynamic? If so, how would you like it to work?
But if all things are to stay the same, this will be very straightforward and I can have this done by the end of the week.
Oh wow I havenāt expect that.
matt:
Question: Do you like how your current mapping configuration works, or would you like anything changed about it? i.e. if you could have it ANY WAY YOU WANT with your web server, how would it be? Just like it is now?
One thing to consider: do you pre-generate this config and then reload it when any of those half-million-lines change, as a stopgap for a static config? (In other words, are you using this huge static config in place of a dynamic config?) Would you rather this mapping be dynamic? If so, how would you like it to work?
But if all things are to stay the same, this will be very straightforward and I can have this done by the end of the week.
Well as the map implementation looks quite fast it would be okay for me to have it as it is in nginx. We pre-generate the map file and run a nginx reload as there is no service interruption at reload time.
The code for the map module is here
When I take a look into the v2 concept it would be nice to have a key value ādatabaseā which can handle the map dynamically. There are several possible options for such a DB GitHub - avelino/awesome-go: A curated list of awesome Go frameworks, libraries and software
matt
August 22, 2019, 10:00pm
4
Cool, Iām working on this today. Iām going to first implement the map module with similar functionality so that you can keep your existing processes and so other users can migrate more easily. But Iām still interested in doing the mapping ābetterā in v2, since making your web server config a read-only database seems like a bit of a hack around lack of dynamic logic.
Yes, I agree ā but what specifically would work best with your setup?
matt
August 22, 2019, 11:20pm
5
@al-caddyserver What do you think of this structure?
{
"handler": "switch",
"on": "{http.vars.foobar}",
"set": ["foo", "bar"],
"map_vars": {
"if this value": ["then foo is this", "and bar is this"]
},
"map_routes": {
"if this value": [
... route ...
]
}
}
It allows you to set variables or execute routes based on the value of a certain input.
Still, Iād like to know if thereās a particular dynamic solution to this that would work really nicely with your setup.
matt:
@al-caddyserver What do you think of this structure?
{
"handler": "switch",
"on": "{http.vars.foobar}",
"set": ["foo", "bar"],
"map_vars": {
"if this value": ["then foo is this", "and bar is this"]
},
"map_routes": {
"if this value": [
... route ...
]
}
}
It allows you to set variables or execute routes based on the value of a certain input.
Looks good but the map is more generic as it sets some variables which makes something like below possible
map $request_method $purge_method {
PURGE $purge_allowed;
default 0;
}
...
fastcgi_cache_purge $purge_method;
or set expire based on content type. This can be handled with https://caddyserver.com/docs/http.expires AFAIK
map $sent_http_content_type $expires {
default off;
text/html +1h;
image/jpeg 5m;
}
expires $expires
In summary the map directive can be used to check some http-header, methodes and a lot more and set the variable to the defined value.
In the original use case we change the root line dynamically like this root /web/data/data$storageid; . Is this possible in Caddyserver v1 or v2 to change the root dynamically as I havenāt seen it in the doc if itās possible or not https://caddyserver.com/docs/root ?
matt
August 23, 2019, 12:54am
7
Iām not sure I follow as to why the map is more generic, what do you mean? The structure I proposed also sets variables which can be used in other places. (Thatās why I intended with "map_vars" - did you see that?)
Yeah, so can this.
Yes, in Caddy 2 the root is dynamic: GitHub - caddyserver/caddy: Fast and extensible multi-platform HTTP/1-2-3 web server with automatic HTTPS - it doesnāt say so explicitly right there, but you can customize it per-request with the matcher, and use placeholders (variables) in the root path itself.
Yes, that is the point.
Well yes but it looks like I havenāt understood it. Letās try a example with my original post.
The variable camid_date is set via the regex on the location line. I donāt know how to do this in caddy for now.
Nginx:
location ~ \/cams\/(\d+\/\d+\/\d+\/\d+)\/\d+-\d+-\d+_\w+\.jpg$ {
set $camid_date $1;
root /web/data/data$storageid;
}
{
"handler": "switch",
"on": "{http.vars.camid_date}",
"set": ["storageid"],
"map_vars": {
'7/2010/08/10': ["10"]
'7/2010/08/11': ["10"]
'7/2010/08/12': ["10"]
...
},
root [<matcher>]? /web/data/data{storageid}
}
Is this what you have in mind?
matt
August 23, 2019, 1:35am
9
Ah, perhaps it would help to read up on how Caddy 2 is configured: spend a few minutes on the readme here, then dive into the docs where you want to learn more: https://github.com/caddyserver/caddy/tree/v2#caddy-2-development-branch
But, YES, that example you posted is pretty close to what I had in mind.
As for a Caddyfile syntax, it could be similar to nginxās, which isnāt too bad really.
Okay thanks for your patience. I have now created the config below. Is this something it can work with caddy v2? I have a json parser error but canāt see where the problem is.
{"apps":{"http":{"servers":{"myserver":{"listen":[":443"],
"routes":[{
"match":[
{"host":["example.com"]},
{ "path_regexp": {
"name": "camid_date",
"pattern": "^/(?P<CaDa>lala)>"
}
}
],
"handle":[
{
"handler": "switch",
"on": "{http.matchers.camid_date.CaDa}",
"set": ["storageid"],
"map_vars": {
"7/2010/08/10": ["10"],
"7/2010/08/11": ["10"],
"7/2010/08/12": ["10"]
},
{
"handler":"file_server",
"root": "/web/data/data{storageid}"
}
}]}]}}}}}
I tried to build caddy v2 today and get the following error, sadly therfore I canāt test the config. It would be nice to have a working example config somewhere
build github.com/caddyserver/caddy/v2/cmd/caddy: cannot load github.com/caddyserver/caddy/caddyconfig: cannot find module providing package github.com/caddyserver/caddy/caddyconfig
matt
August 23, 2019, 5:24pm
11
Awesome, thanks for giving it a go.
Weāre still in the very early stages of documentation (obviously, as Caddy 2 is still pre-beta!) but there are multiple pages of docs for v2 on the wiki: GitHub - caddyserver/caddy: Fast and extensible multi-platform HTTP/1-2-3 web server with automatic HTTPS
One of the docs pages shows you how to build a HTTP server config from scratch: GitHub - caddyserver/caddy: Fast and extensible multi-platform HTTP/1-2-3 web server with automatic HTTPS - you can use that as a starting point.
You can then validate your JSON with a tool like https://jsonlint.com/ .
I fixed a couple errors (extra closing ā}ā and using a ā]ā where ā}ā was needed) for you:
{
"apps": {
"http": {
"servers": {
"myserver": {
"listen": [
":443"
],
"routes": [
{
"match": [
{
"host": [
"example.com"
]
},
{
"path_regexp": {
"name": "camid_date",
"pattern": "^/(?P<CaDa>lala)>"
}
}
],
"handle": [
{
"handler": "switch",
"on": "{http.matchers.camid_date.CaDa}",
"set": [
"storageid"
],
"map_vars": {
"7/2010/08/10": [
"10"
],
"7/2010/08/11": [
"10"
],
"7/2010/08/12": [
"10"
]
}
},
{
"handler": "file_server",
"root": "/web/data/data{storageid}"
}
]
}
]
}
}
}
}
}
Thatās at least valid JSON.
An alternative way to do this is to use the Caddyfile ā you donāt necessarily have to use the native JSON config directly.
1 Like
Cool, thanks for the valid json
I use normally jq but when the json is not valid then jq strikes.
matt
August 24, 2019, 3:18am
13
Soon, yeah! Itās going to take a backseat to fastcgi at this point (which is my next big project starting monday), but Iāll get to work on it sooner or later, or when I have a few minutes here and there.
1 Like
matt:
Soon, yeah! Itās going to take a backseat to fastcgi at this point (which is my next big project starting monday), but Iāll get to work on it sooner or later, or when I have a few minutes here and there.
Sounds great
As we talk about features is/will this use case be possible?
The most commands are quite straight forward, IMHO.
The directive try_files is documented therehttp://nginx.org/en/docs/http/ngx_http_core_module.html#try_files
The line include /home/nginx/server/conf/global_valid-referer.conf; uses the following directive.http://nginx.org/en/docs/http/ngx_http_referer_module.html#valid_referers
add_header can be used with this GitHub - caddyserver/caddy: Fast and extensible multi-platform HTTP/1-2-3 web server with automatic HTTPS , right?
How about the include directive, is this possible/planned into v2?
location /interfaces {
include add_header-Access-Control-Allow-Origin.conf;
include add_header-Access-Control-Allow-Headers-X-Requested-With.conf;
try_files $uri /app.php$is_args$args;
location ~ /interfaces/json/(\S+) {
include add_header-Access-Control-Allow-Origin.conf;
include add_header-Content-Type_application_json;
try_files $uri /app.php$is_args$args;
}
}
location ~ /cams/(126|714|715|720|721|722|1229)/ {
location ~ recent {
add_header Cache-Control "no-cache";
location ~ /cams/(\d+)/\w+\.(jpg|xml)$ {
set $camid $1;
include /home/nginx/server/conf/global_valid-referer.conf;
root /web/data/data$recentstorageid;
}
}
location ~ /cams/(\d+)/\w+\.(jpg|xml)$ {
set $camid $1;
include /home/nginx/server/conf/global_valid-referer.conf;
root /web/data/data$storageid;
if ($request_method = OPTIONS ) {
add_header Access-Control-Allow-Origin "https://admin.DOMAIN.COM";
add_header Access-Control-Allow-Methods "GET, OPTIONS";
add_header Access-Control-Allow-Headers 'Authorization,Keep-Alive,User-Agent,X-Requested-With,If-Modified-Since,Cache-Control,Content-Type,X-CSRF-Token';
add_header Access-Control-Allow-Credentials "true";
add_header Content-Length 0;
add_header Content-Type text/plain;
return 200;
}
}
}
matt
August 24, 2019, 3:26pm
15
Caddy 2 has this, but even better. Its basic equivalent has a Caddyfile counterpart: https://github.com/caddyserver/caddy/wiki/v2:-Documentation#try_files
But the file matcher can match based on other properties as well (not just existence), using the JSON.
Can probably be done with regex in Caddy 2. If not, or if a more convenient config is needed, this is easy to add.
Yep!
See import: The Caddyfile ā Caddy Documentation
1 Like
Thanks. Can I redirect to another section like in nginx?
try_files $uri @php_location
Please can you show us a example.
Can I suggest to create a similar explaining page like the below for nginx also for caddy v2?
http://nginx.org/en/docs/http/request_processing.html
matt
August 25, 2019, 6:25am
17
Yes, with even more flexibility.
I donāt have time right now, but maybe somebody in the community will whip one up for you!
Yes, when Caddy 2 is released weāll have thorough docs and videos.
Any update on this topic?
matt
September 7, 2019, 2:58pm
19
Iāve been working on the reverse proxy lately, but will revisit this soon I hope!
1 Like
Have you found some times to work on this?
