JSON API issues

Jeff_Galbraith · August 21, 2020, 2:55pm

I want to be able to programmatically add reverse proxies to Caddy.

Here is what I have done:

Create a Caddyfile with known reverse proxies (works)
Cut out the reverse proxy object from the routes to use as a template
In the template, change the IP to %%ADDRESS%%, the port to %%PORT%% and the name to `%%SLUG%%, all of which get changed via regex.
At runtime, read in a JSON template of the “good” Caddyfile, without known reverse proxies, and insert modified JSON template and write contents to Caddy Server.
Test with web browser, but it fails with 404.

I have diff’d the JSON created with Caddyfile that contains the reverse proxy with the JSON that I created (fetched from caddy with curl localhost:2019/config/). I can’t see any difference, except GROUP0 increments to GROUP1 etc.

I am wondering if there is any specific order the routes need to be in? I have tried appending to end of routes array, prepending to front, etc. Still doesn’t work for me.

The JavaScript code I am using to load the JSON to Caddy is:

function putConfig (config) {
  console.log(JSON.stringify(config, null, 2))
  return new Promise((resolve, reject) => {
    axios({
      method: 'post',
      url: 'http://localhost:2019/load',
      headers: {
        'Content-Type': 'application/json'
      },
      data: config
    })
      .then(results => {
        console.log(results)
        resolve(results.data)
      })
      .catch(error => {
        console.log(error)
        reject(error.message)
      })
  })
}

and it succeeds.

used to many characters, so rest of message will be in a reply

Jeff_Galbraith · August 21, 2020, 2:56pm

The JSON is still too much

Jeff_Galbraith · August 21, 2020, 2:56pm

Here is the JSON from Caddy /config/ after adding modified template:

{
   "apps":{
      "http":{
         "servers":{
            "srv0":{
               "automatic_https":{
                  "disable":true
               },
               "listen":[
                  ":80"
               ],
               "routes":[
                  {
                     "handle":[
                        {
                           "handler":"subroute",
                           "routes":[
                              {
                                 "handle":[
                                    {
                                       "handler":"subroute",
                                       "routes":[
                                          {
                                             "handle":[
                                                {
                                                   "handler":"rewrite",
                                                   "uri_substring":[
                                                      {
                                                         "find":"/dcam-dev_files/",
                                                         "replace":"/"
                                                      }
                                                   ]
                                                }
                                             ]
                                          },
                                          {
                                             "handle":[
                                                {
                                                   "handler":"reverse_proxy",
                                                   "headers":{
                                                      "request":{
                                                         "set":{
                                                            "Access-Control-Allow-Credentials":[
                                                               "true"
                                                            ],
                                                            "Access-Control-Allow-Headers":[
                                                               "Cache-Control,Content-Type"
                                                            ],
                                                            "Access-Control-Allow-Origin":[
                                                               "*"
                                                            ],
                                                            "X-Forwarded-Proto":[
                                                               "{http.request.scheme}"
                                                            ],
                                                            "X-Real-Ip":[
                                                               "{http.request.remote.host}"
                                                            ]
                                                         }
                                                      }
                                                   },
                                                   "upstreams":[
                                                      {
                                                         "dial":"10.10.2.241:8080"
                                                      }
                                                   ]
                                                }
                                             ]
                                          }
                                       ]
                                    }
                                 ],
                                 "match":[
                                    {
                                       "path":[
                                          "/dcam-dev_files/*"
                                       ]
                                    }
                                 ]
                              },
                              {
                                 "handle":[
                                    {
                                       "handler":"subroute",
                                       "routes":[
                                          {
                                             "handle":[
                                                {
                                                   "handler":"rewrite",
                                                   "uri_substring":[
                                                      {
                                                         "find":"/dcam-dev_socket/socket.io",
                                                         "replace":"/socket.io"
                                                      }
                                                   ]
                                                }
                                             ]
                                          },
                                          {
                                             "handle":[
                                                {
                                                   "handler":"reverse_proxy",
                                                   "headers":{
                                                      "request":{
                                                         "set":{
                                                            "Access-Control-Allow-Credentials":[
                                                               "true"
                                                            ],
                                                            "Access-Control-Allow-Headers":[
                                                               "Cache-Control,Content-Type"
                                                            ],
                                                            "Access-Control-Allow-Origin":[
                                                               "*"
                                                            ],
                                                            "X-Forwarded-Proto":[
                                                               "{http.request.scheme}"
                                                            ],
                                                            "X-Real-Ip":[
                                                               "{http.request.remote.host}"
                                                            ]
                                                         }
                                                      }
                                                   },
                                                   "upstreams":[
                                                      {
                                                         "dial":"10.10.2.241:80"
                                                      }
                                                   ]
                                                }
                                             ]
                                          }
                                       ]
                                    }
                                 ],
                                 "match":[
                                    {
                                       "path":[
                                          "/dcam-dev_socket/*"
                                       ]
                                    }
                                 ]
                              },
                              {
                                 "handle":[
                                    {
                                       "handler":"subroute",
                                       "routes":[
                                          {
                                             "handle":[
                                                {
                                                   "handler":"rewrite",
                                                   "uri_substring":[
                                                      {
                                                         "find":"/dcam-dev_socket_data/socket.io",
                                                         "replace":"/socket.io"
                                                      }
                                                   ]
                                                }
                                             ]
                                          },
                                          {
                                             "handle":[
                                                {
                                                   "handler":"reverse_proxy",
                                                   "headers":{
                                                      "request":{
                                                         "set":{
                                                            "Access-Control-Allow-Credentials":[
                                                               "true"
                                                            ],
                                                            "Access-Control-Allow-Headers":[
                                                               "Cache-Control,Content-Type"
                                                            ],
                                                            "Access-Control-Allow-Origin":[
                                                               "*"
                                                            ],
                                                            "X-Forwarded-Proto":[
                                                               "{http.request.scheme}"
                                                            ],
                                                            "X-Real-Ip":[
                                                               "{http.request.remote.host}"
                                                            ]
                                                         }
                                                      }
                                                   },
                                                   "upstreams":[
                                                      {
                                                         "dial":"10.10.2.241:8080"
                                                      }
                                                   ]
                                                }
                                             ]
                                          }
                                       ]
                                    }
                                 ],
                                 "match":[
                                    {
                                       "path":[
                                          "/dcam-dev_socket_data/*"
                                       ]
                                    }
                                 ]
                              },
                              {
                                 "handle":[
                                    {
                                       "handler":"subroute",
                                       "routes":[
                                          {
                                             "handle":[
                                                {
                                                   "handler":"rewrite",
                                                   "uri_substring":[
                                                      {
                                                         "find":"/dcam-dev/ivt-hvr-web-app/",
                                                         "replace":"/"
                                                      }
                                                   ]
                                                }
                                             ]
                                          },
                                          {
                                             "handle":[
                                                {
                                                   "handler":"reverse_proxy",
                                                   "headers":{
                                                      "request":{
                                                         "set":{
                                                            "Access-Control-Allow-Credentials":[
                                                               "true"
                                                            ],
                                                            "Access-Control-Allow-Headers":[
                                                               "Cache-Control,Content-Type"
                                                            ],
                                                            "Access-Control-Allow-Origin":[
                                                               "*"
                                                            ],
                                                            "X-Forwarded-Proto":[
                                                               "{http.request.scheme}"
                                                            ],
                                                            "X-Real-Ip":[
                                                               "{http.request.remote.host}"
                                                            ]
                                                         }
                                                      }
                                                   },
                                                   "upstreams":[
                                                      {
                                                         "dial":"10.10.2.241:80"
                                                      }
                                                   ]
                                                }
                                             ]
                                          }
                                       ]
                                    }
                                 ],
                                 "match":[
                                    {
                                       "path":[
                                          "/dcam-dev/ivt-hvr-web-app/*"
                                       ]
                                    }
                                 ]
                              },
                              {
                                 "group":"group0",
                                 "handle":[
                                    {
                                       "handler":"vars",
                                       "root":"/opt/ivt/photography"
                                    }
                                 ],
                                 "match":[
                                    {
                                       "path":[
                                          "/files/*"
                                       ]
                                    }
                                 ]
                              },
                              {
                                 "group":"group0",
                                 "handle":[
                                    {
                                       "handler":"vars",
                                       "root":"/opt/ivt/apps/6.0.0/packages/client/spa"
                                    }
                                 ]
                              },
                              {
                                 "handle":[
                                    {
                                       "handler":"subroute",
                                       "routes":[
                                          {
                                             "handle":[
                                                {
                                                   "handler":"rewrite",
                                                   "uri_substring":[
                                                      {
                                                         "find":"/console_socket",
                                                         "replace":"/console/socket.io"
                                                      }
                                                   ]
                                                }
                                             ]
                                          },
                                          {
                                             "handle":[
                                                {
                                                   "handler":"reverse_proxy",
                                                   "headers":{
                                                      "request":{
                                                         "set":{
                                                            "Access-Control-Allow-Credentials":[
                                                               "true"
                                                            ],
                                                            "Access-Control-Allow-Headers":[
                                                               "Cache-Control,Content-Type"
                                                            ],
                                                            "Access-Control-Allow-Origin":[
                                                               "*"
                                                            ],
                                                            "X-Forwarded-Proto":[
                                                               "{http.request.scheme}"
                                                            ],
                                                            "X-Real-Ip":[
                                                               "{http.request.remote.host}"
                                                            ]
                                                         }
                                                      }
                                                   },
                                                   "upstreams":[
                                                      {
                                                         "dial":"localhost:3001"
                                                      }
                                                   ]
                                                }
                                             ]
                                          }
                                       ]
                                    }
                                 ],
                                 "match":[
                                    {
                                       "path":[
                                          "/console_socket/*"
                                       ]
                                    }
                                 ]
                              },
                              {
                                 "handle":[
                                    {
                                       "handler":"subroute",
                                       "routes":[
                                          {
                                             "handle":[
                                                {
                                                   "handler":"rewrite",
                                                   "uri_substring":[
                                                      {
                                                         "find":"/web_app_socket",
                                                         "replace":"/web/socket.io"
                                                      }
                                                   ]
                                                }
                                             ]
                                          },
                                          {
                                             "handle":[
                                                {
                                                   "handler":"reverse_proxy",
                                                   "headers":{
                                                      "request":{
                                                         "set":{
                                                            "Access-Control-Allow-Credentials":[
                                                               "true"
                                                            ],
                                                            "Access-Control-Allow-Headers":[
                                                               "Cache-Control,Content-Type"
                                                            ],
                                                            "Access-Control-Allow-Origin":[
                                                               "*"
                                                            ],
                                                            "X-Forwarded-Proto":[
                                                               "{http.request.scheme}"
                                                            ],
                                                            "X-Real-Ip":[
                                                               "{http.request.remote.host}"
                                                            ]
                                                         }
                                                      }
                                                   },
                                                   "upstreams":[
                                                      {
                                                         "dial":"localhost:3001"
                                                      }
                                                   ]
                                                }
                                             ]
                                          }
                                       ]
                                    }
                                 ],
                                 "match":[
                                    {
                                       "path":[
                                          "/web_app_socket/*"
                                       ]
                                    }
                                 ]
                              },

Jeff_Galbraith · August 21, 2020, 2:57pm

                              {
                                 "handle":[
                                    {
                                       "handler":"subroute",
                                       "routes":[
                                          {
                                             "handle":[
                                                {
                                                   "handler":"rewrite",
                                                   "uri_substring":[
                                                      {
                                                         "find":"/liveview",
                                                         "replace":"/socket.io"
                                                      }
                                                   ]
                                                }
                                             ]
                                          },
                                          {
                                             "handle":[
                                                {
                                                   "handler":"reverse_proxy",
                                                   "headers":{
                                                      "request":{
                                                         "set":{
                                                            "Access-Control-Allow-Credentials":[
                                                               "true"
                                                            ],
                                                            "Access-Control-Allow-Headers":[
                                                               "Cache-Control,Content-Type"
                                                            ],
                                                            "Access-Control-Allow-Origin":[
                                                               "*"
                                                            ],
                                                            "X-Forwarded-Proto":[
                                                               "{http.request.scheme}"
                                                            ],
                                                            "X-Real-Ip":[
                                                               "{http.request.remote.host}"
                                                            ]
                                                         }
                                                      }
                                                   },
                                                   "upstreams":[
                                                      {
                                                         "dial":"localhost:3004"
                                                      }
                                                   ]
                                                }
                                             ]
                                          }
                                       ]
                                    }
                                 ],
                                 "match":[
                                    {
                                       "path":[
                                          "/liveview/*"
                                       ]
                                    }
                                 ]
                              },
                              {
                                 "handle":[
                                    {
                                       "handler":"subroute",
                                       "routes":[
                                          {
                                             "handle":[
                                                {
                                                   "handler":"rewrite",
                                                   "uri_substring":[
                                                      {
                                                         "find":"/weather",
                                                         "replace":"/socket.io"
                                                      }
                                                   ]
                                                }
                                             ]
                                          },
                                          {
                                             "handle":[
                                                {
                                                   "handler":"reverse_proxy",
                                                   "headers":{
                                                      "request":{
                                                         "set":{
                                                            "Access-Control-Allow-Credentials":[
                                                               "true"
                                                            ],
                                                            "Access-Control-Allow-Headers":[
                                                               "Cache-Control,Content-Type"
                                                            ],
                                                            "Access-Control-Allow-Origin":[
                                                               "*"
                                                            ],
                                                            "X-Forwarded-Proto":[
                                                               "{http.request.scheme}"
                                                            ],
                                                            "X-Real-Ip":[
                                                               "{http.request.remote.host}"
                                                            ]
                                                         }
                                                      }
                                                   },
                                                   "upstreams":[
                                                      {
                                                         "dial":"localhost:3010"
                                                      }
                                                   ]
                                                }
                                             ]
                                          }
                                       ]
                                    }
                                 ],
                                 "match":[
                                    {
                                       "path":[
                                          "/weather/*"
                                       ]
                                    }
                                 ]
                              },
                              {
                                 "handle":[
                                    {
                                       "handler":"subroute",
                                       "routes":[
                                          {
                                             "handle":[
                                                {
                                                   "handler":"rewrite",
                                                   "uri_substring":[
                                                      {
                                                         "find":"/archive",
                                                         "replace":"/socket.io"
                                                      }
                                                   ]
                                                }
                                             ]
                                          },
                                          {
                                             "handle":[
                                                {
                                                   "handler":"reverse_proxy",
                                                   "headers":{
                                                      "request":{
                                                         "set":{
                                                            "Access-Control-Allow-Credentials":[
                                                               "true"
                                                            ],
                                                            "Access-Control-Allow-Headers":[
                                                               "Cache-Control,Content-Type"
                                                            ],
                                                            "Access-Control-Allow-Origin":[
                                                               "*"
                                                            ],
                                                            "X-Forwarded-Proto":[
                                                               "{http.request.scheme}"
                                                            ],
                                                            "X-Real-Ip":[
                                                               "{http.request.remote.host}"
                                                            ]
                                                         }
                                                      }
                                                   },
                                                   "upstreams":[
                                                      {
                                                         "dial":"localhost:3003"
                                                      }
                                                   ]
                                                }
                                             ]
                                          }
                                       ]
                                    }
                                 ],
                                 "match":[
                                    {
                                       "path":[
                                          "/archive/*"
                                       ]
                                    }
                                 ]
                              },
                              {
                                 "handle":[
                                    {
                                       "handler":"subroute",
                                       "routes":[
                                          {
                                             "handle":[
                                                {
                                                   "handler":"rewrite",
                                                   "uri_substring":[
                                                      {
                                                         "find":"/alarms",
                                                         "replace":"/socket.io"
                                                      }
                                                   ]
                                                }
                                             ]
                                          },
                                          {
                                             "handle":[
                                                {
                                                   "handler":"reverse_proxy",
                                                   "headers":{
                                                      "request":{
                                                         "set":{
                                                            "Access-Control-Allow-Credentials":[
                                                               "true"
                                                            ],
                                                            "Access-Control-Allow-Headers":[
                                                               "Cache-Control,Content-Type"
                                                            ],
                                                            "Access-Control-Allow-Origin":[
                                                               "*"
                                                            ],
                                                            "X-Forwarded-Proto":[
                                                               "{http.request.scheme}"
                                                            ],
                                                            "X-Real-Ip":[
                                                               "{http.request.remote.host}"
                                                            ]
                                                         }
                                                      }
                                                   },
                                                   "upstreams":[
                                                      {
                                                         "dial":"localhost:3002"
                                                      }
                                                   ]
                                                }
                                             ]
                                          }
                                       ]
                                    }
                                 ],
                                 "match":[
                                    {
                                       "path":[
                                          "/alarms/*"
                                       ]
                                    }
                                 ]
                              },
                              {
                                 "handle":[
                                    {
                                       "handler":"subroute",
                                       "routes":[
                                          {
                                             "handle":[
                                                {
                                                   "handler":"rewrite",
                                                   "uri_substring":[
                                                      {
                                                         "find":"/ptz",
                                                         "replace":"/socket.io"
                                                      }
                                                   ]
                                                }
                                             ]
                                          },
                                          {
                                             "handle":[
                                                {
                                                   "handler":"reverse_proxy",
                                                   "headers":{
                                                      "request":{
                                                         "set":{
                                                            "Access-Control-Allow-Credentials":[
                                                               "true"
                                                            ],
                                                            "Access-Control-Allow-Headers":[
                                                               "Cache-Control,Content-Type"
                                                            ],
                                                            "Access-Control-Allow-Origin":[
                                                               "*"
                                                            ],
                                                            "X-Forwarded-Proto":[
                                                               "{http.request.scheme}"
                                                            ],
                                                            "X-Real-Ip":[
                                                               "{http.request.remote.host}"
                                                            ]
                                                         }
                                                      }
                                                   },
                                                   "upstreams":[
                                                      {
                                                         "dial":"localhost:3006"
                                                      }
                                                   ]
                                                }
                                             ]
                                          }
                                       ]
                                    }
                                 ],
                                 "match":[
                                    {
                                       "path":[
                                          "/ptz/*"
                                       ]
                                    }
                                 ]
                              },
                              {
                                 "handle":[
                                    {
                                       "handler":"subroute",
                                       "routes":[
                                          {
                                             "handle":[
                                                {
                                                   "handler":"reverse_proxy",
                                                   "headers":{
                                                      "request":{
                                                         "set":{
                                                            "Access-Control-Allow-Credentials":[
                                                               "true"
                                                            ],
                                                            "Access-Control-Allow-Headers":[
                                                               "Cache-Control,Content-Type"
                                                            ],
                                                            "Access-Control-Allow-Origin":[
                                                               "*"
                                                            ],
                                                            "X-Forwarded-Proto":[
                                                               "{http.request.scheme}"
                                                            ],
                                                            "X-Real-Ip":[
                                                               "{http.request.remote.host}"
                                                            ]
                                                         }
                                                      }
                                                   },
                                                   "upstreams":[
                                                      {
                                                         "dial":"localhost:3001"
                                                      }
                                                   ]
                                                }
                                             ]
                                          }
                                       ]
                                    }
                                 ],
                                 "match":[
                                    {
                                       "path":[
                                          "/api/*"
                                       ]
                                    }
                                 ]
                              },
                              {
                                 "handle":[
                                    {
                                       "handler":"file_server",
                                       "hide":[
                                          "Caddyfile"
                                       ]
                                    }
                                 ]
                              }
                           ]
                        }
                     ],
                     "match":[
                        {
                           "host":[
                              "localhost"
                           ]
                        }
                     ],
                     "terminal":true
                  }
               ]
            }
         }
      },
      "tls":{
         "automation":{
            "policies":[
               {
                  "issuer":{
                     "module":"internal"
                  }
               }
            ]
         }
      }
   },
   "logging":{
      "logs":{
         "default":{
            "level":"DEBUG"
         }
      }
   }
}

Any insights would be helpful. Thanks.

The reverse proxies added programmatically are the ones that have dcam-dev in them.

Jeff_Galbraith · August 21, 2020, 3:12pm

here is a snippet from above (with less spacing) of one of the proxies added:

{
  "handle":[
    {
      "handler":"subroute",
      "routes":[
        {
          "handle":[
            {
              "handler":"rewrite",
              "uri_substring":[
                {
                  "find":"/dcam-dev/ivt-hvr-web-app/",
                  "replace":"/"
                }
              ]
            }
          ]
        },
        {
          "handle":[
            {
              "handler":"reverse_proxy",
              "headers":{
                "request":{
                  "set":{
                    "Access-Control-Allow-Credentials":[
                      "true"
                    ],
                    "Access-Control-Allow-Headers":[
                      "Cache-Control,Content-Type"
                    ],
                    "Access-Control-Allow-Origin":[
                      "*"
                    ],
                    "X-Forwarded-Proto":[
                      "{http.request.scheme}"
                    ],
                    "X-Real-Ip":[
                      "{http.request.remote.host}"
                    ]
                  }
                }
              },
              "upstreams":[
                {
                  "dial":"10.10.2.241:80"
                }
              ]
            }
          ]
        }
      ]
    }
  ],
  "match":[
    {
      "path":[
        "/dcam-dev/ivt-hvr-web-app/*"
      ]
    }
  ]
},

Jeff_Galbraith · August 21, 2020, 3:27pm

I think I may have discovered an issue here:

Aug 21 09:26:16 FLEX-5 caddy[1037]: {"level":"info","ts":1598023576.419586,"logger":"admin.api","msg":"received request","method":"POST","host":"localhost:2019","uri":"/load","remote_addr":"127.0.0.1:54164","headers":{"Accept":["application/json, text/plain, */*"],"Connection":["close"],"Content-Length":["9050"],"Content-Type":["application/json"],"User-Agent":["axios/0.19.2"]}}
Aug 21 09:26:16 FLEX-5 caddy[1037]: {"level":"info","ts":1598023576.4207368,"logger":"admin.api","msg":"config is unchanged"}
Aug 21 09:26:16 FLEX-5 caddy[1037]: {"level":"info","ts":1598023576.4207559,"logger":"admin.api","msg":"load complete"}

Caddy “thinks” the config hasn’t changed

Jeff_Galbraith · August 21, 2020, 3:31pm

Fixed it by adding:

Cache-Control: must-revalidate

Jeff_Galbraith · August 21, 2020, 3:32pm

Still unsure why Caddy could not detect changes to the config tho

matt · August 21, 2020, 7:53pm

Do you know how to reproduce the problem consistently?

Jeff_Galbraith · August 27, 2020, 3:27pm

No, I cannot get it to be consistent. It seems to happen if I redo the operation within milliseconds. However, like I said above:

'Cache-Control': 'must-revalidate'

seems to work, but I have actually gone with:

'Cache-Control': 'max-age=1'

which forces it each time, regardless

system · November 25, 2020, 3:27pm

This topic was automatically closed 90 days after the last reply. New replies are no longer allowed.