Need a good example of how to use curl & API to "GET" and "PUT"

1. Caddy version (caddy version):

v2.4.3 h1:Y1FaV2N4WO3rBqxSYA8UZsZTQdN+PwcoOcAiZTM8C0I=

2. How I run Caddy:

a. System environment:

ubuntu 16.04

b. Command:

systemctl start caddy

c. Service/unit/compose file:

# caddy.service
#
# For using Caddy with a config file.
#
# Make sure the ExecStart and ExecReload commands are correct
# for your installation.
#
# See https://caddyserver.com/docs/install for instructions.
#
# WARNING: This service does not use the --resume flag, so if you
# use the API to make changes, they will be overwritten by the
# Caddyfile next time the service is restarted. If you intend to
# use Caddy's API to configure it, add the --resume flag to the
# `caddy run` command or use the caddy-api.service file instead.

[Unit]
Description=Caddy
Documentation=https://caddyserver.com/docs/
After=network.target network-online.target
Requires=network-online.target

[Service]
Type=notify
User=caddy
Group=caddy
ExecStart=/usr/bin/caddy run --environ --config /etc/caddy/Caddyfile
ExecReload=/usr/bin/caddy reload --config /etc/caddy/Caddyfile
TimeoutStopSec=5s
LimitNOFILE=1048576
LimitNPROC=512
PrivateTmp=true
ProtectSystem=full
AmbientCapabilities=CAP_NET_BIND_SERVICE

[Install]
WantedBy=multi-user.target

d. My complete Caddyfile or JSON config:

http://192.168.37.153, http://hostname.example.com 
{
	route /auth {
		authp {
			backend local /opt/caddy/assets/conf/local/auth/user_db.json local
			transform user {
				match origin local
				require mfa
				}
			registration {
				dropbox /opt/caddy/assets/conf/local/auth/registrations_db.json
				title "User Registration"
				}
			crypto default token lifetime 21600
			crypto key sign-verify AnExampleSecretKey123
			crypto key token name access_token
			ui {
				theme basic
				template login "/opt/caddy/templates/em20-tmpl/login.template"
				template generic "/opt/caddy/templates/em20-tmpl/generic.template"
				template portal "/opt/caddy/templates/em20-tmpl/portal.template"
				template register "/opt/caddy/templates/em20-tmpl/register.template"
				template whoami "/opt/caddy/templates/em20-tmpl/whoami.template"
				template settings "/opt/caddy/templates/em20-tmpl/settings.template"
				custom css path "/opt/caddy/css/custom.css"
				static_asset "assets/logo.png" "image/png" "/opt/caddy/templates/em20-tmpl/logo.png"
				logo url "assets/logo.png"
				logo description "Elite Manager"
				links {
					"Elite Manager" /
					"My Auth Portal Settings" /auth/settings icon "las la-cog" target_blank
					"who am i check" /auth/whoami target_blank icon "las la-star"
					"Add MFA Authentication App" /auth/settings/mfa/add/app
					"Manage Users" /user-mgr/ target_blank icon "las la-users"
				}
			}
		}
	}
	route /gr/ {
		jwt {
			allow roles user manager superadmin
			inject headers with claims
		}
		reverse_proxy http://localhost:3000
	}	
	route /version {
		respond * "2.0.0-a" 200
	}
	route /ui/ {
		jwt {
			allow roles user manager superadmin
			inject headers with claims	
		}
		reverse_proxy http://localhost:1880
	}	
	route /user-mgr/ {
		jwt {
			allow roles superadmin manager
			inject headers with claims
		}
		reverse_proxy http://localhost:5555
	}
	route / {
			jwt {
				allow roles user manager superadmin
				set auth url /auth?redirect_url=/
  				primary yes
				crypto key token name access_token
				crypto key verify AnExampleSecretKey1234!
				inject headers with claims
			}
			reverse_proxy http://localhost:1081
	}
}

3. The problem I’m having:

I am attempting to learn & start setting caddy configuration for local HTTPS, TLS and PKI settings via the API.

First effort is to be able to setup a set of static host FQDN’s and IP’s that need to be supported over the same config (users connecting from outside firewall as well as inside firewall with diff FQDN’s and IP’s but to the same set of routes in a reverse proxy configuration)

I’m running into issues being able to GET the specific routes/match/host set already.

4. Error messages and/or full log output:

5. What I already tried:

I’m trying to make sure I can “GET” before I try to start doing “POST” or “PUT” or "PATCH. I

 curl -X GET -H 'Content-Type: application/json' localhost:2019/config/apps/http/servers/ | jq

(which returns the following)

{
  "srv0": {
    "listen": [
      ":443"
    ],
    "logs": {
      "logger_names": {
        "192.168.37.122": "log0",
        "agrajag-virtualbox.tor.lab": "log0"
      }
    },
    "routes": [
      {
        "handle": [
          {
            "handler": "subroute",
            "routes": [
              {
                "handle": [
                  {
                    "handler": "subroute",
                    "routes": [
                      {
                        "handle": [
                          {
                            "body": "2.0.0-a",
                            "handler": "static_response",
                            "status_code": 200
                          }
                        ]
                      }
                    ]
                  }
                ],
                "match": [
                  {
                    "path": [
                      "/version*"
                    ]
                  }
                ]
              },
              {
                "handle": [
                  {
                    "handler": "subroute",
                    "routes": [
                      {
                        "handle": [
                          {
                            "handler": "reverse_proxy",
                            "upstreams": [
                              {
                                "dial": "localhost:3000"
                              }
                            ]
                          }
                        ]
                      }
                    ]
                  }
                ],
                "match": [
                  {
                    "path": [
                      "/gr/*"
                    ]
                  }
                ]
              },
              {
                "handle": [
                  {
                    "handler": "subroute",
                    "routes": [
                      {
                        "handle": [
                          {
                            "handler": "reverse_proxy",
                            "upstreams": [
                              {
                                "dial": "localhost:1880"
                              }
                            ]
                          }
                        ]
                      }
                    ]
                  }
                ],
                "match": [
                  {
                    "path": [
                      "/ui/*"
                    ]
                  }
                ]
              },
              {
                "handle": [
                  {
                    "handler": "subroute",
                    "routes": [
                      {
                        "handle": [
                          {
                            "handler": "reverse_proxy",
                            "upstreams": [
                              {
                                "dial": "localhost:1081"
                              }
                            ]
                          }
                        ]
                      }
                    ]
                  }
                ],
                "match": [
                  {
                    "path": [
                      "/*"
                    ]
                  }
                ]
              }
            ]
          }
        ],
        "match": [
          {
            "host": [
              "agrajag-virtualbox.tor.lab",
              "192.168.37.122"
            ]
          }
        ],
        "terminal": true
      }
    ]
  }
}

I am trying to GET (and later PUT or PATCH) an updated set of “host”: values, but cant figure out if there is a GET URI that will target this properly. I cant get much father into the URI path beyond that example above (going int srv0 is a dead end). Specifically I’m trying to GET and then be able to set this portion of that output above:

        "match": [
          {
            "host": [
              "agrajag-virtualbox.tor.lab",
              "192.168.37.122"
            ]
          }
        ]

6. Links to relevant resources:

I’ve been going back and forth in the caddy API documentation as well as caddy API tutorial, iterating over what I think is the path structure, but I’m doing it wrong.

Here is an example of a failed traversal into what I think is the proper path, tracing through the json, but obviously I’m wrong:

agrajag@tor-u64:~$ curl -X GET -H 'Content-Type: application/json' localhost:2019/config/apps/http/servers/routes/handle/handler/routes/match/hosts/
{"error":"invalid traversal path at: config/apps/http/servers/routes/handle"}

I could show a bunch of other pokes in the dark, but at this point I’m lost.

The JSONPath for your host matcher array is apps.http.servers.srv0.routes[0].match[0].host[0]. The URL will be parallel to that (but with slashes instead of . and [ ] obviously.)

(I just got this from my editor, VS Code. Other editors probably have JSON helpers too.)

Oh that’s right, the VS Code editor with caddy’s schema loaded. I’ll refocus on using that to identify the proper API paths. I’ll experiment with dropping the same json I shared as I didn’t realize it could properly handle pathing in json file segments like that. I’ll share as I am successful.

I didn’t even load the caddy schema. The path is just a feature of all JSON files.

SO here is the successful “GET” using curl:

$ curl -X GET -H 'Content-Type: application/json' localhost:2019/config/apps/http/servers/srv0/routes/0/match/0/host

["192.168.37.153","hostname.example.com"]

For those following along from home; In my results there were numbers that I’m assuming (in my ignorance) are object and list “index” values. These are significant for defining the proper path for the “GET” api call. I’m assuming this is probably because my config has multiple routes as a reverse proxy configuration, and complex caddy auth portal and auth jwt plugin configuration.

For the specific target I want to examine and set, which is the “hostnames and IP’s that are handled”: Note the 0’s that are significant in the path. Without those values I get stuck in “null” and then errors for any path I attempt to access beyond that point. I swore I had iterated over the path shared as an example, but without those 0 “index” values there was no way to get a proper response.

As Matt indicated; Getting the autosave.json into the editor is key to discovering the proper pathing. You can gather that file from the [CADDY_HOME]/.config/caddy/autosave.json (or a full dump from the api?) after starting from a known working Caddyfile.

I’m hoping for a reverse proxy configuration that this path to the host values is predictable? The /config/apps/http/servers/srv0/routes/0/match/0/host from the URI for the curl. Is the 0 here in the path representing that this will be “always there”? Beyond that any higher index values would represent additional routes and handers that are being defined? I’ll check this assumption out as I plod forward.

What I’m trying to get my head around is how to code updating a caddy server’s configuration from another locally running application that configures itself (and the rest of the services present on the node) for TLS. FQDN’s and final configured IP addresses are going to be determined (and changed). Prototyping it via curl is where we are at right now.

Next step is on to getting “PUT/PATCH” working, I’ll reread the discussion in the API docs on what they do before next attempt.

Foreshadowing: After all that the next steps are:

  • work out the “point of transition” from starting caddy using the configuration file (attached Caddyfile), to starting caddy-api in a IOT appliance on its firstboot, subsequent setup, and finalization of configuration in the field
  • then laying down our required TLS and PKI configuration objects that have been updated to generate unique local root CA naming and “turning on TLS” in caddy via the api. << the ultimate objective of all of this,

zoidberg-shrug
“What could go wrong?”

Yeah, [ ] are arrays in JSON, and allow you to have multiple entries with guaranteed order, so if you only have one item, you need to reference the 0th item when traversing. For objects, i.e. { }, each member has a string key, so you use those to traverse instead of a numerical index. Objects do not have guaranteed order.

Think of JSON like an upside-town tree, or node graph. Each / in the URL is going one level deeper into this tree, following a branch.

You can also run caddy adapt --pretty on your Caddyfile. Always the fastest way to learn what a working/correct JSON might look like for a feature you don’t quite understand yet.

Not if the array is empty. That depends on what you configured.

Yeah, higher numbers will be the subsequent routes in the routes array. For example, say it looked like "routes": [ <first>, <second>, <third> ], then those route entries would be accessed as 0, 1, and 2 respectively. If it looked like "routes": [] then there’s no entries, and accessing 0 would fail and return an error because there’s nothing there to access. you’d need to push something into routes for 0 to get the first item in the array, cause otherwise there’s no item for 0 to reach.


So, what exactly are you trying to achieve? What exactly are you trying to add to your config, and when?

I haven’t totally dug into your above posts to try and assume what it is you’re trying to do, but if you’re trying to add a new route when a customer is adding their own domain, then you might want to consider using On-Demand TLS instead, which would let you avoid changing Caddy’s config altogether. But I might be wrong and that might not be your goal.

2 Likes

Thanks for the breakdown and confirmation of the config items. I think what you stated clarifies that a PUT of the “first” instance of configuration then would be with no “index” values in the path.

To try and test how to do this, I’m “grafting” a manual JSON config into place for TLS and PKI configuration that we have confirmed works. The TLS config is showing index values, the PKI config is simple and does not.

On Demand sounds very appealing. When I first looked at it I didn’t associate it with being an option for Caddy Local CA, I’ll test with that (it’ll help significantly).

I’ll take a look at backing away from attempting to explicitly set the hosts values for caddy as CA scenario.

We still need to customize the naming of the caddy local CA, which I’ll take a stab at in the PKI configuration block

This is what I’m going to test in the AM to “PUT” the PKI config we want in place via API.

The final piece of the puzzle is how to properly present “objects” to be “PUT” into place. My guess is the following without presenting the “{ }” to wrap the object, but I’ve not tested yet (I’ve shut down the lab for the evening, will try in the AM and then update when I get it working).

curl -X PUT -H 'Content-Type: application/json' localhost:2019/config/apps/pki/certificate_authorities/local/ -d '"install_trust": true,"root_common_name": "Caddy Local Root CA - 54a82d9e-8eac-4e7e-a34d-e6d8cae08319","intermediate_common_name": "Caddy Custom Intermediate CA","name": "Caddy Custom CA"'

What you put in the request data must alays be valid JSON, and that’s not valid JSON. The { } around it is necessary to make it valid.

If you’re setting up mTLS, then it’s probably not what you need. But I’m still not clear on what your goal is so :man_shrugging:

Ok thanks on validating that the {} are necessary. I now need to go read up on mTLS to understand your statement.

Goal is to automate the build of rasberry pi units embedded in a control panel. The software stack running on the pi includes caddy as a reverse proxy to a set of applications wrapped into a single display.

We ran into significant issues with testing where everything was the same CA subject DN across dozens of deployments that would cause trust to fail when connecting from a controller application via VPN to the nodes for configuration & maintenance.

We can’t set up & use ACME or any kind of “central issueing CA” type of model for this as its all internal IP and FQDN’s.

We need the caddy local CA to come up, be unique in its Certificate subject CN, and issue certificates.

For 95% of the sites that will be fine. For the outliers (publicly traded companies with actual IT that has “rules” that require all TLS certs be centrally issued by their authority) we have to cover the edge case of “Bring your own CA and certs” to the config.

The JSON equivalency of the configurations we are attempting to set are:

    "tls":{
      "automation": {
        "policies": [
          {"issuers": [{
            "module":"internal",
            "ca": "local",
            "lifetime": "365d"
            }]
          }
        ]
      }
    },
    "pki": {
      "certificate_authorities": {
        "local":{
          "install_trust": true,
          "root_common_name": "Test Local Root CA - [unique generated string]",
          "intermediate_common_name": "Test Intermediate CA",
          "name": "Test CA"
        }
      }

So the exercise is to understand how to use the API (as an integrator) to then be able to go explain this to the actual developers writing our code to go automate this from runtime application, rather than our setup & build scripts that I maintain.

I’m not sure how mTLS (or if what we are doing above) precludes using On Demand, I’ll keep reading. From what I’m seeng we would need to set on demand in the TLS configuration block for it to work…

Well mTLS (in Caddy, anyways) is essentially the idea that an ACME client (either another Caddy instance, or some script like acme.sh or whatever) could use a particular authoritative Caddy instance (probably the one receiving all the public requests) as an ACME server to have certificates issued for it, using that Caddy instance’s CA instead of a public CA; and this can be automated with short cert lifetimes etc. Then since the authoritative/front Caddy instance has its own CA in its trust store, then it trusts connections to upstreams if they got a certificate issued by that CA. This involves making sure the upstream has Caddy’s CA’s root cert in its trust store as well. Can be slightly tedious for initial setup, but once set up it should work well.

On-Demand TLS is mainly for users who are running a SaaS and want to allow their customers to point their own domains to the SaaS’s servers for branding (CNAME), but the SaaS doesn’t control those domains and doesn’t necessarily know ahead of time the full list of domains that customers would point to their Caddy server. So this opt-in feature enables Caddy to fetch a certificate for that domain (from the configured CA) on the fly, during the first TLS handshake that is requesting that domain (usually using an ask endpoint for authorizing that domain, to prevent abuse).

Neither of these sound like they apply to what you’re trying to do, so you can probably ignore what I said earlier about that.

Ok thank you very much for the thorough walkthrough on the API URL structure and placing objects/lists and the insight on options around TLS. I was successful with using PUT to set the PKI configuration we wanted, and I’ll play around with other targets now that I understand how to determine the pathing. Here is the last test I mentioned, thanks for the clarification that we need to provide the { } around the json object being written.

$ curl -X GET -H 'Content-Type: application/json' localhost:2019/config/apps/pki/certificate_authorities/local/

{"error":"invalid traversal path at: config/apps/pki/certificate_authorities"}

$ curl -X PUT -H 'Content-Type: application/json' localhost:2019/config/apps/pki/certificate_authorities/local/ -d '{"install_trust": true,"root_common_name": "Caddy Local Root CA - 54a82d9e-8eac-4e7e-a34d-e6d8cae08319","intermediate_common_name": "Caddy Custom Intermediate CA","name": "Caddy Custom CA"}'

$ curl -X GET -H 'Content-Type: application/json' localhost:2019/config/apps/pki/certificate_authorities/local/

{"install_trust":true,"intermediate_common_name":"Caddy Custom Intermediate CA","name":"Caddy Custom CA","root_common_name":"Caddy Local Root CA - 54a82d9e-8eac-4e7e-a34d-e6d8cae08319"}

This what I was trying to figure out, thanks much!

1 Like

(more scratchpad notes for types of API edits)…

When dealing with an array rather than an object; Appending items to the list of host entries (host values, note ellipse in URI for the POST)

$ curl -X GET -H 'Content-Type: application/json' localhost:2019/config/apps/http/servers/srv0/routes/0/match/0/host/
["192.168.37.153","host.example.com"]

$ curl -X POST -H 'Content-Type: application/json' "localhost:2019/config/apps/http/servers/srv0/routes/0/match/0/host/..." -d '["caddytest.tor.lab", "10.252.252.21"]'

$ curl -X GET -H 'Content-Type: application/json' localhost:2019/config/apps/http/servers/srv0/routes/0/match/0/host/
["192.168.37.153","host.example.com","caddytest.tor.lab","10.252.252.21"]

This topic was automatically closed after 30 days. New replies are no longer allowed.