towk/makeshift
1
0
Fork 0
Code Issues 1 Pull requests Projects Releases Packages Wiki Activity Actions

[DEV] Redesign and Create New Flexible and Simplified API #1

New issue
Open
opened 2025-04-26 20:57:32 -06:00 by towk · 0 comments
towk commented 2025-04-26 20:57:32 -06:00
Owner
Copy link

The Problem

The way that the configurator is currently designed is a bit cumbersome and can be confusing to use. For example, to generate a config file currently, one must have a config file and

Config file:

...
targets:
  coredhcp:
    plugin: "lib/coredhcp.so"  # optional, if we want to use an external plugin instead
    templates:
      - templates/coredhcp.j2
...

Then we invoke the generate command:

export ACCESS_TOKEN=eyJhbGciOiJIUzI1NiIs...
./configurator generate --config config.yaml --target coredhcp -o coredhcp.conf --cacert ochami.pem

And then, if we want to run configurator as a service:

./configurator serve --config config.yaml

We can use curl or the configurator as a client to generate a new config, but it isn't exactly obvious how this should work.

The Proposed Solution

It would be better to simplify configurator's setup and API to make it much easier to use. Instead of using "targets", the configurator will be path-based routing (like an FTP server) and just point to a root directory to serve files which can include any kind of file or directory. The user should be able to download files exactly like they are, download directories and archives, and render the Jinja 2 template files. This is now done using plugins rather than explicitly calling a "render" function.

For example, I should be able to download a file or directory directly:

# download directory at `/configs/compute/base` with rendered local and SMD data
configurator download https://my.server:3000/configs/compute/base -d@data.json --plugin smd --plugin jinja2

# download directory at `/config/compute/x3000s7b0n0` with no rendering or data
configurator download https://my.server:3000/configs/compute/x3000s7b0n0

# download shared host file at `/config/compute/x3000s7b0n0` with rendered data from SMD 
configurator download https://my.server:3000/configs/shared/hosts --plugin smd --plugin jinja2

The user should also be able to upload files to the configurator using an /upload endpoint. This endpoint can optionally be protected with JWT verification if needed.

The configurator client will also be able to execute plugins remotely as well as download an execute plugins locally. This allows more customization when populating template variables. Additionally, templates can be populated using "profile" variables.

Proposed New/revamped features:

  • Remove "target" system and use directory routing to serve files
  • Allow downloading and uploading files to directory root
  • Remote and local generator plugins
  • Profiles used to manage paths to file/directories, variable data, and plugins

New proposed Rest API:

This is the new proposed API served when running configurator as a microservice.

  • GET /download/{path} - download a file or directory (as archive) with optional templating

  • POST /upload - upload a file, directory, or template to configurator root path

  • GET /list - show the available files, directories, and templates

  • POST /profile - create a new profile with specified data

  • GET /profile - get all existing profile data

  • GET /profile/{id} - get a profile by ID

  • POST /profile/{id} - get profile data by ID

  • POST /profile/{id}/data/{varname} - set profile variable value

  • DELETE /profile/{id}/data/{varname} - remove profile variable

  • GET /profile/{id}/data - get all profile data

  • GET /profile/{id}/data/{varname} - get a profile variable value

  • POST /profile/{id}/paths/{path} - add path to file or directory for rendering

  • DELETE /profile/{id}/paths/{path} - remove path to file or directory for rendering

  • GET /profile/{id}/paths - get all paths for specified profile

  • GET /plugins - list available plugins on the server

  • POST /plugins - add a new plugin to the server

  • DELETE /plugins/{id} - remove a plugin by its ID

Other Miscellaneous Stuff

Profiles are represented with the following struct:

type Profile struct {
  ID string
  Description string
  Tags []string
  Paths []string
  Plugins []string
  Data Mappings // map[string]any
}

New proposed example usage:

# setup configurator environment variables
export CONFIGURATOR_HOST_URI=https://configurator.towk2.me
export ACCESS_TOKEN=...

# upload files to server (overrides URI used)
configurator upload templates/hosts.j2 templates/nodes.j2 config.yaml --uri https://configurator.towk2.me --path /example
configurator upload config.yaml --uri https://configurator.towk2.me --path /configs

# show what has been uploaded already (may require ACCESS_TOKEN)
configurator list
configurator list /example
configurator list /configs
configurator list /example /configs

# render and download the templates with plugins
configurator render example/hosts.j2 example/nodes.j2 --plugins=time,custom

# download files and/or directories directly (no rendering)
configurator download /examples /configs/config.yaml

# create a new profile with data
configurator profiles add example --description "An minimal example profile"
configurator profiles add kubernetes --name kubernetes --description "Kubernetes deployment profile" --tags=system
configurator profiles add slurm --description "Slurm node profile" --tags=system
configurator profiles add compute

# list all existing profiles
configurator profiles list

# edit existing profile data (non-destructive; only overwrites specified variables)
configurator profiles edit compute --description "General purpose compute node profile"
configurator profiles edit kubernetes --id k8s --tags=manage --plugins kubernetes --data '{"example": "test"}'

# download rendered files using profile details (i.e. paths, vars, and plugins)
configurator download profile example
configurator download profile kubernetes
configurator download profile slurm
configurator download profile compute

# download specific rendered files only using specified profile variable data
configurator download /configs/configs.yaml --profile k8s 
configurator download /configs/configs.yaml --profile.data k8s --profile.plugins example
configurator download /configs/configs.yaml --profile.data example --profile.plugins test

delete existing profile

configurator profile delete example

Alternatively, we can use curl to do the same as above.

# upload using HTTP multipart POST
curl -X POST $CONFIGURATOR_HOST_URI/upload/example -H "Authorization: $ACCESS_TOKEN" -F "path=/example/hosts.j2" -F "@/templates/hosts.j2"

curl -X GET $CONFIGURATOR_HOST_URI/list -H "Authorization: $ACCESS_TOKEN"
curl -X GET $CONFIGURATOR_HOST_URI/list?path=example -H "Authorization: $ACCESS_TOKEN"

curl -X GET $CONFIGURATOR_HOST_URI/download?render=true -H "Authorization: $ACCESS_TOKEN"

curl -X GET $CONFIGURATOR_HOST_URI/download/example -H "Authorization: $ACCESS_TOKEN"

Revised Plugin System

Plugins are used to fetch, create, modify, etc. data that can be used for creating new mappings provided by the service. Plugins typically live on the server and can called with requests to the /render endpoint. When starting the service, you must point to a directory that stores the plugins as well as any individual plugins stored elsewhere.

configurator serve --plugin-dir=$PWD/plugins --plugin-dir=$HOME/configurator/plugins --root $PWD

A plugin is defined by a simple interface and must be implemented to be used with the configurator. Each plugin returns a setting of mappings used for rendering of a Jinja 2 template.

type Mappings map[string]any
type Plugin interface {
    Name() string                                                                            # plugin name to display
    Description() string                                                                  # plugin description
    Version() string                                                                         # plugin version
    Run(args []string) (Mappings, error)     # plugin "program" that is executed
}

An example implementation would look like this:

type ExamplePlugin struct {}
func (p *ExamplePlugin) Name() { return "example" }
func (p *ExamplePlugin) Description() { return "This is an example plugin" }
func (p *ExamplePlugin) Version() { return "v0.0.1" }
func (p *ExamplePlugin) Run(args []string) ([]byte, error) {

    /* do something something to create mappings from server */
    
    return Mappings{
        "plugin_name": p.Name(),
        "plugin_version": p.Version(),
        "plugin_description": p.Description(),
        "host": "192.168.0.101",
        "date": "2025-05-10",
        "hostname": "apex",
        "banner": "This was generated using the 'example' plugin.",
    }, nil
}
Local Plugins

In additional to rendering files with remote plugins, you can also use local plugins stored on your file system. If the -o option is specified, the output will be saved to a file. Otherwise, the output is printed to stdout.

# run a plugin at the specified path and write output to a file
configurator run --plugin.path /usr/lib/configurator/plugins/jinja.so nodes.j2 -o nodes.yaml

# run a plugin that's found under `plugins` in config and print to stdout
configurator run --config config.yaml --plugin jinja2 nodes.j2
Configuration

These are some of the configuration options in YAML for the server component:

# root path to serve files and 
root: /opt/work/configurator/serve

# list of plugins (can be a directory or specific file)
plugins:
  - /usr/lib/configurator/plugins/extra
  - /usr/lib/configurator/plugins/jinja2.so
  # - /usr/lib/configurator/plugins/jinja3.so
  - /home/$USER/.configurator/plugins/smd.so

routes:
  # routes that require auth (relative to the root path specified above)
  protect:
    - /admin
### The Problem The way that the `configurator` is currently designed is a bit cumbersome and can be confusing to use. For example, to generate a config file currently, one must have a config file and Config file: ```yaml ... targets: coredhcp: plugin: "lib/coredhcp.so" # optional, if we want to use an external plugin instead templates: - templates/coredhcp.j2 ... ``` Then we invoke the `generate` command: ```bash export ACCESS_TOKEN=eyJhbGciOiJIUzI1NiIs... ./configurator generate --config config.yaml --target coredhcp -o coredhcp.conf --cacert ochami.pem ``` And then, if we want to run `configurator` as a service: ```bash ./configurator serve --config config.yaml ``` We can use `curl` or the `configurator` as a client to generate a new config, but it isn't exactly obvious how this should work. ### The Proposed Solution It would be better to simplify `configurator`'s setup and API to make it much easier to use. Instead of using "targets", the `configurator` will be path-based routing (like an FTP server) and just point to a root directory to serve files which can include any kind of file or directory. The user should be able to download files exactly like they are, download directories and archives, and render the Jinja 2 template files. This is now done using plugins rather than explicitly calling a "render" function. For example, I should be able to download a file or directory directly: ```bash # download directory at `/configs/compute/base` with rendered local and SMD data configurator download https://my.server:3000/configs/compute/base -d@data.json --plugin smd --plugin jinja2 # download directory at `/config/compute/x3000s7b0n0` with no rendering or data configurator download https://my.server:3000/configs/compute/x3000s7b0n0 # download shared host file at `/config/compute/x3000s7b0n0` with rendered data from SMD configurator download https://my.server:3000/configs/shared/hosts --plugin smd --plugin jinja2 ``` The user should also be able to upload files to the `configurator` using an `/upload` endpoint. This endpoint can optionally be protected with JWT verification if needed. The `configurator` client will also be able to execute plugins remotely as well as download an execute plugins locally. This allows more customization when populating template variables. Additionally, templates can be populated using "profile" variables. #### Proposed New/revamped features: - Remove "target" system and use directory routing to serve files - Allow downloading and uploading files to directory root - Remote and local generator plugins - Profiles used to manage paths to file/directories, variable data, and plugins #### New proposed Rest API: This is the new proposed API served when running `configurator` as a microservice. - GET `/download/{path}` - download a file or directory (as archive) with optional templating - POST `/upload` - upload a file, directory, or template to `configurator` root path - GET `/list` - show the available files, directories, and templates - POST `/profile` - create a new profile with specified data - GET `/profile` - get all existing profile data - GET `/profile/{id}` - get a profile by ID - POST `/profile/{id}` - get profile data by ID - POST `/profile/{id}/data/{varname}` - set profile variable value - DELETE `/profile/{id}/data/{varname}` - remove profile variable - GET `/profile/{id}/data` - get all profile data - GET `/profile/{id}/data/{varname}` - get a profile variable value - POST `/profile/{id}/paths/{path}` - add path to file or directory for rendering - DELETE `/profile/{id}/paths/{path}` - remove path to file or directory for rendering - GET `/profile/{id}/paths` - get all paths for specified profile - GET `/plugins` - list available plugins on the server - POST `/plugins` - add a new plugin to the server - DELETE `/plugins/{id}` - remove a plugin by its ID #### Other Miscellaneous Stuff Profiles are represented with the following struct: ```golang type Profile struct { ID string Description string Tags []string Paths []string Plugins []string Data Mappings // map[string]any } ``` New proposed example usage: ```bash # setup configurator environment variables export CONFIGURATOR_HOST_URI=https://configurator.towk2.me export ACCESS_TOKEN=... # upload files to server (overrides URI used) configurator upload templates/hosts.j2 templates/nodes.j2 config.yaml --uri https://configurator.towk2.me --path /example configurator upload config.yaml --uri https://configurator.towk2.me --path /configs # show what has been uploaded already (may require ACCESS_TOKEN) configurator list configurator list /example configurator list /configs configurator list /example /configs # render and download the templates with plugins configurator render example/hosts.j2 example/nodes.j2 --plugins=time,custom # download files and/or directories directly (no rendering) configurator download /examples /configs/config.yaml # create a new profile with data configurator profiles add example --description "An minimal example profile" configurator profiles add kubernetes --name kubernetes --description "Kubernetes deployment profile" --tags=system configurator profiles add slurm --description "Slurm node profile" --tags=system configurator profiles add compute # list all existing profiles configurator profiles list # edit existing profile data (non-destructive; only overwrites specified variables) configurator profiles edit compute --description "General purpose compute node profile" configurator profiles edit kubernetes --id k8s --tags=manage --plugins kubernetes --data '{"example": "test"}' # download rendered files using profile details (i.e. paths, vars, and plugins) configurator download profile example configurator download profile kubernetes configurator download profile slurm configurator download profile compute # download specific rendered files only using specified profile variable data configurator download /configs/configs.yaml --profile k8s configurator download /configs/configs.yaml --profile.data k8s --profile.plugins example configurator download /configs/configs.yaml --profile.data example --profile.plugins test ``` # delete existing profile configurator profile delete example Alternatively, we can use `curl` to do the same as above. ```bash # upload using HTTP multipart POST curl -X POST $CONFIGURATOR_HOST_URI/upload/example -H "Authorization: $ACCESS_TOKEN" -F "path=/example/hosts.j2" -F "@/templates/hosts.j2" curl -X GET $CONFIGURATOR_HOST_URI/list -H "Authorization: $ACCESS_TOKEN" curl -X GET $CONFIGURATOR_HOST_URI/list?path=example -H "Authorization: $ACCESS_TOKEN" curl -X GET $CONFIGURATOR_HOST_URI/download?render=true -H "Authorization: $ACCESS_TOKEN" curl -X GET $CONFIGURATOR_HOST_URI/download/example -H "Authorization: $ACCESS_TOKEN" ``` #### Revised Plugin System Plugins are used to fetch, create, modify, etc. data that can be used for creating new mappings provided by the service. Plugins typically live on the server and can called with requests to the `/render` endpoint. When starting the service, you must point to a directory that stores the plugins as well as any individual plugins stored elsewhere. ```bash configurator serve --plugin-dir=$PWD/plugins --plugin-dir=$HOME/configurator/plugins --root $PWD ``` A plugin is defined by a simple interface and must be implemented to be used with the `configurator`. Each plugin returns a setting of mappings used for rendering of a Jinja 2 template. ```go type Mappings map[string]any type Plugin interface { Name() string # plugin name to display Description() string # plugin description Version() string # plugin version Run(args []string) (Mappings, error) # plugin "program" that is executed } ``` An example implementation would look like this: ```go type ExamplePlugin struct {} func (p *ExamplePlugin) Name() { return "example" } func (p *ExamplePlugin) Description() { return "This is an example plugin" } func (p *ExamplePlugin) Version() { return "v0.0.1" } func (p *ExamplePlugin) Run(args []string) ([]byte, error) { /* do something something to create mappings from server */ return Mappings{ "plugin_name": p.Name(), "plugin_version": p.Version(), "plugin_description": p.Description(), "host": "192.168.0.101", "date": "2025-05-10", "hostname": "apex", "banner": "This was generated using the 'example' plugin.", }, nil } ``` ##### Local Plugins In additional to rendering files with remote plugins, you can also use local plugins stored on your file system. If the `-o` option is specified, the output will be saved to a file. Otherwise, the output is printed to stdout. ```bash # run a plugin at the specified path and write output to a file configurator run --plugin.path /usr/lib/configurator/plugins/jinja.so nodes.j2 -o nodes.yaml # run a plugin that's found under `plugins` in config and print to stdout configurator run --config config.yaml --plugin jinja2 nodes.j2 ``` ##### Configuration These are some of the configuration options in YAML for the server component: ```yaml # root path to serve files and root: /opt/work/configurator/serve # list of plugins (can be a directory or specific file) plugins: - /usr/lib/configurator/plugins/extra - /usr/lib/configurator/plugins/jinja2.so # - /usr/lib/configurator/plugins/jinja3.so - /home/$USER/.configurator/plugins/smd.so routes: # routes that require auth (relative to the root path specified above) protect: - /admin ```
towk changed title from [FEATURE] Redesign and Create new simplied API to [DEV] Redesign and Create new simplied API 2025-05-10 18:50:01 -06:00
towk changed title from [DEV] Redesign and Create new simplied API to [DEV] Redesign and Create New Flexible and Simplied API 2025-05-10 19:56:03 -06:00
towk changed title from [DEV] Redesign and Create New Flexible and Simplied API to [DEV] Redesign and Create New Flexible and Simplified API 2025-08-27 09:44:27 -06:00
Sign in to join this conversation.
No Branch/Tag specified
Branches Tags
main
major-rewrite
No results found.
Labels
Clear labels
No items
No labels
Milestone
Clear milestone
No items
No milestone
Projects
Clear projects
No items
No project
Assignees
Clear assignees
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference: towk/makeshift#1
No description provided.
Delete branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?