chore: updated README with makeshift info
This commit is contained in:
parent
f917d2b6f8
commit
d1e7b275a6
GPG key ID:
0430CDBE22619155
1 changed files with 146 additions and 174 deletions
320
README.md
320
README.md
|
|
@ -1,219 +1,191 @@
|
|||
# OpenCHAMI Configurator
|
||||
# << Makeshift >>
|
||||
|
||||
The `configurator` is an extensible tool that is capable of dynamically generating files on the fly. The tool includes a built-in generator that fetchs data from an instance of [SMD](https://github.com/OpenCHAMI/smd) to generate files based on Jinja 2 template files. The tool and generator plugins are written in Go and plugins can be written by following the ["Creating Generator Plugins"](#creating-generator-plugins) section of this README.
|
||||
The `makeshift` tool is a service that serves files and CLI that downloads them with a couple of handy features baked-in. Although the CLI and server component function more like a glorified FTP, the power of this tool comes from the plugin system. For example, the file cobbler is built to run external plugins for more advanced processing files before serving them (e.g. fetching from a data source, rendering Jinja 2 templates, etc.).
|
||||
|
||||
## Building and Usage
|
||||
|
||||
The `configurator` is built using standard `go` build tools. The project separates the client, server, and generator components using build tags. To get started, clone the project, download the dependencies, and build the project:
|
||||
## Building the Tool
|
||||
|
||||
|
||||
The `makeshift` tool is built using standard `go` build tools. To get started, clone the project, download the dependencies, and build the project:
|
||||
|
||||
```bash
|
||||
git clone https://github.com/OpenCHAMI/configurator.git
|
||||
git clone https://git.towk2.me/towk/makeshift.git
|
||||
go mod tidy
|
||||
go build --tags all # equivalent to `go build --tags client,server``
|
||||
go build
|
||||
```
|
||||
|
||||
This will build the main driver program with the default generators that are found in the `pkg/generators` directory.
|
||||
|
||||
> [!WARNING]
|
||||
> Not all of the plugins have completed generation implementations and are a WIP.
|
||||
> [!NOTE]
|
||||
> The project does not current separate the client, server, and plugin components using build tags, but will eventually. This will allow users to only compile and distribute specific parts of the tool with limited functionality.
|
||||
|
||||
### Running Configurator with CLI
|
||||
|
||||
After you build the program, run the following command to use the tool:
|
||||
## Basic Examples
|
||||
|
||||
Here are some of the common commands you may want to try right off the bat (aside from `makeshift help` of course). The commands below that do not include the `--host`, `--path`, or `--root` flags are set using the environment variables.
|
||||
|
||||
```bash
|
||||
export ACCESS_TOKEN=eyJhbGciOiJIUzI1NiIs...
|
||||
./configurator generate --config config.yaml --target coredhcp -o coredhcp.conf --cacert ochami.pem
|
||||
export MAKESHIFT_HOST=localhost
|
||||
export MAKESHIFT_PATH=/test
|
||||
export MAKESHIFT_SERVER_ROOT=./test
|
||||
```
|
||||
|
||||
This will generate a new `coredhcp` config file based on the Jinja 2 template specified in the config file for "coredhcp". The files will be written to `coredhcp.conf` as specified with the `-o/--output` flag. The `--target` flag specifies the type of config file to generate by its name (see the [`Creating Generator Plugins`](#creating-generator-plugins) section for details).
|
||||
Start the server. The `--init` flag with create the default files and directory to get started at the `--root` path.
|
||||
|
||||
In other words, there should be an entry in the config file that looks like this:
|
||||
```bash
|
||||
makeshift serve --root $HOME/apps/makeshift/server --init
|
||||
```
|
||||
|
||||
```yaml
|
||||
...
|
||||
targets:
|
||||
coredhcp:
|
||||
plugin: "lib/coredhcp.so" # optional, if we want to use an external plugin instead
|
||||
templates:
|
||||
- templates/coredhcp.j2
|
||||
...
|
||||
From here, you might want to see what files are available by default.
|
||||
|
||||
```bash
|
||||
# list the files in the root directory
|
||||
makeshift list
|
||||
|
||||
# list files store in the template directory with a specified host
|
||||
makeshift list --host http://localhost:5050 --path templates
|
||||
```
|
||||
|
||||
Then, we can start downloading some files or directories (as archives).
|
||||
|
||||
```bash
|
||||
# download all data (notice --host and --port are not set here)
|
||||
makeshift download
|
||||
|
||||
# download the 'help.txt' file without processing (i.e. using plugins)
|
||||
makeshift download --host http://localhost:5050 --path help.txt
|
||||
|
||||
# download files with rendering using Jinja 2 plugin and default profile
|
||||
makeshift download -p help.txt --plugins jinja2 --profile default
|
||||
|
||||
# download directory with rendering using plugins to fetch data and render
|
||||
# using a custom 'compute' profile
|
||||
makeshift download -p templates --plugins smd,jinja2 --profile compute
|
||||
|
||||
# do everything in the above example but extract and remove archive
|
||||
makeshift download -p templates --plugins smd,jinja2 --profile compute -xr
|
||||
```
|
||||
|
||||
> [!NOTE]
|
||||
> The `configurator` tool requires a valid access token when making requests to an instance of SMD that has protected routes.
|
||||
> Plugins are ran in order specified with the `--plugins` flag, which means if you're creating a plugin to write to a data store and then read in a subsequent plugin, the order specified with the CLI matters!
|
||||
|
||||
### Running Configurator as a Service
|
||||
|
||||
The tool can also run as a service to generate files for clients:
|
||||
(WIP) Files, directories, profiles, and plugins will eventually be able to be uploaded to the server.
|
||||
|
||||
```bash
|
||||
export MAKESHIFT_JWKS_URL="http://my.openchami.cluster:8443/key"
|
||||
./configurator serve --config config.yaml
|
||||
```
|
||||
|
||||
Once the server is up and listening for HTTP requests, you can try making a request to it with `curl` or `configurator fetch`. Both commands below are essentially equivalent:
|
||||
|
||||
```bash
|
||||
export ACCESS_TOKEN=eyJhbGciOiJIUzI1NiIs...
|
||||
curl http://127.0.0.1:3334/generate?target=coredhcp -X GET -H "Authorization: Bearer $ACCESS_TOKEN" --cacert ochami.pem
|
||||
# ...or...
|
||||
./configurator fetch --target coredhcp --host http://127.0.0.1:3334 --cacert ochami.pem
|
||||
```
|
||||
|
||||
This will do the same thing as the `generate` subcommand, but through a GET request where the file contents is returned in the response. The access token is only required if the `MAKESHIFT_JWKS_URL` environment variable is set when starting the server with `serve`. The `ACCESS_TOKEN` environment variable is passed to `curl` using the `Authorization` header and expects a token as a JWT.
|
||||
|
||||
### Docker
|
||||
|
||||
New images can be built and tested using the `Dockerfile` provided in the project. However, the binary executable and the generator plugins must first be built before building the image since the Docker build copies the binary over. Therefore, build all of the binaries first by following the first section of ["Building and Usage"](#building-and-usage). Running `make docker` from the Makefile will automate this process. Otherwise, run the `docker build` command after building the executable and libraries.
|
||||
|
||||
```bash
|
||||
docker build -t configurator:testing path/to/configurator/Dockerfile
|
||||
```
|
||||
|
||||
If you want to easily include your own external generator plugins, you can build it and copy the `lib.so` file to `lib/`. Make sure that the `Generator` interface is implemented correctly as described in the ["Creating Generator Plugins"](#creating-generator-plugins) or the plugin will not load (you should get an error that specifically says this). Additionally, the name string returned from the `GetName()` method is used for looking up the plugin with the `--target` flag by the main driver program.
|
||||
|
||||
Alternatively, pull the latest existing image/container from the GitHub container repository.
|
||||
|
||||
```bash
|
||||
docker pull ghcr.io/openchami/configurator:latest
|
||||
```
|
||||
|
||||
Then, run the Docker container similarly to running the binary.
|
||||
|
||||
```bash
|
||||
export ACCESS_TOKEN=eyJhbGciOiJIUzI1NiIs...
|
||||
docker run ghcr.io/openchami/configurator:latest configurator generate --config config.yaml --target coredhcp -o coredhcp.conf --cacert configurator.pem
|
||||
```
|
||||
|
||||
### Creating Generator Plugins
|
||||
|
||||
The `configurator` uses built-in and user-defined generators that implement the `Generator` interface to describe how config files should be generated. The interface is defined like so:
|
||||
|
||||
```go
|
||||
// maps the file path to its contents
|
||||
type FileMap = map[string][]byte
|
||||
|
||||
// interface for generator plugins
|
||||
type Generator interface {
|
||||
GetName() string
|
||||
GetVersion() string
|
||||
GetDescription() string
|
||||
Generate(config *configurator.Config, opts ...util.Option) (FileMap, error)
|
||||
}
|
||||
```
|
||||
|
||||
A new plugin can be created by implementing the methods from interface and exporting a symbol with `Generator` as the name and the plugin struct as the type. The `GetName()` function returns the name that is used for looking up the corresponding target set in your config file. It can also be included in the templated files with the default plugins using the `{{ plugin_name }}` in your template. The `GetVersion()` and `GetDescription()` functions returns the version and description of the plugin which can be included in the templated files using `{{ plugin_version }}` and `{{ plugin_description }}` respectively with the default plugins. The `Generate` function is where the magic happens to build the config file from a template.
|
||||
|
||||
```go
|
||||
package main
|
||||
|
||||
type MyGenerator struct {
|
||||
PluginInfo map[string]any
|
||||
}
|
||||
|
||||
var pluginInfo map[string]any
|
||||
|
||||
// this function is not a part of the `Generator` interface
|
||||
func (g *MyGenerator) LoadFromFile() map[string]any{ /*...*/ }
|
||||
|
||||
func (g *MyGenerator) GetName() string {
|
||||
// just an example...this can be done however you want
|
||||
g.PluginInfo := LoadFromFile("path/to/plugin/info.json")
|
||||
return g.PluginInfo["name"]
|
||||
}
|
||||
|
||||
func (g *MyGenerator) GetVersion() string {
|
||||
return g.PluginInfo["version"] // "v1.0.0"
|
||||
}
|
||||
|
||||
func (g *MyGenerator) GetDescription() string {
|
||||
return g.PluginInfo["description"] // "This is an example plugin."
|
||||
}
|
||||
|
||||
func (g *MyGenerator) Generate(config *configurator.Config, opts ...util.Option) (generator.FileMap, error) {
|
||||
// do config generation stuff here...
|
||||
var (
|
||||
params = generator.GetParams(opts...)
|
||||
client = generator.GetClient(params)
|
||||
output = ""
|
||||
)
|
||||
if client {
|
||||
eths, err := client.FetchEthernetInterfaces(opts...)
|
||||
// ... blah, blah, blah, check error, format output, and so on...
|
||||
|
||||
|
||||
// apply the substitutions to Jinja template and return output as FileMap (i.e. path and it's contents)
|
||||
return generator.ApplyTemplate(path, generator.Mappings{
|
||||
"plugin_name": g.GetName(),
|
||||
"plugin_version": g.GetVersion(),
|
||||
"plugin_description": g.GetDescription(),
|
||||
"dhcp_hosts": output,
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
// this MUST be named "Generator" for symbol lookup in main driver
|
||||
var Generator MyGenerator
|
||||
# upload a file or directory (recursively)</br>
|
||||
makeshift upload</br>
|
||||
makeshift upload --host http://localhost:5050 --path help.txt</br>
|
||||
```
|
||||
|
||||
> [!NOTE]
|
||||
> The keys in `generator.ApplyTemplate` must not contain illegal characters such as a `-` or else the templates will not apply correctly.
|
||||
> Although every command has a `curl` equivalent, it is better to use the CLI since it has other features such as extracting and remove archives after downloading and saving archives as files automatically.
|
||||
|
||||
Finally, build the plugin and put it somewhere specified by `plugins` in your config. Make sure that the package is `main` before building.
|
||||
## Creating Plugins
|
||||
|
||||
```bash
|
||||
go build -buildmode=plugin -o lib/mygenerator.so path/to/mygenerator.go
|
||||
The `makeshift` tool defines a plugin as an interface that can be implemented and compiled.
|
||||
|
||||
```go
|
||||
type Plugin interface {
|
||||
Name() string
|
||||
Version() string
|
||||
Description() string
|
||||
Metadata() Metadata
|
||||
|
||||
Init() error
|
||||
Run(data storage.KVStore, args []string) error
|
||||
Cleanup() error
|
||||
}
|
||||
```
|
||||
|
||||
Now your plugin should be available to use with the `configurator` main driver program. If you get an error about not loading the correct symbol type, make sure that your generator function definitions match the `Generator` interface entirely and that you don't have a partially implemented interface.
|
||||
Plugins can *literally* contain whatever you want and is written in Go. Here is a simple example implementation to demonstrate how that is done which we will save at `src/example.go`.
|
||||
|
||||
```go
|
||||
type Example struct{}
|
||||
|
||||
func (p *Example) Name() string { return "example" }
|
||||
func (p *Example) Version() string { return "v0.0.1-alpha" }
|
||||
func (p *Example) Description() string { return "An example plugin" }
|
||||
func (p *Example) Metadata() map[string]string {
|
||||
return makeshift.Metadata{
|
||||
"author": map[string]any{
|
||||
"name": "John Smith",
|
||||
"email": "john.smith@example",
|
||||
"links": []string{
|
||||
"https://example.com",
|
||||
},
|
||||
},
|
||||
}
|
||||
}
|
||||
|
||||
func (p *Example) Init() error {
|
||||
// Initialize the plugin if necessary.
|
||||
return nil
|
||||
}
|
||||
|
||||
func (p *Example) Run(data storage.KVStore, args []string) error {
|
||||
// Plugins can read and write to a data stores passed in.
|
||||
// See the 'jinja2' plugin for reading and 'smd' plugin for writing.
|
||||
return nil
|
||||
}
|
||||
|
||||
func (p *Example) Clean() error {
|
||||
// Clean up resources if necessary.
|
||||
return nil
|
||||
}
|
||||
|
||||
// This MUST be included to find the symbol in the main driver executable.
|
||||
var Makeshift Example
|
||||
```
|
||||
|
||||
Then, we can use the built-in `makeshift plugins compile` command to compile it.
|
||||
|
||||
```bash
|
||||
makeshift plugins compile src/example.go -o $MAKESHIFT_ROOT/plugins/example.so
|
||||
```
|
||||
|
||||
> [!TIP]
|
||||
> See the `examples/test.go` file for a plugin and template example.
|
||||
> Make sure you move all of your plugins to `$MAKESHIFT_ROOT/plugins` to use them and should have an `*.so` name for lookup. For example, to use a custom plugin with `makeshift download -p templates/hosts.j2 --plugins my-plugin`, there has to a plugin `$MAKESHIFT_ROOT/plugins/my-plugin.so`.
|
||||
|
||||
## Configuration
|
||||
## Creating Profiles
|
||||
|
||||
Here is an example config file to start using configurator:
|
||||
On the other hand, profiles are simply objects that contain data used to populate data stores. The `makeshift` tool does not currently use all fields of a profile which will likely be removed in the near future.
|
||||
|
||||
```yaml
|
||||
server: # Server-related parameters when using as service
|
||||
host: 127.0.0.1
|
||||
port: 3334
|
||||
jwks: # Set the JWKS uri for protected routes
|
||||
uri: ""
|
||||
retries: 5
|
||||
smd: # SMD-related parameters
|
||||
host: http://127.0.0.1:27779
|
||||
plugins: # path to plugin directories
|
||||
- "lib/"
|
||||
targets: # targets to call with --target flag
|
||||
coredhcp:
|
||||
templates:
|
||||
- templates/coredhcp.j2
|
||||
files: # files to be copied without templating
|
||||
- extra/nodes.conf
|
||||
targets: # additional targets to run (does not run recursively)
|
||||
- dnsmasq
|
||||
```go
|
||||
type Profile struct {
|
||||
ID string `json:"id"` // profile ID
|
||||
Description string `json:"description,omitempty"` // profile description
|
||||
Tags []string `json:"tags,omitempty"` // tags used for ...
|
||||
Paths []string `json:"paths,omitempty"` // paths to download
|
||||
Plugins []string `json:"plugins,omitempty"` // plugins to run
|
||||
Data map[string]any `json:"data,omitempty"` // include render data
|
||||
}
|
||||
```
|
||||
|
||||
The `server` section sets the properties for running the `configurator` tool as a service and is not required if you're only using the CLI. Also note that the `jwks.uri` parameter is only needed for protecting endpoints. If it is not set, then all API routes are entirely public. The `smd` section tells the `configurator` tool where to find the SMD service to pull state management data used internally by the client's generator. The `templates` section is where the paths are mapped to each generator by its name (see the [`Creating Generator Plugins`](#creating-generator-plugins) section for details). The `plugins` is a list of paths to search for and load external generator plugins.
|
||||
Profiles can be created using JSON. See the example in `$MAKESHIFT_ROOT/profiles/default.json`.
|
||||
|
||||
## Running the Tests
|
||||
|
||||
The `configurator` project includes a collection of tests focused on verifying plugin behavior and generating files. The tests do not include fetching information from any remote sources, can be ran with the following command:
|
||||
|
||||
```bash
|
||||
go test ./tests/generate_test.go --tags=all
|
||||
```json
|
||||
{
|
||||
"id": "default",
|
||||
"description": "Makeshift default profile",
|
||||
"data": {
|
||||
"host": "localhost",
|
||||
"path": "/test",
|
||||
"server_root": "./test"
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
## Known Issues
|
||||
> [!TIP]
|
||||
> Make sure that you store your custom profiles in `$MAKESHIFT_ROOT/profiles` and that you set the name you want to use for lookup with a `*.json` extension (e.g. `compute.json`).
|
||||
|
||||
- Adds a new `OAuthClient` with every token request
|
||||
- Plugins are being loaded each time a file is generated
|
||||
## TODO: Missing Features
|
||||
|
||||
## TODO
|
||||
There are some features still missing that will be added later.
|
||||
|
||||
- Add group functionality to create by files by groups
|
||||
- Extend SMD client functionality (or make extensible?)
|
||||
- Handle authentication with `OAuthClient`'s correctly
|
||||
1. Uploading files and directories
|
||||
2. Uploading new profiles and plugins
|
||||
3. Running `makeshift` locally with profiles and plugins
|
||||
4. Plugin to add user data for one-time use without creating a profile
|
||||
5. Protected routes that require authentication
|
||||
6. Configuration file for persistent runs
|
||||
7. `Dockerfile` and `docker-compose.yml` files
|
||||
Loading…
Add table
Add a link
Reference in a new issue