243 lines
8.4 KiB
Markdown
243 lines
8.4 KiB
Markdown
# << Makeshift >>
|
|
|
|
The `makeshift` tool is a service that serves files and CLI that downloads them with a couple of handy features baked-in. Although the CLI and server component function more like a glorified FTP, the power of this tool comes from the plugin system. For example, the file cobbler is built to run external plugins for more advanced processing files before serving them (e.g. fetching from a data source, rendering Jinja 2 templates, etc.).
|
|
|
|
|
|
## Build and Go!
|
|
|
|
|
|
The `makeshift` tool is built using standard `go` build tools. To get started, clone the project, download the dependencies, and build the project:
|
|
|
|
```bash
|
|
git clone https://git.towk2.me/towk/makeshift.git
|
|
go mod tidy
|
|
go build
|
|
```
|
|
|
|
This will build the main driver program with the default generators that are found in the `pkg/generators` directory.
|
|
|
|
> [!NOTE]
|
|
> The project does not current separate the client, server, and plugin components using build tags, but will eventually. This will allow users to only compile and distribute specific parts of the tool with limited functionality.
|
|
|
|
|
|
## Basic Examples
|
|
|
|
Here are some of the common commands you may want to try right off the bat (aside from `makeshift help` of course). The commands below that do not include the `--host`, `--path`, or `--root` flags are set using the environment variables.
|
|
|
|
```bash
|
|
export MAKESHIFT_HOST=localhost
|
|
export MAKESHIFT_PATH=/test
|
|
export MAKESHIFT_ROOT=./test
|
|
export MAKESHIFT_LOG_FILE=logs/makeshift.log
|
|
export MAKESHIFT_LOG_LEVEL=debug
|
|
```
|
|
|
|
Start the server. The `--init` flag with create the default files and directory to get started at the `--root` path.
|
|
|
|
```bash
|
|
makeshift serve --root $HOME/apps/makeshift/server --init
|
|
```
|
|
|
|
From here, you might want to see what files, plugins, and profiles that are available by default.
|
|
|
|
```bash
|
|
# list the files in the root directory
|
|
makeshift list
|
|
|
|
# list files store in the template directory with a specified host
|
|
makeshift list --host http://localhost:5050 --path templates
|
|
|
|
# list all available plugins
|
|
makeshift list plugins
|
|
|
|
# list specific plugin (same as 'makeshift plugins info jinja2')
|
|
makeshift list plugins jinja2
|
|
|
|
# list all available profiles
|
|
makeshift list profiles
|
|
|
|
# list specific profile information
|
|
makeshift list profiles default
|
|
```
|
|
|
|
Then, we can start downloading some files or directories (as archives) both with and without running plugins.
|
|
|
|
```bash
|
|
# download all data (notice --host and --port are not set here)
|
|
makeshift download
|
|
|
|
# download the 'help.txt' file without processing (i.e. using plugins)
|
|
makeshift download --host http://localhost:5050 --path help.txt
|
|
|
|
# download files with rendering using Jinja 2 plugin and default profile
|
|
makeshift download -p help.txt --plugins jinja2 --profile default
|
|
|
|
# download directory with rendering using plugins to fetch data and render
|
|
# using a custom 'compute' profile
|
|
makeshift download -p templates --plugins smd,jinja2 --profile compute
|
|
|
|
# do everything in the above example but extract and remove archive
|
|
makeshift download -p templates --plugins smd,jinja2 --profile compute -xr
|
|
|
|
# download a raw plugin
|
|
makeshift download plugin jinja2
|
|
|
|
# download a profile
|
|
makeshift download profile default
|
|
```
|
|
|
|
> [!NOTE]
|
|
> Plugins are ran in order specified with the `--plugins` flag, which means if you're creating a plugin to write to a data store and then read in a subsequent plugin, the order specified with the CLI matters!
|
|
|
|
(WIP) Files, directories, profiles, and plugins will eventually be able to be uploaded to the server.
|
|
|
|
```bash
|
|
# upload a single file in root directory
|
|
makeshift upload -d @compute-base.yaml
|
|
|
|
# upload a directory (not working yet...)
|
|
makeshift upload -d @setup/
|
|
|
|
# upload an archive (extracted and saved on server - not working yet...)
|
|
makeshift upload -d @setup.tar.gz -t archive
|
|
|
|
# upload a new profile
|
|
makeshift upload profile -d @compute.json kubernetes.json
|
|
|
|
# upload a new profile from file and directory with `-d`
|
|
makeshift upload profile -d @kubernetes.json
|
|
makeshift upload profile -d '{"id": "custom", "data": {}}' kubernetes.json
|
|
|
|
# upload a new plugin
|
|
makeshift upload plugin -d @slurm.so
|
|
makeshift upload plugin slurm.so
|
|
```
|
|
|
|
> [!NOTE]
|
|
> Although every command has a `curl` equivalent, it is better to use the CLI since it has other features such as extracting and remove archives after downloading and saving archives as files automatically.
|
|
|
|
## Server Root Structure
|
|
|
|
The `makeshift` server serves files at the specified `--root` path (also set with `MAKESHIFT_ROOT` environment variable). The directory structure looks like the following by default with initializing with `makeshift init $MAKESHIFT_ROOT`.
|
|
|
|
```bash
|
|
├── data
|
|
├── plugins
|
|
└── profiles
|
|
```
|
|
|
|
Each directory holds specific files for different purposes:
|
|
|
|
- `data` - Stores any and all miscellaenous files and directories.
|
|
- `plugins` - Stores plugins defined in the ["Creating Plugins"](#creating-plugins) section.
|
|
- `profiles` - Stores profiles in JSON format as defined in the ["Creating Profiles"](#creating-profiles) section.
|
|
|
|
|
|
## Creating Plugins
|
|
|
|
The `makeshift` tool defines a plugin as an interface that can be implemented and compiled.
|
|
|
|
```go
|
|
type Plugin interface {
|
|
Name() string
|
|
Version() string
|
|
Description() string
|
|
Metadata() Metadata
|
|
|
|
Init() error
|
|
Run(data storage.KVStore, args []string) error
|
|
Cleanup() error
|
|
}
|
|
```
|
|
|
|
Plugins can *literally* contain whatever you want and is written in Go. Here is a simple example implementation to demonstrate how that is done which we will save at `src/example.go`.
|
|
|
|
```go
|
|
type Example struct{}
|
|
|
|
func (p *Example) Name() string { return "example" }
|
|
func (p *Example) Version() string { return "v0.0.1-alpha" }
|
|
func (p *Example) Description() string { return "An example plugin" }
|
|
func (p *Example) Metadata() map[string]any {
|
|
return makeshift.Metadata{
|
|
"author": map[string]any{
|
|
"name": "John Smith",
|
|
"email": "john.smith@example",
|
|
"links": []string{
|
|
"https://example.com",
|
|
},
|
|
},
|
|
}
|
|
}
|
|
|
|
func (p *Example) Init() error {
|
|
// Initialize the plugin if necessary.
|
|
return nil
|
|
}
|
|
|
|
func (p *Example) Run(data storage.KVStore, args []string) error {
|
|
// Plugins can read and write to a data stores passed in.
|
|
// See the 'jinja2' plugin for reading and 'smd' plugin for writing.
|
|
return nil
|
|
}
|
|
|
|
func (p *Example) Clean() error {
|
|
// Clean up resources if necessary.
|
|
return nil
|
|
}
|
|
|
|
// This MUST be included to find the symbol in the main driver executable.
|
|
var Makeshift Example
|
|
```
|
|
|
|
Then, we can use the built-in `makeshift plugins compile` command to compile it.
|
|
|
|
```bash
|
|
makeshift plugins compile src/example.go -o $MAKESHIFT_ROOT/plugins/example.so
|
|
```
|
|
|
|
> [!TIP]
|
|
> Make sure you move all of your plugins to `$MAKESHIFT_ROOT/plugins` to use them and should have an `*.so` name for lookup. For example, to use a custom plugin with `makeshift download -p templates/hosts.j2 --plugins my-plugin`, there has to a plugin `$MAKESHIFT_ROOT/plugins/my-plugin.so`.
|
|
|
|
### Built-in Plugins
|
|
|
|
There is a collection of built-in plugins that can be compiled using the helper script in `bin/compile-plugins.so` and saved to `$MAKESHIFT_ROOT/plugins`.
|
|
|
|
See the README for [`jinja`](pkg/plugins/jinja2/README.md) and [`smd`](pkg/plugins/smd/README.md) for more details.
|
|
|
|
## Creating Profiles
|
|
|
|
On the other hand, profiles are simply objects that contain data used to populate data stores. The `makeshift` tool does not currently use all fields of a profile which will likely be removed in the near future.
|
|
|
|
```go
|
|
type Profile struct {
|
|
ID string `json:"id"` // profile ID
|
|
Description string `json:"description,omitempty"` // profile description
|
|
Tags []string `json:"tags,omitempty"` // tags used for filtering (not implemented yet)
|
|
Data map[string]any `json:"data,omitempty"` // include render data
|
|
}
|
|
```
|
|
|
|
Profiles can be created using JSON and only require an `id` with optional `data`. See the example in `$MAKESHIFT_ROOT/profiles/default.json`.
|
|
|
|
```json
|
|
{
|
|
"id": "default",
|
|
"description": "Makeshift default profile",
|
|
"data": {
|
|
"host": "localhost",
|
|
"path": "/test",
|
|
"server_root": "./test"
|
|
}
|
|
}
|
|
```
|
|
|
|
> [!TIP]
|
|
> Make sure that you store your custom profiles in `$MAKESHIFT_ROOT/profiles` and that you set the name you want to use for lookup with a `*.json` extension (e.g. `compute.json`).
|
|
|
|
## TODO: Missing Features
|
|
|
|
There are some features still missing that will be added later.
|
|
|
|
1. Optionally build plugins directly into the main driver
|
|
2. Protected routes that require authentication
|