<< Makeshift >>

The makeshift tool is a service that serves files and CLI that downloads them with a couple of handy features baked-in. Although the CLI and server component function more like a glorified FTP, the power of this tool comes from the plugin system. For example, the file cobbler is built to run external plugins for more advanced processing files before serving them (e.g. fetching from a data source, rendering Jinja 2 templates, etc.).

Build and Go!

The makeshift tool is built using standard go build tools. To get started, clone the project, download the dependencies, and build the project:

git clone https://git.towk2.me/towk/makeshift.git
go mod tidy
go build

This will build the main driver program with the default generators that are found in the pkg/generators directory.

Note

The project does not current separate the client, server, and plugin components using build tags, but will eventually. This will allow users to only compile and distribute specific parts of the tool with limited functionality.

Basic Examples

Here are some of the common commands you may want to try right off the bat (aside from makeshift help of course). The commands below that do not include the --host, --path, or --root flags are set using the environment variables.

export MAKESHIFT_HOST=localhost
export MAKESHIFT_PATH=/test
export MAKESHIFT_ROOT=./test
export MAKESHIFT_LOG_FILE=logs/makeshift.log
export MAKESHIFT_LOG_LEVEL=debug

Start the server. The --init flag with create the default files and directory to get started at the --root path.

makeshift serve --root $HOME/apps/makeshift/server --init

From here, you might want to see what files, plugins, and profiles that are available by default.

# list the files in the root directory
makeshift list

# list files store in the template directory with a specified host
makeshift list --host http://localhost:5050 --path templates

# list all available plugins
makeshift list plugins

# list specific plugin (same as 'makeshift plugins info jinja2')
makeshift list plugins jinja2

# list all available profiles
makeshift list profiles

# list specific profile information
makeshift list profiles default

Then, we can start downloading some files or directories (as archives) both with and without running plugins.

# download all data (notice --host and --port are not set here)
makeshift download

# download the 'help.txt' file without processing (i.e. using plugins)
makeshift download --host http://localhost:5050 --path help.txt

# download files with rendering using Jinja 2 plugin and default profile 
makeshift download -p help.txt --plugins jinja2 --profile default

# download directory with rendering using plugins to fetch data and render
# using a custom 'compute' profile
makeshift download -p templates --plugins smd,jinja2 --profile compute

# do everything in the above example but extract and remove archive
makeshift download -p templates --plugins smd,jinja2 --profile compute -xr

# download a raw plugin
makeshift download plugin jinja2

# download a profile
makeshift download profile default

Note

Plugins are ran in order specified with the --plugins flag, which means if you're creating a plugin to write to a data store and then read in a subsequent plugin, the order specified with the CLI matters!

(WIP) Files, directories, profiles, and plugins will eventually be able to be uploaded to the server.

# upload a single file in root directory
makeshift upload -d @compute-base.yaml

# upload a directory (not working yet...)
makeshift upload -d @setup/

# upload an archive (extracted and saved on server - not working yet...)
makeshift upload -d @setup.tar.gz -t archive

# upload a new profile
makeshift upload profile -d @compute.json kubernetes.json

# upload a new profile from file and directory with `-d`
makeshift upload profile -d @kubernetes.json
makeshift upload profile -d '{"id": "custom", "data": {}}' kubernetes.json

# upload a new plugin
makeshift upload plugin -d @slurm.so
makeshift upload plugin slurm.so

Note

Although every command has a curl equivalent, it is better to use the CLI since it has other features such as extracting and remove archives after downloading and saving archives as files automatically.

Server Root Structure

The makeshift server serves files at the specified --root path (also set with MAKESHIFT_ROOT environment variable). The directory structure looks like the following by default with initializing with makeshift init $MAKESHIFT_ROOT.

├── data
├── plugins
└── profiles

Each directory holds specific files for different purposes:

• data - Stores any and all miscellaenous files and directories.
• plugins - Stores plugins defined in the "Creating Plugins" section.
• profiles - Stores profiles in JSON format as defined in the "Creating Profiles" section.

Creating Plugins

The makeshift tool defines a plugin as an interface that can be implemented and compiled.

type Plugin interface {
	Name() string
	Version() string
	Description() string
	Metadata() Metadata

	Init() error
	Run(data storage.KVStore, args []string) error
	Cleanup() error
}

Plugins can literally contain whatever you want and is written in Go. Here is a simple example implementation to demonstrate how that is done which we will save at src/example.go.

type Example struct{}

func (p *Example) Name() string        { return "example" }
func (p *Example) Version() string     { return "v0.0.1-alpha" }
func (p *Example) Description() string { return "An example plugin" }
func (p *Example) Metadata() map[string]any {
	return makeshift.Metadata{
		"author": map[string]any{
			"name":  "John Smith",
			"email": "john.smith@example",
			"links": []string{
				"https://example.com",
			},
		},
	}
}

func (p *Example) Init() error {
	// Initialize the plugin if necessary.
	return nil
}

func (p *Example) Run(data storage.KVStore, args []string) error {
  // Plugins can read and write to a data stores passed in.
  // See the 'jinja2' plugin for reading and 'smd' plugin for writing.
	return nil
}

func (p *Example) Clean() error {
  // Clean up resources if necessary.
	return nil
}

// This MUST be included to find the symbol in the main driver executable.
var Makeshift Example

Then, we can use the built-in makeshift plugins compile command to compile it.

makeshift plugins compile src/example.go -o $MAKESHIFT_ROOT/plugins/example.so

Tip

Make sure you move all of your plugins to $MAKESHIFT_ROOT/plugins to use them and should have an *.so name for lookup. For example, to use a custom plugin with makeshift download -p templates/hosts.j2 --plugins my-plugin, there has to a plugin $MAKESHIFT_ROOT/plugins/my-plugin.so.

Built-in Plugins

There is a collection of built-in plugins that can be compiled using the helper script in bin/compile-plugins.so and saved to $MAKESHIFT_ROOT/plugins.

See the README for jinja and smd for more details.

Creating Profiles

On the other hand, profiles are simply objects that contain data used to populate data stores. The makeshift tool does not currently use all fields of a profile which will likely be removed in the near future.

type Profile struct {
	ID          string         `json:"id"`                    // profile ID
	Description string         `json:"description,omitempty"` // profile description
	Tags        []string       `json:"tags,omitempty"`        // tags used for filtering (not implemented yet)
	Data        map[string]any `json:"data,omitempty"`        // include render data
}

Profiles can be created using JSON and only require an id with optional data. See the example in $MAKESHIFT_ROOT/profiles/default.json.

{
  "id": "default",
  "description": "Makeshift default profile",
  "data": {
        "host": "localhost",
        "path": "/test",
        "server_root": "./test"
  }
}

Tip

Make sure that you store your custom profiles in $MAKESHIFT_ROOT/profiles and that you set the name you want to use for lookup with a *.json extension (e.g. compute.json).

TODO: Missing Features

There are some features still missing that will be added later.

1. Running makeshift locally with profiles and plugins
2. Plugin to add user data for one-time use without creating a profile
3. Optionally build plugins directly into the main driver
4. Protected routes that require authentication
5. Configuration file for persistent runs
6. Dockerfile and docker-compose.yml files to build containers