From 0c87336d5823a3da0abae48b9b86797823d216dc Mon Sep 17 00:00:00 2001 From: David Allen Date: Sun, 31 Aug 2025 22:53:41 -0600 Subject: [PATCH 01/34] chore: added section about default plugins to README --- README.md | 6 ++++++ 1 file changed, 6 insertions(+) diff --git a/README.md b/README.md index 380d435..0d217ed 100644 --- a/README.md +++ b/README.md @@ -199,6 +199,12 @@ makeshift plugins compile src/example.go -o $MAKESHIFT_ROOT/plugins/example.so > [!TIP] > Make sure you move all of your plugins to `$MAKESHIFT_ROOT/plugins` to use them and should have an `*.so` name for lookup. For example, to use a custom plugin with `makeshift download -p templates/hosts.j2 --plugins my-plugin`, there has to a plugin `$MAKESHIFT_ROOT/plugins/my-plugin.so`. +### Built-in Plugins + +There is a collection of built-in plugins that can be compiled using the helper script in `bin/compile-plugins.so` and saved to `$MAKESHIFT_ROOT/plugins`. + +See the README for [`jinja`](pkg/plugins/jinja2/README.md) and [`smd`](pkg/plugins/smd/README.md) for more details. + ## Creating Profiles On the other hand, profiles are simply objects that contain data used to populate data stores. The `makeshift` tool does not currently use all fields of a profile which will likely be removed in the near future. From fb6f6933a2b3070ca73840bc47fbb7b1cf9a736f Mon Sep 17 00:00:00 2001 From: David Allen Date: Sun, 31 Aug 2025 22:54:18 -0600 Subject: [PATCH 02/34] chore: added individual READMEs to default plugins --- pkg/plugins/jinja2/README.md | 34 ++++++++++++++++++++++++++++++++++ pkg/plugins/smd/README.md | 13 +++++++++++++ 2 files changed, 47 insertions(+) create mode 100644 pkg/plugins/jinja2/README.md create mode 100644 pkg/plugins/smd/README.md diff --git a/pkg/plugins/jinja2/README.md b/pkg/plugins/jinja2/README.md new file mode 100644 index 0000000..717cb3e --- /dev/null +++ b/pkg/plugins/jinja2/README.md @@ -0,0 +1,34 @@ +# Makeshift Jinja 2 Plugin + +This is a `makeshift` plugin that allows rendering files with Jinja 2 grammar. The plugin reads from a `storage.KVStore` passed to the `Run()` function and substitute the values in files provided with key `file`. Profiles can also be used to inject data into the store as well. + +Compile the plugin if not done already. + +```bash +makeshift compile pkg/plugins/jinja2/jinja2.go -o $MAKESHIFT_ROOT/plugins/jinja.so +``` + +Then, try download a file that contains Jinja2 variables. The `makeshift` server includes a `help.txt` and `default.json` profile to show how rendering can be done with downloads, profiles, and plugins. + +First, try downloading the `help.txt` file *without* rendering. + +```bash +makeshift download -p help.txt +``` + +Notice that there are Jinja2 variables with no substitutions done here. Now, try it again with the `jinja2` plugin and the `default` profile. + +```bash +makeshift download -p help.txt --plugins jinja2 --profile default +``` + +You should see the same file, but now the values are populated with the content stored in the profile. Remember...the data can also be populated using other plugins as well (see the [`smd`](../smd/README.md) plugin regarding this). + +Also, keep in mind that rendering happens across directories as well. + +```bash +makeshift download -p templates --plugins jinja2 --profile +default +``` + +This will render all Jinja 2 templates before archiving and downloading the files with the client. diff --git a/pkg/plugins/smd/README.md b/pkg/plugins/smd/README.md new file mode 100644 index 0000000..ef49a20 --- /dev/null +++ b/pkg/plugins/smd/README.md @@ -0,0 +1,13 @@ +# Makeshift SMD Plugin + +This `makeshift` plugin fetchs data from [SMD](https://github.com/OpenCHAMI/smd) and stores it in the `storage.KVStore` passed to the `Run()` function. This plugin can be used in conjunction with other plugins like the `jinja2` plugin to render files using the data stored. + +Here's a simple example. + +```bash +makeshift download -p /etc/hosts --plugins smd,jinja2 -xr +``` + +This example downloads all the files in the `/etc/hosts` directory and renders the Jinja 2 templates with the data fetched from SMD. Then, the `-x` and `-r` flags tells the CLI to extract and remove the archive respectively. + +Since plugins are ran in the order specified with the CLI, be sure to always call the `smd` plugin before reading from the `storage.KVStore` like in the example above or else the fetched data will not exist! From ee8318bc2c63527dd5b78cc1f4d4c6fd65ae2ae1 Mon Sep 17 00:00:00 2001 From: David Allen Date: Sun, 31 Aug 2025 22:56:01 -0600 Subject: [PATCH 03/34] fix: typo in README --- README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/README.md b/README.md index 0d217ed..eb89462 100644 --- a/README.md +++ b/README.md @@ -3,7 +3,7 @@ The `makeshift` tool is a service that serves files and CLI that downloads them with a couple of handy features baked-in. Although the CLI and server component function more like a glorified FTP, the power of this tool comes from the plugin system. For example, the file cobbler is built to run external plugins for more advanced processing files before serving them (e.g. fetching from a data source, rendering Jinja 2 templates, etc.). -## Building and Go! +## Build and Go! The `makeshift` tool is built using standard `go` build tools. To get started, clone the project, download the dependencies, and build the project: From 1171b4f5385e79af370cff1e05054c7aaa0408ad Mon Sep 17 00:00:00 2001 From: David Allen Date: Sun, 31 Aug 2025 22:59:20 -0600 Subject: [PATCH 04/34] fix: spacing with upload examples in REAMDE --- README.md | 28 ++++++++++++++-------------- 1 file changed, 14 insertions(+), 14 deletions(-) diff --git a/README.md b/README.md index eb89462..5c4e701 100644 --- a/README.md +++ b/README.md @@ -93,24 +93,24 @@ makeshift download profile default ```bash # upload a single file in root directory - makeshift upload -d @compute-base.yaml - - # upload a directory (not working yet...) - makeshift upload -d @setup/ +makeshift upload -d @compute-base.yaml - # upload an archive (extracted and saved on server - not working yet...) - makeshift upload -d @setup.tar.gz -t archive +# upload a directory (not working yet...) +makeshift upload -d @setup/ - # upload a new profile - makeshift upload profile -d @compute.json kubernetes.json +# upload an archive (extracted and saved on server - not working yet...) +makeshift upload -d @setup.tar.gz -t archive - # upload a new profile with a specific path - makeshift upload profile -d @kubernetes.json - makeshift upload profile -d '{"id": "custom", "data": {}}' kubernetes.json + # upload a new profile +makeshift upload profile -d @compute.json kubernetes.json - # upload a new plugin - makeshift upload plugin -d @slurm.so - makeshift upload plugin slurm.so +# upload a new profile with a specific path +makeshift upload profile -d @kubernetes.json +makeshift upload profile -d '{"id": "custom", "data": {}}' kubernetes.json + +# upload a new plugin +makeshift upload plugin -d @slurm.so +makeshift upload plugin slurm.so ``` > [!NOTE] From b0f8adef2a8e7aa4375707b90c189de9b53e2039 Mon Sep 17 00:00:00 2001 From: David Allen Date: Sun, 31 Aug 2025 23:01:25 -0600 Subject: [PATCH 05/34] fix: minor typos and information --- README.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/README.md b/README.md index 5c4e701..2610cb8 100644 --- a/README.md +++ b/README.md @@ -101,10 +101,10 @@ makeshift upload -d @setup/ # upload an archive (extracted and saved on server - not working yet...) makeshift upload -d @setup.tar.gz -t archive - # upload a new profile +# upload a new profile makeshift upload profile -d @compute.json kubernetes.json -# upload a new profile with a specific path +# upload a new profile from file and directory with `-d` makeshift upload profile -d @kubernetes.json makeshift upload profile -d '{"id": "custom", "data": {}}' kubernetes.json From dc6a141ca1dcb6f8ea130eddc714d4cc872e0496 Mon Sep 17 00:00:00 2001 From: David Allen Date: Mon, 1 Sep 2025 18:57:31 -0600 Subject: [PATCH 06/34] chore: minor changes to README --- README.md | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/README.md b/README.md index 2610cb8..2ce09ce 100644 --- a/README.md +++ b/README.md @@ -38,7 +38,7 @@ Start the server. The `--init` flag with create the default files and directory makeshift serve --root $HOME/apps/makeshift/server --init ``` -From here, you might want to see what files are available by default. +From here, you might want to see what files, plugins, and profiles that are available by default. ```bash # list the files in the root directory @@ -60,7 +60,7 @@ makeshift list profiles makeshift list profiles default ``` -Then, we can start downloading some files or directories (as archives). +Then, we can start downloading some files or directories (as archives) both with and without running plugins. ```bash # download all data (notice --host and --port are not set here) @@ -158,7 +158,7 @@ type Example struct{} func (p *Example) Name() string { return "example" } func (p *Example) Version() string { return "v0.0.1-alpha" } func (p *Example) Description() string { return "An example plugin" } -func (p *Example) Metadata() map[string]string { +func (p *Example) Metadata() map[string]any { return makeshift.Metadata{ "author": map[string]any{ "name": "John Smith", From 42c8fd7c1aecb9e31ffdb302ab4e466b692b99ac Mon Sep 17 00:00:00 2001 From: David Allen Date: Mon, 1 Sep 2025 18:59:12 -0600 Subject: [PATCH 07/34] feat: added kwargs and flags to pass to plugins --- cmd/download.go | 13 +++++++++++++ internal/kwargs/kwargs.go | 34 ++++++++++++++++++++++++++++++++++ pkg/service/routes.go | 17 +++++++++++++---- pkg/storage/disk.go | 15 +++++++++++++-- pkg/storage/memory.go | 18 +++++++++++++++++- pkg/storage/storage.go | 5 +++++ 6 files changed, 95 insertions(+), 7 deletions(-) create mode 100644 internal/kwargs/kwargs.go diff --git a/cmd/download.go b/cmd/download.go index 03bf061..0111b53 100644 --- a/cmd/download.go +++ b/cmd/download.go @@ -9,11 +9,16 @@ import ( "strings" "git.towk2.me/towk/makeshift/internal/archive" + "git.towk2.me/towk/makeshift/internal/kwargs" "git.towk2.me/towk/makeshift/pkg/client" "github.com/rs/zerolog/log" "github.com/spf13/cobra" ) +var ( + pluginArgs []string + pluginKWArgs kwargs.KWArgs = kwargs.KWArgs{} +) var downloadCmd = cobra.Command{ Use: "download", Example: ` @@ -64,6 +69,12 @@ var downloadCmd = cobra.Command{ if len(profileIDs) > 0 { query += "&profiles=" + url.QueryEscape(strings.Join(profileIDs, ",")) } + if len(pluginArgs) > 0 { + query += "&args=" + url.QueryEscape(strings.Join(pluginArgs, ",")) + } + if len(pluginKWArgs) > 0 { + query += "&kwargs=" + url.QueryEscape(pluginKWArgs.String()) + } log.Debug(). Str("host", host). @@ -287,6 +298,8 @@ func init() { downloadCmd.Flags().StringP("path", "p", ".", "Set the path to list files (can be set with MAKESHIFT_PATH)") downloadCmd.Flags().StringSlice("profiles", []string{}, "Set the profile(s) to use to populate data store") downloadCmd.Flags().StringSlice("plugins", []string{}, "Set the plugin(s) to run before downloading files") + downloadCmd.Flags().StringSlice("plugin-args", []string{}, "Set the argument list to pass to plugin(s)") + downloadCmd.Flags().Var(&pluginKWArgs, "plugin-kwargs", "Set the argument map to pass to plugin(s)") downloadCmd.Flags().BoolP("extract", "x", false, "Set whether to extract archive locally after downloading") downloadCmd.Flags().BoolP("remove-archive", "r", false, "Set whether to remove the archive after extracting (used with '--extract' flag)") diff --git a/internal/kwargs/kwargs.go b/internal/kwargs/kwargs.go new file mode 100644 index 0000000..1da80bd --- /dev/null +++ b/internal/kwargs/kwargs.go @@ -0,0 +1,34 @@ +package kwargs + +import ( + "encoding/json" + "fmt" + + "git.towk2.me/towk/makeshift/internal/format" +) + +const RESERVED_KEY = "kwargs" + +type KWArgs map[string]any + +func (kw KWArgs) String() string { + b, _ := json.Marshal(kw) + return string(b) +} + +func (kw *KWArgs) Set(v string /* should be JSON object*/) error { + var ( + newArgs KWArgs + err error + ) + err = format.Unmarshal([]byte(v), &newArgs, format.JSON) + if err != nil { + return fmt.Errorf("failed to unmarshal value for %s: %w", kw.Type(), err) + } + *kw = newArgs + return nil +} + +func (kw *KWArgs) Type() string { + return "KWArgs" +} diff --git a/pkg/service/routes.go b/pkg/service/routes.go index 7d371c9..555c115 100644 --- a/pkg/service/routes.go +++ b/pkg/service/routes.go @@ -11,18 +11,23 @@ import ( "strings" "git.towk2.me/towk/makeshift/internal/archive" + "git.towk2.me/towk/makeshift/internal/kwargs" makeshift "git.towk2.me/towk/makeshift/pkg" "git.towk2.me/towk/makeshift/pkg/storage" + "github.com/go-chi/chi/v5" "github.com/rs/zerolog/log" ) func (s *Service) Download() http.HandlerFunc { return func(w http.ResponseWriter, r *http.Request) { var ( - path = s.PathForData() + strings.TrimPrefix(r.URL.Path, "/download") - pluginNames = strings.Split(r.URL.Query().Get("plugins"), ",") - profileIDs = strings.Split(r.URL.Query().Get("profiles"), ",") + path = s.PathForData() + strings.TrimPrefix(r.URL.Path, "/download") + pluginKWArgs = chi.URLParam(r, "kwargs") + pluginArgs = strings.Split(r.URL.Query().Get("args"), ",") + pluginNames = strings.Split(r.URL.Query().Get("plugins"), ",") + profileIDs = strings.Split(r.URL.Query().Get("profiles"), ",") + kw *kwargs.KWArgs fileInfo os.FileInfo out *os.File store *storage.MemoryStorage = new(storage.MemoryStorage) @@ -32,8 +37,12 @@ func (s *Service) Download() http.HandlerFunc { err error ) + // parse the KWArgs from request + kw.Set(pluginKWArgs) + // initialize storage store.Init() + store.SetKWArgs(kw) log.Debug(). Str("path", path). @@ -78,7 +87,7 @@ func (s *Service) Download() http.HandlerFunc { log.Debug().Strs("files", filenamesToArchive).Send() // prepare plugins - hooks, errs = s.loadPlugins(pluginNames, store, nil, errs) + hooks, errs = s.loadPlugins(pluginNames, store, pluginArgs, errs) if len(errs) > 0 { log.Error().Errs("errs", errs).Msg("errors occurred loading plugins") errs = []error{} diff --git a/pkg/storage/disk.go b/pkg/storage/disk.go index 46d3f1b..f464056 100644 --- a/pkg/storage/disk.go +++ b/pkg/storage/disk.go @@ -1,5 +1,7 @@ package storage +import "git.towk2.me/towk/makeshift/internal/kwargs" + type DiskStorage struct{} func (ds DiskStorage) Init() error { @@ -10,8 +12,17 @@ func (ds DiskStorage) Cleanup() error { return nil } -func (ds DiskStorage) Get(k string) error { - return nil +func (ds *DiskStorage) SetKWArgs(kw *kwargs.KWArgs) error { + return ds.Set(kwargs.RESERVED_KEY, kw) +} + +func (ds *DiskStorage) GetKWArgs() (*kwargs.KWArgs, error) { + kw, err := ds.Get(kwargs.RESERVED_KEY) + return kw.(*kwargs.KWArgs), err +} + +func (ds DiskStorage) Get(k string) (any, error) { + return nil, nil } func (ds DiskStorage) Set(k string, v any) error { diff --git a/pkg/storage/memory.go b/pkg/storage/memory.go index e554fbd..a3b1abb 100644 --- a/pkg/storage/memory.go +++ b/pkg/storage/memory.go @@ -1,6 +1,10 @@ package storage -import "fmt" +import ( + "fmt" + + "git.towk2.me/towk/makeshift/internal/kwargs" +) type MemoryStorage struct { Data map[string]any `json:"data"` @@ -15,6 +19,15 @@ func (ms *MemoryStorage) Cleanup() error { return nil } +func (ms *MemoryStorage) SetKWArgs(kw *kwargs.KWArgs) error { + return ms.Set(kwargs.RESERVED_KEY, kw) +} + +func (ms *MemoryStorage) GetKWArgs() (*kwargs.KWArgs, error) { + kw, err := ms.Get(kwargs.RESERVED_KEY) + return kw.(*kwargs.KWArgs), err +} + func (ms *MemoryStorage) Get(k string) (any, error) { v, ok := ms.Data[k] if ok { @@ -24,6 +37,9 @@ func (ms *MemoryStorage) Get(k string) (any, error) { } func (ms *MemoryStorage) Set(k string, v any) error { + if k == "kwargs" { + return fmt.Errorf("cannot set reserved key '%s' (use SetKWArgs() instead)", k) + } ms.Data[k] = v return nil } diff --git a/pkg/storage/storage.go b/pkg/storage/storage.go index 3bc9877..82908f6 100644 --- a/pkg/storage/storage.go +++ b/pkg/storage/storage.go @@ -1,9 +1,14 @@ package storage +import "git.towk2.me/towk/makeshift/internal/kwargs" + type KVStore interface { Init() error Cleanup() error + SetKWArgs(kwargs *kwargs.KWArgs) error + GetKWArgs() (*kwargs.KWArgs, error) + Get(k string) (any, error) Set(k string, v any) error GetData() any From 64801014843e9963d4574017be62bdfad47d52eb Mon Sep 17 00:00:00 2001 From: David Allen Date: Mon, 1 Sep 2025 18:59:37 -0600 Subject: [PATCH 08/34] chore: minor changes to format package --- internal/format/format.go | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/internal/format/format.go b/internal/format/format.go index b753268..bdbfea0 100644 --- a/internal/format/format.go +++ b/internal/format/format.go @@ -36,11 +36,11 @@ func (df DataFormat) Type() string { return "DataFormat" } -// MarshalData marshals arbitrary data into a byte slice formatted as outFormat. +// Marshal marshals arbitrary data into a byte slice formatted as outFormat. // If a marshalling error occurs or outFormat is unknown, an error is returned. // // Supported values are: json, list, yaml -func Marshal(data interface{}, outFormat DataFormat) ([]byte, error) { +func Marshal(data any, outFormat DataFormat) ([]byte, error) { switch outFormat { case JSON: if bytes, err := json.MarshalIndent(data, "", " "); err != nil { @@ -61,12 +61,12 @@ func Marshal(data interface{}, outFormat DataFormat) ([]byte, error) { } } -// UnmarshalData unmarshals a byte slice formatted as inFormat into an interface +// Unmarshal unmarshals a byte slice formatted as inFormat into an interface // v. If an unmarshalling error occurs or inFormat is unknown, an error is // returned. // // Supported values are: json, list, yaml -func Unmarshal(data []byte, v interface{}, inFormat DataFormat) error { +func Unmarshal(data []byte, v any, inFormat DataFormat) error { switch inFormat { case JSON: if err := json.Unmarshal(data, v); err != nil { From f0e27192c82a4d455780cd4552b4a1c37e3e8a03 Mon Sep 17 00:00:00 2001 From: David Allen Date: Mon, 1 Sep 2025 19:00:21 -0600 Subject: [PATCH 09/34] feat: updated default plugins implementation --- pkg/plugins/mapper/mapper.go | 25 +++++++++++++++++-------- pkg/plugins/user/user.go | 36 ++++++++++++++++++++++++++++++++++++ 2 files changed, 53 insertions(+), 8 deletions(-) create mode 100644 pkg/plugins/user/user.go diff --git a/pkg/plugins/mapper/mapper.go b/pkg/plugins/mapper/mapper.go index 1fded1f..66b4a5d 100644 --- a/pkg/plugins/mapper/mapper.go +++ b/pkg/plugins/mapper/mapper.go @@ -1,16 +1,25 @@ package main -import "git.towk2.me/towk/makeshift/pkg/storage" +import ( + makeshift "git.towk2.me/towk/makeshift/pkg" + "git.towk2.me/towk/makeshift/pkg/storage" +) type Mapper struct{} -func (p *Mapper) Name() string { return "jinja2" } -func (p *Mapper) Version() string { return "test" } -func (p *Mapper) Description() string { return "Renders Jinja 2 templates" } -func (p *Mapper) Metadata() map[string]string { - return map[string]string{ - "author.name": "David J. Allen", - "author.email": "davidallendj@gmail.com", +func (p *Mapper) Name() string { return "mapper" } +func (p *Mapper) Version() string { return "v0.0.1-alpha" } +func (p *Mapper) Description() string { return "Directly maps data to store" } +func (p *Mapper) Metadata() makeshift.Metadata { + return makeshift.Metadata{ + "author": map[string]any{ + "name": "David J. Allen", + "email": "davidallendj@gmail.com", + "links": []string{ + "https://github.com/davidallendj", + "https://git.towk2.me/towk", + }, + }, } } diff --git a/pkg/plugins/user/user.go b/pkg/plugins/user/user.go new file mode 100644 index 0000000..ff59640 --- /dev/null +++ b/pkg/plugins/user/user.go @@ -0,0 +1,36 @@ +package main + +import ( + makeshift "git.towk2.me/towk/makeshift/pkg" + "git.towk2.me/towk/makeshift/pkg/storage" +) + +type User struct{} + +func (p *User) Name() string { return "user" } +func (p *User) Version() string { return "v0.0.1-alpha" } +func (p *User) Description() string { return "Get user information" } +func (p *User) Metadata() makeshift.Metadata { + return makeshift.Metadata{ + "author": map[string]any{ + "name": "David J. Allen", + "email": "davidallendj@gmail.com", + "links": []string{ + "https://github.com/davidallendj", + "https://git.towk2.me/towk", + }, + }, + } +} + +func (p *User) Init() error { + return nil +} + +func (p *User) Run(store storage.KVStore, args []string) error { + return nil +} + +func (p *User) Cleanup() error { + return nil +} From aa62158ee270aab9a097a4f4cd7fb0cd56a6d4c9 Mon Sep 17 00:00:00 2001 From: David Allen Date: Wed, 3 Sep 2025 17:37:05 -0600 Subject: [PATCH 10/34] feat: added new alt Jinja2 plugin implementation --- pkg/plugins/pyjinja2/pyjinja2.go | 121 +++++++++++++++++++++++++++++++ 1 file changed, 121 insertions(+) create mode 100644 pkg/plugins/pyjinja2/pyjinja2.go diff --git a/pkg/plugins/pyjinja2/pyjinja2.go b/pkg/plugins/pyjinja2/pyjinja2.go new file mode 100644 index 0000000..fe823f2 --- /dev/null +++ b/pkg/plugins/pyjinja2/pyjinja2.go @@ -0,0 +1,121 @@ +package main + +import ( + "encoding/json" + "fmt" + + makeshift "git.towk2.me/towk/makeshift/pkg" + "git.towk2.me/towk/makeshift/pkg/storage" + jinja2 "github.com/kluctl/kluctl/lib/go-jinja2" + "github.com/rs/zerolog/log" +) + +type Jinja2 struct{} + +func (p *Jinja2) Name() string { return "pyjinja2" } +func (p *Jinja2) Version() string { return "v0.0.1-alpha" } +func (p *Jinja2) Description() string { + return "Renders Jinja 2 templates using embedded Python renderer" +} +func (p *Jinja2) Metadata() makeshift.Metadata { + return makeshift.Metadata{ + "author": map[string]any{ + "name": "David J. Allen", + "email": "davidallendj@gmail.com", + "links": []string{ + "https://github.com/davidallendj", + "https://git.towk2.me/towk", + }, + }, + } +} + +func (p *Jinja2) Init() error { + // nothing to initialize + log.Debug().Str("plugin", p.Name()).Msg("pyjinja2.Init()") + return nil +} + +func (p *Jinja2) Run(store storage.KVStore, args []string) error { + // render the files using Jinja 2 from args + var ( + mappings struct { + Data map[string]any `json:"data"` + } + profiles any // makeshift.ProfileMap + input any // []byte + output string + err error + ) + log.Debug(). + Str("plugin", p.Name()). + Any("store", store). + Strs("args", args). + Int("arg_count", len(args)). + Msg("(pyjinja2) Run()") + + profiles, err = store.Get("profiles") + if err != nil { + return fmt.Errorf("(pyjinja2) failed to get profiles: %v", err) + } + + input, err = store.Get("file") + if err != nil { + return fmt.Errorf("(pyjinja2) failed to get input data: %v", err) + } + + // get mappings from shared data (optional) + shared, err := store.Get("shared") + if err != nil { + log.Warn().Err(err).Msg("(pyjinja2) could not retrieve shared data") + } else { + err = json.Unmarshal(shared.([]byte), &mappings) + if err != nil { + return fmt.Errorf("(pyjinja2) failed to unmarshal mappings from shared data: %v", err) + } + } + + var ps = make(map[string]any) + for profileID, profile := range profiles.(makeshift.ProfileMap) { + ps[profileID] = map[string]any{ + "id": profile.ID, + "description": profile.Description, + "data": profile.Data, + } + } + + j2, err := jinja2.NewJinja2("pyjinja", 1, + jinja2.WithGlobal("makeshift", map[string]any{ + "profiles": ps, + "plugin": map[string]any{ + "name": p.Name(), + "version": p.Version(), + "description": p.Description(), + "metadata": p.Metadata(), + }, + }), + ) + if err != nil { + return fmt.Errorf("(pyjinja2) failed to create new Jinja 2 instance: %v", err) + } + defer j2.Close() + + output, err = j2.RenderString(string(input.([]byte))) + if err != nil { + return fmt.Errorf("(pyjinja2) failed to render template: %v", err) + } + + log.Debug().Any("mappings", mappings).Send() + + // write render templates to data store output + store.Set("out", []byte(output)) + return nil +} + +func (p *Jinja2) Cleanup() error { + // nothing to clean up + log.Debug().Str("plugin", p.Name()).Msg("(pyjinja2) Cleanup()") + return nil +} + +var Makeshift Jinja2 From eef69c7d42d25daea974b9afaace9b64383650f4 Mon Sep 17 00:00:00 2001 From: David Allen Date: Wed, 3 Sep 2025 17:37:19 -0600 Subject: [PATCH 11/34] chore: updated go deps --- go.mod | 25 +++++++++++++++----- go.sum | 74 ++++++++++++++++++++++++++++++++++++++++------------------ 2 files changed, 70 insertions(+), 29 deletions(-) diff --git a/go.mod b/go.mod index 4b47295..5f74103 100644 --- a/go.mod +++ b/go.mod @@ -8,30 +8,40 @@ require ( github.com/OpenCHAMI/jwtauth/v5 v5.0.0-20240321222802-e6cb468a2a18 github.com/cavaliergopher/grab/v3 v3.0.1 github.com/go-chi/chi/v5 v5.1.0 + github.com/kluctl/kluctl/lib v0.0.0-20250903095056-d99ecc263d5d github.com/lestrrat-go/jwx/v2 v2.1.1 github.com/nikolalohinski/gonja/v2 v2.3.5 github.com/rs/zerolog v1.34.0 github.com/spf13/cobra v1.8.0 github.com/tidwall/sjson v1.2.5 - go.yaml.in/yaml/v3 v3.0.4 golang.org/x/exp v0.0.0-20240719175910-8a7402abbf56 gopkg.in/yaml.v3 v3.0.1 ) require ( + github.com/Masterminds/semver/v3 v3.4.0 // indirect github.com/decred/dcrd/dcrec/secp256k1/v4 v4.3.0 // indirect github.com/dustin/go-humanize v1.0.1 // indirect + github.com/go-git/gcfg v1.5.1-0.20230307220236-3a3c6141e376 // indirect + github.com/go-git/go-billy/v5 v5.6.2 // indirect + github.com/go-git/go-git/v5 v5.16.2 // indirect + github.com/gobwas/glob v0.2.3 // indirect github.com/goccy/go-json v0.10.3 // indirect + github.com/gofrs/flock v0.12.1 // indirect + github.com/hashicorp/errwrap v1.1.0 // indirect + github.com/hashicorp/go-multierror v1.1.1 // indirect github.com/inconshreveable/mousetrap v1.1.0 // indirect + github.com/jbenet/go-context v0.0.0-20150711004518-d14ea06fba99 // indirect + github.com/jinzhu/copier v0.4.0 // indirect github.com/json-iterator/go v1.1.12 // indirect - github.com/kr/pretty v0.3.1 // indirect + github.com/kluctl/go-embed-python v0.0.0-3.11.11-20241219-1 // indirect github.com/lestrrat-go/blackmagic v1.0.2 // indirect github.com/lestrrat-go/httpcc v1.0.1 // indirect github.com/lestrrat-go/httprc v1.0.6 // indirect github.com/lestrrat-go/iter v1.0.2 // indirect github.com/lestrrat-go/option v1.0.1 // indirect github.com/mattn/go-colorable v0.1.13 // indirect - github.com/mattn/go-isatty v0.0.19 // indirect + github.com/mattn/go-isatty v0.0.20 // indirect github.com/modern-go/concurrent v0.0.0-20180306012644-bacd9c7ef1dd // indirect github.com/modern-go/reflect2 v1.0.2 // indirect github.com/pkg/errors v0.9.1 // indirect @@ -41,7 +51,10 @@ require ( github.com/tidwall/gjson v1.14.2 // indirect github.com/tidwall/match v1.1.1 // indirect github.com/tidwall/pretty v1.2.0 // indirect - golang.org/x/crypto v0.25.0 // indirect - golang.org/x/sys v0.32.0 // indirect - golang.org/x/text v0.23.0 // indirect + golang.org/x/crypto v0.39.0 // indirect + golang.org/x/net v0.41.0 // indirect + golang.org/x/sync v0.15.0 // indirect + golang.org/x/sys v0.33.0 // indirect + golang.org/x/text v0.26.0 // indirect + gopkg.in/warnings.v0 v0.1.2 // indirect ) diff --git a/go.sum b/go.sum index 91a82d0..fe24c0b 100644 --- a/go.sum +++ b/go.sum @@ -1,37 +1,62 @@ github.com/MakeNowJust/heredoc v1.0.0 h1:cXCdzVdstXyiTqTvfqk9SDHpKNjxuom+DOlyEeQ4pzQ= github.com/MakeNowJust/heredoc v1.0.0/go.mod h1:mG5amYoWBHf8vpLOuehzbGGw0EHxpZZ6lCpQ4fNJ8LE= +github.com/Masterminds/semver/v3 v3.4.0 h1:Zog+i5UMtVoCU8oKka5P7i9q9HgrJeGzI9SA1Xbatp0= +github.com/Masterminds/semver/v3 v3.4.0/go.mod h1:4V+yj/TJE1HU9XfppCwVMZq3I84lprf4nC11bSS5beM= github.com/OpenCHAMI/jwtauth/v5 v5.0.0-20240321222802-e6cb468a2a18 h1:oBPtXp9RVm9lk5zTmDLf+Vh21yDHpulBxUqGJQjwQCk= github.com/OpenCHAMI/jwtauth/v5 v5.0.0-20240321222802-e6cb468a2a18/go.mod h1:ggNHWgLfW/WRXcE8ZZC4S7UwHif16HVmyowOCWdNSN8= github.com/cavaliergopher/grab/v3 v3.0.1 h1:4z7TkBfmPjmLAAmkkAZNX/6QJ1nNFdv3SdIHXju0Fr4= github.com/cavaliergopher/grab/v3 v3.0.1/go.mod h1:1U/KNnD+Ft6JJiYoYBAimKH2XrYptb8Kl3DFGmsjpq4= github.com/coreos/go-systemd/v22 v22.5.0/go.mod h1:Y58oyj3AT4RCenI/lSvhwexgC+NSVTIJ3seZv2GcEnc= github.com/cpuguy83/go-md2man/v2 v2.0.3/go.mod h1:tgQtvFlXSQOSOSIRvRPT7W67SCa46tRHOmNcaadrF8o= -github.com/creack/pty v1.1.9/go.mod h1:oKZEueFk5CKHvIhNR5MUki03XCEU+Q6VDXinZuGJ33E= github.com/davecgh/go-spew v1.1.0/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSsI+c5H38= -github.com/davecgh/go-spew v1.1.1 h1:vj9j/u1bqnvCEfJOwUhtlOARqs3+rkHYY13jYWTU97c= github.com/davecgh/go-spew v1.1.1/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSsI+c5H38= +github.com/davecgh/go-spew v1.1.2-0.20180830191138-d8f796af33cc h1:U9qPSI2PIWSS1VwoXQT9A3Wy9MM3WgvqSxFWenqJduM= +github.com/davecgh/go-spew v1.1.2-0.20180830191138-d8f796af33cc/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSsI+c5H38= github.com/decred/dcrd/dcrec/secp256k1/v4 v4.3.0 h1:rpfIENRNNilwHwZeG5+P150SMrnNEcHYvcCuK6dPZSg= github.com/decred/dcrd/dcrec/secp256k1/v4 v4.3.0/go.mod h1:v57UDF4pDQJcEfFUCRop3lJL149eHGSe9Jvczhzjo/0= github.com/dustin/go-humanize v1.0.1 h1:GzkhY7T5VNhEkwH0PVJgjz+fX1rhBrR7pRT3mDkpeCY= github.com/dustin/go-humanize v1.0.1/go.mod h1:Mu1zIs6XwVuF/gI1OepvI0qD18qycQx+mFykh5fBlto= github.com/go-chi/chi/v5 v5.1.0 h1:acVI1TYaD+hhedDJ3r54HyA6sExp3HfXq7QWEEY/xMw= github.com/go-chi/chi/v5 v5.1.0/go.mod h1:DslCQbL2OYiznFReuXYUmQ2hGd1aDpCnlMNITLSKoi8= +github.com/go-git/gcfg v1.5.1-0.20230307220236-3a3c6141e376 h1:+zs/tPmkDkHx3U66DAb0lQFJrpS6731Oaa12ikc+DiI= +github.com/go-git/gcfg v1.5.1-0.20230307220236-3a3c6141e376/go.mod h1:an3vInlBmSxCcxctByoQdvwPiA7DTK7jaaFDBTtu0ic= +github.com/go-git/go-billy/v5 v5.6.2 h1:6Q86EsPXMa7c3YZ3aLAQsMA0VlWmy43r6FHqa/UNbRM= +github.com/go-git/go-billy/v5 v5.6.2/go.mod h1:rcFC2rAsp/erv7CMz9GczHcuD0D32fWzH+MJAU+jaUU= +github.com/go-git/go-git/v5 v5.16.2 h1:fT6ZIOjE5iEnkzKyxTHK1W4HGAsPhqEqiSAssSO77hM= +github.com/go-git/go-git/v5 v5.16.2/go.mod h1:4Ge4alE/5gPs30F2H1esi2gPd69R0C39lolkucHBOp8= github.com/go-logr/logr v1.4.2 h1:6pFjapn8bFcIbiKo3XT4j/BhANplGihG6tvd+8rYgrY= github.com/go-logr/logr v1.4.2/go.mod h1:9T104GzyrTigFIr8wt5mBrctHMim0Nb2HLGrmQ40KvY= github.com/go-task/slim-sprig/v3 v3.0.0 h1:sUs3vkvUymDpBKi3qH1YSqBQk9+9D/8M2mN1vB6EwHI= github.com/go-task/slim-sprig/v3 v3.0.0/go.mod h1:W848ghGpv3Qj3dhTPRyJypKRiqCdHZiAzKg9hl15HA8= +github.com/gobwas/glob v0.2.3 h1:A4xDbljILXROh+kObIiy5kIaPYD8e96x1tgBhUI5J+Y= +github.com/gobwas/glob v0.2.3/go.mod h1:d3Ez4x06l9bZtSvzIay5+Yzi0fmZzPgnTbPcKjJAkT8= github.com/goccy/go-json v0.10.3 h1:KZ5WoDbxAIgm2HNbYckL0se1fHD6rz5j4ywS6ebzDqA= github.com/goccy/go-json v0.10.3/go.mod h1:oq7eo15ShAhp70Anwd5lgX2pLfOS3QCiwU/PULtXL6M= github.com/godbus/dbus/v5 v5.0.4/go.mod h1:xhWf0FNVPg57R7Z0UbKHbJfkEywrmjJnf7w5xrFpKfA= +github.com/gofrs/flock v0.12.1 h1:MTLVXXHf8ekldpJk3AKicLij9MdwOWkZ+a/jHHZby9E= +github.com/gofrs/flock v0.12.1/go.mod h1:9zxTsyu5xtJ9DK+1tFZyibEV7y3uwDxPPfbxeeHCoD0= github.com/google/go-cmp v0.7.0 h1:wk8382ETsv4JYUZwIsn6YpYiWiBsYLSJiTsyBybVuN8= github.com/google/go-cmp v0.7.0/go.mod h1:pXiqmnSA92OHEEa9HXL2W4E7lf9JzCmGVUdgjX3N/iU= github.com/google/gofuzz v1.0.0/go.mod h1:dBl0BpW6vV/+mYPU4Po3pmUjxk6FQPldtuIdl/M65Eg= github.com/google/pprof v0.0.0-20250403155104-27863c87afa6 h1:BHT72Gu3keYf3ZEu2J0b1vyeLSOYI8bm5wbJM/8yDe8= github.com/google/pprof v0.0.0-20250403155104-27863c87afa6/go.mod h1:boTsfXsheKC2y+lKOCMpSfarhxDeIzfZG1jqGcPl3cA= +github.com/hashicorp/errwrap v1.0.0/go.mod h1:YH+1FKiLXxHSkmPseP+kNlulaMuP3n2brvKWEqk/Jc4= +github.com/hashicorp/errwrap v1.1.0 h1:OxrOeh75EUXMY8TBjag2fzXGZ40LB6IKw45YeGUDY2I= +github.com/hashicorp/errwrap v1.1.0/go.mod h1:YH+1FKiLXxHSkmPseP+kNlulaMuP3n2brvKWEqk/Jc4= +github.com/hashicorp/go-multierror v1.1.1 h1:H5DkEtf6CXdFp0N0Em5UCwQpXMWke8IA0+lD48awMYo= +github.com/hashicorp/go-multierror v1.1.1/go.mod h1:iw975J/qwKPdAO1clOe2L8331t/9/fmwbPZ6JB6eMoM= github.com/inconshreveable/mousetrap v1.1.0 h1:wN+x4NVGpMsO7ErUn/mUI3vEoE6Jt13X2s0bqwp9tc8= github.com/inconshreveable/mousetrap v1.1.0/go.mod h1:vpF70FUmC8bwa3OWnCshd2FqLfsEA9PFc4w1p2J65bw= +github.com/jbenet/go-context v0.0.0-20150711004518-d14ea06fba99 h1:BQSFePA1RWJOlocH6Fxy8MmwDt+yVQYULKfN0RoTN8A= +github.com/jbenet/go-context v0.0.0-20150711004518-d14ea06fba99/go.mod h1:1lJo3i6rXxKeerYnT8Nvf0QmHCRC1n8sfWVwXF2Frvo= +github.com/jinzhu/copier v0.4.0 h1:w3ciUoD19shMCRargcpm0cm91ytaBhDvuRpz1ODO/U8= +github.com/jinzhu/copier v0.4.0/go.mod h1:DfbEm0FYsaqBcKcFuvmOZb218JkPGtvSHsKg8S8hyyg= github.com/json-iterator/go v1.1.12 h1:PV8peI4a0ysnczrg+LtxykD8LfKY9ML6u2jnxaEnrnM= github.com/json-iterator/go v1.1.12/go.mod h1:e30LSqwooZae/UwlEbR2852Gd8hjQvJoHmT4TnhNGBo= +github.com/kluctl/go-embed-python v0.0.0-3.11.11-20241219-1 h1:yVXFTxa6cvhLC6TwkDEHmNaiWlDyaFEGiXd1DQdoeOU= +github.com/kluctl/go-embed-python v0.0.0-3.11.11-20241219-1/go.mod h1:3ebNU9QBrNpUO+Hj6bHaGpkh5pymDHQ+wwVPHTE4mCE= +github.com/kluctl/kluctl/lib v0.0.0-20250903095056-d99ecc263d5d h1:++9mQstGMwSb9ofOSM8FqG5WrKso6VXBqwpcAsra7vc= +github.com/kluctl/kluctl/lib v0.0.0-20250903095056-d99ecc263d5d/go.mod h1:Yf0GI0evAyhH0YpS8Rud+ROVmT30qpK4z1PCeS3BcNU= github.com/kr/pretty v0.3.1 h1:flRD4NNwYAUpkphVc1HcthR4KEIFJ65n8Mw5qdRn3LE= github.com/kr/pretty v0.3.1/go.mod h1:hoEshYVHaxMs3cyo3Yncou5ZscifuDolrwPKZanG3xk= github.com/kr/text v0.2.0 h1:5Nx0Ya0ZqY2ygV366QzturHI13Jq95ApcVaJBhpS+AY= @@ -51,8 +76,9 @@ github.com/lestrrat-go/option v1.0.1/go.mod h1:5ZHFbivi4xwXxhxY9XHDe2FHo6/Z7WWmt github.com/mattn/go-colorable v0.1.13 h1:fFA4WZxdEF4tXPZVKMLwD8oUnCTTo08duU7wxecdEvA= github.com/mattn/go-colorable v0.1.13/go.mod h1:7S9/ev0klgBDR4GtXTXX8a3vIGJpMovkB8vQcUbaXHg= github.com/mattn/go-isatty v0.0.16/go.mod h1:kYGgaQfpe5nmfYZH+SKPsOc2e4SrIfOl2e/yFXSvRLM= -github.com/mattn/go-isatty v0.0.19 h1:JITubQf0MOLdlGRuRq+jtsDlekdYPia9ZFsB8h/APPA= github.com/mattn/go-isatty v0.0.19/go.mod h1:W+V8PltTTMOvKvAeJH7IuucS94S2C6jfK/D7dTCTo3Y= +github.com/mattn/go-isatty v0.0.20 h1:xfD0iDuEKnDkl03q4limB+vH+GxLEtL/jb4xVJSWWEY= +github.com/mattn/go-isatty v0.0.20/go.mod h1:W+V8PltTTMOvKvAeJH7IuucS94S2C6jfK/D7dTCTo3Y= github.com/modern-go/concurrent v0.0.0-20180228061459-e0a39a4cb421/go.mod h1:6dJC0mAP4ikYIbvyc7fijjWJddQyLn8Ig3JB5CqoB9Q= github.com/modern-go/concurrent v0.0.0-20180306012644-bacd9c7ef1dd h1:TRLaZ9cD/w8PVh93nsPXa1VrQ6jlwL5oN8l14QlcNfg= github.com/modern-go/concurrent v0.0.0-20180306012644-bacd9c7ef1dd/go.mod h1:6dJC0mAP4ikYIbvyc7fijjWJddQyLn8Ig3JB5CqoB9Q= @@ -64,13 +90,13 @@ github.com/onsi/ginkgo/v2 v2.23.4 h1:ktYTpKJAVZnDT4VjxSbiBenUjmlL/5QkBEocaWXiQus github.com/onsi/ginkgo/v2 v2.23.4/go.mod h1:Bt66ApGPBFzHyR+JO10Zbt0Gsp4uWxu5mIOTusL46e8= github.com/onsi/gomega v1.37.0 h1:CdEG8g0S133B4OswTDC/5XPSzE1OeP29QOioj2PID2Y= github.com/onsi/gomega v1.37.0/go.mod h1:8D9+Txp43QWKhM24yyOBEdpkzN8FvJyAwecBgsU4KU0= -github.com/pkg/diff v0.0.0-20210226163009-20ebb0f2a09e/go.mod h1:pJLUxLENpZxwdsKMEsNbx1VGcRFpLqf3715MtcvvzbA= github.com/pkg/errors v0.9.1 h1:FEBLx1zS214owpjy7qsBeixbURkuhQAwrK5UwLGTwt4= github.com/pkg/errors v0.9.1/go.mod h1:bwawxfHBFNV+L2hUp1rHADufV3IMtnDRdf1r5NINEl0= -github.com/pmezard/go-difflib v1.0.0 h1:4DBwDE0NGyQoBHbLQYPwSUPoCMWR5BEzIk/f1lZbAQM= github.com/pmezard/go-difflib v1.0.0/go.mod h1:iKH77koFhYxTK1pcRnkKkqfTogsbg7gZNVY4sRDYZ/4= -github.com/rogpeppe/go-internal v1.9.0 h1:73kH8U+JUqXU8lRuOHeVHaa/SZPifC7BkcraZVejAe8= -github.com/rogpeppe/go-internal v1.9.0/go.mod h1:WtVeX8xhTBvf0smdhujwtBcq4Qrzq/fJaraNFVN+nFs= +github.com/pmezard/go-difflib v1.0.1-0.20181226105442-5d4384ee4fb2 h1:Jamvg5psRIccs7FGNTlIRMkT8wgtp5eCXdBlqhYGL6U= +github.com/pmezard/go-difflib v1.0.1-0.20181226105442-5d4384ee4fb2/go.mod h1:iKH77koFhYxTK1pcRnkKkqfTogsbg7gZNVY4sRDYZ/4= +github.com/rogpeppe/go-internal v1.14.1 h1:UQB4HGPB6osV0SQTLymcB4TgvyWu6ZyliaW0tI/otEQ= +github.com/rogpeppe/go-internal v1.14.1/go.mod h1:MaRKkUm5W0goXpeCfT7UZI6fk/L7L7so1lCWt35ZSgc= github.com/rs/xid v1.6.0/go.mod h1:7XoLgs4eV+QndskICGsho+ADou8ySMSjJKDIan90Nz0= github.com/rs/zerolog v1.34.0 h1:k43nTLIwcTVQAncfCw4KZ2VY6ukYoZaBPNOE8txlOeY= github.com/rs/zerolog v1.34.0/go.mod h1:bJsvje4Z08ROH4Nhs5iH600c3IkWhwp44iRc54W6wYQ= @@ -88,8 +114,8 @@ github.com/stretchr/testify v1.3.0/go.mod h1:M5WIy9Dh21IEIfnGCwXGc5bZfKNJtfHm1UV github.com/stretchr/testify v1.6.1/go.mod h1:6Fq8oRcR53rry900zMqJjRRixrwX3KX962/h/Wwjteg= github.com/stretchr/testify v1.7.0/go.mod h1:6Fq8oRcR53rry900zMqJjRRixrwX3KX962/h/Wwjteg= github.com/stretchr/testify v1.7.1/go.mod h1:6Fq8oRcR53rry900zMqJjRRixrwX3KX962/h/Wwjteg= -github.com/stretchr/testify v1.9.0 h1:HtqpIVDClZ4nwg75+f6Lvsy/wHu+3BoSGCbBAcpTsTg= -github.com/stretchr/testify v1.9.0/go.mod h1:r2ic/lqez/lEtzL7wO/rwa5dbSLXVDPFyf8C91i36aY= +github.com/stretchr/testify v1.10.0 h1:Xv5erBjTwe/5IxqUQTdXv5kgmIvbHo3QQyRwhJsOfJA= +github.com/stretchr/testify v1.10.0/go.mod h1:r2ic/lqez/lEtzL7wO/rwa5dbSLXVDPFyf8C91i36aY= github.com/tidwall/gjson v1.14.2 h1:6BBkirS0rAHjumnjHF6qgy5d2YAJ1TLIaFE2lzfOLqo= github.com/tidwall/gjson v1.14.2/go.mod h1:/wbyibRr2FHMks5tjHJ5F8dMZh3AcwJEMf5vlfC0lxk= github.com/tidwall/match v1.1.1 h1:+Ho715JplO36QYgwN9PGYNhgZvoUSc9X2c80KVTi+GA= @@ -100,27 +126,29 @@ github.com/tidwall/sjson v1.2.5 h1:kLy8mja+1c9jlljvWTlSazM7cKDRfJuR/bOJhcY5NcY= github.com/tidwall/sjson v1.2.5/go.mod h1:Fvgq9kS/6ociJEDnK0Fk1cpYF4FIW6ZF7LAe+6jwd28= go.uber.org/automaxprocs v1.6.0 h1:O3y2/QNTOdbF+e/dpXNNW7Rx2hZ4sTIPyybbxyNqTUs= go.uber.org/automaxprocs v1.6.0/go.mod h1:ifeIMSnPZuznNm6jmdzmU3/bfk01Fe2fotchwEFJ8r8= -go.yaml.in/yaml/v3 v3.0.4 h1:tfq32ie2Jv2UxXFdLJdh3jXuOzWiL1fo0bu/FbuKpbc= -go.yaml.in/yaml/v3 v3.0.4/go.mod h1:DhzuOOF2ATzADvBadXxruRBLzYTpT36CKvDb3+aBEFg= -golang.org/x/crypto v0.25.0 h1:ypSNr+bnYL2YhwoMt2zPxHFmbAN1KZs/njMG3hxUp30= -golang.org/x/crypto v0.25.0/go.mod h1:T+wALwcMOSE0kXgUAnPAHqTLW+XHgcELELW8VaDgm/M= +golang.org/x/crypto v0.39.0 h1:SHs+kF4LP+f+p14esP5jAoDpHU8Gu/v9lFRK6IT5imM= +golang.org/x/crypto v0.39.0/go.mod h1:L+Xg3Wf6HoL4Bn4238Z6ft6KfEpN0tJGo53AAPC632U= golang.org/x/exp v0.0.0-20240719175910-8a7402abbf56 h1:2dVuKD2vS7b0QIHQbpyTISPd0LeHDbnYEryqj5Q1ug8= golang.org/x/exp v0.0.0-20240719175910-8a7402abbf56/go.mod h1:M4RDyNAINzryxdtnbRXRL/OHtkFuWGRjvuhBJpk2IlY= -golang.org/x/net v0.37.0 h1:1zLorHbz+LYj7MQlSf1+2tPIIgibq2eL5xkrGk6f+2c= -golang.org/x/net v0.37.0/go.mod h1:ivrbrMbzFq5J41QOQh0siUuly180yBYtLp+CKbEaFx8= +golang.org/x/net v0.41.0 h1:vBTly1HeNPEn3wtREYfy4GZ/NECgw2Cnl+nK6Nz3uvw= +golang.org/x/net v0.41.0/go.mod h1:B/K4NNqkfmg07DQYrbwvSluqCJOOXwUjeb/5lOisjbA= +golang.org/x/sync v0.15.0 h1:KWH3jNZsfyT6xfAfKiz6MRNmd46ByHDYaZ7KSkCtdW8= +golang.org/x/sync v0.15.0/go.mod h1:1dzgHSNfp02xaA81J2MS99Qcpr2w7fw1gpm99rleRqA= golang.org/x/sys v0.0.0-20220715151400-c0bba94af5f8/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg= golang.org/x/sys v0.0.0-20220811171246-fbc7d0a398ab/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg= golang.org/x/sys v0.6.0/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg= golang.org/x/sys v0.12.0/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg= -golang.org/x/sys v0.32.0 h1:s77OFDvIQeibCmezSnk/q6iAfkdiQaJi4VzroCFrN20= -golang.org/x/sys v0.32.0/go.mod h1:BJP2sWEmIv4KK5OTEluFJCKSidICx8ciO85XgH3Ak8k= -golang.org/x/text v0.23.0 h1:D71I7dUrlY+VX0gQShAThNGHFxZ13dGLBHQLVl1mJlY= -golang.org/x/text v0.23.0/go.mod h1:/BLNzu4aZCJ1+kcD0DNRotWKage4q2rGVAg4o22unh4= -golang.org/x/tools v0.31.0 h1:0EedkvKDbh+qistFTd0Bcwe/YLh4vHwWEkiI0toFIBU= -golang.org/x/tools v0.31.0/go.mod h1:naFTU+Cev749tSJRXJlna0T3WxKvb1kWEx15xA4SdmQ= +golang.org/x/sys v0.33.0 h1:q3i8TbbEz+JRD9ywIRlyRAQbM0qF7hu24q3teo2hbuw= +golang.org/x/sys v0.33.0/go.mod h1:BJP2sWEmIv4KK5OTEluFJCKSidICx8ciO85XgH3Ak8k= +golang.org/x/text v0.26.0 h1:P42AVeLghgTYr4+xUnTRKDMqpar+PtX7KWuNQL21L8M= +golang.org/x/text v0.26.0/go.mod h1:QK15LZJUUQVJxhz7wXgxSy/CJaTFjd0G+YLonydOVQA= +golang.org/x/tools v0.33.0 h1:4qz2S3zmRxbGIhDIAgjxvFutSvH5EfnsYrRBj0UI0bc= +golang.org/x/tools v0.33.0/go.mod h1:CIJMaWEY88juyUfo7UbgPqbC8rU2OqfAV1h2Qp0oMYI= gopkg.in/check.v1 v0.0.0-20161208181325-20d25e280405/go.mod h1:Co6ibVJAznAaIkqp8huTwlJQCZ016jof/cbN4VW5Yz0= -gopkg.in/check.v1 v1.0.0-20190902080502-41f04d3bba15 h1:YR8cESwS4TdDjEe65xsg0ogRM/Nc3DYOhEAlW+xobZo= -gopkg.in/check.v1 v1.0.0-20190902080502-41f04d3bba15/go.mod h1:Co6ibVJAznAaIkqp8huTwlJQCZ016jof/cbN4VW5Yz0= +gopkg.in/check.v1 v1.0.0-20201130134442-10cb98267c6c h1:Hei/4ADfdWqJk1ZMxUNpqntNwaWcugrBjAiHlqqRiVk= +gopkg.in/check.v1 v1.0.0-20201130134442-10cb98267c6c/go.mod h1:JHkPIbrfpd72SG/EVd6muEfDQjcINNoR0C8j2r3qZ4Q= +gopkg.in/warnings.v0 v0.1.2 h1:wFXVbFY8DY5/xOe1ECiWdKCzZlxgshcYVNkBHstARME= +gopkg.in/warnings.v0 v0.1.2/go.mod h1:jksf8JmL6Qr/oQM2OXTHunEvvTAsrWBLb6OOjuVWRNI= gopkg.in/yaml.v3 v3.0.0-20200313102051-9f266ea9e77c/go.mod h1:K4uyk7z7BCEPqu6E+C64Yfv1cQ7kz7rIZviUmN+EgEM= gopkg.in/yaml.v3 v3.0.1 h1:fxVm/GzAzEWqLHuvctI91KS9hhNmmWOoWu0XTYJS7CA= gopkg.in/yaml.v3 v3.0.1/go.mod h1:K4uyk7z7BCEPqu6E+C64Yfv1cQ7kz7rIZviUmN+EgEM= From 760b386f2d14cd5bb1d635cdd67db6a472fd1d16 Mon Sep 17 00:00:00 2001 From: David Allen Date: Wed, 3 Sep 2025 17:37:29 -0600 Subject: [PATCH 12/34] chore: updated .gitignore --- .gitignore | 1 + 1 file changed, 1 insertion(+) diff --git a/.gitignore b/.gitignore index cc2fb4d..3c16074 100644 --- a/.gitignore +++ b/.gitignore @@ -11,3 +11,4 @@ tests/data tests/downloads tests/profiles tests/plugins +logs/ From 3ced4e4fd484e1fc3c2bfa5cca101c693cad1cc3 Mon Sep 17 00:00:00 2001 From: David Allen Date: Wed, 3 Sep 2025 17:38:00 -0600 Subject: [PATCH 13/34] docs: fixed examples and help output --- cmd/plugins.go | 10 +++++----- 1 file changed, 5 insertions(+), 5 deletions(-) diff --git a/cmd/plugins.go b/cmd/plugins.go index 99b8908..abda305 100644 --- a/cmd/plugins.go +++ b/cmd/plugins.go @@ -17,7 +17,7 @@ import ( var pluginsCmd = &cobra.Command{ Use: "plugins", - Short: "Manage, inspect, and compile plugins (requires Go build tools)", + Short: "Manage, inspect, and compile plugins (requires Go build tools and C compiler)", PersistentPreRun: func(cmd *cobra.Command, args []string) { setenv(cmd, "host", "MAKESHIFT_HOST") }, @@ -31,13 +31,13 @@ var pluginsCompileCmd = &cobra.Command{ # try to compile all plugins in current directory cd src/plugins - makeshift plugin compile + makeshift plugins compile # try to compile all plugins in specified directory - makeshift plugin compile src/plugins + makeshift plugins compile src/plugins # compile 'src/plugins/myplugin.go' and save to 'lib/myplugin.so' - makeshift plugin compile src/plugins/myplugin.go -o lib/myplugin.so + makeshift plugins compile src/plugins/myplugin.go -o lib/myplugin.so `, Args: cobra.MaximumNArgs(1), Run: func(cmd *cobra.Command, args []string) { @@ -97,7 +97,7 @@ var pluginsInspectCmd = &cobra.Command{ Args: cobra.MinimumNArgs(1), Example: ` # inspect a plugin and print its information - makeshift plugin inspect lib/jinja2.so + makeshift plugins inspect lib/jinja2.so `, Run: func(cmd *cobra.Command, args []string) { for _, path := range args { From cd0058dfa4ba2e2bfdd7219fe8912e12136db7b8 Mon Sep 17 00:00:00 2001 From: David Allen Date: Wed, 3 Sep 2025 17:38:41 -0600 Subject: [PATCH 14/34] fix: added logic to make directory for log file --- pkg/log/log.go | 7 +++++++ 1 file changed, 7 insertions(+) diff --git a/pkg/log/log.go b/pkg/log/log.go index 90ffa8d..95735b6 100644 --- a/pkg/log/log.go +++ b/pkg/log/log.go @@ -4,6 +4,7 @@ import ( "fmt" "io" "os" + "path/filepath" "slices" "strings" @@ -74,6 +75,12 @@ func InitWithLogLevel(logLevel LogLevel, logPath string) error { Level: level, }) + // make the log directory if necessary + err = os.MkdirAll(filepath.Dir(logPath), 0o777) + if err != nil { + return fmt.Errorf("failed to create log path directory: %v", err) + } + // add another writer to write to a log file if logPath != "" { LogFile, err = os.OpenFile(logPath, os.O_APPEND|os.O_CREATE|os.O_WRONLY, 0664) From 870107d680426b63205c6464a322553c22612496 Mon Sep 17 00:00:00 2001 From: David Allen Date: Wed, 3 Sep 2025 17:39:20 -0600 Subject: [PATCH 15/34] chore: updated help.txt to use spaces instead of tabs --- pkg/service/constants.go | 12 ++++++------ 1 file changed, 6 insertions(+), 6 deletions(-) diff --git a/pkg/service/constants.go b/pkg/service/constants.go index 68c2953..b94b34a 100644 --- a/pkg/service/constants.go +++ b/pkg/service/constants.go @@ -21,14 +21,14 @@ const (

Plugin Information: - Name: {{ makeshift.plugin.name }} - Version: {{ makeshift.plugin.version }} - Description: {{ makeshift.plugin.description }} - Author: {{ makeshift.plugin.metadata.name }} ({{ makeshift.plugin.metadata.email }}) + Name: {{ makeshift.plugin.name }} + Version: {{ makeshift.plugin.version }} + Description: {{ makeshift.plugin.description }} + Author: {{ makeshift.plugin.metadata.name }} ({{ makeshift.plugin.metadata.email }}) Profile Information: - ID: {{ makeshift.profiles.default.id }} - Description: {{ makeshift.profiles.default.description }} + ID: {{ makeshift.profiles.default.id }} + Description: {{ makeshift.profiles.default.description }} # setup environment variables
export MAKESHIFT_HOST={{ makeshift.profiles.default.data.host }}
From 2eee847205842a414080632f6fcdff3753b91053 Mon Sep 17 00:00:00 2001 From: David Allen Date: Wed, 3 Sep 2025 17:40:07 -0600 Subject: [PATCH 16/34] feat: added special case for --profile=all --- pkg/service/routes.go | 20 ++++++++++++++++++++ pkg/service/service.go | 36 ++++++++++++++++++++++++++++++++++++ 2 files changed, 56 insertions(+) diff --git a/pkg/service/routes.go b/pkg/service/routes.go index 555c115..5f552d1 100644 --- a/pkg/service/routes.go +++ b/pkg/service/routes.go @@ -8,6 +8,7 @@ import ( "net/http" "os" "path/filepath" + "slices" "strings" "git.towk2.me/towk/makeshift/internal/archive" @@ -315,6 +316,25 @@ func (s *Service) GetStatus(w http.ResponseWriter, r *http.Request) { } func (s *Service) loadProfiles(profileIDs []string, store storage.KVStore, errs []error) []error { + // check for special case profile ID (e.g. '*' or 'all') + useAll := slices.ContainsFunc(profileIDs, func(profileID string) bool { + return profileID == "*" || profileID == "all" + }) + if useAll { + var ( + dirpath = s.PathForProfiles() + profiles []*makeshift.Profile + err error + ) + profiles, err = LoadProfilesFromDir(dirpath) + if err != nil { + errs = append(errs, err) + return errs + } + store.Set("profiles", profiles) + return nil + } + // load data from profiles into the data store var profiles = make(makeshift.ProfileMap, len(profileIDs)) for i, profileID := range profileIDs { diff --git a/pkg/service/service.go b/pkg/service/service.go index 7197876..f0e0bae 100644 --- a/pkg/service/service.go +++ b/pkg/service/service.go @@ -136,6 +136,42 @@ func LoadProfileFromFile(path string) (*makeshift.Profile, error) { return loadFromJSONFile[makeshift.Profile](path) } +func LoadProfilesFromDir(dirpath string) ([]*makeshift.Profile, error) { + var ( + profiles []*makeshift.Profile + err error + ) + + // walk profiles directory to load all profiles + err = filepath.Walk(dirpath, func(path string, info fs.FileInfo, err error) error { + if err != nil { + return err + } + + // skip directories + if info.IsDir() { + return nil + } + + // read file contents + var profile *makeshift.Profile + profile, err = LoadProfileFromFile(path) + if err != nil { + return err + } + + profiles = append(profiles, profile) + + fmt.Println(path, info.Size()) + return nil + }) + if err != nil { + return nil, fmt.Errorf("failed to walk directory '%s': %v", dirpath, err) + } + + return profiles, nil +} + // LoadPluginFromFile loads a single plugin given a single file path func LoadPluginFromFile(path string) (makeshift.Plugin, error) { var ( From 215dbe8eff2fc4f342d1104bddde37f916f61b7e Mon Sep 17 00:00:00 2001 From: "David J. Allen" Date: Thu, 4 Sep 2025 10:07:10 -0600 Subject: [PATCH 17/34] feat: add util function to format error list --- pkg/util/util.go | 10 ++++++++++ 1 file changed, 10 insertions(+) diff --git a/pkg/util/util.go b/pkg/util/util.go index fc53b67..6f259a8 100644 --- a/pkg/util/util.go +++ b/pkg/util/util.go @@ -4,6 +4,7 @@ import ( "bytes" "cmp" "crypto/tls" + "errors" "fmt" "io" "net/http" @@ -97,3 +98,12 @@ func CopyIf[T comparable](s []T, condition func(t T) bool) []T { } return f } + +func FormatErrors(message string, prefix string, errs []error) error { + var errMessage = prefix + message + "\n" + for _, err := range errs { + errMessage = fmt.Sprintf("%s %v\n", prefix, err) + } + + return errors.New(errMessage) +} From 62b58f4cbb3b0a42e8c1ae67ae241efc41ec30ae Mon Sep 17 00:00:00 2001 From: "David J. Allen" Date: Thu, 4 Sep 2025 10:08:09 -0600 Subject: [PATCH 18/34] feat: change how incorrect profiles and plugins behave --- pkg/service/constants.go | 3 +-- pkg/service/routes.go | 22 ++++++++++++++++++---- 2 files changed, 19 insertions(+), 6 deletions(-) diff --git a/pkg/service/constants.go b/pkg/service/constants.go index b94b34a..649fa86 100644 --- a/pkg/service/constants.go +++ b/pkg/service/constants.go @@ -24,8 +24,7 @@ const ( Name: {{ makeshift.plugin.name }} Version: {{ makeshift.plugin.version }} Description: {{ makeshift.plugin.description }} - Author: {{ makeshift.plugin.metadata.name }} ({{ makeshift.plugin.metadata.email }}) - + Author: {{ makeshift.plugin.metadata }} Profile Information: ID: {{ makeshift.profiles.default.id }} Description: {{ makeshift.profiles.default.description }} diff --git a/pkg/service/routes.go b/pkg/service/routes.go index 5f552d1..1cbfc7f 100644 --- a/pkg/service/routes.go +++ b/pkg/service/routes.go @@ -15,6 +15,7 @@ import ( "git.towk2.me/towk/makeshift/internal/kwargs" makeshift "git.towk2.me/towk/makeshift/pkg" "git.towk2.me/towk/makeshift/pkg/storage" + "git.towk2.me/towk/makeshift/pkg/util" "github.com/go-chi/chi/v5" "github.com/rs/zerolog/log" ) @@ -56,7 +57,11 @@ func (s *Service) Download() http.HandlerFunc { // prepare profiles errs = s.loadProfiles(profileIDs, store, errs) if len(errs) > 0 { - log.Error().Errs("errs", errs).Msg("errors occurred loading profiles") + log.Error(). + Errs("errs", errs). + Msg("errors occurred loading profiles") + err = util.FormatErrors("failed to load plugins", "", errs) + http.Error(w, err.Error(), http.StatusInternalServerError) errs = []error{} } @@ -90,8 +95,13 @@ func (s *Service) Download() http.HandlerFunc { // prepare plugins hooks, errs = s.loadPlugins(pluginNames, store, pluginArgs, errs) if len(errs) > 0 { - log.Error().Errs("errs", errs).Msg("errors occurred loading plugins") + log.Error(). + Errs("errs", errs). + Msg("errors occurred loading plugins") + err = util.FormatErrors("failed to load plugins", "", errs) + http.Error(w, err.Error(), http.StatusInternalServerError) errs = []error{} + return } // create an archive of the directory, run hooks, and download @@ -132,12 +142,16 @@ func (s *Service) Download() http.HandlerFunc { } // prepare plugins - store.Set("file", contents) hooks, errs = s.loadPlugins(pluginNames, store, nil, errs) if len(errs) > 0 { - log.Error().Errs("errs", errs).Msg("errors occurred loading plugins") + log.Error(). + Errs("errs", errs). + Msg("errors occurred loading plugins") + err = util.FormatErrors("failed to load plugins", "", errs) + http.Error(w, err.Error(), http.StatusInternalServerError) errs = []error{} + return } if len(hooks) > 0 { // run pre-hooks to modify the contents of the file before archiving From 23d43061fb88515c009bdc156948a35f88c5ab2d Mon Sep 17 00:00:00 2001 From: "David J. Allen" Date: Thu, 4 Sep 2025 10:08:40 -0600 Subject: [PATCH 19/34] refactor: update pyjinja2 implementation --- pkg/plugins/pyjinja2/pyjinja2.go | 12 +++++++++--- 1 file changed, 9 insertions(+), 3 deletions(-) diff --git a/pkg/plugins/pyjinja2/pyjinja2.go b/pkg/plugins/pyjinja2/pyjinja2.go index fe823f2..673f1a1 100644 --- a/pkg/plugins/pyjinja2/pyjinja2.go +++ b/pkg/plugins/pyjinja2/pyjinja2.go @@ -75,6 +75,7 @@ func (p *Jinja2) Run(store storage.KVStore, args []string) error { } } + // get mappings from provided profiles var ps = make(map[string]any) for profileID, profile := range profiles.(makeshift.ProfileMap) { ps[profileID] = map[string]any{ @@ -84,8 +85,9 @@ func (p *Jinja2) Run(store storage.KVStore, args []string) error { } } - j2, err := jinja2.NewJinja2("pyjinja", 1, - jinja2.WithGlobal("makeshift", map[string]any{ + // get mappings supplied by this plugin + mappings.Data = map[string]any{ + "makeshift": map[string]any{ "profiles": ps, "plugin": map[string]any{ "name": p.Name(), @@ -93,7 +95,11 @@ func (p *Jinja2) Run(store storage.KVStore, args []string) error { "description": p.Description(), "metadata": p.Metadata(), }, - }), + }, + } + + j2, err := jinja2.NewJinja2("makeshift", 1, + jinja2.WithGlobals(mappings.Data), ) if err != nil { return fmt.Errorf("(pyjinja2) failed to create new Jinja 2 instance: %v", err) From 224df9ef7adb90a56a7e3752c7439ca46cf201cf Mon Sep 17 00:00:00 2001 From: "David J. Allen" Date: Thu, 4 Sep 2025 10:18:15 -0600 Subject: [PATCH 20/34] refactor: changed where index.html is created --- pkg/service/constants.go | 2 +- pkg/service/service.go | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/pkg/service/constants.go b/pkg/service/constants.go index 649fa86..3f99e16 100644 --- a/pkg/service/constants.go +++ b/pkg/service/constants.go @@ -5,7 +5,7 @@ const ( RELPATH_PROFILES = "/profiles" RELPATH_DATA = "/data" RELPATH_METADATA = "/.makeshift" - RELPATH_HELP = RELPATH_DATA + "/index.html" + RELPATH_HELP = RELPATH_DATA + "/www/index.html" RELPATH_PROFILE = RELPATH_PROFILES + "/default.json" PATH_CONFIG = "$HOME/.config/makeshift/config.yaml" diff --git a/pkg/service/service.go b/pkg/service/service.go index f0e0bae..1eb26f3 100644 --- a/pkg/service/service.go +++ b/pkg/service/service.go @@ -57,7 +57,7 @@ func (s *Service) Init() error { if err != nil { return fmt.Errorf("failed to make service profile path: %v", err) } - err = os.MkdirAll(s.PathForData(), 0o777) + err = os.MkdirAll(s.PathForData()+"/www", 0o777) if err != nil { return fmt.Errorf("failed to make service data path: %v", err) } From 568d3e21a6b0cee6d0806053b29576b59e3deae8 Mon Sep 17 00:00:00 2001 From: David Allen Date: Thu, 4 Sep 2025 18:23:17 -0600 Subject: [PATCH 21/34] chore: updated license file --- LICENSE | 30 ++++++++++++------------------ 1 file changed, 12 insertions(+), 18 deletions(-) diff --git a/LICENSE b/LICENSE index 85c3d9c..2c909b6 100644 --- a/LICENSE +++ b/LICENSE @@ -1,21 +1,15 @@ -MIT License +Copyright (C) 2015 David J. Allen -Copyright © 2025 David J. Allen +This program is free software; you can redistribute it and/or +modify it under the terms of the GNU General Public License +as published by the Free Software Foundation; either version 2 +of the License, or (at your option) any later version. -Permission is hereby granted, free of charge, to any person obtaining a -copy of this software and associated documentation files (the "Software"), -to deal in the Software without restriction, including without limitation -the rights to use, copy, modify, merge, publish, distribute, sublicense, -and/or sell copies of the Software, and to permit persons to whom the -Software is furnished to do so, subject to the following conditions: +This program is distributed in the hope that it will be useful, +but WITHOUT ANY WARRANTY; without even the implied warranty of +MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +GNU General Public License for more details. -The above copyright notice and this permission notice shall be included -in all copies or substantial portions of the Software. - -THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR -IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, -FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL -THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR -OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, -ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR -OTHER DEALINGS IN THE SOFTWARE. +You should have received a copy of the GNU General Public License +along with this program; if not, see +. \ No newline at end of file From d408893389098e23f9ec252ff24a07c53c8f595e Mon Sep 17 00:00:00 2001 From: David Allen Date: Fri, 5 Sep 2025 22:21:00 -0600 Subject: [PATCH 22/34] refactor: small changes and tweaks --- cmd/root.go | 9 +++++++++ pkg/service/routes.go | 10 +++++----- 2 files changed, 14 insertions(+), 5 deletions(-) diff --git a/cmd/root.go b/cmd/root.go index fb354f0..f5b1fc6 100644 --- a/cmd/root.go +++ b/cmd/root.go @@ -118,3 +118,12 @@ func handleResponseError(res *http.Response, host, query string, err error) { os.Exit(1) } } + +// helper to write downloaded files +func writeFiles(path string, body []byte) { + var err = os.WriteFile(path, body, 0o755) + if err != nil { + log.Error().Err(err).Msg("failed to write file(s) from download") + os.Exit(1) + } +} diff --git a/pkg/service/routes.go b/pkg/service/routes.go index 1cbfc7f..e4fdb04 100644 --- a/pkg/service/routes.go +++ b/pkg/service/routes.go @@ -55,7 +55,7 @@ func (s *Service) Download() http.HandlerFunc { Msg("Service.Download()") // prepare profiles - errs = s.loadProfiles(profileIDs, store, errs) + errs = s.LoadProfiles(profileIDs, store, errs) if len(errs) > 0 { log.Error(). Errs("errs", errs). @@ -93,7 +93,7 @@ func (s *Service) Download() http.HandlerFunc { log.Debug().Strs("files", filenamesToArchive).Send() // prepare plugins - hooks, errs = s.loadPlugins(pluginNames, store, pluginArgs, errs) + hooks, errs = s.LoadPlugins(pluginNames, store, pluginArgs, errs) if len(errs) > 0 { log.Error(). Errs("errs", errs). @@ -143,7 +143,7 @@ func (s *Service) Download() http.HandlerFunc { // prepare plugins store.Set("file", contents) - hooks, errs = s.loadPlugins(pluginNames, store, nil, errs) + hooks, errs = s.LoadPlugins(pluginNames, store, pluginArgs, errs) if len(errs) > 0 { log.Error(). Errs("errs", errs). @@ -329,7 +329,7 @@ func (s *Service) GetStatus(w http.ResponseWriter, r *http.Request) { } } -func (s *Service) loadProfiles(profileIDs []string, store storage.KVStore, errs []error) []error { +func (s *Service) LoadProfiles(profileIDs []string, store storage.KVStore, errs []error) []error { // check for special case profile ID (e.g. '*' or 'all') useAll := slices.ContainsFunc(profileIDs, func(profileID string) bool { return profileID == "*" || profileID == "all" @@ -381,7 +381,7 @@ func (s *Service) loadProfiles(profileIDs []string, store storage.KVStore, errs return errs } -func (s *Service) loadPlugins(pluginNames []string, store storage.KVStore, args []string, errs []error) ([]makeshift.Hook, []error) { +func (s *Service) LoadPlugins(pluginNames []string, store storage.KVStore, args []string, errs []error) ([]makeshift.Hook, []error) { // create hooks to run from provided plugins specified var hooks []makeshift.Hook for i, pluginName := range pluginNames { From 4d55a3edc21b2a6382a9dbdc264b5b1e601a537a Mon Sep 17 00:00:00 2001 From: David Allen Date: Fri, 5 Sep 2025 22:21:27 -0600 Subject: [PATCH 23/34] feat: initial implementation of run cmd --- cmd/run.go | 277 ++++++++++++++++++++++++++++++++++++++++++++++++++++- 1 file changed, 275 insertions(+), 2 deletions(-) diff --git a/cmd/run.go b/cmd/run.go index ab698e7..846e560 100644 --- a/cmd/run.go +++ b/cmd/run.go @@ -1,6 +1,22 @@ package cmd -import "github.com/spf13/cobra" +import ( + "fmt" + "io/fs" + "net/url" + "os" + "path/filepath" + "strings" + "time" + + "git.towk2.me/towk/makeshift/internal/archive" + makeshift "git.towk2.me/towk/makeshift/pkg" + "git.towk2.me/towk/makeshift/pkg/service" + "git.towk2.me/towk/makeshift/pkg/storage" + "git.towk2.me/towk/makeshift/pkg/util" + "github.com/rs/zerolog/log" + "github.com/spf13/cobra" +) var runCmd = &cobra.Command{ Use: "run", @@ -13,16 +29,273 @@ var runCmd = &cobra.Command{ export MAKESHIFT_ROOT=/opt/makeshift # run locally similar to 'download' - makeshift run --plugins jinja2 --profiles default + makeshift run -p help.txt --plugins jinja2 --profiles default makeshift run --root $HOME/apps/makeshift -p help.txt --plugins jinja2 --profiles default `, Args: cobra.NoArgs, Short: "Run locally with plugins and profiles", Run: func(cmd *cobra.Command, args []string) { + var ( + host, _ = cmd.Flags().GetString("host") + path, _ = cmd.Flags().GetString("path") + rootPath, _ = cmd.Flags().GetString("root") + outputPath, _ = cmd.Flags().GetString("output") + cacertPath, _ = cmd.Flags().GetString("cacert") + keyfile, _ = cmd.Flags().GetString("keyfile") + timeout, _ = cmd.Flags().GetInt("timeout") + pluginNames, _ = cmd.Flags().GetStringSlice("plugins") + profileIDs, _ = cmd.Flags().GetStringSlice("profiles") + extract, _ = cmd.Flags().GetBool("extract") + removeArchive, _ = cmd.Flags().GetBool("remove-archive") + contents []byte + parsed *url.URL + localServer *service.Service + fileInfo os.FileInfo + out *os.File + store *storage.MemoryStorage = new(storage.MemoryStorage) + hooks []makeshift.Hook + errs []error + err error + ) + + // parse the host to remove scheme if needed + parsed, err = url.Parse(host) + if err != nil { + log.Warn().Err(err). + Str("host", host). + Msg("could not parse host") + } + + // set the server values + localServer = service.New() + localServer.Addr = parsed.Host + localServer.RootPath = rootPath + localServer.CACertFile = cacertPath + localServer.CACertKeyfile = keyfile + localServer.Timeout = time.Duration(timeout) * time.Second + + // initialize storage + store.Init() + store.SetKWArgs(&pluginKWArgs) + + // prepare the profiles + errs = localServer.LoadProfiles(profileIDs, store, errs) + if len(errs) > 0 { + log.Error(). + Errs("errs", errs). + Msg("errors occurred loading profiles") + err = util.FormatErrors("failed to load plugins", "", errs) + errs = []error{} + } + + // prepare the plugins + // determine if path is directory, file, or exists + if fileInfo, err = os.Stat(path); err == nil { + if fileInfo.IsDir() { + // get the final archive path + archivePath := fmt.Sprintf("%s.tar.gz", path) + + log.Debug(). + Str("archive_path", archivePath). + Str("type", "directory"). + Msg("Service.Download()") + + out, err = os.Create(archivePath) + if err != nil { + log.Error(). + Err(err). + Str("path", archivePath). + Msg("failed to create named file") + return + } + + // get a list of filenames to archive + filenamesToArchive := []string{} + filepath.WalkDir(path, func(path string, d fs.DirEntry, err error) error { + if !d.IsDir() { + filenamesToArchive = append(filenamesToArchive, path) + } + return nil + }) + log.Debug().Strs("files", filenamesToArchive).Send() + + // prepare plugins + hooks, errs = localServer.LoadPlugins(pluginNames, store, pluginArgs, errs) + if len(errs) > 0 { + log.Error(). + Errs("errs", errs). + Strs("plugins", pluginNames). + Strs("args", pluginArgs). + Msg("errors occurred loading plugins") + errs = []error{} + return + } + + // create an archive of the directory, run hooks, and download + err = archive.Create(filenamesToArchive, out, hooks) + if err != nil { + log.Error(). + Err(err). + Str("path", archivePath). + Msg("failed to create archive") + return + } + + // load the final archive + contents, err = os.ReadFile(archivePath) + if err != nil { + log.Error(). + Err(err). + Str("path", archivePath). + Msg("failed to read archive contents") + return + } + + // clean up the temporary archive + err = os.Remove(archivePath) + if err != nil { + log.Error().Err(err).Msg("failed to remove temporary archive") + return + } + + // extract files if '-x' flag is passed + if extract { + var ( + dir = filepath.Dir(outputPath) + base = strings.TrimSuffix(filepath.Base(outputPath), ".tar.gz") + ) + err = archive.Expand(outputPath, fmt.Sprintf("%s/%s", dir, base)) + if err != nil { + log.Error().Err(err). + Str("path", outputPath). + Msg("failed to expand archive") + os.Exit(1) + } + } + + // optionally, remove archive if '-r' flag is passed + // NOTE: this can only be used if `-x` flag is set + if removeArchive { + if !extract { + log.Warn().Msg("requires '-x/--extract' flag to be set to 'true'") + } else { + err = os.Remove(outputPath) + if err != nil { + log.Error().Err(err). + Str("path", outputPath). + Msg("failed to remove archive") + } + } + } + + } else { + // download individual file + log.Debug(). + Str("type", "file"). + Msg("Service.Download()") + + contents, err = os.ReadFile(path) + if err != nil { + log.Error(). + Err(err). + Msg("failed to read file to download") + return + } + + // prepare plugins + store.Set("file", contents) + hooks, errs = localServer.LoadPlugins(pluginNames, store, pluginArgs, errs) + if len(errs) > 0 { + log.Error(). + Strs("plugins", pluginNames). + Strs("args", pluginArgs). + Errs("errs", errs). + Msg("errors occurred loading plugins") + errs = []error{} + return + } + if len(hooks) > 0 { + // run pre-hooks to modify the contents of the file before archiving + log.Debug().Int("hook_count", len(hooks)).Msg("running hooks") + for _, hook := range hooks { + log.Debug().Any("hook", map[string]any{ + "store": hook.Data, + "args": hook.Args, + "plugin": map[string]string{ + "name": hook.Plugin.Name(), + "description": hook.Plugin.Description(), + "version": hook.Plugin.Version(), + }, + }).Send() + err = hook.Init() + if err != nil { + log.Error(). + Err(err). + Str("plugin", hook.Plugin.Name()). + Msg("failed to initialize plugin") + continue + } + err = hook.Run() + if err != nil { + log.Error(). + Err(err). + Str("plugin", hook.Plugin.Name()). + Msg("failed to run plugin") + continue + } + err = hook.Cleanup() + if err != nil { + log.Error(). + Err(err). + Str("plugin", hook.Plugin.Name()). + Msg("failed to cleanup plugin") + continue + } + } + + // take the contents from the last hook and update files + var ( + hook = hooks[len(hooks)-1] + data any + ) + data, err = hook.Data.Get("out") + if err != nil { + log.Error(). + Err(err). + Str("plugin", hook.Plugin.Name()). + Msg("failed to get data from hook") + return + } + + // write to file if '-o' specified otherwise stdout + if outputPath != "" { + writeFiles(outputPath, data.([]byte)) + log.Debug().Str("path", outputPath).Msg("wrote file to specified path") + } else { + fmt.Println(string(data.([]byte))) + } + + } else { + // write contents to file + // send non-processed file back as response + + } + } + } }, } func init() { + runCmd.PersistentFlags().String("host", "http://localhost:5050", "Set the makeshift remote host (can be set with MAKESHIFT_HOST)") + runCmd.PersistentFlags().StringP("output", "o", "", "Set the output path to write files") + runCmd.PersistentFlags().String("cacert", "", "Set the CA certificate path to load") + runCmd.Flags().StringP("path", "p", ".", "Set the path to list files (can be set with MAKESHIFT_PATH)") + runCmd.Flags().StringSlice("profiles", []string{}, "Set the profile(s) to use to populate data store") + runCmd.Flags().StringSlice("plugins", []string{}, "Set the plugin(s) to run before downloading files") + runCmd.Flags().StringSlice("plugin-args", []string{}, "Set the argument list to pass to plugin(s)") + runCmd.Flags().Var(&pluginKWArgs, "plugin-kwargs", "Set the argument map to pass to plugin(s)") + runCmd.Flags().BoolP("extract", "x", false, "Set whether to extract archive locally after downloading") + runCmd.Flags().BoolP("remove-archive", "r", false, "Set whether to remove the archive after extracting (used with '--extract' flag)") rootCmd.AddCommand(runCmd) } From 773dc556cdac864cb6fdabc0ae4895861a715bd8 Mon Sep 17 00:00:00 2001 From: David Allen Date: Sun, 7 Sep 2025 14:43:45 -0600 Subject: [PATCH 24/34] feat: added config implementation using viper --- cmd/download.go | 9 -------- cmd/root.go | 61 +++++++++++++++++++++++++++++++++++++++++++++++++ 2 files changed, 61 insertions(+), 9 deletions(-) diff --git a/cmd/download.go b/cmd/download.go index 0111b53..fdb2233 100644 --- a/cmd/download.go +++ b/cmd/download.go @@ -307,12 +307,3 @@ func init() { rootCmd.AddCommand(&downloadCmd) } - -// helper to write downloaded files -func writeFiles(path string, body []byte) { - var err = os.WriteFile(path, body, 0o755) - if err != nil { - log.Error().Err(err).Msg("failed to write file(s) from download") - os.Exit(1) - } -} diff --git a/cmd/root.go b/cmd/root.go index f5b1fc6..9f233f6 100644 --- a/cmd/root.go +++ b/cmd/root.go @@ -4,10 +4,13 @@ import ( "fmt" "net/http" "os" + "strings" logger "git.towk2.me/towk/makeshift/pkg/log" "github.com/rs/zerolog/log" "github.com/spf13/cobra" + "github.com/spf13/pflag" + "github.com/spf13/viper" ) var ( @@ -30,6 +33,9 @@ var rootCmd = cobra.Command{ log.Error().Err(err).Msg("failed to initialize logger") os.Exit(1) } + + // You can bind cobra and viper in a few locations, but PersistencePreRunE on the root command works well + err = initializeConfig(cmd) }, Run: func(cmd *cobra.Command, args []string) { // try and set flags using env vars @@ -127,3 +133,58 @@ func writeFiles(path string, body []byte) { os.Exit(1) } } + +// Bind each cobra flag to its associated viper configuration (config file and environment variable) +func bindFlags(cmd *cobra.Command, v *viper.Viper) { + cmd.Flags().VisitAll(func(f *pflag.Flag) { + // Determine the naming convention of the flags when represented in the config file + configName := f.Name + + // Apply the viper config value to the flag when the flag is not set and viper has a value + if !f.Changed && v.IsSet(configName) { + val := v.Get(configName) + cmd.Flags().Set(f.Name, fmt.Sprintf("%v", val)) + } + }) +} + +func initializeConfig(cmd *cobra.Command) error { + v := viper.New() + + // Set the base name of the config file, without the file extension. + v.SetConfigName("makeshift") + + // Set as many paths as you like where viper should look for the + // config file. We are only looking in the current working directory. + v.AddConfigPath(".") + + // Attempt to read the config file, gracefully ignoring errors + // caused by a config file not being found. Return an error + // if we cannot parse the config file. + if err := v.ReadInConfig(); err != nil { + // It's okay if there isn't a config file + if _, ok := err.(viper.ConfigFileNotFoundError); !ok { + return err + } + } + + // When we bind flags to environment variables expect that the + // environment variables are prefixed, e.g. a flag like --number + // binds to an environment variable STING_NUMBER. This helps + // avoid conflicts. + v.SetEnvPrefix("MAKESHIFT") + + // Environment variables can't have dashes in them, so bind them to their equivalent + // keys with underscores, e.g. --favorite-color to STING_FAVORITE_COLOR + v.SetEnvKeyReplacer(strings.NewReplacer("-", "_")) + + // Bind to environment variables + // Works great for simple config names, but needs help for names + // like --favorite-color which we fix in the bindFlags function + v.AutomaticEnv() + + // Bind the current command's flags to viper + bindFlags(cmd, v) + + return nil +} From a08f9ce5a090545dbe5e30813697c039447cdc68 Mon Sep 17 00:00:00 2001 From: David Allen Date: Sun, 7 Sep 2025 14:43:58 -0600 Subject: [PATCH 25/34] chore: updated go deps --- go.mod | 13 ++++++++++++- go.sum | 27 ++++++++++++++++++++++++++- 2 files changed, 38 insertions(+), 2 deletions(-) diff --git a/go.mod b/go.mod index 5f74103..f1b8bd6 100644 --- a/go.mod +++ b/go.mod @@ -13,6 +13,8 @@ require ( github.com/nikolalohinski/gonja/v2 v2.3.5 github.com/rs/zerolog v1.34.0 github.com/spf13/cobra v1.8.0 + github.com/spf13/pflag v1.0.6 + github.com/spf13/viper v1.20.1 github.com/tidwall/sjson v1.2.5 golang.org/x/exp v0.0.0-20240719175910-8a7402abbf56 gopkg.in/yaml.v3 v3.0.1 @@ -22,9 +24,11 @@ require ( github.com/Masterminds/semver/v3 v3.4.0 // indirect github.com/decred/dcrd/dcrec/secp256k1/v4 v4.3.0 // indirect github.com/dustin/go-humanize v1.0.1 // indirect + github.com/fsnotify/fsnotify v1.8.0 // indirect github.com/go-git/gcfg v1.5.1-0.20230307220236-3a3c6141e376 // indirect github.com/go-git/go-billy/v5 v5.6.2 // indirect github.com/go-git/go-git/v5 v5.16.2 // indirect + github.com/go-viper/mapstructure/v2 v2.2.1 // indirect github.com/gobwas/glob v0.2.3 // indirect github.com/goccy/go-json v0.10.3 // indirect github.com/gofrs/flock v0.12.1 // indirect @@ -44,13 +48,20 @@ require ( github.com/mattn/go-isatty v0.0.20 // indirect github.com/modern-go/concurrent v0.0.0-20180306012644-bacd9c7ef1dd // indirect github.com/modern-go/reflect2 v1.0.2 // indirect + github.com/pelletier/go-toml/v2 v2.2.3 // indirect github.com/pkg/errors v0.9.1 // indirect + github.com/sagikazarmark/locafero v0.7.0 // indirect github.com/segmentio/asm v1.2.0 // indirect github.com/sirupsen/logrus v1.9.3 // indirect - github.com/spf13/pflag v1.0.5 // indirect + github.com/sourcegraph/conc v0.3.0 // indirect + github.com/spf13/afero v1.12.0 // indirect + github.com/spf13/cast v1.7.1 // indirect + github.com/subosito/gotenv v1.6.0 // indirect github.com/tidwall/gjson v1.14.2 // indirect github.com/tidwall/match v1.1.1 // indirect github.com/tidwall/pretty v1.2.0 // indirect + go.uber.org/atomic v1.9.0 // indirect + go.uber.org/multierr v1.9.0 // indirect golang.org/x/crypto v0.39.0 // indirect golang.org/x/net v0.41.0 // indirect golang.org/x/sync v0.15.0 // indirect diff --git a/go.sum b/go.sum index fe24c0b..74f2e52 100644 --- a/go.sum +++ b/go.sum @@ -16,6 +16,10 @@ github.com/decred/dcrd/dcrec/secp256k1/v4 v4.3.0 h1:rpfIENRNNilwHwZeG5+P150SMrnN github.com/decred/dcrd/dcrec/secp256k1/v4 v4.3.0/go.mod h1:v57UDF4pDQJcEfFUCRop3lJL149eHGSe9Jvczhzjo/0= github.com/dustin/go-humanize v1.0.1 h1:GzkhY7T5VNhEkwH0PVJgjz+fX1rhBrR7pRT3mDkpeCY= github.com/dustin/go-humanize v1.0.1/go.mod h1:Mu1zIs6XwVuF/gI1OepvI0qD18qycQx+mFykh5fBlto= +github.com/frankban/quicktest v1.14.6 h1:7Xjx+VpznH+oBnejlPUj8oUpdxnVs4f8XU8WnHkI4W8= +github.com/frankban/quicktest v1.14.6/go.mod h1:4ptaffx2x8+WTWXmUCuVU6aPUX1/Mz7zb5vbUoiM6w0= +github.com/fsnotify/fsnotify v1.8.0 h1:dAwr6QBTBZIkG8roQaJjGof0pp0EeF+tNV7YBP3F/8M= +github.com/fsnotify/fsnotify v1.8.0/go.mod h1:8jBTzvmWwFyi3Pb8djgCCO5IBqzKJ/Jwo8TRcHyHii0= github.com/go-chi/chi/v5 v5.1.0 h1:acVI1TYaD+hhedDJ3r54HyA6sExp3HfXq7QWEEY/xMw= github.com/go-chi/chi/v5 v5.1.0/go.mod h1:DslCQbL2OYiznFReuXYUmQ2hGd1aDpCnlMNITLSKoi8= github.com/go-git/gcfg v1.5.1-0.20230307220236-3a3c6141e376 h1:+zs/tPmkDkHx3U66DAb0lQFJrpS6731Oaa12ikc+DiI= @@ -28,6 +32,8 @@ github.com/go-logr/logr v1.4.2 h1:6pFjapn8bFcIbiKo3XT4j/BhANplGihG6tvd+8rYgrY= github.com/go-logr/logr v1.4.2/go.mod h1:9T104GzyrTigFIr8wt5mBrctHMim0Nb2HLGrmQ40KvY= github.com/go-task/slim-sprig/v3 v3.0.0 h1:sUs3vkvUymDpBKi3qH1YSqBQk9+9D/8M2mN1vB6EwHI= github.com/go-task/slim-sprig/v3 v3.0.0/go.mod h1:W848ghGpv3Qj3dhTPRyJypKRiqCdHZiAzKg9hl15HA8= +github.com/go-viper/mapstructure/v2 v2.2.1 h1:ZAaOCxANMuZx5RCeg0mBdEZk7DZasvvZIxtHqx8aGss= +github.com/go-viper/mapstructure/v2 v2.2.1/go.mod h1:oJDH3BJKyqBA2TXFhDsKDGDTlndYOZ6rGS0BRZIxGhM= github.com/gobwas/glob v0.2.3 h1:A4xDbljILXROh+kObIiy5kIaPYD8e96x1tgBhUI5J+Y= github.com/gobwas/glob v0.2.3/go.mod h1:d3Ez4x06l9bZtSvzIay5+Yzi0fmZzPgnTbPcKjJAkT8= github.com/goccy/go-json v0.10.3 h1:KZ5WoDbxAIgm2HNbYckL0se1fHD6rz5j4ywS6ebzDqA= @@ -90,6 +96,8 @@ github.com/onsi/ginkgo/v2 v2.23.4 h1:ktYTpKJAVZnDT4VjxSbiBenUjmlL/5QkBEocaWXiQus github.com/onsi/ginkgo/v2 v2.23.4/go.mod h1:Bt66ApGPBFzHyR+JO10Zbt0Gsp4uWxu5mIOTusL46e8= github.com/onsi/gomega v1.37.0 h1:CdEG8g0S133B4OswTDC/5XPSzE1OeP29QOioj2PID2Y= github.com/onsi/gomega v1.37.0/go.mod h1:8D9+Txp43QWKhM24yyOBEdpkzN8FvJyAwecBgsU4KU0= +github.com/pelletier/go-toml/v2 v2.2.3 h1:YmeHyLY8mFWbdkNWwpr+qIL2bEqT0o95WSdkNHvL12M= +github.com/pelletier/go-toml/v2 v2.2.3/go.mod h1:MfCQTFTvCcUyyvvwm1+G6H/jORL20Xlb6rzQu9GuUkc= github.com/pkg/errors v0.9.1 h1:FEBLx1zS214owpjy7qsBeixbURkuhQAwrK5UwLGTwt4= github.com/pkg/errors v0.9.1/go.mod h1:bwawxfHBFNV+L2hUp1rHADufV3IMtnDRdf1r5NINEl0= github.com/pmezard/go-difflib v1.0.0/go.mod h1:iKH77koFhYxTK1pcRnkKkqfTogsbg7gZNVY4sRDYZ/4= @@ -101,14 +109,25 @@ github.com/rs/xid v1.6.0/go.mod h1:7XoLgs4eV+QndskICGsho+ADou8ySMSjJKDIan90Nz0= github.com/rs/zerolog v1.34.0 h1:k43nTLIwcTVQAncfCw4KZ2VY6ukYoZaBPNOE8txlOeY= github.com/rs/zerolog v1.34.0/go.mod h1:bJsvje4Z08ROH4Nhs5iH600c3IkWhwp44iRc54W6wYQ= github.com/russross/blackfriday/v2 v2.1.0/go.mod h1:+Rmxgy9KzJVeS9/2gXHxylqXiyQDYRxCVz55jmeOWTM= +github.com/sagikazarmark/locafero v0.7.0 h1:5MqpDsTGNDhY8sGp0Aowyf0qKsPrhewaLSsFaodPcyo= +github.com/sagikazarmark/locafero v0.7.0/go.mod h1:2za3Cg5rMaTMoG/2Ulr9AwtFaIppKXTRYnozin4aB5k= github.com/segmentio/asm v1.2.0 h1:9BQrFxC+YOHJlTlHGkTrFWf59nbL3XnCoFLTwDCI7ys= github.com/segmentio/asm v1.2.0/go.mod h1:BqMnlJP91P8d+4ibuonYZw9mfnzI9HfxselHZr5aAcs= github.com/sirupsen/logrus v1.9.3 h1:dueUQJ1C2q9oE3F7wvmSGAaVtTmUizReu6fjN8uqzbQ= github.com/sirupsen/logrus v1.9.3/go.mod h1:naHLuLoDiP4jHNo9R0sCBMtWGeIprob74mVsIT4qYEQ= +github.com/sourcegraph/conc v0.3.0 h1:OQTbbt6P72L20UqAkXXuLOj79LfEanQ+YQFNpLA9ySo= +github.com/sourcegraph/conc v0.3.0/go.mod h1:Sdozi7LEKbFPqYX2/J+iBAM6HpqSLTASQIKqDmF7Mt0= +github.com/spf13/afero v1.12.0 h1:UcOPyRBYczmFn6yvphxkn9ZEOY65cpwGKb5mL36mrqs= +github.com/spf13/afero v1.12.0/go.mod h1:ZTlWwG4/ahT8W7T0WQ5uYmjI9duaLQGy3Q2OAl4sk/4= +github.com/spf13/cast v1.7.1 h1:cuNEagBQEHWN1FnbGEjCXL2szYEXqfJPbP2HNUaca9Y= +github.com/spf13/cast v1.7.1/go.mod h1:ancEpBxwJDODSW/UG4rDrAqiKolqNNh2DX3mk86cAdo= github.com/spf13/cobra v1.8.0 h1:7aJaZx1B85qltLMc546zn58BxxfZdR/W22ej9CFoEf0= github.com/spf13/cobra v1.8.0/go.mod h1:WXLWApfZ71AjXPya3WOlMsY9yMs7YeiHhFVlvLyhcho= -github.com/spf13/pflag v1.0.5 h1:iy+VFUOCP1a+8yFto/drg2CJ5u0yRoB7fZw3DKv/JXA= github.com/spf13/pflag v1.0.5/go.mod h1:McXfInJRrz4CZXVZOBLb0bTZqETkiAhM9Iw0y3An2Bg= +github.com/spf13/pflag v1.0.6 h1:jFzHGLGAlb3ruxLB8MhbI6A8+AQX/2eW4qeyNZXNp2o= +github.com/spf13/pflag v1.0.6/go.mod h1:McXfInJRrz4CZXVZOBLb0bTZqETkiAhM9Iw0y3An2Bg= +github.com/spf13/viper v1.20.1 h1:ZMi+z/lvLyPSCoNtFCpqjy0S4kPbirhpTMwl8BkW9X4= +github.com/spf13/viper v1.20.1/go.mod h1:P9Mdzt1zoHIG8m2eZQinpiBjo6kCmZSKBClNNqjJvu4= github.com/stretchr/objx v0.1.0/go.mod h1:HFkY916IF+rwdDfMAkV7OtwuqBVzrE8GR6GFx+wExME= github.com/stretchr/testify v1.3.0/go.mod h1:M5WIy9Dh21IEIfnGCwXGc5bZfKNJtfHm1UVUgZn+9EI= github.com/stretchr/testify v1.6.1/go.mod h1:6Fq8oRcR53rry900zMqJjRRixrwX3KX962/h/Wwjteg= @@ -116,6 +135,8 @@ github.com/stretchr/testify v1.7.0/go.mod h1:6Fq8oRcR53rry900zMqJjRRixrwX3KX962/ github.com/stretchr/testify v1.7.1/go.mod h1:6Fq8oRcR53rry900zMqJjRRixrwX3KX962/h/Wwjteg= github.com/stretchr/testify v1.10.0 h1:Xv5erBjTwe/5IxqUQTdXv5kgmIvbHo3QQyRwhJsOfJA= github.com/stretchr/testify v1.10.0/go.mod h1:r2ic/lqez/lEtzL7wO/rwa5dbSLXVDPFyf8C91i36aY= +github.com/subosito/gotenv v1.6.0 h1:9NlTDc1FTs4qu0DDq7AEtTPNw6SVm7uBMsUCUjABIf8= +github.com/subosito/gotenv v1.6.0/go.mod h1:Dk4QP5c2W3ibzajGcXpNraDfq2IrhjMIvMSWPKKo0FU= github.com/tidwall/gjson v1.14.2 h1:6BBkirS0rAHjumnjHF6qgy5d2YAJ1TLIaFE2lzfOLqo= github.com/tidwall/gjson v1.14.2/go.mod h1:/wbyibRr2FHMks5tjHJ5F8dMZh3AcwJEMf5vlfC0lxk= github.com/tidwall/match v1.1.1 h1:+Ho715JplO36QYgwN9PGYNhgZvoUSc9X2c80KVTi+GA= @@ -124,8 +145,12 @@ github.com/tidwall/pretty v1.2.0 h1:RWIZEg2iJ8/g6fDDYzMpobmaoGh5OLl4AXtGUGPcqCs= github.com/tidwall/pretty v1.2.0/go.mod h1:ITEVvHYasfjBbM0u2Pg8T2nJnzm8xPwvNhhsoaGGjNU= github.com/tidwall/sjson v1.2.5 h1:kLy8mja+1c9jlljvWTlSazM7cKDRfJuR/bOJhcY5NcY= github.com/tidwall/sjson v1.2.5/go.mod h1:Fvgq9kS/6ociJEDnK0Fk1cpYF4FIW6ZF7LAe+6jwd28= +go.uber.org/atomic v1.9.0 h1:ECmE8Bn/WFTYwEW/bpKD3M8VtR/zQVbavAoalC1PYyE= +go.uber.org/atomic v1.9.0/go.mod h1:fEN4uk6kAWBTFdckzkM89CLk9XfWZrxpCo0nPH17wJc= go.uber.org/automaxprocs v1.6.0 h1:O3y2/QNTOdbF+e/dpXNNW7Rx2hZ4sTIPyybbxyNqTUs= go.uber.org/automaxprocs v1.6.0/go.mod h1:ifeIMSnPZuznNm6jmdzmU3/bfk01Fe2fotchwEFJ8r8= +go.uber.org/multierr v1.9.0 h1:7fIwc/ZtS0q++VgcfqFDxSBZVv/Xo49/SYnDFupUwlI= +go.uber.org/multierr v1.9.0/go.mod h1:X2jQV1h+kxSjClGpnseKVIxpmcjrj7MNnI0bnlfKTVQ= golang.org/x/crypto v0.39.0 h1:SHs+kF4LP+f+p14esP5jAoDpHU8Gu/v9lFRK6IT5imM= golang.org/x/crypto v0.39.0/go.mod h1:L+Xg3Wf6HoL4Bn4238Z6ft6KfEpN0tJGo53AAPC632U= golang.org/x/exp v0.0.0-20240719175910-8a7402abbf56 h1:2dVuKD2vS7b0QIHQbpyTISPd0LeHDbnYEryqj5Q1ug8= From 7713c2e55d6296f3f0553b29c0fdb209fa9ea6cf Mon Sep 17 00:00:00 2001 From: David Allen Date: Mon, 8 Sep 2025 11:07:53 -0600 Subject: [PATCH 26/34] feat: updated dockerfile and added compose --- Dockerfile | 31 ++++++++++++++++++++++--------- docker-compose.yaml | 15 +++++++++++++++ 2 files changed, 37 insertions(+), 9 deletions(-) create mode 100644 docker-compose.yaml diff --git a/Dockerfile b/Dockerfile index 3a1e3a4..09456c3 100644 --- a/Dockerfile +++ b/Dockerfile @@ -1,14 +1,27 @@ -FROM cgr.dev/chainguard/wolfi-base +FROM alpine:latest AS builder -RUN apk add --no-cache tini bash -RUN mkdir -p /configurator +RUN apk add go git gcc binutils bash -# nobody 65534:65534 -USER 65534:65534 +WORKDIR /tmp +RUN git clone https://git.towk2.me/towk/makeshift.git -# copy the binary and all of the default plugins -COPY configurator /configurator/configurator +WORKDIR /tmp/makeshift -CMD ["/configurator/configurator"] +RUN go mod tidy && \ + go build && \ + mkdir tmp && \ + ./makeshift plugins compile ./pkg/plugins/jinja2/jinja2.go -o ./tmp/plugins/jinja.so -ENTRYPOINT [ "/sbin/tini", "--" ] \ No newline at end of file + +FROM alpine:latest + + +COPY --from=builder /tmp/makeshift/makeshift /usr/local/bin +COPY --from=builder /tmp/makeshift/tmp/plugins/* /makeshift/server/plugins/ + +RUN chmod +x /usr/local/bin/makeshift + +RUN mkdir -p /makeshift/logs && \ + touch /makeshift/logs/makeshift.log + +ENTRYPOINT ["/usr/local/bin/makeshift"] \ No newline at end of file diff --git a/docker-compose.yaml b/docker-compose.yaml new file mode 100644 index 0000000..942fa01 --- /dev/null +++ b/docker-compose.yaml @@ -0,0 +1,15 @@ +services: + makeshift: + # build: ./Dockerfile + image: makeshift:latest + container_name: makeshift + network: + - internal + volumes: + - /tmp/makeshift/makeshift:/usr/local/bin + - /tmp/makeshift/plugins/*:/makeshift/server/plugins + ports: + - 5050:5050 + +networks: + internal: From 481a0782c5dbe871bd429266a59a3720281b6fbf Mon Sep 17 00:00:00 2001 From: David Allen Date: Mon, 15 Sep 2025 15:52:42 -0600 Subject: [PATCH 27/34] feat: added config implementation using viper --- .gitmodules | 3 ++ cmd/config.go | 115 +++++++++++++++++++++++++++++++++++++++ cmd/download.go | 3 ++ cmd/root.go | 141 ++++++++++++++++++++++++++++-------------------- cmd/serve.go | 16 +++--- makeshift.wiki | 1 + 6 files changed, 215 insertions(+), 64 deletions(-) create mode 100644 .gitmodules create mode 100644 cmd/config.go create mode 160000 makeshift.wiki diff --git a/.gitmodules b/.gitmodules new file mode 100644 index 0000000..395de78 --- /dev/null +++ b/.gitmodules @@ -0,0 +1,3 @@ +[submodule "makeshift.wiki"] + path = makeshift.wiki + url = https://git.towk2.me/towk/makeshift.wiki.git diff --git a/cmd/config.go b/cmd/config.go new file mode 100644 index 0000000..14b8ede --- /dev/null +++ b/cmd/config.go @@ -0,0 +1,115 @@ +package cmd + +import ( + "os" + + "git.towk2.me/towk/makeshift/pkg/util" + "github.com/rs/zerolog/log" + "github.com/spf13/cobra" +) + +const CONFIG_FILE = `--- +# +# Makeshift Config +# +# Repository: https://git.towk2.me/towk/makeshift +# +# Default configuration file for 'makeshift' CLI and service. +# This file was autogenerated using 'makeshift config new' command. +# + +# Set the service host. +host: http://localhost:5050 + +# Set the path to the file or directory to download. +path: help.txt + +# Set the log file path. Logs will be written to this location. +log-file: logs/makeshift.log + +# Set the log level. Possible values include 'debug', 'info', 'warn', +# 'error', 'disabled', and 'trace'. +log-level: info + +# Set the plugins to use when downloading files and/or directories. +plugins: + - smd + - jinja2 + +# Set the positional arguments to pass to ALL plugins when downloading +# files and directories. NOTE: These arguments may be ignored or not +# used within certain plugins. +plugin-args: + +# Set the key-word arguments stored in the data that is passed to ALL +# plugins when downloading files and directories. NOTE: These arguments +# may be ignored or not used within certain plugins. +plugin-kwargs: + +# Set the profiles to use when downloading files and/or directories. +profiles: + - default + +# Set whether to extract the archive when downloading directories. +extract: false + +# Set whether to remove an archive after downloading and extracting. +# This requires that the 'extract' flag be set to 'true'. +remove-archive: false + +# Set the path to a CA certificate. +cacert: "" + +# Set the path to a CA key file. +keyfile: "" +` + +var configCmd = &cobra.Command{ + Use: "config", + Short: "Manage makeshift config file", +} + +var configNewCmd = &cobra.Command{ + Use: "new [path]", + Example: ` + # create a new default config at specified path + makeshift config new configs/makeshift.yaml +`, + Args: cobra.ExactArgs(1), + Short: "Create a new config file", + Run: func(cmd *cobra.Command, args []string) { + for _, path := range args { + var ( + overwrite, _ = cmd.Flags().GetBool("overwrite") + + exists bool + err error + ) + if exists, err = util.PathExists(path); err != nil { + log.Error(). + Err(err). + Str("path", path). + Msg("failed to determine if path exists") + } else { + if exists && !overwrite { + log.Error().Msg("file exists and '--overwrite' flag not passed") + return + } + err := os.WriteFile(path, []byte(CONFIG_FILE), 0o644) + if err != nil { + log.Error(). + Err(err). + Str("path", path). + Msg("failed to write diefault config file to path") + continue + } + } + } + }, +} + +func init() { + configNewCmd.Flags().BoolP("overwrite", "f", false, "Set whether to overwrite an existing file") + configCmd.AddCommand(configNewCmd) + rootCmd.AddCommand(configCmd) +} diff --git a/cmd/download.go b/cmd/download.go index fdb2233..6c50c3f 100644 --- a/cmd/download.go +++ b/cmd/download.go @@ -49,6 +49,7 @@ var downloadCmd = cobra.Command{ host, _ = cmd.Flags().GetString("host") path, _ = cmd.Flags().GetString("path") outputPath, _ = cmd.Flags().GetString("output") + configPath, _ = cmd.Flags().GetString("config") cacertPath, _ = cmd.Flags().GetString("cacert") pluginNames, _ = cmd.Flags().GetStringSlice("plugins") profileIDs, _ = cmd.Flags().GetStringSlice("profiles") @@ -62,6 +63,7 @@ var downloadCmd = cobra.Command{ err error ) + // build download query URL query = fmt.Sprintf("/download/%s?", path) if len(pluginNames) > 0 { query += "plugins=" + url.QueryEscape(strings.Join(pluginNames, ",")) @@ -79,6 +81,7 @@ var downloadCmd = cobra.Command{ log.Debug(). Str("host", host). Str("path", path). + Str("config", configPath). Str("query", query). Str("output", outputPath). Strs("profiles", profileIDs). diff --git a/cmd/root.go b/cmd/root.go index 9f233f6..c55700c 100644 --- a/cmd/root.go +++ b/cmd/root.go @@ -4,6 +4,7 @@ import ( "fmt" "net/http" "os" + "path/filepath" "strings" logger "git.towk2.me/towk/makeshift/pkg/log" @@ -20,14 +21,14 @@ var ( var rootCmd = cobra.Command{ Use: "makeshift", Short: "Extensible file cobbler", - PersistentPreRun: func(cmd *cobra.Command, args []string) { + PersistentPreRunE: func(cmd *cobra.Command, args []string) error { var ( - logFile string - err error + logFile, _ = cmd.Flags().GetString("log-file") + configPath, _ = cmd.Flags().GetString("config") + err error ) // initialize the logger - logFile, _ = cmd.Flags().GetString("log-file") err = logger.InitWithLogLevel(loglevel, logFile) if err != nil { log.Error().Err(err).Msg("failed to initialize logger") @@ -35,12 +36,13 @@ var rootCmd = cobra.Command{ } // You can bind cobra and viper in a few locations, but PersistencePreRunE on the root command works well - err = initializeConfig(cmd) + return initConfig(cmd, configPath) }, Run: func(cmd *cobra.Command, args []string) { // try and set flags using env vars setenv(cmd, "log-file", "MAKESHIFT_LOG_FILE") setenv(cmd, "log-level", "MAKESHIFT_LOG_LEVEL") + setenv(cmd, "config", "MAKESHIFT_CONFIG_FILE") if len(args) == 0 { err := cmd.Help() if err != nil { @@ -54,6 +56,7 @@ var rootCmd = cobra.Command{ err := logger.LogFile.Close() if err != nil { log.Error().Err(err).Msg("failed to close log file") + os.Exit(1) } }, } @@ -71,8 +74,83 @@ func init() { initLogger, ) // initialize the config a single time - rootCmd.PersistentFlags().VarP(&loglevel, "log-level", "l", "Set the log level output") + rootCmd.PersistentFlags().VarP(&loglevel, "log-level", "l", "Set the log level output (can be set with MAKESHIFT_LOG_LEVEL)") rootCmd.PersistentFlags().String("log-file", "", "Set the log file path (can be set with MAKESHIFT_LOG_FILE)") + rootCmd.PersistentFlags().StringP("config", "c", "", "Set the config file path (can be set with MAKESHIFT_CONFIG_FILE)") +} + +func initLogger() { + // initialize the logger + logfile, _ := rootCmd.PersistentFlags().GetString("log-file") + err := logger.InitWithLogLevel(loglevel, logfile) + if err != nil { + log.Error().Err(err).Msg("failed to initialize logger") + os.Exit(1) + } +} + +func initConfig(cmd *cobra.Command, path string) error { + + // Dissect the path to separate config name from its directory + var ( + isFlagSet = cmd.Flags().Changed("config") + filename = filepath.Base(path) + ext = filepath.Ext(filename) + directory = filepath.Dir(path) + v = viper.New() + ) + + // The 'config' flag not set, so don't continue + if !isFlagSet { + return nil + } + + // Only use specified YAML file from --config or -c flag + v.SetConfigName(strings.TrimSuffix(filename, ext)) + v.SetConfigType("yaml") + v.AddConfigPath(directory) + + // Attempt to read the config file. Return an error if we cannot parse + // the config file or if it is not found. + if err := v.ReadInConfig(); err != nil { + if isFlagSet { + // It's okay if there isn't a config file when no path is provided + // if _, ok := err.(viper.ConfigFileNotFoundError); !ok { + // return err + // } + switch err.(type) { + case viper.ConfigFileNotFoundError: + log.Error(). + Err(err). + Str("path", path). + Msg("failed to read config") + os.Exit(1) + default: + log.Error().Err(err).Msg("failed to read config") + os.Exit(1) + } + } + } + + // When we bind flags to environment variables expect that the + // environment variables are prefixed, e.g. a flag like --number + // binds to an environment variable STING_NUMBER. This helps + // avoid conflicts. + v.SetEnvPrefix("MAKESHIFT") + + // Environment variables can't have dashes in them, so bind them to their equivalent + // keys with underscores, e.g. --favorite-color to STING_FAVORITE_COLOR + v.SetEnvKeyReplacer(strings.NewReplacer("-", "_")) + + // Bind to environment variables + // Works great for simple config names, but needs help for names + // like --favorite-color which we fix in the bindFlags function + v.AutomaticEnv() + + // Bind the current command's flags to viper + bindFlags(cmd, v) + + return nil } func setenv(cmd *cobra.Command, varname string, envvar string) { @@ -95,16 +173,6 @@ func setenvp(cmd *cobra.Command, varname string, envvar string) { } } -func initLogger() { - // initialize the logger - logfile, _ := rootCmd.PersistentFlags().GetString("log-file") - err := logger.InitWithLogLevel(loglevel, logfile) - if err != nil { - log.Error().Err(err).Msg("failed to initialize logger") - os.Exit(1) - } -} - func handleResponseError(res *http.Response, host, query string, err error) { if err != nil { log.Error().Err(err). @@ -147,44 +215,3 @@ func bindFlags(cmd *cobra.Command, v *viper.Viper) { } }) } - -func initializeConfig(cmd *cobra.Command) error { - v := viper.New() - - // Set the base name of the config file, without the file extension. - v.SetConfigName("makeshift") - - // Set as many paths as you like where viper should look for the - // config file. We are only looking in the current working directory. - v.AddConfigPath(".") - - // Attempt to read the config file, gracefully ignoring errors - // caused by a config file not being found. Return an error - // if we cannot parse the config file. - if err := v.ReadInConfig(); err != nil { - // It's okay if there isn't a config file - if _, ok := err.(viper.ConfigFileNotFoundError); !ok { - return err - } - } - - // When we bind flags to environment variables expect that the - // environment variables are prefixed, e.g. a flag like --number - // binds to an environment variable STING_NUMBER. This helps - // avoid conflicts. - v.SetEnvPrefix("MAKESHIFT") - - // Environment variables can't have dashes in them, so bind them to their equivalent - // keys with underscores, e.g. --favorite-color to STING_FAVORITE_COLOR - v.SetEnvKeyReplacer(strings.NewReplacer("-", "_")) - - // Bind to environment variables - // Works great for simple config names, but needs help for names - // like --favorite-color which we fix in the bindFlags function - v.AutomaticEnv() - - // Bind the current command's flags to viper - bindFlags(cmd, v) - - return nil -} diff --git a/cmd/serve.go b/cmd/serve.go index 9c8c046..b723a02 100644 --- a/cmd/serve.go +++ b/cmd/serve.go @@ -19,17 +19,18 @@ var serveCmd = &cobra.Command{ makeshift serve --root ./test --init -l debug `, Args: cobra.NoArgs, - PreRun: func(cmd *cobra.Command, args []string) { - setenv(cmd, "host", "MAKESHIFT_HOST") - setenv(cmd, "root", "MAKESHIFT_ROOT") - setenv(cmd, "timeout", "MAKESHIFT_TIMEOUT") - setenv(cmd, "cacert", "MAKESHIFT_CACERT") - setenv(cmd, "keyfile", "MAKESHIFT_KEYFILE") - }, + // PreRun: func(cmd *cobra.Command, args []string) { + // setenv(cmd, "host", "MAKESHIFT_HOST") + // setenv(cmd, "root", "MAKESHIFT_ROOT") + // setenv(cmd, "timeout", "MAKESHIFT_TIMEOUT") + // setenv(cmd, "cacert", "MAKESHIFT_CACERT") + // setenv(cmd, "keyfile", "MAKESHIFT_KEYFILE") + // }, Run: func(cmd *cobra.Command, args []string) { var ( host, _ = cmd.Flags().GetString("host") rootPath, _ = cmd.Flags().GetString("root") + configPath, _ = cmd.Flags().GetString("config") cacertPath, _ = cmd.Flags().GetString("cacert") keyfile, _ = cmd.Flags().GetString("keyfile") timeout, _ = cmd.Flags().GetInt("timeout") @@ -60,6 +61,7 @@ var serveCmd = &cobra.Command{ Str("host", parsed.Host). Any("paths", map[string]string{ "root": rootPath, + "config": configPath, "cacert": cacertPath, "keyfile": keyfile, "data": server.PathForData(), diff --git a/makeshift.wiki b/makeshift.wiki new file mode 160000 index 0000000..6409089 --- /dev/null +++ b/makeshift.wiki @@ -0,0 +1 @@ +Subproject commit 64090892b46ffea8d1775cc37b995182aeacf31d From 5429a4147e83c310982f87c85a40430c4bf13d3f Mon Sep 17 00:00:00 2001 From: David Allen Date: Mon, 15 Sep 2025 15:53:50 -0600 Subject: [PATCH 28/34] feat: added example config --- examples/config.yaml | 54 ++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 54 insertions(+) create mode 100644 examples/config.yaml diff --git a/examples/config.yaml b/examples/config.yaml new file mode 100644 index 0000000..de62da6 --- /dev/null +++ b/examples/config.yaml @@ -0,0 +1,54 @@ +--- +# +# Makeshift Config +# +# Repository: https://git.towk2.me/towk/makeshift +# +# Default configuration file for 'makeshift' CLI and service. +# This file was autogenerated using 'makeshift config new' command. +# + +# Set the service host. +host: http://localhost:5050 + +# Set the path to the file or directory to download. +path: help.txt + +# Set the log file path. Logs will be written to this location. +log-file: logs/makeshift.log + +# Set the log level. Possible values include 'debug', 'info', 'warn', +# 'error', 'disabled', and 'trace'. +log-level: info + +# Set the plugins to use when downloading files and/or directories. +plugins: + - smd + - jinja2 + +# Set the positional arguments to pass to ALL plugins when downloading +# files and directories. NOTE: These arguments may be ignored or not +# used within certain plugins. +plugin-args: + +# Set the key-word arguments stored in the data that is passed to ALL +# plugins when downloading files and directories. NOTE: These arguments +# may be ignored or not used within certain plugins. +plugin-kwargs: + +# Set the profiles to use when downloading files and/or directories. +profiles: + - default + +# Set whether to extract the archive when downloading directories. +extract: false + +# Set whether to remove an archive after downloading and extracting. +# This requires that the 'extract' flag be set to 'true'. +remove-archive: false + +# Set the path to a CA certificate. +cacert: "" + +# Set the path to a CA key file. +keyfile: "" From d1e892dd98dc4ab5ff48c38a0ee708c7ec6845e5 Mon Sep 17 00:00:00 2001 From: David Allen Date: Sat, 20 Sep 2025 15:25:20 -0600 Subject: [PATCH 29/34] docs: removed some TODO entries --- README.md | 8 ++------ 1 file changed, 2 insertions(+), 6 deletions(-) diff --git a/README.md b/README.md index 2ce09ce..b6b6a6b 100644 --- a/README.md +++ b/README.md @@ -239,9 +239,5 @@ Profiles can be created using JSON and only require an `id` with optional `data` There are some features still missing that will be added later. -1. Running `makeshift` locally with profiles and plugins -2. Plugin to add user data for one-time use without creating a profile -3. Optionally build plugins directly into the main driver -4. Protected routes that require authentication -5. Configuration file for persistent runs -6. `Dockerfile` and `docker-compose.yml` files to build containers \ No newline at end of file +1. Optionally build plugins directly into the main driver +2. Protected routes that require authentication From 47c9d48735e2b0c7701189f685e1dd63e064d977 Mon Sep 17 00:00:00 2001 From: David Allen Date: Sat, 20 Sep 2025 15:26:09 -0600 Subject: [PATCH 30/34] refactor: changed how plugin kwargs are sent and other minor changes --- cmd/download.go | 11 ++++++----- cmd/run.go | 1 + cmd/serve.go | 6 ++++-- 3 files changed, 11 insertions(+), 7 deletions(-) diff --git a/cmd/download.go b/cmd/download.go index 6c50c3f..c19a783 100644 --- a/cmd/download.go +++ b/cmd/download.go @@ -1,6 +1,7 @@ package cmd import ( + "encoding/base64" "fmt" "net/http" "net/url" @@ -15,10 +16,7 @@ import ( "github.com/spf13/cobra" ) -var ( - pluginArgs []string - pluginKWArgs kwargs.KWArgs = kwargs.KWArgs{} -) +var pluginKWArgs kwargs.KWArgs = kwargs.KWArgs{} var downloadCmd = cobra.Command{ Use: "download", Example: ` @@ -52,6 +50,7 @@ var downloadCmd = cobra.Command{ configPath, _ = cmd.Flags().GetString("config") cacertPath, _ = cmd.Flags().GetString("cacert") pluginNames, _ = cmd.Flags().GetStringSlice("plugins") + pluginArgs, _ = cmd.Flags().GetStringSlice("plugin-args") profileIDs, _ = cmd.Flags().GetStringSlice("profiles") extract, _ = cmd.Flags().GetBool("extract") removeArchive, _ = cmd.Flags().GetBool("remove-archive") @@ -75,7 +74,7 @@ var downloadCmd = cobra.Command{ query += "&args=" + url.QueryEscape(strings.Join(pluginArgs, ",")) } if len(pluginKWArgs) > 0 { - query += "&kwargs=" + url.QueryEscape(pluginKWArgs.String()) + query += "&kwargs=" + base64.RawURLEncoding.EncodeToString(pluginKWArgs.Bytes()) } log.Debug(). @@ -86,6 +85,8 @@ var downloadCmd = cobra.Command{ Str("output", outputPath). Strs("profiles", profileIDs). Strs("plugins", pluginNames). + Strs("args", pluginArgs). + Any("kwargs", pluginKWArgs). Send() if cacertPath != "" { diff --git a/cmd/run.go b/cmd/run.go index 846e560..e738e26 100644 --- a/cmd/run.go +++ b/cmd/run.go @@ -44,6 +44,7 @@ var runCmd = &cobra.Command{ keyfile, _ = cmd.Flags().GetString("keyfile") timeout, _ = cmd.Flags().GetInt("timeout") pluginNames, _ = cmd.Flags().GetStringSlice("plugins") + pluginArgs, _ = cmd.Flags().GetStringSlice("plugin-args") profileIDs, _ = cmd.Flags().GetStringSlice("profiles") extract, _ = cmd.Flags().GetBool("extract") removeArchive, _ = cmd.Flags().GetBool("remove-archive") diff --git a/cmd/serve.go b/cmd/serve.go index b723a02..fba854b 100644 --- a/cmd/serve.go +++ b/cmd/serve.go @@ -94,8 +94,10 @@ func init() { serveCmd.Flags().String("host", "localhost:5050", "Set the configurator server host (can be set with MAKESHIFT_HOST)") serveCmd.Flags().String("root", "./", "Set the root path to serve files (can be set with MAKESHIFT_ROOT)") serveCmd.Flags().IntP("timeout", "t", 60, "Set the timeout in seconds for requests (can be set with MAKESHIFT_TIMEOUT)") - serveCmd.Flags().String("cacert", "", "Set the CA certificate path to load (can be set with MAKESHIFT_CACERT)") - serveCmd.Flags().String("keyfile", "", "Set the CA key file to use (can be set with MAKESHIFT_KEYFILE)") + serveCmd.Flags().String("cacert", "", "Set the CA certificate path to load (can be set with MAKESHIFT_CACERT, only used if set with '--keyfile' flag)") + serveCmd.Flags().String("keyfile", "", "Set the CA key file to use (can be set with MAKESHIFT_KEYFILE, only used if set with '--cacert' flag)") + serveCmd.Flags().String("keyurl", "", "Set the JWKS remote host for JWT verification") + serveCmd.Flags().StringSlice("protect-routes", []string{}, "Set the routes to require authentication (uses default routes if not set with '--keyurl' flag)") serveCmd.MarkFlagsRequiredTogether("cacert", "keyfile") From 505dbefb67f9a5f174356c77afd01ef767563181 Mon Sep 17 00:00:00 2001 From: David Allen Date: Sat, 20 Sep 2025 15:27:20 -0600 Subject: [PATCH 31/34] refactor: minor changes and fixes --- internal/format/format.go | 7 ++++--- internal/kwargs/kwargs.go | 17 +++++++++++++++-- 2 files changed, 19 insertions(+), 5 deletions(-) diff --git a/internal/format/format.go b/internal/format/format.go index bdbfea0..11cfa56 100644 --- a/internal/format/format.go +++ b/internal/format/format.go @@ -4,6 +4,7 @@ import ( "encoding/json" "fmt" "path/filepath" + "strings" "gopkg.in/yaml.v3" ) @@ -92,11 +93,11 @@ func Unmarshal(data []byte, v any, inFormat DataFormat) error { // both the standard default format (JSON) and any command line // change to that provided by options. func DataFormatFromFileExt(path string, defaultFmt DataFormat) DataFormat { - switch filepath.Ext(path) { - case ".json", ".JSON": + switch strings.TrimLeft(strings.ToLower(filepath.Ext(path)), ".") { + case JSON.String(): // The file is a JSON file return JSON - case ".yaml", ".yml", ".YAML", ".YML": + case YAML.String(), "yml": // The file is a YAML file return YAML } diff --git a/internal/kwargs/kwargs.go b/internal/kwargs/kwargs.go index 1da80bd..0c8911f 100644 --- a/internal/kwargs/kwargs.go +++ b/internal/kwargs/kwargs.go @@ -5,15 +5,28 @@ import ( "fmt" "git.towk2.me/towk/makeshift/internal/format" + "github.com/rs/zerolog/log" ) const RESERVED_KEY = "kwargs" type KWArgs map[string]any +func New() KWArgs { + return KWArgs{} +} + func (kw KWArgs) String() string { - b, _ := json.Marshal(kw) - return string(b) + return string(kw.Bytes()) +} + +func (kw KWArgs) Bytes() []byte { + b, err := json.Marshal(kw) + if err != nil { + log.Error().Err(err).Msg("failed to marshal kwargs") + return []byte("{}") + } + return b } func (kw *KWArgs) Set(v string /* should be JSON object*/) error { From 72c52fbac653c96a88cf204ee377f79d24e4752a Mon Sep 17 00:00:00 2001 From: David Allen Date: Sat, 20 Sep 2025 15:35:39 -0600 Subject: [PATCH 32/34] refactor: updated plugin implementations --- pkg/plugins/jinja2/jinja2.go | 18 ++++++++++++--- pkg/plugins/mapper/mapper.go | 39 -------------------------------- pkg/plugins/pyjinja2/pyjinja2.go | 13 ++++++++++- pkg/plugins/user/user.go | 36 ----------------------------- 4 files changed, 27 insertions(+), 79 deletions(-) delete mode 100644 pkg/plugins/mapper/mapper.go delete mode 100644 pkg/plugins/user/user.go diff --git a/pkg/plugins/jinja2/jinja2.go b/pkg/plugins/jinja2/jinja2.go index daf6b20..6282d7c 100644 --- a/pkg/plugins/jinja2/jinja2.go +++ b/pkg/plugins/jinja2/jinja2.go @@ -5,6 +5,7 @@ import ( "encoding/json" "fmt" + "git.towk2.me/towk/makeshift/internal/kwargs" makeshift "git.towk2.me/towk/makeshift/pkg" "git.towk2.me/towk/makeshift/pkg/storage" "github.com/nikolalohinski/gonja/v2" @@ -42,10 +43,11 @@ func (p *Jinja2) Run(store storage.KVStore, args []string) error { mappings struct { Data map[string]any `json:"data"` } + userdata *kwargs.KWArgs context *exec.Context template *exec.Template profiles any // makeshift.ProfileMap - input any // []byte + contents any // []byte output bytes.Buffer err error ) @@ -56,18 +58,26 @@ func (p *Jinja2) Run(store storage.KVStore, args []string) error { Int("arg_count", len(args)). Msg("(jinja2) Run()") + // get profile data used as variable `{{ makeshift.profiles }}` profiles, err = store.Get("profiles") if err != nil { return fmt.Errorf("(jinja2) failed to get profiles: %v", err) } - input, err = store.Get("file") + // get userdata used as variable `{{ makeshift.userdata }}` + userdata, err = store.GetKWArgs() + if err != nil { + return fmt.Errorf("(jinja2) failed to get key-word arguments: %v", err) + } + + // get file contents used for templating + contents, err = store.Get("file") if err != nil { return fmt.Errorf("(jinja2) failed to get input data: %v", err) } // get the templates provided as args to the plugin - template, err = gonja.FromBytes(input.([]byte)) + template, err = gonja.FromBytes(contents.([]byte)) if err != nil { return fmt.Errorf("(jinja2) failed to get template from args: %v", err) } @@ -83,6 +93,7 @@ func (p *Jinja2) Run(store storage.KVStore, args []string) error { } } + // get mappings from provided profiles `{{ makeshift.plugin.*}}` var ps = make(map[string]any) for profileID, profile := range profiles.(makeshift.ProfileMap) { ps[profileID] = map[string]any{ @@ -96,6 +107,7 @@ func (p *Jinja2) Run(store storage.KVStore, args []string) error { mappings.Data = map[string]any{ "makeshift": map[string]any{ "profiles": ps, + "userdata": userdata, "plugin": map[string]any{ "name": p.Name(), "version": p.Version(), diff --git a/pkg/plugins/mapper/mapper.go b/pkg/plugins/mapper/mapper.go deleted file mode 100644 index 66b4a5d..0000000 --- a/pkg/plugins/mapper/mapper.go +++ /dev/null @@ -1,39 +0,0 @@ -package main - -import ( - makeshift "git.towk2.me/towk/makeshift/pkg" - "git.towk2.me/towk/makeshift/pkg/storage" -) - -type Mapper struct{} - -func (p *Mapper) Name() string { return "mapper" } -func (p *Mapper) Version() string { return "v0.0.1-alpha" } -func (p *Mapper) Description() string { return "Directly maps data to store" } -func (p *Mapper) Metadata() makeshift.Metadata { - return makeshift.Metadata{ - "author": map[string]any{ - "name": "David J. Allen", - "email": "davidallendj@gmail.com", - "links": []string{ - "https://github.com/davidallendj", - "https://git.towk2.me/towk", - }, - }, - } -} - -func (p *Mapper) Init() error { - // nothing to initialize - return nil -} - -func (p *Mapper) Run(data storage.KVStore, args []string) error { - return nil -} - -func (p *Mapper) Clean() error { - return nil -} - -var Makeshift Mapper diff --git a/pkg/plugins/pyjinja2/pyjinja2.go b/pkg/plugins/pyjinja2/pyjinja2.go index 673f1a1..f625c26 100644 --- a/pkg/plugins/pyjinja2/pyjinja2.go +++ b/pkg/plugins/pyjinja2/pyjinja2.go @@ -4,6 +4,7 @@ import ( "encoding/json" "fmt" + "git.towk2.me/towk/makeshift/internal/kwargs" makeshift "git.towk2.me/towk/makeshift/pkg" "git.towk2.me/towk/makeshift/pkg/storage" jinja2 "github.com/kluctl/kluctl/lib/go-jinja2" @@ -42,6 +43,7 @@ func (p *Jinja2) Run(store storage.KVStore, args []string) error { mappings struct { Data map[string]any `json:"data"` } + userdata *kwargs.KWArgs profiles any // makeshift.ProfileMap input any // []byte output string @@ -54,11 +56,19 @@ func (p *Jinja2) Run(store storage.KVStore, args []string) error { Int("arg_count", len(args)). Msg("(pyjinja2) Run()") + // get profile data used as variable `{{ makeshift.profiles }}` profiles, err = store.Get("profiles") if err != nil { return fmt.Errorf("(pyjinja2) failed to get profiles: %v", err) } + // get userdata used as variable `{{ makeshift.userdata }}` + userdata, err = store.GetKWArgs() + if err != nil { + return fmt.Errorf("(pyjinja2) failed to get key-word arguments: %v", err) + } + + // get file contents used for templating input, err = store.Get("file") if err != nil { return fmt.Errorf("(pyjinja2) failed to get input data: %v", err) @@ -75,7 +85,7 @@ func (p *Jinja2) Run(store storage.KVStore, args []string) error { } } - // get mappings from provided profiles + // get mappings from provided profiles `{{ makeshift.plugin.*}}` var ps = make(map[string]any) for profileID, profile := range profiles.(makeshift.ProfileMap) { ps[profileID] = map[string]any{ @@ -89,6 +99,7 @@ func (p *Jinja2) Run(store storage.KVStore, args []string) error { mappings.Data = map[string]any{ "makeshift": map[string]any{ "profiles": ps, + "userdata": userdata, "plugin": map[string]any{ "name": p.Name(), "version": p.Version(), diff --git a/pkg/plugins/user/user.go b/pkg/plugins/user/user.go deleted file mode 100644 index ff59640..0000000 --- a/pkg/plugins/user/user.go +++ /dev/null @@ -1,36 +0,0 @@ -package main - -import ( - makeshift "git.towk2.me/towk/makeshift/pkg" - "git.towk2.me/towk/makeshift/pkg/storage" -) - -type User struct{} - -func (p *User) Name() string { return "user" } -func (p *User) Version() string { return "v0.0.1-alpha" } -func (p *User) Description() string { return "Get user information" } -func (p *User) Metadata() makeshift.Metadata { - return makeshift.Metadata{ - "author": map[string]any{ - "name": "David J. Allen", - "email": "davidallendj@gmail.com", - "links": []string{ - "https://github.com/davidallendj", - "https://git.towk2.me/towk", - }, - }, - } -} - -func (p *User) Init() error { - return nil -} - -func (p *User) Run(store storage.KVStore, args []string) error { - return nil -} - -func (p *User) Cleanup() error { - return nil -} From c1c5ec1625b33a2cde58b9ebd14e821f9f774c37 Mon Sep 17 00:00:00 2001 From: David Allen Date: Sat, 20 Sep 2025 15:44:08 -0600 Subject: [PATCH 33/34] refactor: minor changes and fixes --- pkg/service/routes.go | 44 +++++++++++++++++++++++++++----------- pkg/storage/memory.go | 8 +++++-- pkg/storage/memory_test.go | 1 + 3 files changed, 39 insertions(+), 14 deletions(-) create mode 100644 pkg/storage/memory_test.go diff --git a/pkg/service/routes.go b/pkg/service/routes.go index e4fdb04..faa0236 100644 --- a/pkg/service/routes.go +++ b/pkg/service/routes.go @@ -1,6 +1,7 @@ package service import ( + "encoding/base64" "encoding/json" "fmt" "io" @@ -16,44 +17,63 @@ import ( makeshift "git.towk2.me/towk/makeshift/pkg" "git.towk2.me/towk/makeshift/pkg/storage" "git.towk2.me/towk/makeshift/pkg/util" - "github.com/go-chi/chi/v5" "github.com/rs/zerolog/log" ) func (s *Service) Download() http.HandlerFunc { return func(w http.ResponseWriter, r *http.Request) { var ( - path = s.PathForData() + strings.TrimPrefix(r.URL.Path, "/download") - pluginKWArgs = chi.URLParam(r, "kwargs") - pluginArgs = strings.Split(r.URL.Query().Get("args"), ",") - pluginNames = strings.Split(r.URL.Query().Get("plugins"), ",") - profileIDs = strings.Split(r.URL.Query().Get("profiles"), ",") + path = s.PathForData() + strings.TrimPrefix(r.URL.Path, "/download") + pluginArgs = strings.Split(r.URL.Query().Get("args"), ",") + pluginNames = strings.Split(r.URL.Query().Get("plugins"), ",") + profileIDs = strings.Split(r.URL.Query().Get("profiles"), ",") - kw *kwargs.KWArgs + kw *kwargs.KWArgs = new(kwargs.KWArgs) fileInfo os.FileInfo out *os.File store *storage.MemoryStorage = new(storage.MemoryStorage) hooks []makeshift.Hook contents []byte + decoded []byte errs []error err error ) // parse the KWArgs from request - kw.Set(pluginKWArgs) - - // initialize storage - store.Init() - store.SetKWArgs(kw) + decoded, err = base64.RawURLEncoding.DecodeString(r.URL.Query().Get("kwargs")) + if err != nil { + s.writeErrorResponse(w, err.Error(), http.StatusBadRequest) + return + } log.Debug(). Str("path", path). Str("client_host", r.Host). Strs("plugins", pluginNames). Strs("profiles", profileIDs). + Strs("args", pluginArgs). + Str("kwargs", string(decoded)). Any("query", r.URL.Query()). Msg("Service.Download()") + err = kw.Set(string(decoded)) + if err != nil { + s.writeErrorResponse(w, err.Error(), http.StatusBadRequest) + return + } + + // initialize storage + err = store.Init() + if err != nil { + s.writeErrorResponse(w, err.Error(), http.StatusInternalServerError) + return + } + err = store.SetKWArgs(kw) + if err != nil { + s.writeErrorResponse(w, err.Error(), http.StatusInternalServerError) + return + } + // prepare profiles errs = s.LoadProfiles(profileIDs, store, errs) if len(errs) > 0 { diff --git a/pkg/storage/memory.go b/pkg/storage/memory.go index a3b1abb..76874c1 100644 --- a/pkg/storage/memory.go +++ b/pkg/storage/memory.go @@ -20,11 +20,15 @@ func (ms *MemoryStorage) Cleanup() error { } func (ms *MemoryStorage) SetKWArgs(kw *kwargs.KWArgs) error { - return ms.Set(kwargs.RESERVED_KEY, kw) + ms.Data[kwargs.RESERVED_KEY] = kw + return nil } func (ms *MemoryStorage) GetKWArgs() (*kwargs.KWArgs, error) { kw, err := ms.Get(kwargs.RESERVED_KEY) + if err != nil { + return nil, err + } return kw.(*kwargs.KWArgs), err } @@ -37,7 +41,7 @@ func (ms *MemoryStorage) Get(k string) (any, error) { } func (ms *MemoryStorage) Set(k string, v any) error { - if k == "kwargs" { + if k == kwargs.RESERVED_KEY { return fmt.Errorf("cannot set reserved key '%s' (use SetKWArgs() instead)", k) } ms.Data[k] = v diff --git a/pkg/storage/memory_test.go b/pkg/storage/memory_test.go new file mode 100644 index 0000000..82be054 --- /dev/null +++ b/pkg/storage/memory_test.go @@ -0,0 +1 @@ +package storage From 5d9612d5b424f5baa404285b45154b2e02167a37 Mon Sep 17 00:00:00 2001 From: David Allen Date: Mon, 3 Nov 2025 20:48:19 -0700 Subject: [PATCH 34/34] refactor: changed base image from alpine to archlinux --- Dockerfile | 14 ++++++++------ 1 file changed, 8 insertions(+), 6 deletions(-) diff --git a/Dockerfile b/Dockerfile index 09456c3..38ff8d7 100644 --- a/Dockerfile +++ b/Dockerfile @@ -1,6 +1,7 @@ -FROM alpine:latest AS builder +FROM archlinux:latest AS builder -RUN apk add go git gcc binutils bash +RUN pacman -Sy +RUN pacman -S go git gcc binutils bash --noconfirm WORKDIR /tmp RUN git clone https://git.towk2.me/towk/makeshift.git @@ -9,19 +10,20 @@ WORKDIR /tmp/makeshift RUN go mod tidy && \ go build && \ - mkdir tmp && \ - ./makeshift plugins compile ./pkg/plugins/jinja2/jinja2.go -o ./tmp/plugins/jinja.so + mkdir -p /makeshift +RUN ./makeshift init /makeshift +RUN ./makeshift plugins compile ./pkg/plugins/jinja2/jinja2.go -o ./tmp/plugins/jinja.so FROM alpine:latest COPY --from=builder /tmp/makeshift/makeshift /usr/local/bin -COPY --from=builder /tmp/makeshift/tmp/plugins/* /makeshift/server/plugins/ +COPY --from=builder /tmp/makeshift/tmp/plugins/* /makeshift/plugins/ RUN chmod +x /usr/local/bin/makeshift RUN mkdir -p /makeshift/logs && \ touch /makeshift/logs/makeshift.log -ENTRYPOINT ["/usr/local/bin/makeshift"] \ No newline at end of file +ENTRYPOINT ["/usr/local/bin/makeshift"]