implement grid-compose #1130

eyad-hussein · 2024-07-24T15:08:22Z

Description

Building the grid-compose cli tool by implementing several commands adapted by docker-compose(up, down, version, etc)

Changes

ps
up
down
version

Related Issues

Grid-Compose #1022

Checklist

Tests included
Build pass
Documentation
Code format and docstring

xmonader · 2024-07-24T20:49:33Z

grid-compose/cmd/root.go

+	rootCmd.AddCommand(downCmd)
+}
+
+type App struct {


Not the right place, should be in internal/app.go or similar

xmonader · 2024-07-24T20:50:52Z

grid-compose/cmd/root.go

+	Specs  types.Specs
+}
+
+func NewApp(net, mne, filePath string) (App, error) {


factory function should be in internal/app.go and use better names eg mnemonic instead of mne

xmonader · 2024-07-24T20:51:47Z

grid-compose/cmd/root.go

+	}, nil
+}
+
+func LoadSpecsFromFile(filePath string) (types.Specs, error) {


create a loadSpecsFromReader with io.Reader and don't bother about paths, opening/closing the files.

and of course, not the right place, internal/app.go

xmonader · 2024-07-24T20:52:35Z

grid-compose/cmd/root.go

+	return specs, nil
+}
+
+func Execute() {


move this to the beginning of the file

xmonader · 2024-07-24T20:53:06Z

grid-compose/cmd/root.go

+
+var (
+	app        App
+	configFile string


xmonader · 2024-07-24T20:53:54Z

grid-compose/cmd/root.go

+	Short: "Grid-Compose is a tool for running multi-vm applications on TFGrid defined using a Yaml formatted file.",
+	PersistentPreRun: func(cmd *cobra.Command, args []string) {
+		var err error
+		app, err = NewApp(network, mnemonic, configFile)


if app, err := ... ; err != nil {
log.Fatal ...
}

xmonader · 2024-07-24T20:54:21Z

grid-compose/cmd/up.go

+	},
+}
+
+func up(ctx context.Context) error {


move that into internal/app.go and make it a method

and same for the rest..

xmonader · 2024-07-24T20:55:00Z

grid-compose/cmd/up.go

+	}
+
+	for key, val := range results {
+		fmt.Printf("%s vm addresses:\n", key)


why mixing logging with printing?

xmonader · 2024-07-24T20:56:12Z

grid-compose/internal/utils.go

+	"github.com/threefoldtech/zos/pkg/gridtypes/zos"
+)
+
+func GetProjectName(key string, twinId uint32) string {


method on the app

xmonader · 2024-07-24T20:56:38Z

grid-compose/internal/utils.go

+	return fmt.Sprintf("compose/%v/%v", twinId, key)
+}
+
+func GetVmAddresses(vm workloads.VM) string {


move into vmaddresses.go in internal

xmonader · 2024-07-24T20:57:03Z

grid-compose/internal/utils.go

+	return res
+}
+
+func GetRandomMyceliumIPSeed() ([]byte, error) {


move into internal/mycelium_seed.go

xmonader · 2024-07-24T20:58:19Z

grid-compose/pkg/types/types.go

+type Service struct {
+	Flist       string       `yaml:"flist"`
+	Entrypoint  string       `yaml:"entrypoint,omitempty"`
+	Environment KVMap        `yaml:"environment"`


no need for the new type here map[string]string, specially if you don't have any new methods reacting on that type and doesn't provide any type safety

xmonader · 2024-07-24T20:58:34Z

grid-compose/pkg/types/types.go

+
+type Service struct {
+	Flist       string       `yaml:"flist"`
+	Entrypoint  string       `yaml:"entrypoint,omitempty"`


this is required

xmonader · 2024-07-24T20:59:24Z

grid-compose/pkg/types/types.go

+	Test     []string `yaml:"test"`
+	Interval string   `yaml:"interval"`
+	Timeout  string   `yaml:"timeout"`
+	Retries  int      `yaml:"retries"`


think of the type used, is that retries going to be a negative at anypoint? use uint or uint8 instead

xmonader · 2024-07-24T20:59:47Z

grid-compose/pkg/types/types.go

+}
+
+type HealthCheck struct {
+	Test     []string `yaml:"test"`


Is that a test command or what exactly?add a comment next to it

xmonader · 2024-07-24T21:00:36Z

grid-compose/pkg/types/types.go

+
+type KVMap map[string]string
+
+func (m *KVMap) UnmarshalYAML(value *yaml.Node) error {


remove this

xmonader · 2024-07-24T21:02:57Z

grid-compose/cmd/down.go

+		projectName := internal.GetProjectName(key, app.Client.TwinID)
+		log.Info().Str("projectName", projectName).Msg("canceling deployments")
+		if err := app.Client.CancelByProjectName(projectName); err != nil {
+			return err


wrap the error here was e.g fmt.Errorf("failed to cancel project %s: %w", projectName, err)

move to to a method on App struct in app.go

xmonader · 2024-07-24T21:04:12Z

grid-compose/cmd/down.go

+	},
+}
+
+func down() error {


move to a method on App struct in internal/app.go

xmonader · 2024-07-24T21:04:56Z

grid-compose/cmd/ps.go

+	},
+}
+
+func ps(ctx context.Context) error {


move to a method on the app

xmonader · 2024-07-24T21:06:53Z

grid-compose/cmd/ps.go

+	// try to get contracts with the old project name format "<name>"
+	contracts, err := t.ContractsGetter.ListContractsOfProjectName(projectName, true)
+	if err != nil {
+		return workloads.Deployment{}, err


consider returning a wrapper error with more context

xmonader · 2024-07-24T21:09:18Z

grid-compose/cmd/version.go

+	"github.com/spf13/cobra"
+)
+
+var version = "v0.0.1"


check how to properly do the versioning https://github.com/threefoldtech/tfgrid-sdk-go/blob/0c4a82b2e3fe68c8b70d4fac89bb3e31fecf8566/grid-cli/cmd/version.go

xmonader · 2024-07-24T21:11:25Z

grid-compose/pkg/types/types.go

@@ -0,0 +1,70 @@
+package types
+


All of these fields require validation e.g valid flist url, positive disk space, valid node id, depends_on a service and so on.

xmonader · 2024-07-24T21:13:20Z

grid-compose/internal/utils.go

+}
+
+func GetVmAddresses(vm workloads.VM) string {
+	var res string


use a string Builder for this https://pkg.go.dev/strings#Builder

xmonader · 2024-07-24T21:15:08Z

Please review the project structure, cobra commands should ONLY invoke methods defined on the app, don't leak any code logic into your commands. make sure the input is validated with decent error messages as well.

xmonader · 2024-08-04T12:15:30Z

still needs to handle the dependency resolution "and guard against cycles"

grid-compose/internal/utils.go

…nstraints

Omarabdul3ziz

good job ya eyad

i have some comments on the project structure

Omarabdul3ziz · 2024-08-15T11:11:01Z

grid-compose/cmd/down.go

+	},
+}
+
+func init() {


move registering commands to the init() in root.go

Omarabdul3ziz · 2024-08-15T11:12:24Z

grid-compose/cmd/ps.go

+
+func init() {
+	psCmd.PersistentFlags().BoolP("verbose", "v", false, "all information about deployed services")
+	psCmd.PersistentFlags().StringP("output", "o", "", "output result to a file")


we can drop this and simply use bash redirecting. grid-compose ps > dep.json

Omarabdul3ziz · 2024-08-15T11:13:51Z

grid-compose/cmd/ps.go

+	Run: func(cmd *cobra.Command, args []string) {
+		flags := cmd.Flags()
+
+		if err := app.Ps(cmd.Context(), flags); err != nil {


invoke the flags here, and pass to the method only the values it needs

Omarabdul3ziz · 2024-08-15T11:16:30Z

grid-compose/cmd/root.go

+
+var (
+	app        *internal.App
+	configPath string


to avoid the global variable, we could add the app to the context. it will shared between the commands

Omarabdul3ziz · 2024-08-15T11:17:44Z

grid-compose/cmd/root.go

+}
+
+func init() {
+	network = os.Getenv("NETWORK")


move getting the env to the Run in rootCmd

Omarabdul3ziz · 2024-08-15T11:34:26Z

grid-compose/internal/app.go

@@ -0,0 +1,372 @@
+package internal


a better structure for this file could be a separate package internal/app have the app.go file for the app struct & factory method. and a file for each command with its utils ps.go, down.go, ...

Omarabdul3ziz · 2024-08-15T11:34:37Z

grid-compose/internal/app.go

+
+// App is the main application struct that holds the client and the config data
+type App struct {
+	Client *deployer.TFPluginClient


is it needed to be public?

Omarabdul3ziz · 2024-08-15T11:35:00Z

grid-compose/internal/app.go

+// App is the main application struct that holds the client and the config data
+type App struct {
+	Client *deployer.TFPluginClient
+	Config *config.Config


i think it is not used anywhere? if so remove it

Omarabdul3ziz · 2024-08-15T11:38:43Z

grid-compose/internal/app.go

+
+				if !verbose {
+					if !dlAdded {
+						output.WriteString(fmt.Sprintf("%-15s | %-15s | %-15s | %-15s | %-10s | %s\n", contractDlData.Name, vm.NetworkName, vm.Name, vm.Mounts[0].DiskName, wl.Result.State, utils.GetVmAddresses(vm)))


could this formatting be abstracted? get the result as a map/array then pass it to some method maybe in pkg/logging/print_resutl.go it will be easier to make changes or even use a different pkg for formatting

Omarabdul3ziz · 2024-08-15T11:40:48Z

grid-compose/internal/app.go

+		}
+
+		// freeCRU is not in NodeFilter?
+		var freeMRU, freeSRU uint64


yes freeCRU is not a filter. due to cores overprovisioning mulitple vms can use the same cpu cores, use totalCRU instead just making sure the node has the cores you need

refactor: refactor folder structure and add more examples

refactor: refcator project structure

tests: test each example manually docs: populate readme file, add future_work, add config.md, and extend cases

…e and move its components to app

…dependency_resolution implementation of a docker-compose-like cli for managing deployment of services simultaneously

…into development_compose_init

refactor: move config to pkg and create storage parser validation: add extensive validation for services and config

…hboard

xmonader · 2024-09-29T20:51:57Z

Let's unblock this

Omarabdul3ziz and others added 4 commits June 25, 2024 15:11

init grid-compose

7503ba9

add network support

84cbd10

use yaml.v3 to fix the custom unmarshal

7f0d28b

feat: implement init logic for ps command

9e70865

eyad-hussein changed the title ~~implementation of a docker-compose-like cli interface for managing deployment of services simultaneously~~ implementation of a docker-compose-like cli for managing deployment of services simultaneously Jul 24, 2024

xmonader reviewed Jul 24, 2024

View reviewed changes

grid-compose/cmd/root.go Outdated

return specs, nil

}

func Execute() {

Copy link

Contributor

xmonader Jul 24, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

move this to the beginning of the file

xmonader reviewed Jul 24, 2024

View reviewed changes

grid-compose/cmd/root.go Outdated

var (

app App

configFile string

Copy link

Contributor

xmonader Jul 24, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

configPath

xmonader reviewed Jul 24, 2024

View reviewed changes

grid-compose/cmd/down.go Outdated

},

}

func down() error {

Copy link

Contributor

xmonader Jul 24, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

move to a method on App struct in internal/app.go

xmonader reviewed Jul 24, 2024

View reviewed changes

grid-compose/cmd/ps.go Outdated

},

}

func ps(ctx context.Context) error {

Copy link

Contributor

xmonader Jul 24, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

move to a method on the app

xmonader reviewed Jul 24, 2024

View reviewed changes

eyad-hussein mentioned this pull request Jul 25, 2024

INTERNSHIP - EYAD HUSSEIN codescalersinternships/home#118

Open

eyad-hussein added 2 commits August 4, 2024 19:50

refactor: refactor up and ps commands

40e67a2

feat: implement deployment of two or more services on same network

9fcedd2

github-advanced-security bot found potential problems Aug 4, 2024

View reviewed changes

grid-compose/internal/utils.go Fixed Show fixed Hide fixed

grid-compose/internal/utils.go Fixed Show fixed Hide fixed

grid-compose/internal/utils.go Fixed Show fixed Hide fixed

grid-compose/internal/utils.go Fixed Show fixed Hide fixed

eyad-hussein added 4 commits August 6, 2024 17:01

exmple: add more example files to cover test cases

09d55e2

refactor: refactor logic for all commands to comply with cases and co…

3b095a7

…nstraints

docs: edit readme file

9e227e1

docs: add supported cases for deployment, update readme

b0f3b22

Omarabdul3ziz requested changes Aug 15, 2024

View reviewed changes

eyad-hussein added 3 commits August 17, 2024 09:23

feat: add support for depends_on optoin(dependency resolution)

4bf7aaf

refactor: refactor folder structure and add more examples

feat: add support for health check option

352ef37

refactor: refcator project structure

docs: add future work file

de9f63a

eyad-hussein mentioned this pull request Aug 18, 2024

implementation of a docker-compose-like cli for managing deployment of services simultaneously #1155

Merged

8 tasks

eyad-hussein and others added 3 commits August 21, 2024 05:20

refactor: refactor project structure and code

07c5da9

tests: test each example manually docs: populate readme file, add future_work, add config.md, and extend cases

fix: add check for app type assertion, refactor: remove deploy packag…

124fe56

…e and move its components to app

Merge pull request #1155 from threefoldtech/development_compose_init_…

bd2aa64

…dependency_resolution implementation of a docker-compose-like cli for managing deployment of services simultaneously

eyad-hussein changed the title ~~implementation of a docker-compose-like cli for managing deployment of services simultaneously~~ implementation grid-compose Aug 22, 2024

eyad-hussein changed the title ~~implementation grid-compose~~ implement grid-compose Aug 22, 2024

eyad-hussein added 4 commits August 22, 2024 12:49

Merge branch 'development' of github.com:threefoldtech/tfgrid-sdk-go …

8ddab7b

…into development_compose_init

feat: add support for different storage units for service resources

deb6275

refactor: move config to pkg and create storage parser validation: add extensive validation for services and config

fix: fix storage parser

f21cf97

fix: change project name so type becomes Micro Virtual Machine in das…

bf550b0

…hboard

xmonader marked this pull request as ready for review September 29, 2024 20:51

xmonader requested review from MarioBassem, ashraffouda, rawdaGastan, AbdelrahmanElawady, sameh-farouk and Eslam-Nawara as code owners September 29, 2024 20:51


		type KVMap map[string]string

		func (m KVMap) UnmarshalYAML(value yaml.Node) error {

implement grid-compose #1130

Are you sure you want to change the base?

implement grid-compose #1130

Conversation

eyad-hussein commented Jul 24, 2024 • edited Loading

Description

Changes

Related Issues

Checklist

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

xmonader commented Jul 24, 2024

xmonader commented Aug 4, 2024

Omarabdul3ziz left a comment

Choose a reason for hiding this comment

Omarabdul3ziz Aug 15, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

xmonader commented Sep 29, 2024

eyad-hussein commented Jul 24, 2024 •

edited

Loading

Omarabdul3ziz Aug 15, 2024 •

edited

Loading