Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

The way to Go: Project Future Rewrite #33

Open
arcticicestudio opened this issue Jun 20, 2019 · 2 comments
Open

The way to Go: Project Future Rewrite #33

arcticicestudio opened this issue Jun 20, 2019 · 2 comments
Assignees
@arcticicestudio
Labels
context-api priority-high scope-maintainability scope-performance scope-security scope-stability scope-ux type-epic

Comments

@arcticicestudio
Copy link
Owner

arcticicestudio commented Jun 20, 2019 •
edited
Loading

The title should hint and summarize what this document is all about: The future of the snowsaw project formed by a complete rewrite in the awesome Go language.

Even though the project is still in a very early development state with only two release versions, this rewrite is a large step forward a way more stable project foundation and better designed code base.

🚧 This is a living document which means it is work in progress, not completed yet and will be extended!

All implementation details and requirements are documented and tracked in the corresponding issues:

To test the current development state or keep track of the completed tickets check out the epic/gh-33-the-way-to-go branch. See the linked ticket above and the development workflow section below for more details.

Please report every bug to help making the project more stable. Every feedback is always welcome! 💪

A Small Excerpt From The Project History

The origin of the project is a port of the great Dotbot.
I've searched for a tool to manage my .dotfiles and found long-time and stable projects like GNU Stow, Ansible or homesick as well as many more through great resources like GitHub's official .dotfiles website and awesome lists like awesome-dotfiles, but unfortunately none of them could fulfil all my requirements:

  1. KISS, DRY and the UNIX philosophy — Many developers can not resist the temptation to create monolith tools that are overloaded with solutions for multiple use cases and different targets. In my opinion development should always focus on the UNIX philosophy (“Do One Thing and Do It Well“) as well as the KISS (“keep it simple stupid“) and DRY (“Don't repeat yourself“) principles to create tools that are easy to develop, maintain, use, scale and provide a high reliability. In the next list points these principles will also match other parts of my requirements.
  2. Don't wrap Git commands — Many existing .dotfile manager try to provide more features to edit, update and persist the tracked files than necessary. They add CLI interfaces with commands like add, update or commit that are nothing else than wrapper around the Git add/commit core commands. Such features only add unnecessary complexity to the tool, reducing the transparency of what is really happening “under the hood“ and destroying the purpose of the UNIX philosophy (“Do One Thing and Do It Well“) as well as the KISS (“keep it simple stupid“) and DRY (“Don't repeat yourself“) principles. The only reason for such features might be that users don't need to know some simple Git basics (or Git at all), but if you're creating and tracking .dofiles the chance that you're not familiar with Git is close to zero. If you're modifying your .dotfiles in any way, Git provides you with all necessary tools and even if you're new to Git there are fantastic resources like Atlassian's Git guides and documentations that'll teach you the basics within several hours.
  3. No automatic Git actions — There are also many tools that automatically react to changes in existing files or new ones like adding, committing and pushing them to the connected repository. Like describes in the Nr.1 above, such features just blur the power of Git and reducing the transparency of what really gets executed: multiple sub-processes calling the actual Git commands.
  4. Easy integration and high portability — .dotfiles are the toolbox of every developer and often one of the main buttress for productivity, at least that's what I've experienced many times for me as well as for others. Everyone can work and develop thousand times faster and more fluidly when the apps and tools you use are configured to work without problems and fit your needs, your shell and terminal are set up with all the goodies like aliases and your favorite CLI apps are right at your hand. .dotfiles are one instrument to achieve this no matter if you're freshly setting up your new machine, working remotely via SSH or on someone else: Clone/Copy the .dotfile repository from your server or GitHub, download your .dotfile manager and let it bootstrap your files should be the only steps to get you up and running. That's a small example why portability of a .dotfile manager is important: it shouldn't require external dependencies, runtimes, interpreters, libraries or anything else. Just download the (binary) executable and you're done for 1/3 of these small steps. To learn more about the fact that snowsaw is written in Python, that in contrast to the listing indeed requires a interpreter/runtime (Python 3), please read the section below about the previous design decisions.
  5. Modular .dotfile structuring — Most tools require all dotfiles to be placed into one directory of the root of the Git repository which does not allow the user to structure the files and folders to match the personal preferences. The files must match the exact names and directory structures like their target symlinks or copy paths that doesn't allow customization or the sorting of files into some kind of category folders to aggregate them based on the target application, system or whatever ways users like to arrange them.
  6. Dynamics and configurability can go hand-in-hand with simplicity — The UNIX philosophy as well as the KISS principle both match the argument that the resulting app should “do one thing and do it well“, but this doesn't limit it's functionality to predefined tasks. The app should always provide sane defaults, but should also allows users to configure them if they doesn't match their personal preferences. This targets various aspects like the definition of conditions to only process files when they match, e.g. only for specific hostname(s) or when running on specific OS type(s). This requirement is related to the previous point (Nr.5), where users can use such configurations to achieve custom file and directory repository structures as well as making it easy to process only specific files based on conditions.

Based on these requirements I tested a lot of the existing tools and the ones that matched the most were Ansible and Dotbot, but unfortunately both also couldn't fulfil the requirements of being portable.
Ansible can convince with large ecosystem, a granular configurability and the usage and extensibility with modules, but also comes with a lot of overhead for small projects like a .dotfile repository. It is mainly targeted for the commercial administration of large, distributed systems and the setup is way too over engineered for such a use case.
Dotbot also provides flexible configuration features and can also convince through it's modularized design by using dedicated plugins for tasks like linking, copying or execution of commands through a shell process, but there were also features missing that were a must-have for me. The day after the evaluation was the birth of snowsaw.

Previous Design Decisions

Even though Dotbot is written in Python, that pulls in the dependency to the Python 3 interpreter and runtime, I've decided to base snowsaw on it. The decision was quite easy because to the time I've evaluated existing tools there was no stable and reliable project that was written in a portable language like Go, C/C++, Rust or anything else that (statically) compiles into a single (binary) artifact and also fulfils most of my requirements listed above.
In comparison to other similar projects like Dotbot that are written in Python, there is only one external Python library dependency next to the Python 2/3 runtime itself. It is not essential and only adds support to optionally write configuration files in YAML instead of only using JSON, but this is not a must-have requirement for snowsaw.

The facts described above lead to the decision to port Dotbot and implement the missing features. Even though I'm a long-time Linux user and have some experience in Python (wrote some scripts where a shell script might be too complicated), I don't like script languages at all and always prefer type-safe compile-time languages like Go, Rust or Java. The only exception is JavaScript when used for websites or Electron/Web apps with React which is in my opinion the best way to build a UI since web technologies like CSS were invented for it.
Next to this, Python also comes with the Python 2 to Python 3 ecosystem split-up and a likely broken package management, global vs. local package installations with pip (that also has a slightly complicated installation process itself) that can cause problems with native OS package managers (apt, yum, pacman etc.) because pip bypasses their tracking logic.

However, since Dotbot provides most of my desired features I decided to stay with Python.

snowsaw Goes Its Way

Like described in the project history above, the only reason to use Python was because of its Dotbot origins. During the development of more (requested) features and the fixing of bugs I often faced some problems everybody faces when writing in a language in which one is not so experienced. I always see such problems as opportunities to learn more about something new and gain experience, but after a while I unfortunately lost the interest iun Python for many reasons also described in the previous sections above.

In the meantime I expanded my knowledge in Go and until today I get more and more into love with this awesome language with each line of code. Some days ago I decided to take a few days off from porting all of Nord's port project to the shiny new website and wandering through my currently over 620 (!!!) notifications about open issues and PRs which are scattered in all my projects and other contributed repositories. After landing at snowsaw and trying to wrap my head around some of the pending tasks and how to solve them (with Python skills that are already dusted again 😄), I had the lightning thought (and wish) that it would be awesome if snowsaw would be written in my favorite language: Go. And that's the reason I'm currently writing this wall of text 😄

What To Expect

Before rewriting and reviving the project from its kind of „hyper sleep“ I want to make the process clear to all snowsaw users. Even though this started as a project for my personal use, it got some more attention and quite and larger user base. This means simply implementing everything and pushing it to the develop and master branches with a new version will break many users expectations and maybe their .dotfile setup too.

In order to carry out the project rewrite I want to clarify some general aspects and details:

  • No Breaking Changes For Existing Configurations — Changing the schema for the JSON snowblock.json configuration files means all existing setups will break if the changes are not adapted manually. There will be changes to the schema, but they will be handled through a schema version similar to the version field of docker-compose. This will help to differentiate between “legacy“ configurations and new ones, allowing to use a new Go language based snowsaw version with “legacy“ configurations. It will be made possible through a specialized handler that convert these configurations internally.
  • Backwards Compatibility — Like described in the bullet point above the new snowsaw implementation will be able to handle and process existing setups and will therefore be fully backwards compatible. The important note here is that this will be only valid for versions <1.0.0! As of v1.0.0 all code related to legacy support will be removed in order to achieve a clean and maintainable code base. For users who like to stay with a legacy snowsaw version, every version <1.0.0 will be suitable while the current Python-based snowsaw can also be used.
  • Drop Support For Python Based Implementation — As soon as the new Go implementation is merged into the develop branch and released in master through a new tagged version, the support for the Python based implementation will be dropped in aspects like feature requests, bug fixes or support/questions regarding the setup. This might sound a bit drastic, but my free time is really limited and the time I spend for the open source community shoots far beyond a normal volume (even though I will always enjoy every second of if 💚) so I can't effort to support code that only exists in the Git repository history anymore.

Bye Bye Loose Plugin Architecture

One of the larger features of snowsaw was the plugin architecture that allows extend snowsaw's functionality by dropping a Python script into the plugins directory in order to let snowsaw handle other tasks defined in any snowblock configuration file. By default snowsaw came with the three core plugins clean, link and shell to provide basic and most of the time completely sufficient tasks to handle almost everything needed to manage .dotfiles.
As far as I can tell (information only based on public repositories on GitHub!) most users of snowsaw never used custom plugins since the bundled ones served all necessary functions. This is a more or less relevant information since this means the omission of this feature for the new Go implementation will have almost no impact on the usability.
Adding a new plugin to handle other tasks was possible by satisfying the snowsaw.Plugin interface that requires the plugin to implement the can_handle() and handle methods. This more or less unstable pattern is the reason why this section's headline uses the „loose“ plugin architecture wording since Python is not designed for type safety as well as concepts like strict interface implementations. snowsaw was instructed to assume that the plugin author has read the documentations regarding the required behavior and return values of these functions.

Luckily Go is a type safe language and it's language design makes heavy use of interfaces that require correct implementations, but due to it's nature of being a compilation language it is not that easy to introduce a plugin system.
I've spend a lot of time to think about a way to keep the previous plugin-driven architecture up for the rewrite and evaluated the following possible solutions:

Go Standard Library plugin Package

Go comes with the plugin package by default that allows to load and resolve symbols of other Go artifacts, a so called „Go plugin“. It allows to load the files from anywhere on the same filesystem and make use of any exported type or function. It was first introduced in Go 1.8 and at the time sounded like the perfect solution to build modular and dynamic applications with endless expandability. Anyway, one downside was the restriction to be only compatible with Linux. Later on, Go 1.11 added support for macOS and support for Windows is on it's way.
A Go plugin can be easily compiled by simply using go build with the specific -buildmode=plugin flag in order to compile the target packages to a .so file. There are also more supported build modes, e.g. to create a shared library that can be imported into any other language like C or Python (buildmode=shared or buildmode=c-shared) or also to create position independent executables (PIE) through the -buildmode=pie flag. Anyway, I don't want to go into details here, but if you want to take a deep dive into this topic please take a look at the official plugin package documentations, go help buildmode and go help build as well as many other references and tutorials out there.

As beautiful as that sounds, there are also several difficulties when using Go plugins making it too hard to maintain and develop for such a small project like snowsaw. This is not the marching solution to let users add in their own code, they need to adhere to many rules, configurations/setups and conventions when building a custom Go plugin due to the following points:

  • The plugin compiler version must exactly match the application's compiler version. If the application was compiled with 1.11.4, it won't work to compile the plugin with 1.11.5. When distributing a program binary, you must communicate what the compiler version you used is.
  • Any packages outside of the standard library that are used by both the plugin and the application must have their versions match exactly. This means that when distributing a application binary, you must communicate the exact versions of all dependencies. This might be accomplished by publishing the output of GOOS=linux GOARCH=amd64 CGO_ENABLED=1 go list -deps -f='{{if not .Standard}}{{.Module}}{{end}}' <APPLICATION>, but plugin authors will need to also pin those versions (go.mod, Gopkg.toml etc.)
  • If there are any packages outside of the standard library that are used by both the plugin and the application, then the plugin must be compiled with the same GOPATH setting as the application (even if using modules). This means when using CircleCI as CI/CD service, users who use plugins must set GOPATH=/home/circleci/go, even though they don't have a circleci user.
  • In order to work on things like Alpine Linux's musl libc6-compat, everything must be compiled for compatibility with LSB 3. Setting _FORTIFY_SOURCE=2 with GNU libc causes the CGO 1.12 runtime to require LSB 4. Several distributions (including Ubuntu 14.04 used by CircleCI) patch their GCC to define _FORTIFY_SOURCE=2 by default. When compiling plugins, users may need to fuss with setting CGO_CPPFLAGS to make things not fall over.
  • Because plugins use the libc dynamic linker, they force CGO_ENABLED on sp cross-compiling is no longer easy to do. Someone wanting to compile a plugin for the GNU/Linux program binary from their macOS workstation must compile the plugin in Docker or any other VM.

Hopefully all these bullet points will be obsolete later on when the plugin packages gets improved with future Go versions, but in the meantime this is not the desired solution.

There are other plugin systems designs out there, e.g. by using Go's net/rpc package that allows the main application to communicate with plugins through remote procedure calls. A more advanced solution is the awesome go-plugin project by Hashicorp that brings all these functionalities out-of-the-box with a easy-to-use API, many additional feature and also full support to use the awesome gRPC project instead of the more basic (and limited) net/rpc package. They're using their own package in famous and busniess-critical projects like Terraform and Vault and I've also used it in some other private/public/dayjob projects. It's performance can not be compared to native Go plugins, but even in production with really heavy throughput there is no noticeable problem or bottleneck.
Anyway, even though a gRPC based solution for plugins for snowsaw would work really well, it is too over engineered and only brings in unnecessary complexity for such a small project that aims to lightweight and tries to follow the KISS princicle and Unix philosophy.

These are some facts which must be considered when snowsaw would use Go plugins and these are also all reasons why snowsaw won't adapt to this concept.
For more details, please read the official Go plugin package documentations, join the official Gophers Slack workspace and take a look at posts like this in the official /r/golang subreddit.

Long story short: The initial Go implementation of snowsaw won't use a plugin architecture anymore, but will come with necessary functionalities out-of-the-box to handle almost every use case for dotfile management. There will be a kind of „task“ API with interfaces that'll be implemented by snowsaw's core features and it will be exposed as exported types, allowing users to implement custom task handlers to extend snowsaw's capabilities. Later on a detailed documentation will be added plus resources to simplify the process of compiling the project together with custom task handlers, e.g. a Dockerfile that can be used to automatically place custom code in the correct package folder, build the project and copy the resulting artifact from the container to the host while leaving the host system in a clean state without the requirement to even clone and set up snowsaw's repository.

Next Steps

This document will serve as the epic issue and keeps track of all the sub-tickets that are listed at the top below the introduction paragraph. Before starting the actual implementation I will create the design concept tickets that'll be used to build the repository, documentation and code base from scratch. Note that this might take some time since it is not a high priority task and will be done step-by-step when there is some time left from the more urgent tasks like the Nord port project data transitions.

Development Workflow

Since this issue represents the main epic there will be a branch all results of the sub-tickets and stories will be merged into. As soon as everything is finally completed this branch will be merged into the main develop branch and later on into master to create a new version tag and deploy it. This way the rewrite can live together in parallel with the current code base without leaving it in an unusable state.

Build With & For The Community

Even though snowsaw was mainly developed for my personal use cases it is a open source project that means everyone can contribute to push the project forward and help to form its future.

If you like to test the new rewrite or keep track of the actual development state you can check out the epic/gh-33-the-way-to-go branch and follow the design concept documents and linked implementation ticket listed above.

Please report every bug to help making the project more stable. Every feedback is always welcome! 💪
@arcticicestudio arcticicestudio self-assigned this Jun 20, 2019
This was referenced Jun 22, 2019
Closed
Closed
Closed
Closed
Closed
Closed
Closed
Closed
Closed
Closed
Closed
Closed
Closed
Closed
arcticicestudio added a commit that referenced this issue Jun 22, 2019
@arcticicestudio
Initial repository clean up for Go rewrite
b311c41 
In order to start the Go project rewrite (1) from scratch the current
repository structure and files have been reset to a clean state to
remove all references to the previous implementations, documentations
and project structure/layout.

Starting from a "fresh" state allows to build the project up with the
correct structure and design pattern as if there were leftovers from
the previous repository data resulting in mixed files and folders.

This commit must be pushed first before all other blocked tickets can be
resolved that are also bound to the epic GH-33!
See the corresponding milestone (2) for more details about the
implementation/resolve order.

>>>> Tasks

- `.idea/` (3) - Deleted the whole folder, the files were scoped for
  "Pycharm Community Edition" and will be replaced with the correct
  files for "IntelliJ Ultimate Edition" with the official Go plugin (4)
  (Goland (5)) later on.
- `assets/` (6) - Deleted the whole folder, all assets will be
  redesigned and added again later on.
- `bin/` (7) - Deleted the whole folder, the script was part of the
  Python implementation and represented the entry point of the app.
- `snowsaw/` (8) - Deleted the whole folder, included the main Python
  app and API implementations.
- `.editorconfig` (9) - Deleted the file, it will be recreated in GH-38
  to match the new project layout and latest "Arctic Ice Studio" project
  design standards/guidelines.
- `.gitignore` (10) - Deleted the file, it will be recreated in GH-35 to
  match the new project layout and latest "Arctic Ice Studio" project
  design standards/guidelines.
- `CHANGELOG.md` (11) - Deleted the file, it will be recreated later on to
  match the new project layout and latest "Arctic Ice Studio" project
  design standards/guidelines.
- `README.md` (12) - Deleted the file, it will be recreated later on to
  match the new project layout and latest "Arctic Ice Studio" project
  design standards/guidelines including the new project assets
  (logo, repository hero etc.).

References:
  (1) #33
  (2) https://github.com/arcticicestudio/snowsaw/milestone/5
  (3) https://github.com/arcticicestudio/snowsaw/tree/bc54e5136be27f8037de5bbc2f046f37eb036274/.idea
  (4) https://plugins.jetbrains.com/plugin/9568-go
  (5) https://www.jetbrains.com/go
  (6) https://github.com/arcticicestudio/snowsaw/tree/bc54e5136be27f8037de5bbc2f046f37eb036274/assets
  (7) https://github.com/arcticicestudio/snowsaw/tree/bc54e5136be27f8037de5bbc2f046f37eb036274/bin
  (8) https://github.com/arcticicestudio/snowsaw/tree/bc54e5136be27f8037de5bbc2f046f37eb036274/snowsaw
  (9) https://github.com/arcticicestudio/snowsaw/blob/bc54e5136be27f8037de5bbc2f046f37eb036274/.editorconfig
  (10) https://github.com/arcticicestudio/snowsaw/blob/bc54e5136be27f8037de5bbc2f046f37eb036274/.gitignore
  (11) https://github.com/arcticicestudio/snowsaw/blob/bc54e5136be27f8037de5bbc2f046f37eb036274/CHANGELOG.md
  (12) https://github.com/arcticicestudio/snowsaw/blob/bc54e5136be27f8037de5bbc2f046f37eb036274/README.md

Epic: GH-33
Blocked GH-34 GH-35 GH-36 GH-37 GH-38 GH-39 GH-42 GH-43 GH-44 GH-45
        GH-46 GH-47 GH-48
Resolves GH-49
arcticicestudio added a commit that referenced this issue Jun 22, 2019
@arcticicestudio
MIT license
47ea77b 
Added the `LICENSE.md` file for the MIT license (1).

References:
  (1) https://opensource.org/licenses/MIT

Epic: GH-33
Depends on GH-49
GH-49
arcticicestudio added a commit that referenced this issue Jun 22, 2019
@arcticicestudio
MIT license
015227e 
Added the `LICENSE.md` file for the MIT license (1).

References:
  (1) https://opensource.org/licenses/MIT

Epic: GH-33
Depends on GH-49
GH-34
arcticicestudio added a commit that referenced this issue Jun 22, 2019
@arcticicestudio
MIT license (#51)
8093595 
Added the `LICENSE.md` file for the MIT license (1).

References:
  (1) https://opensource.org/licenses/MIT

Epic: GH-33
Depends on GH-49
GH-34
arcticicestudio added a commit that referenced this issue Jun 22, 2019
@arcticicestudio
Add Git ignore and attribute pattern
329a754 
Added the `.gitattributes` (1) and `.gitignore` (2) configuration files
for pattern handling that are matching the latest "Arctic Ice Studio"
project design standards/guidelines.

References:
  (1) https://git-scm.com/docs/gitattributes
  (2) https://git-scm.com/docs/gitignore

Epic: GH-33
Blocks GH-48
Depends on GH-49
Resolves GH-35
arcticicestudio added a commit that referenced this issue Jun 22, 2019
@arcticicestudio
Add Git mail mapping
162057c 
Added a Git mailmap (1) file to link to in documentations to allow
contributors to send mails regarding security issues. This prevents
unnecessary overhead of updating all documents when new core team and
members and contributors are added and additionally adds the main
functionality of the file:
Mapping commits when someone uses a different email address.

References:
  (1) https://git-scm.com/docs/git-shortlog#_mapping_authors

Epic: GH-33
Depends on GH-49
Resolves GH-46
arcticicestudio added a commit that referenced this issue Jun 22, 2019
@arcticicestudio
GitHub code owners
934bdac 
The project adapted to GitHub's code owners  feature. This allows to
define matching pattern for project paths to automatically add all
required reviewers of the core team and contributors to new PRs.

See GitHub Help (2) for more details.

References:
  (1) https://github.com/blog/2392-introducing-code-owners
  (2) https://help.github.com/articles/about-codeowners

Epic: GH-33
Depends on GH-49
Resolves GH-43
arcticicestudio added a commit that referenced this issue Jul 13, 2019
@arcticicestudio
Link task runner API implementation
4121393 
Implemented the `snowblock.TaskRunner` API interface to handle `link`
tasks from the original Python implementation (1).

References:
  (1) https://github.com/arcticicestudio/snowsaw/blob/3e3840824bf6f3d5cc09573b9505737473c7ed95/README.md#link

Epic GH-33
Resolves GH-74
@arcticicestudio arcticicestudio mentioned this issue Jul 14, 2019
Closed
arcticicestudio added a commit that referenced this issue Jul 15, 2019
@arcticicestudio
Clean task runner API implementation
c511fa1 
Implemented the `snowblock.TaskRunner` API interface to handle `clean`
tasks from the original Python implementation (1).

References:
  (1) https://github.com/arcticicestudio/snowsaw/blob/3e3840824bf6f3d5cc09573b9505737473c7ed95/README.md#clean

Epic: GH-33
Resolves GH-75
@arcticicestudio arcticicestudio mentioned this issue Jul 15, 2019
Closed
arcticicestudio added a commit that referenced this issue Jul 15, 2019
@arcticicestudio
Fix multiple tasks of same type only handled once per snowblock config
006ae99 
When a task is defined multiple times within the same snowblock
configuration file, only the last object was processed while any object
before has been ignored.
The root cause was the `pkg/snowblock.TaskRunnerMapping` (1) custom type
that only accepted one `pkg/api.TaskConfiguration` (2) object.
Therefore any parsed task object of the same type was overriden (2) by
tasks that are parsed after that task resulted in missing tasks.

Before this commit, running this example configuration has not processed
the first `clean` task but only the second one:

```json
[
  {
    "clean": ["~/desktop/failure"]
  },
  {
    "link": {
      "~/desktop/success/config.json": {
        "create": true,
        "path": "config.json"
      }
    }
  },
  {
    "clean": ["~/desktop/success"]
  },
]
```

To fix the problem the `pkg/snowblock.TaskRunnerMapping` type now
accepts multiple `pkg/api.TaskConfiguration` (2) objects
(`TaskRunnerMapping map[api.TaskRunner][]api.TaskConfiguration`) instead
of only one so the previous object won't be overridden.

References:
  (1) https://github.com/arcticicestudio/snowsaw/blob/efdff96ec01f26bbf0a0d75bb9aab4cb86f023e8/pkg/snowblock/snowblock.go#L46
  (2) https://github.com/arcticicestudio/snowsaw/blob/988073b1bde8d7db4b40f259e99d218c959bba8f/pkg/api/snowblock/task.go#L18
  (3) https://github.com/arcticicestudio/snowsaw/blob/efdff96ec01f26bbf0a0d75bb9aab4cb86f023e8/pkg/snowblock/snowblock.go#L112

Epic: GH-33
Fixes GH-76
@arcticicestudio arcticicestudio mentioned this issue Jul 15, 2019
Closed
arcticicestudio added a commit that referenced this issue Jul 15, 2019
@arcticicestudio
Global basedirs flag uses wrong default value
ed6226d 
The default value of the global `basedirs` flag (1) used the
`config.AppConfig.Snowblocks.Paths` array instead of
`config.AppConfig.Snowblocks.BaseDirs` that resulted in invalid paths
being processed by task runners and the configuration validators.

References:
  (1) https://github.com/arcticicestudio/snowsaw/blob/145a4c36caf960fc9a43b16c105df997e819a04d/cmd/snowsaw/snowsaw.go#L74

Epic: GH-33
Fixes GH-77
@arcticicestudio arcticicestudio mentioned this issue Jul 15, 2019
Closed
arcticicestudio added a commit that referenced this issue Jul 15, 2019
@arcticicestudio
Pass individual snowblock paths as arguments instead of flag values
9366c4a 
To simplify the CLI usage and removing the requirement to pass correctly
formatted (comma-separated) values to the `--snowblocks`/`-s` flag (1)
to process individual snowblocks the `bootstrap` command now accepts one
or more space-separated snowblock directory paths.

References:
  (1) https://github.com/arcticicestudio/snowsaw/blob/145a4c36caf960fc9a43b16c105df997e819a04d/cmd/snowsaw/bootstrap/bootstrap.go#L46

Epic: GH-33
Resolves GH-78
@arcticicestudio arcticicestudio mentioned this issue Jul 18, 2019
Closed
arcticicestudio added a commit that referenced this issue Jul 18, 2019
@arcticicestudio
Shell task runner API implementation
a78810b 
Implemented the `snowblock.TaskRunner` API interface to handle `shell`
tasks from the original Python implementation (1).

References:
  (1) https://github.com/arcticicestudio/snowsaw/blob/3e3840824bf6f3d5cc09573b9505737473c7ed95/README.md#shell

Epic: GH-33
Resolves GH-79
arcticicestudio added a commit that referenced this issue Jul 20, 2019
@arcticicestudio
Handle problems detected by used linters
8136905 
The problems in the code base detected by the linters that have been
integrated in GH-62 through GolangCI have been handled by refactoring
the affected implementations.
This helps to improve the overall code quality and prevents possible
errors.

1. Removed unused function parameters detected by unparam (1).
   1. `(*cmdOptions).prepare` - `cmd` is unused:
      cmd/snowsaw/bootstrap/bootstrap.go:51:30 (2)
      ```go
      func (o *cmdOptions) prepare(cmd *cobra.Command, args []string) {
                                   ^
      ```
   2. `(*cmdOptions).run` - `cmd` is unused:
      cmd/snowsaw/bootstrap/bootstrap.go:100:26 (2)
      ```go
      func (o *cmdOptions) run(cmd *cobra.Command, args []string) {
                               ^
      ```
   3. `(*cmdOptions).run` - `args` is unused:
      cmd/snowsaw/bootstrap/bootstrap.go:100:46 (2)
      ```go
      func (o *cmdOptions) run(cmd *cobra.Command, args []string) {
                                                   ^
      ```
2. Improved function names and code flows detected by golint (3).
   1. func `NewJsonEncoder` should be `NewJSONEncoder`:
      pkg/config/encoder/json/json.go:34:6 (4)
      ```go
      func NewJsonEncoder() Encoder {
           ^
      ```
   2. var `ExtensionsJson` should be `ExtensionsJSON`:
   pkg/config/encoder/constants.go:26:2 (5)
      ```go
      ExtensionsJson = "json"
      ^
      ```
   3. if block ends with a return statement, so drop this else and
      outdent its block (move short variable declaration to its own line
      if necessary): pkg/prt/printer.go:121:9 (6)
      ```go
      } else {
             ^
      ```
   4. exported func Load returns unexported type *builder.builder, which
      can be annoying to use: pkg/config/builder/builder.go:39:32 (7)
      ```go
      func Load(files ...*file.File) *builder {
                                     ^
      ```
3. Improved code style smells detected by gocritic (8).
   1. assignOp: replace `format = format + "\n"` with `format += "\n"`:
   pkg/prt/printer.go:179:4 (9)
      ```go
      format = format + "\n"
      ^
      ```
   2. paramTypeCombine: `func(v Verbosity, w io.Writer, prefix string,
      format string, args ...interface{})` could be replaced with
      `func(v Verbosity, w io.Writer, prefix, format string, args
      ...interface{})`: pkg/prt/printer.go:176:1 (10)
      ```go
      func (p *printerConfig) withNewLine(v Verbosity, w io.Writer,
      ^
      prefix string, format string, args ...interface{}) {
      ```
   3. emptyStringTest: replace `len(parts[0]) == 0` with `parts[0] ==
      ""`: pkg/snowblock/task/shell/shell.go:165:5 (11)
      ```go
      if len(parts[0]) == 0 {
         ^
      ```
   4. elseif: can replace 'else {if cond {}}' with 'else if cond {}':
   cmd/snowsaw/bootstrap/bootstrap.go:57:9 (12)
      ```go
      } else {
             ^
      ```
4. Remove unnecessary type conversions detected by unconvert (13).
   1. unnecessary conversion: pkg/prt/printer.go:132:16 (14)
      ```go
      *v = Verbosity(l)
                    ^
      ```

References:
  (1) https://github.com/mvdan/unparam
  (2) https://github.com/arcticicestudio/snowsaw/blob/9366c4a9c6d59dd0fccad12fbc413842ea751fa6/cmd/snowsaw/bootstrap/bootstrap.go#L51
  (3) https://github.com/golang/lint
  (4) https://github.com/arcticicestudio/snowsaw/blob/5aa483e7e5e45888254aa4d0143d2afb898b4332/pkg/config/encoder/json/json.go#L34
  (5) https://github.com/arcticicestudio/snowsaw/blob/008edbcb509af2cb5ced942d679fa3845a4ec1e1/pkg/config/encoder/constants.go#L26
  (6) https://github.com/arcticicestudio/snowsaw/blob/79afc12ebc15620fd94e78416e0b49a68bbf2eb6/pkg/prt/printer.go#L121
  (7) https://github.com/arcticicestudio/snowsaw/blob/dea6ab56b7410a8cbc8901818703d5ab1ace5c87/pkg/config/builder/builder.go#L39
  (8) https://github.com/go-critic/go-critic
  (9) https://github.com/arcticicestudio/snowsaw/blob/79afc12ebc15620fd94e78416e0b49a68bbf2eb6/pkg/prt/printer.go#L179
  (10) https://github.com/arcticicestudio/snowsaw/blob/79afc12ebc15620fd94e78416e0b49a68bbf2eb6/pkg/prt/printer.go#L176
  (11) https://github.com/arcticicestudio/snowsaw/blob/a78810b7ccb5ddb8e80929d54fb7c461a1b80a1c/pkg/snowblock/task/shell/shell.go#L165
  (12) https://github.com/arcticicestudio/snowsaw/blob/9366c4a9c6d59dd0fccad12fbc413842ea751fa6/cmd/snowsaw/bootstrap/bootstrap.go#L57
  (13) https://github.com/mdempsky/unconvert
  (14) https://github.com/arcticicestudio/snowsaw/blob/79afc12ebc15620fd94e78416e0b49a68bbf2eb6/pkg/prt/printer.go#L132

Epic: GH-33
Resolves GH-80
arcticicestudio added a commit that referenced this issue Oct 12, 2019
@arcticicestudio
Switch from npm to Yarn
b7de6db 
Since Yarn (1) is fully compatible with npm (2), but provides way more
useful features and uses a stable dependency solution strategy,
snowsaw now uses Yarn by default. Therefore the npm specific
`package-lock.json` file has been removed and replaced with Yarn's
`yarn.lock` file.

References:
  (1) https://yarnpkg.com
  (2) https://www.npmjs.com

Epic GH-33
Resolves GH-81
arcticicestudio added a commit that referenced this issue Oct 12, 2019
@arcticicestudio
Drop cross-compilation of FreeBSD binary artifacts
37ceafa 
Drop cross-compilation of FreeBSD binary artifacts

There were too many compatibility problems with some dependencies due to
native bindings. Therefore only builds for the main OS and architectures
(Linux, macOS and Windows) are compiled for now which covers almost all
users.

Epic GH-33
GH-83
arcticicestudio added a commit that referenced this issue Oct 12, 2019
@arcticicestudio
Drop cross-compilation of FreeBSD binary artifacts (#84)
3bb0b89 
There were too many compatibility problems with some dependencies due to
native bindings. Therefore only builds for the main OS and architectures
(Linux, macOS and Windows) are compiled for now which covers almost all
users.

Epic GH-33
Resolves GH-83
arcticicestudio added a commit that referenced this issue Oct 12, 2019
@arcticicestudio
Prevent go.mod file pollution with development dependencies
a8f34bc 
Previously when installing development dependencies through "mage",
the `go.mod` file was updated to include the installed packages since
this the default behavior of the `go get` command when running in
"module" mode.
To prevent the pollution of the project's Go module definition the
"module" mode has been disabled when installing the dev/build packages.
This is a necessary workaround until the Go toolchain is able to install
packages globally without updating the module file when the `go get`
command is run from within the project root directory.

See golang/go#30515 for more details and
proposed solutions that might be added to Go's build tools in future
versions.

Epic GH-33
GH-82
arcticicestudio added a commit that referenced this issue Oct 12, 2019
@arcticicestudio
Prevent go.mod file pollution with development dependencies (#85)
278d08e 
Previously when installing development dependencies through "mage",
the `go.mod` file was updated to include the installed packages since
this the default behavior of the `go get` command when running in
"module" mode.
To prevent the pollution of the project's Go module definition the
"module" mode has been disabled when installing the dev/build packages.
This is a necessary workaround until the Go toolchain is able to install
packages globally without updating the module file when the `go get`
command is run from within the project root directory.

See golang/go#30515 for more details and
proposed solutions that might be added to Go's build tools in future
versions.

Epic GH-33
Resolves GH-82
arcticicestudio added a commit that referenced this issue Oct 12, 2019
@arcticicestudio
Update to Go 1.13 and latest dependency versions
d7d88dc 
Go 1.13 has been released (1) over a month ago that comes with some
great features and a lot stability, performance and security
improvements and bug fixes. The new `os.UserConfigDir()` function (2) is
a great addition for the handling for snowsaw's configuration files that
will be implemented late on. See the Go 1.13 official release notes (3)
for more details.
Since there are no breaking changes snowsaw now requires Go 1.13 as
minimum version.

With the update to Go 1.13.x all outdated dependencies have also been
updated to their latest versions to prevent possible module
ncompatibilities as well as including the latest improvements and bug
fixes.

References:
  (1) https://blog.golang.org/go1.13
  (2) https://golang.org/pkg/os/#UserConfigDir
  (3) https://golang.org/doc/go1.13

Epic GH-33
GH-86
arcticicestudio added a commit that referenced this issue Oct 12, 2019
@arcticicestudio
Update to Go 1.13 and latest dependency versions (#87)
7fb34ca 
Go 1.13 has been released (1) over a month ago that comes with some
great features and a lot stability, performance and security
improvements and bug fixes. The new `os.UserConfigDir()` function (2) is
a great addition for the handling for snowsaw's configuration files that
will be implemented late on. See the Go 1.13 official release notes (3)
for more details.
Since there are no breaking changes snowsaw now requires Go 1.13 as
minimum version.

With the update to Go 1.13.x all outdated dependencies have also been
updated to their latest versions to prevent possible module
ncompatibilities as well as including the latest improvements and bug
fixes.

References:
  (1) https://blog.golang.org/go1.13
  (2) https://golang.org/pkg/os/#UserConfigDir
  (3) https://golang.org/doc/go1.13

Epic GH-33
Resolves GH-86
arcticicestudio added a commit that referenced this issue Oct 12, 2019
@arcticicestudio
Development dependency global installation workaround
2089def 
The workaround implemented in GH-82 (PR GH-85) worked fine, but due to
the explicitly disabled "module" mode it was not possible to define
pinned dependency versions but only using the normal `go get` behavior
to build the repositories default branch.

A better workaround is to run the `go get` command for development &
build dependencies/packages outside of the project's root directory.
Therefore the `go.mod` file is not in scope for the `go get` command and
is therefore not updated. In order to use pinned versions the
`GO1111MODULE=on` environment variable is explicitly set when running
the `go get` command.

See golang/go#30515 for more details and
proposed solutions that might be added to Go's build tools in future
versions.

Epic GH-33
GH-88
arcticicestudio added a commit that referenced this issue Oct 12, 2019
@arcticicestudio
Development dependency global installation workaround (#89)
213111d 
The workaround implemented in GH-82 (PR GH-85) worked fine, but due to
the explicitly disabled "module" mode it was not possible to define
pinned dependency versions but only using the normal `go get` behavior
to build the repositories default branch.

A better workaround is to run the `go get` command for development &
build dependencies/packages outside of the project's root directory.
Therefore the `go.mod` file is not in scope for the `go get` command and
is therefore not updated. In order to use pinned versions the
`GO1111MODULE=on` environment variable is explicitly set when running
the `go get` command.

See golang/go#30515 for more details and
proposed solutions that might be added to Go's build tools in future
versions.

Epic GH-33
Resolves GH-88
arcticicestudio added a commit that referenced this issue Oct 17, 2019
@arcticicestudio
Assemble app version with pure Go Git and SemVer libraries
bcd921b 
Previously the version of the application was determined by calling
`git` commands in a new shell process. This works in most cases,
but might fail if Git is not installed on the running system.
To prevent further enlargement of the required development environment
setup dependencies by adding more checks for external dependencies,
the `go-git` library v4 [1] (`github.com/src-d/go-git/v4`) has been
added:
"A highly extensible Git implementation in pure Go"

It allows to interact with the project repository and extrac required
information like the latest tag and commit of the current branch to
assemble the application version. To simplify the processing and parsing
of the version, the `semver` library v3 [2]
(`github.com/Masterminds/semver/v3`) has also been added.

The new `getAppVersionFromGit()` function assembles the version of the
application from the metadata of the Git repository.
It searches for the latest "SemVer" [3] compatible version tag in the
current branch and falls back to the default version from the
application configuration if none is found.
If at least one tag is found but it is not the latest commit of the
current branch, the build metadata will be appended, consisting of the
amount of commits ahead and the shortened reference hash (8 digits) of
the latest commit from the current branch.
The function is a early implementation of the Git `describe` command
because support in `go-git` [1] has not been implemented yet. See the
full compatibility comparison documentation with Git [4] as well as the
proposed Git `describe` command implementation [5] for more details.

[1]: https://github.com/src-d/go-git/v4
[2]: https://github.com/Masterminds/semver/v3
[3]: https://semver.org
[4]: https://github.com/src-d/go-git/blob/master/COMPATIBILITY.md
[5]: src-d/go-git#816

Epic GH-33
GH-92
svengreb pushed a commit that referenced this issue Oct 18, 2019
@arcticicestudio
Use composed struct to store application version information and meta…
d1b5f25 
…data

The function implemented in the parent commit returned a string which
results in loss of addiional data when used for other tasks.

To allow to use the version metadata for other tasks, like adding more
information to the compiled binary artifact, the function now returns a
custom struct that is composed of a `semver.Version` struct and
additional fields that stores Git related information:

- `GitCommitHash` (type `plumbing.Hash`) - The latest commit hash of the
  current branch.
- `GitCommitsAhead` (type `int`) - The count of commits ahead to the
  latest Git tag from the current branch.
- `GitLatestVersionTag` (type `plumbing.Reference`) - The latest Git
  version tag in the current branch.

Epic GH-33
GH-92
arcticicestudio added a commit that referenced this issue Oct 18, 2019
@arcticicestudio
App version with pure Go Git and SemVer libraries (#93)
532e680 
App version with pure Go Git and SemVer libraries

Previously the version of the application was determined by calling
`git` commands in a new shell process. This works in most cases,
but might fail if Git is not installed on the running system.
To prevent further enlargement of the required development environment
setup dependencies by adding more checks for external dependencies,
the `go-git` library v4 [1] (`github.com/src-d/go-git/v4`) has been
added:
"A highly extensible Git implementation in pure Go"

It allows to interact with the project repository and extrac required
information like the latest tag and commit of the current branch to
assemble the application version. To simplify the processing and parsing
of the version, the `semver` library v3 [2]
(`github.com/Masterminds/semver/v3`) has also been added.

The new `getAppVersionFromGit()` function assembles the version of the
application from the metadata of the Git repository.
It searches for the latest "SemVer" [3] compatible version tag in the
current branch and falls back to the default version from the
application configuration if none is found.
If at least one tag is found but it is not the latest commit of the
current branch, the build metadata will be appended, consisting of the
amount of commits ahead and the shortened reference hash (8 digits) of
the latest commit from the current branch.
The function is a early implementation of the Git `describe` command
because support in `go-git` [1] has not been implemented yet. See the
full compatibility comparison documentation with Git [4] as well as the
proposed Git `describe` command implementation [5] for more details.

The function returned a composed struct to store application version
information and metadata that allows to use the version metadata for
other tasks. It consists of a embedded `semver.Version` struct and
additional fields that stores Git related information:

- `GitCommitHash` (type `plumbing.Hash`) - The latest commit hash
   of the current branch.
- `GitCommitsAhead` (type `int`) - The count of commits ahead to the
  latest Git tag from the current branch.
- `GitLatestVersionTag` (type `plumbing.Reference`) - The latest Git
  version tag in the current branch.

[1]: https://github.com/src-d/go-git/v4
[2]: https://github.com/Masterminds/semver/v3
[3]: https://semver.org
[4]: https://github.com/src-d/go-git/blob/master/COMPATIBILITY.md
[5]: src-d/go-git#816

Epic GH-33
Resolves GH-92
svengreb pushed a commit that referenced this issue Oct 18, 2019
@arcticicestudio
Refactor info command and rename to version
2cd8e0a 
Previously the `info` command printed more detailed application
information compared to the compact and machine-friendly `--version`
global flag. The name of command was not optimal as it could give the
impression that it provides more "info"rmation about one or more
snowblocks that might be passed as argument(s). Therefore the command
has been renamed to `version` which also matches the naming of many
other Go CLI apps like Kubernetes [1] `kubectl` [2].

To also enhance the provided information the command now prints more
application version details using the function implemented in GH-93.

[1]: https://kubernetes.io
[2]: https://github.com/kubernetes/kubernetes/tree/master/pkg/kubectl

Epic GH-33
Relates to GH-93
GH-94
@arcticicestudio arcticicestudio mentioned this issue Oct 18, 2019
Merged
arcticicestudio added a commit that referenced this issue Oct 18, 2019
@arcticicestudio
Refactor info command and rename to version (#95)
64a432e 
Previously the `info` command printed more detailed application
information compared to the compact and machine-friendly `--version`
global flag. The name of command was not optimal as it could give the
impression that it provides more "info"rmation about one or more
snowblocks that might be passed as argument(s). Therefore the command
has been renamed to `version` which also matches the naming of many
other Go CLI apps like Kubernetes [1] `kubectl` [2].

To also enhance the provided information the command now prints more
application version details using the function implemented in GH-93.

[1]: https://kubernetes.io
[2]: https://github.com/kubernetes/kubernetes/tree/master/pkg/kubectl

Epic GH-33
Relates to GH-93
Resolves GH-94
@yaron2 yaron2 mentioned this issue Nov 3, 2019
Closed
@wren
Copy link

wren commented Aug 4, 2021

Hi! This project looks really interesting. Did it ever get off the ground?

@arcticicestudio
Copy link
Owner Author

arcticicestudio commented Aug 4, 2021 •
edited
Loading

@wren I'm still dogfooding the existing implementation (epic/gh-33-the-way-to-go branch) to see if it fits and works fine with an existing setup. So far I am satisfied and I think it's going in the right direction. The current blocker is just that I'm short on time to work on my all of my OSS and private projects. The Nord project continues to occupy a large part of my time, but some of the upcoming changes will (hopefully) reduce some maintenance overhead so that I can spend more time on other projects again.
Next up will be some clean ups of the current implementation and improvements I planned while building other Go projects, e.g. setting up a build system with my wand project, migrating the repository to the GitHub flow and the master branch to main and adapting my new repository organization for Go projects. Actual features and improvements like the YAML support and a way to use “wildcard bootstrapping“ are then on the list.

In summary, that means that the plans documented in this issue are still up-to-date and, as soon as time permits, more improvements will follow. I'll keep the entry post up-to-date and post comments here so subscribers get notified 😄

I'm also always interested in feedback from others so feel free to clone the epic/gh-33-the-way-to-go branch and run go build -o snowsaw . from within the repository root to build the binary. There is currently no managed installation process so the binary must be manually placed somewhere in your PATH, but support for package managers like AUR, Homebrew and Scoop will also follow later on.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@arcticicestudio arcticicestudio

Labels
context-api priority-high scope-maintainability scope-performance scope-security scope-stability scope-ux type-epic
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@arcticicestudio @wren