updates to naming and versioning of internal packages and tools

[indirectlylit]

Status

This issue is currently a draft and open for feedback.

Current state

The Kolibri product ecosystem has a number of "internal" Node packages. These are currently tracked monorepo-style in the Kolibri repo, and some are used by other products. For example, Studio and KDS both use kolibri-tools.

These packages are currently defined within the packages directory:

package name	description
kolibri-tools	Utilities for building Kolibri plugins; (2) Linting code in Kolibri products; (3) i18n string handling
browserslist-config-kolibri	Supported browsers config used by kolibri-tools
eslint-plugin-kolibri	Custom eslint rules used by kolibri-tools
kolibri-core-for-export a.k.a. kolibri	Exposes importable interfaces for kolibri-tools
hashi (unpublished)	Untrusted JS sandbox with communication API

These are all implemented as Yarn 1.0 workspaces, along with kolibri/core and kolibri/plugins/*:

kolibri/package.json

Lines 5 to 9 in a7d1361

	"workspaces": [
	"packages/*",
	"kolibri/core",
	"kolibri/plugins/*"
	],

Currently, all of these plugins except for Hashi are supposed track Kolibri's version number, and get re-published to NPM on a regular basis. This occurs with the packages/publish.js tool. In practice, this process occurs rarely and is not documented. Currently all NPM dependencies are dev versions.

Some additional notes:

• Kolibri depends on kolibri-tools as a workspace-based package using symlinks
• Kolibri depends on hashi both as a workspace-based package and also through custom webpack build tooling in hashi that targets Kolibri's static folder as the output directory
• Functionality provided by kolibri-core-for-export could be replaced by an automocker in the jest configuration of kolibri-tools and bundling of these four dependency modules

Challenges

1. Lack of documentation
2. Circular dependency: Kolibri -> Design System -> kolibri-tools -> kolibri-core-for-export -> Kolibri
3. NPM packages are not being kept up-to-date, and it's not clear whether this matters or not. Most are at dev versions and use the next rather than latest tag
4. Unclear which packages are private to the Kolibri repo and which are public APIs for other products to use. Sometimes packages are being imported when they shouldn't be, e.g. in KDP, the OIDC client, and even Kolibri itself.

Proposal

Overview

new package name	code from	versioning	description
kolibri	kolibri-core-for-export	Tracking	The Kolibri core JS API
kolibri-build	kolibri-tools	Independent	Utilities for building Kolibri plugins
kolibri-test	kolibri-tools	Independent	Utilities for testing Kolibri plugins
kolibri-format	kolibri-tools	Independent	Linting and auto-formatting code in Kolibri products
kolibri-i18n	kolibri-tools	Independent	i18n string handling in Kolibri products
N/A	browserslist-config-kolibri	Independent	Supported browsers config, used by kolibri-plugin-builder
N/A	eslint-plugin-kolibri	Independent	Custom eslint rules, used by kolibri-code-formatting
kolibri-sandbox	hashi	Independent	Untrusted JS sandbox with communication API

N.B. any renaming of hashi will have to not be a global change, as this could cause significant breakage. Gradual, minimal updates are required, starting only with the name of the package.

Versioning

Most of the internal packages can be independently versioned and published because they change at different cadences and are highly decoupled from each other. Any package versioning labeled Independent above can bump its first official version to 1.0.0, and then increment minor, patch, and major versions as appropriate using semantic versioning conventions.

On the other hand, some tooling related to building and testing plugins has a tighter coupling to the Kolibri code-base, while still providing some degree of abstraction to allow plugins to be built and tested independently. Currently the kolibri-tools and kolibri-core-for-export packages fall into this category, and in this proposal only kolibri would. Here it's the versioning is labeled Tracking. It does not need to follow Kolibri patch releases, but the major and minor version numbers should always match.

Having independently versioned packages within the Kolibri repo is a bit strange because we use git tags that refer to the Kolibri version, while the other package versions are specified only in package.json files. In theory (and in practice) these packages can be released from any branch including develop or even forked feature branch. This can make it harder to associate particular released NPM versions with a particular git commit, branch, or tag. We will need to be careful not to abuse this ambiguity, and be diligent about documentation and communication for package updates.

Refactoring

There are four separable tools within kolibri-tools. Separating them will simplify the process of regularly publishing updates for the packages that need it while pinning stable versions of the packages that don't need regular updates.

• kolibri-build – Building Kolibri plugins without needing the entire Kolibri codebase at build-time.
• kolibri-test – Testing Kolibri plugins without needing the entire Kolibri codebase at test-time.
• kolibri-format – Linting and and autoformatting code.
• kolibri-i18n – Managing strings, context, crowdin, and other i18n concerns

Documentation

The Kolibri Dev Docs should have a new page covering how all of this works, and each internal package should have a top-level readme.md file with a high-level description of the package's purpose within the Kolibri product ecosystem.

[rtibbles]

It's not a blocker, but one slight complexity in separating the linting and building code, is our use of a custom module resolver that stops the linter complaining about our Kolibri API spec imports.

What this would mean is that the linting configuration for the kolibri-plugin-tools would have to enhance the configuration provided by kolibri-formatting-tools in order to substitute in the custom import resolver that handles this: https://github.com/learningequality/kolibri/blob/release-v0.15.x/packages/kolibri-tools/.eslintrc.js#L63

Another possibility would be to make kolibri-plugin-tools an unrequired peer dependency of kolibri-format-tools and set it in the case that it is present. A little interconnection, but might be simpler in the long run.

[indirectlylit]

in a thread @rtibbles mentioned that

maybe one day, we can make it happen automatically (although that might not jive with your proposed new versioning scheme)

This would be very helpful for the kolibri-plugin-tools/kolibri-tools/kolibri-core-for-export package.

Whether it makes sense for the other packages is an open question we should discuss. One issue is that it would make it harder to resolve the circular dependency between KDS and Kolibri if kolibri-formatting-tools upgrades in lock-step with Kolibri versioning.

[rtibbles]

Another minor tweak here is that we can't prefix all the packages with kolibri, as eslint plugins and browserslist configs have very specific naming requirements, which is why they are prefixed in this way.

[rtibbles]

This was only partially fixed by #12847 - wasn't intended to close it!