Update version handling

[jameslamb]

Description

As of this writing, updating the package version involves manually modifying a VERSION file:

rapids-cli/VERSION

Line 1 in 4a846b9

0.1.1

And that file's value then flows into wheels

rapids-cli/pyproject.toml

Lines 47 to 48 in 4a846b9

	[tool.setuptools.dynamic]
	version = {file = "VERSION"}

and conda packages

rapids-cli/ci/build_python.sh

Lines 10 to 11 in 4a846b9

	RAPIDS_PACKAGE_VERSION=$(head -1 ./VERSION)
	export RAPIDS_PACKAGE_VERSION

rapids-cli/conda/recipes/rapids-cli/recipe.yaml

Line 5 in 4a846b9

version: ${{ env.get("RAPIDS_PACKAGE_VERSION") }}

rapids-cli/conda/recipes/rapids-cli/recipe.yaml

Line 14 in 4a846b9

version: ${{ version }}

This totally works!

For example, it's how pynvjitlink has been managing its version. Development happens on main, and then when you're ready to release a batch of functionality, you put up a PR changing the version number in VERSION, like rapidsai/pynvjitlink#139

But there are some reasons we might not want to stay with this approach:

• not aligned with git tags / GitHub releases, which makes cloning and building from source a bit more difficult
• doesn't easily support publishing nightly versions on merge
  • (not sure if we care about that for this project, we might not)
• creates extra development and reviewing churn
  • (because of the need to open PRs that update VERSION)

This documents the current state and some reasons we might want to change it.

Benefits of this work

• addresses some of the concerns listed above

Acceptance Criteria

• version-management for the repo is in a long-term state we're comfortable with
• documentation has been added explaining how to update the version and publish releases

Approach

The approach we set up in the rapidsai/legate-* packages could be a good starting point:

• https://github.com/rapidsai/legate-boost/blob/7b94a53c608cb0cb58847af514eadd12536e214d/contributing.md#change-the-project-version
• https://github.com/rapidsai/legate-boost/blob/7b94a53c608cb0cb58847af514eadd12536e214d/contributing.md#releasing

The main idea is to set git tags with versions, and then pull those in with rapids-generate-version at runtime, like this:

rapids-generate-version > ./VERSION

(rapidsai/legateboost/ci/build_python.sh)

Notes

Merges to main here don't automatically publish new packages to nightly conda / pip repos if the content of VERSION has not changed relative to the previous upload.

Because we pass --skip-existing to the Anaconda CLI:

https://github.com/rapidsai/gha-tools/blob/d4c9117c231314272b0444173dc0623d4ba3bb90/tools/rapids-upload-to-anaconda-github#L56

https://github.com/rapidsai/gha-tools/blob/d4c9117c231314272b0444173dc0623d4ba3bb90/tools/rapids-wheels-anaconda-github#L69

[jacobtomlinson]

In many projects I work on we just use the hatchling build backend and the hatch-vcs plugin to pull version information from git tags. Configuring this is very minimal, is there a reason why this approach isn't suitable?

[jameslamb]

The rapids-generate-version approach is in some ways even more lightweight... it uses dunamai (mtkennerly/dunamai, gha-tools code link) which is purpose-built for dynamic versioning, while hatchling is a much bigger, faster-moving project that is built to do a lot more than that.

For example, look at all these dependencies hatchling pulls into build environment... it's a lot: https://github.com/pypa/hatch/blob/4ebce0e1fe8bf0fcdef587a704c207a063d72575/pyproject.toml#L41-L59

In contrast, for the Python versions rapids-cli supports, dunamai only pulls in a dependency on packaging (code link).

The biggest downside of the rapids-generate-version approach that I see, relative to hatch-vcs, setuptools_scm, etc., is that it happens outside of the package build system... it's an invocation in a shell script instead of something that runs at pip wheel / python -m build time.

But in RAPIDS that hasn't been a practical problem as far as I know... we only distribute wheels (not sdists), and haven't really needed to worry about precise versions in local builds.

Doing this the same way across all of RAPIDS also makes it easier to work on these libraries... once you've learned how things work for 1, you've learned most of what you need to work on the others.

All that said... if I picked up this issue, I'd use the rapids-generate-version approach I described above. But "get version information from tags" is more important here than the exact mechanism, so I wouldn't automatically reject a PR that proposed using hatchling.

[jacobtomlinson]

You're thinking of hatch the CLI project management tool, not hatchling the build backend. They are two separate packages that are part of the same overall project. The hatchling dependency is also very small and only pulls in packaging, pathspec, tomli and trove-classifiers. https://github.com/pypa/hatch/blob/4ebce0e1fe8bf0fcdef587a704c207a063d72575/backend/pyproject.toml#L37-L43

The biggest downside of the rapids-generate-version approach that I see, relative to hatch-vcs, setuptools_scm, etc., is that it happens outside of the package build system

This is what concerns me. I've seen this go wrong many times in other projects. Any time you have to run a magic bash script before you do anything leaves you open to user error.

Also with this approach what happens if you pip install directly from a git url? Does it think it's installing whatever version was last committed in the VERSION file? With a build based version like hatch-vcs and setuptools_scm it calculates the correct version at install time.

Doing this the same way across all of RAPIDS also makes it easier to work on these libraries... once you've learned how things work for 1, you've learned most of what you need to work on the others.

I do understand the appeal of this though.

[jameslamb]

You're thinking of hatch the CLI project management tool, not hatchling the build backend.

Blegh sorry! Thanks for correcting me.

Also with this approach what happens if you pip install directly from a git url? Does it think it's installing whatever version was last committed in the VERSION file?

Right, it would. My contention is that that doesn't really matter much for this project... it's unlikely you'll have anything else locally that imposes some other constraint on the version of rapids-cli and particularly cares about the exact version.

That said, the patterns in RAPIDS are not really optimized for small, pure-Python projects like this... they're optimized for much more complex projects, where "install from a git URL" isn't a thing we really even try to support (today). So if you wanted to try hatchling + hatch-vcs here, like I mentioned I wouldn't automatically resist it.