Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Apply new versioning scheme #398

Merged
merged 2 commits into from
Oct 31, 2024
Merged
Show file tree
Hide file tree
Changes from 1 commit
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
44 changes: 25 additions & 19 deletions .github/workflows/build_packages.yml
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Did you test this workflow? Should be possible with workflow_dispatch over at https://github.com/marbre/SHARK-Platform/actions/workflows/build_packages.yml

... which reminds me, we may want to disable it in forks somehow. I tested at https://github.com/ScottTodd/SHARK-Platform/actions/workflows/build_packages.yml, generating these: https://github.com/ScottTodd/SHARK-Platform/releases/tag/dev-wheels

I see that several forks are running the action too, which is failing because of the missing token: https://github.com/nithinsubbiah/sharktank/actions/workflows/build_packages.yml but some runs did succeed at one point: https://github.com/nithinsubbiah/sharktank/releases/tag/dev-wheels

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks promising :)
https://github.com/marbre/SHARK-Platform/actions/runs/11615896057/job/32347586648#step:4:1434

******************** BUILD COMPLETE ********************
Generated binaries:
total 2512
-rw-r--r-- 1 root root 2568751 Oct 31 16:31 shortfin-3.0.0rc20241031-cp312-cp312-manylinux_2_17_x86_64.manylinux2014_x86_64.whl

Original file line number Diff line number Diff line change
Expand Up @@ -21,25 +21,31 @@ jobs:
outputs:
shark_package_version: ${{ steps.version.outputs.shark_package_version }}
steps:
# For now the version is just a calendar date + an automatically
# incrementing value. We may want different versions for nightly/dev
# builds and stable releases published to official places like pypi.
- name: Compute version
id: version
- name: Checkout repository
uses: actions/checkout@11bd71901bbe5b1630ceea73d27597364c9af683 # v4.2.2
with:
submodules: false
- name: Setup Python
uses: actions/setup-python@0b93645e9fea7318ecaed2b359559ac225c90a2b # v5.3.3
with:
python-version: 3.12
cache: "pip"
- name: Install Python packages
run: |
pip install packaging
pip freeze
- name: Generate release candidate versions
id: version_rc
run: |
shark_package_version="$(printf '%(%Y%m%d)T.${{ github.run_number }}')"
echo "shark_package_version=${shark_package_version}" >> $GITHUB_OUTPUT
cat << EOF > ./version_info.json
{
"package-version": "${shark_package_version}"
}
EOF
cat ./version_info.json
- name: Upload version_info.json
sharktank_package_version=$(python3 build_tools/gen_version_info_rc.py sharktank)
shortfin_package_version=$(python3 build_tools/gen_version_info_rc.py shortfin)
- name: Upload version_info_rc.json
uses: actions/upload-artifact@50769540e7f4bd5e21e526ee35c689e35e0d6874 # v4.4.0
with:
name: version_info
path: version_info.json
name: version_info_rc
path: |
sharktank/version_info_rc.json
shortfin/version_info_rc.json

build_packages:
name: "${{ matrix.package }} :: ${{ matrix.platform }} :: ${{ matrix.python-version }}"
Expand Down Expand Up @@ -74,11 +80,11 @@ jobs:
path: "c" # Windows can hit path length limits, so use a short path.
submodules: false

- name: Download version_info.json
- name: Download version_info_rc.json
uses: actions/download-artifact@fa0a91b85d4f404e444e00e005971372dc801d16 # v4.1.8
with:
name: version_info
path: ./c/shortfin/
name: version_info_rc
path: ./c/
merge-multiple: true

- name: Build shortfin (Linux x86_64, ${{ matrix.python-version }})
Expand Down
48 changes: 48 additions & 0 deletions build_tools/gen_version_info_rc.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,48 @@
# Copyright 2024 Advanced Micro Devices, Inc.
#
# Licensed under the Apache License v2.0 with LLVM Exceptions.
# See https://llvm.org/LICENSE.txt for license information.
# SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception

# This scripts grabs the X.Y.Z[.dev]` version identifier from a
# `version_info.json` and writes the corresponding
# `X.Y.ZrcYYYYMMDD` version identifier to `version_rc_info.json`.

import argparse
from pathlib import Path
import json
from datetime import datetime

from packaging.version import Version


parser = argparse.ArgumentParser()
parser.add_argument("path", type=Path)
args = parser.parse_args()

VERSION_INFO_FILE = args.path / "version_info.json"
VERSION_INFO_RC_FILE = args.path / "version_info_rc.json"


def load_version_info():
with open(VERSION_INFO_FILE, "rt") as f:
return json.load(f)


def write_version_info():
with open(VERSION_INFO_RC_FILE, "w") as f:
json.dump(version_info_rc, f, indent=2)
f.write("\n")


version_info = load_version_info()

PACKAGE_VERSION = version_info.get("package-version")
PACKAGE_BASE_VERSION = Version(PACKAGE_VERSION).base_version
PACKAGE_RC_VERSION = PACKAGE_BASE_VERSION + "rc" + datetime.today().strftime("%Y%m%d")

version_info_rc = {"package-version": PACKAGE_RC_VERSION}

write_version_info()

print(PACKAGE_RC_VERSION)
2 changes: 2 additions & 0 deletions sharktank/.gitignore
Original file line number Diff line number Diff line change
@@ -0,0 +1,2 @@
# Local-only config options
version_info_rc.json
15 changes: 11 additions & 4 deletions sharktank/setup.py
Original file line number Diff line number Diff line change
Expand Up @@ -14,15 +14,22 @@

# Setup and get version information.
VERSION_INFO_FILE = os.path.join(SETUPPY_DIR, "version_info.json")
VERSION_INFO_RC_FILE = os.path.join(SETUPPY_DIR, "version_info_rc.json")


def load_version_info():
with open(VERSION_INFO_FILE, "rt") as f:
def load_version_info(version_file):
with open(version_file, "rt") as f:
return json.load(f)


version_info = load_version_info()
PACKAGE_VERSION = version_info["package-version"]
try:
version_info = load_version_info(VERSION_INFO_RC_FILE)
except FileNotFoundError:
print("version_info_rc.json not found. Default to dev build")
version_info = load_version_info(VERSION_INFO_FILE)
Comment on lines +25 to +29
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Oh, interesting choice to have two different files. Did you consider other approaches? I can think of a few:

  • Overwrite the file (makes git index dirty for development though)
  • Have setup.py load from version_info.json and do the PACKAGE_RC_VERSION = PACKAGE_BASE_VERSION + "rc" + datetime.today().strftime("%Y%m%d") here

Multiple ways to do it, and what you have works.

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think there are even more:

  • Read the version from a version_info (text) and override with the updated one.
    • Use the version info file in the pyproject.toml making the setup.py obsolete:
      version = {file = version_info}
    • Drawback: Git gets dirty, limited flexibility.

With regards to the ones you listed:

  • I would like to avoid that the index gets dirty.
  • This introduces a dependency on packaging which isn't strictly needed. Furthermore, with this every builder calculates the version instead of computing this once in the CI.

But I am happy to iterate in a follow up but this one should also be compatible with what we will need for IREE.


PACKAGE_VERSION = version_info.get("package-version")
print(f"Using PACKAGE_VERSION: '{PACKAGE_VERSION}'")

setup(
version=f"{PACKAGE_VERSION}",
Expand Down
2 changes: 1 addition & 1 deletion sharktank/version_info.json
Original file line number Diff line number Diff line change
@@ -1,3 +1,3 @@
{
"package-version": "0.1.dev3"
"package-version": "3.0.0.dev"
}
2 changes: 1 addition & 1 deletion shortfin/.gitignore
Original file line number Diff line number Diff line change
@@ -1,2 +1,2 @@
# Local-only config options
version_info.json
version_info_rc.json
28 changes: 7 additions & 21 deletions shortfin/setup.py
Original file line number Diff line number Diff line change
Expand Up @@ -141,36 +141,22 @@ def copy_extensions_to_source(self, *args, **kwargs):


# Setup and get version information.
VERSION_INFO_FILE = os.path.join(SOURCE_DIR, "version_info.json")
VERSION_INFO_FILE = os.path.join(REL_SOURCE_DIR, "version_info.json")
VERSION_INFO_RC_FILE = os.path.join(REL_SOURCE_DIR, "version_info_rc.json")


def load_version_info():
with open(VERSION_INFO_FILE, "rt") as f:
def load_version_info(version_file):
with open(version_file, "rt") as f:
return json.load(f)


def find_git_version():
try:
return (
subprocess.check_output(["git", "rev-parse", "HEAD"], cwd=SOURCE_DIR)
.decode("utf-8")
.strip()
)
except subprocess.SubprocessError as e:
print(f"ERROR: Could not get git revision: {e}", file=sys.stderr)
return None


try:
version_info = load_version_info()
version_info = load_version_info(VERSION_INFO_RC_FILE)
except FileNotFoundError:
print("version_info.json not found. Using defaults", file=sys.stderr)
version_info = {}
git_version = find_git_version()
print("version_info_rc.json not found. Default to dev build")
version_info = load_version_info(VERSION_INFO_FILE)

PACKAGE_VERSION = version_info.get("package-version")
if not PACKAGE_VERSION:
PACKAGE_VERSION = f"0.dev0+{git_version or '0'}"
print(f"Using PACKAGE_VERSION: '{PACKAGE_VERSION}'")


Expand Down
3 changes: 3 additions & 0 deletions shortfin/version_info.json
Original file line number Diff line number Diff line change
@@ -0,0 +1,3 @@
{
"package-version": "3.0.0.dev"
}
Loading