Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add rapids-wheels-anaconda tool #86

Merged
merged 1 commit into from
Nov 13, 2023
Merged
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
56 changes: 56 additions & 0 deletions tools/rapids-wheels-anaconda
Original file line number Diff line number Diff line change
@@ -0,0 +1,56 @@
#!/bin/bash
# A utility script to upload Python wheel packages to Anaconda repository using anaconda-client.

# Positional Arguments:
# 1) wheel name

set -exou pipefail
source rapids-constants
export RAPIDS_SCRIPT_NAME="rapids-wheels-anaconda"

if [ -z "$1" ]; then
rapids-echo-stderr "Must specify input arguments: WHEEL_NAME"
exit 1
fi
WHEEL_NAME="$1"

WHEEL_SEARCH_KEY="wheel_python_${WHEEL_NAME}"

WHEEL_DIR="./dist"
mkdir -p "${WHEEL_DIR}"

S3_PATH=$(rapids-s3-path)
BUCKET_PREFIX=${S3_PATH/s3:\/\/${RAPIDS_DOWNLOADS_BUCKET}\//} # removes s3://rapids-downloads/ from s3://rapids-downloads/ci/rmm/...

# shellcheck disable=SC2016
WHEEL_TARBALLS=$(
set -eo pipefail;
aws \
--output json \
s3api list-objects \
--bucket "${RAPIDS_DOWNLOADS_BUCKET}" \
--prefix "${BUCKET_PREFIX}" \
--page-size 100 \
--query "Contents[?contains(Key, '${WHEEL_SEARCH_KEY}')].Key" \
| jq -c
)
export WHEEL_TARBALLS

# first untar them all
for OBJ in $(jq -nr 'env.WHEEL_TARBALLS | fromjson | .[]'); do
FILENAME=$(basename "${OBJ}")
S3_URI="${S3_PATH}${FILENAME}"

rapids-echo-stderr "Untarring ${S3_URI} into ${WHEEL_DIR}"
aws s3 cp --only-show-errors "${S3_URI}" - | tar xzf - -C "${WHEEL_DIR}"
done

export RAPIDS_RETRY_SLEEP=180
# shellcheck disable=SC2086
rapids-retry anaconda \
-t "${RAPIDS_CONDA_TOKEN}" \
upload \
--skip-existing \
--no-progress \
"${WHEEL_DIR}"/*.whl
echo ""