Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Enable a smoke test for UCX in pre-merge #3200

Merged
merged 5 commits into from
Aug 13, 2021
Merged

Enable a smoke test for UCX in pre-merge #3200

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
5 commits
Select commit Hold shift + click to select a range
ac5aec6
Enable smoke test with new marker to run with UCX in pre-merge
abellina Aug 11, 2021
1dd5bdf
Do not clean for the ucx component of it
abellina Aug 12, 2021
e0a0746
Remove "mvn package" call, and set a default shim value from SPARK_VER
abellina Aug 12, 2021
56bc8ba
Adjust pattern to trim out -SNAPSHOT as well
abellina Aug 13, 2021
7dba924
Replace -SNAPSHOT in a different statement, instead of attempting an or
abellina Aug 13, 2021
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions integration_tests/pytest.ini
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -24,3 +24,4 @@ markers =
cudf_udf: Mark udf cudf test
rapids_udf_example_native: test UDFs that require custom cuda compilation
validate_execs_in_gpu_plan([execs]): Exec class names to validate they exist in the GPU plan.
shuffle_test: Mark to include test in the RAPIDS Shuffle Manager
1 change: 1 addition & 0 deletions integration_tests/src/main/python/hash_aggregate_test.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -208,6 +208,7 @@ def get_params(init_list, marked_params=[]):
_init_list_no_nans_with_decimal = _init_list_no_nans + [
_grpkey_small_decimals]

@shuffle_test
@approximate_float
@ignore_order
@incompat
Expand Down
1 change: 1 addition & 0 deletions integration_tests/src/main/python/marks.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -24,3 +24,4 @@
qarun = pytest.mark.qarun
cudf_udf = pytest.mark.cudf_udf
rapids_udf_example_native = pytest.mark.rapids_udf_example_native
shuffle_test = pytest.mark.shuffle_test
29 changes: 29 additions & 0 deletions jenkins/spark-premerge-build.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -63,8 +63,37 @@ mvn_verify() {
jar xf $FILE
rm -rf com/nvidia/shaded/ org/openucx/
popd

# Triggering here until we change the jenkins file
rapids_shuffle_smoke_test
}

rapids_shuffle_smoke_test() {
echo "Run rapids_shuffle_smoke_test..."

# basic ucx check
ucx_info -d

# run in standalone mode
export SPARK_MASTER_HOST=localhost
export SPARK_MASTER=spark://$SPARK_MASTER_HOST:7077
$SPARK_HOME/sbin/start-master.sh -h $SPARK_MASTER_HOST
$SPARK_HOME/sbin/spark-daemon.sh start org.apache.spark.deploy.worker.Worker 1 $SPARK_MASTER

PYSP_TEST_spark_master=$SPARK_MASTER \
TEST_PARALLEL=0 \
PYSP_TEST_spark_cores_max=2 \
PYSP_TEST_spark_executor_cores=1 \
SPARK_SUBMIT_FLAGS="--conf spark.executorEnv.UCX_ERROR_SIGNALS=" \
PYSP_TEST_spark_shuffle_manager=com.nvidia.spark.rapids.$SHUFFLE_SPARK_SHIM.RapidsShuffleManager \
PYSP_TEST_spark_rapids_memory_gpu_minAllocFraction=0 \
PYSP_TEST_spark_rapids_memory_gpu_maxAllocFraction=0.1 \
PYSP_TEST_spark_rapids_memory_gpu_allocFraction=0.1 \
./integration_tests/run_pyspark_from_build.sh -m shuffle_test

$SPARK_HOME/sbin/spark-daemon.sh stop org.apache.spark.deploy.worker.Worker 1
$SPARK_HOME/sbin/stop-master.sh
jlowe marked this conversation as resolved.
Show resolved Hide resolved
}

unit_test() {
echo "Run unit testings..."
Expand Down
6 changes: 6 additions & 0 deletions jenkins/version-def.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -31,6 +31,12 @@ CUDA_CLASSIFIER=${CUDA_CLASSIFIER:-"cuda11"}
PROJECT_VER=${PROJECT_VER:-"21.10.0-SNAPSHOT"}
PROJECT_TEST_VER=${PROJECT_TEST_VER:-"21.10.0-SNAPSHOT"}
SPARK_VER=${SPARK_VER:-"3.0.1"}
# Make a best attempt to set the default value for the shuffle shim.
# Note that SPARK_VER for non-Apache Spark flavors (i.e. databricks,
# cloudera, and others) may not be a simple as just the version number, so
# this variable should be set accordingly.
SHUFFLE_SPARK_SHIM=${SHUFFLE_SPARK_SHIM:-spark${SPARK_VER//./}}
pxLi marked this conversation as resolved.
Show resolved Hide resolved
SHUFFLE_SPARK_SHIM=${SHUFFLE_SPARK_SHIM//\-SNAPSHOT/}
SCALA_BINARY_VER=${SCALA_BINARY_VER:-"2.12"}
SERVER_ID=${SERVER_ID:-"snapshots"}

Expand Down