Enable a smoke test for UCX in pre-merge

[abellina]

This PR runs test_hash_grpby_sum with UCX enabled, as UCX is now installed in the docker image.

This adds ~1-2 minutes to the build from the logs. It starts the smoke test from the mvn_verify function so it could be merged without disturbing the jenkins file. The mvn_verify and now rapids_shuffle_smoke_test assume spark 3.0.1 for the pre-merge build. There will be a follow on PR to clean up how rapids_shuffle_smoke_test gets invoked, though right now I piggy back from mvn_verify binaries.

I added a variable so the correct shuffle manager shim can be picked up (SHUFFLE_SPARK_SHIM). This should be kept in sync with SPARK_VER.

There is a local-cluster issue I noted which happens even without the shuffle manager and will file a follow up, as it seems unrelated. I am inclined to let this use standalone mode for now. I could use some more eyes on it to see if it makes sense.

Also I chose a simple test, it does use the UCX shuffle. I can enable others in case it's not going to change that 1-2 minute time addition too much, or pick something else. Just pick the first thing that I knew would use UCX.

[abellina]

build

[abellina]

build

[abellina]

build

[abellina]

build

[abellina]

This seems to mostly be working, but I am getting an issue in local-cluster mode where executors get created after the tests (and I mean 100 executors or so). The logs show UCX is utilized, but something is happening later. Note this issue with the runaway executors is with or without the manager, so likely a pytest config issue on my side.

[abellina]

build

[abellina]

build

[abellina]

build

[abellina]

Waiting on: #3218 to be merged and image pushed.

[abellina]

build

[abellina]

build

[jlowe]

build

[abellina]

build

[pxLi]

SHUFFLE_SPARK_SHIM=spark3.0.1
...
java.lang.ClassNotFoundException: com.nvidia.spark.rapids.spark3.0.1.RapidsShuffleManager

Looks like the pattern matching does not work consistently in different shell envs, I don't know if we have better way to do replace in bash default_value syntax. At least we can trim first, then assign a default value

[abellina]

Looks like the pattern matching does not work consistently in different shell envs

I tested this with the command line, instead of within bash, so I was able to reproduce the issue locally. I got it to work with 2 statements, but was not able to with a single statement.

[abellina]

build