Spawn multiple pytest workers to speedup CI

[lijinf2]

with pytest -n 4 on a V100 32GB GPU,
run_tests.sh got passed in 20 mins.
run_tests.sh --runslow got passed in 40 mins.
peak CPU memory usage was around 2GB.

[lijinf2]

build

[lijinf2]

pre-merge CI took about 50 mins

[eordentlich]

Wasn't it taking 50 min before?

[eordentlich]

@pxLi @YanxuanLiu Any potential issues with this?
Seems to give 2x speedup in premerge.

[lijinf2]

build

[lijinf2]

build

[pxLi]

@pxLi @YanxuanLiu Any potential issues with this? Seems to give 2x speedup in premerge.

It may cause some GPU OOM or host memory OOM as in the CI pool, parts of machines have only 16/24GB GPU memory.

It would be nice for the ML, the test script could detect the gpu type (mem size) and then dynamically set the parallelism like e.g. https://github.com/NVIDIA/spark-rapids/blob/branch-24.12/integration_tests/run_pyspark_from_build.sh#L113-L134 instead of hardcode the numbers thanks

[lijinf2]

build

[lijinf2]

build

[eordentlich]

Do these reflect container limits or VM limits? And where did denominators come from?

[lijinf2]

denominator is inherited from the script that Peixin shared. The script expects 8GB peak memory usage during the whole testing. Ours was 2.5 GB I observed. 8GB is much larger than that so also works for us.

@pxLi Do you have an idea on whether the the /proc/meminfo reflects container limits or VM limits?

[lijinf2]

Revised back to -n 3 instances.

[pxLi]

Do you have an idea on whether the the /proc/meminfo reflects container limits or VM limits?

the info is for local machine but not the correct value in docker.

please give it a hard upper bound if possible as the GPU memory size is always correct~

like

if dynamical parallelism > upper_bound then go with upper bound. The link that I gave above would also covered the developer local run, and spark have a hard limit in the same script https://github.com/NVIDIA/spark-rapids/blob/branch-24.12/integration_tests/run_pyspark_from_build.sh#L174-L186

[lijinf2]

@pxLi @YanxuanLiu Any potential issues with this? Seems to give 2x speedup in premerge.

It may cause some GPU OOM or host memory OOM as in the CI pool, parts of machines have only 16/24GB GPU memory.

It would be nice for the ML, the test script could detect the gpu type (mem size) and then dynamically set the parallelism like e.g. https://github.com/NVIDIA/spark-rapids/blob/branch-24.12/integration_tests/run_pyspark_from_build.sh#L113-L134 instead of hardcode the numbers thanks

Thanks for sharing the link.
Added the dynamic parallelism with a few modifications:
(1) calculate GPU_PARALLEL with the minimum device memory.
(2) support CUDA_VISIBLE_DEVICES.

[lijinf2]

build

[lijinf2]

build

[lijinf2]

@pxLi @YanxuanLiu Any potential issues with this? Seems to give 2x speedup in premerge.

It may cause some GPU OOM or host memory OOM as in the CI pool, parts of machines have only 16/24GB GPU memory.

It would be nice for the ML, the test script could detect the gpu type (mem size) and then dynamically set the parallelism like e.g. https://github.com/NVIDIA/spark-rapids/blob/branch-24.12/integration_tests/run_pyspark_from_build.sh#L113-L134 instead of hardcode the numbers thanks

The script gave 9 on CI. It seems we need only 3 at this stage after discussing with Erik. No OOM was observed when I tested pre-merge and nightly tests locally. Should be safe.

[pxLi]

@pxLi @YanxuanLiu Any potential issues with this? Seems to give 2x speedup in premerge.

It may cause some GPU OOM or host memory OOM as in the CI pool, parts of machines have only 16/24GB GPU memory.
It would be nice for the ML, the test script could detect the gpu type (mem size) and then dynamically set the parallelism like e.g. https://github.com/NVIDIA/spark-rapids/blob/branch-24.12/integration_tests/run_pyspark_from_build.sh#L113-L134 instead of hardcode the numbers thanks

The script gave 9 on CI. It seems we need only 3 at this stage after discussing with Erik. No OOM was observed when I tested pre-merge and nightly tests locally. Should be safe.

I replied above, the important part is the GPU memory, feel free to remove the host memory detect part, just give GPU memory calculation (e.g. spark test required ~1.5g GPU mem for each instance)
and put 3 as a upper bound for parallelism if possible thanks.

If you can also confirm hardcode 3 would not consume more than 15g (minimal GPU mem size in CI pool), we should be good too~

[lijinf2]

build

[lijinf2]

@pxLi @YanxuanLiu Any potential issues with this? Seems to give 2x speedup in premerge.

It may cause some GPU OOM or host memory OOM as in the CI pool, parts of machines have only 16/24GB GPU memory.
It would be nice for the ML, the test script could detect the gpu type (mem size) and then dynamically set the parallelism like e.g. https://github.com/NVIDIA/spark-rapids/blob/branch-24.12/integration_tests/run_pyspark_from_build.sh#L113-L134 instead of hardcode the numbers thanks

The script gave 9 on CI. It seems we need only 3 at this stage after discussing with Erik. No OOM was observed when I tested pre-merge and nightly tests locally. Should be safe.

I replied above, the important part is the GPU memory, feel free to remove the host memory detect part, just give GPU memory calculation (e.g. spark test required ~1.5g GPU mem for each instance) and put 3 as a upper bound for parallelism if possible thanks.

If you can also confirm hardcode 3 would not consume more than 15g (minimal GPU mem size in CI pool), we should be good too~

@pxLi Got you. Just added back the dynamic parallelism, with host memory constraint removed yet a maximum parallelism 3 added. Our pre-merge and nightly tests took < 4 GB device memory tested in my workstation. Hardcode 3 should consume no more than 12g.

[lijinf2]

build

[eordentlich]

👍