GH-33697: [CI][Python] Nightly test for PySpark 3.2.0 fail with AttributeError on numpy.bool

[AlenkaF]

Rationale for this change

Fix for nightly integration tests with PySpark 3.2.0 failure.

What changes are included in this PR?

NumPy version pin in docker-compose.yml.

Are these changes tested?

Will test on the open PR with the CI.

Are there any user-facing changes?

No.

• Closes: [CI][Python] Nightly test for PySpark 3.2.0 fail with AttributeError on numpy.bool #33697

[github-actions]

• Closes: [CI][Python] Nightly test for PySpark 3.2.0 fail with AttributeError on numpy.bool #33697

[github-actions]

⚠️ GitHub issue #33697 has been automatically assigned in GitHub to PR creator.

[AlenkaF]

@github-actions crossbow submit test-conda-python-3.8-spark-v3.2.0

[github-actions]

Revision: 8e3ab26

Submitted crossbow builds: ursacomputing/crossbow @ actions-94e388e574

Task	Status
test-conda-python-3.8-spark-v3.2.0

[AlenkaF]

@github-actions crossbow submit test-conda-python-3.8-spark-v3.2.0

[github-actions]

Revision: 70430e9

Submitted crossbow builds: ursacomputing/crossbow @ actions-7fd30999ea

Task	Status
test-conda-python-3.8-spark-v3.2.0

[AlenkaF]

@github-actions crossbow submit test-conda-python-3.8-spark-v3.2.0

[AlenkaF]

@github-actions crossbow submit test-conda-python-3.8-spark-v3.2.0

[github-actions]

Revision: 427191d

Submitted crossbow builds: ursacomputing/crossbow @ actions-431f284696

Task	Status
test-conda-python-3.8-spark-v3.2.0

[raulcd]

Thanks @AlenkaF for the PR! I have some questions because the nightly build with both spark master and 3.1.2 are already working without pinning numpy. I've pinged @kiszk as he's a spark committer because I am not sure if this will ever get backported to 3.2.0

[raulcd]

we already test spark master nightly, this is the current testing combination:

{% for python_version, spark_version, test_pyarrow_only in [("3.7", "v3.1.2", "false"),
                                                            ("3.8", "v3.2.0", "false"),
                                                            ("3.9", "master", "false")] %}

And the build for spark master is currently passing: https://github.com/ursacomputing/crossbow/actions/runs/3934958561/jobs/6730195747#step:5:10
Maybe we can add the numpy version to the task definition only for 3.2.0 and if it is different than latest install the pinned version. I am thinking on something like:

{% for python_version, spark_version, test_pyarrow_only, numpy_version in [("3.7", "v3.1.2", "false", "latest"),
                                                            ("3.8", "v3.2.0", "false", "1.23"),
                                                            ("3.9", "master", "false", "latest")] %}

And the corresponding if to validate if we have to install numpy or not?
@kiszk you are spark committer. I suppose this fix won't get backported to spark 3.2.0 and we have to pin numpy always for it? Should we update the tasks for our nightlies to test with spark 3.3.0 maybe remove 3.2.0?

[AlenkaF]

@github-actions crossbow submit test-conda-python-3.8-spark-v3.2.0

[github-actions]

Revision: d86c6a9

Submitted crossbow builds: ursacomputing/crossbow @ actions-94d502001c

Task	Status
test-conda-python-3.8-spark-v3.2.0

[AlenkaF]

@github-actions crossbow submit test-conda-python-3.8-spark-v3.2.0

[github-actions]

Revision: 7f76899

Submitted crossbow builds: ursacomputing/crossbow @ actions-b7b5dc6d5d

Task	Status
test-conda-python-3.8-spark-v3.2.0

[AlenkaF]

@github-actions crossbow submit test-conda-python--spark-

[github-actions]

Revision: 7f76899

Submitted crossbow builds: ursacomputing/crossbow @ actions-8c12467062

Task	Status
test-conda-python-3.7-spark-v3.1.2
test-conda-python-3.8-spark-v3.2.0
test-conda-python-3.9-spark-master

[AlenkaF]

@github-actions crossbow submit test-conda-python--spark-

[github-actions]

Revision: b1b776d

Submitted crossbow builds: ursacomputing/crossbow @ actions-5192a832e2

Task	Status
test-conda-python-3.7-spark-v3.1.2
test-conda-python-3.8-spark-v3.2.0
test-conda-python-3.9-spark-master

[AlenkaF]

@github-actions crossbow submit test-conda-python--spark-

[github-actions]

Revision: 9743842

Submitted crossbow builds: ursacomputing/crossbow @ actions-d30511e746

Task	Status
test-conda-python-3.7-spark-v3.1.2
test-conda-python-3.8-spark-v3.2.0
test-conda-python-3.9-spark-master

[AlenkaF]

@github-actions crossbow submit test-conda-python--spark-

[github-actions]

Revision: efdf9fc

Submitted crossbow builds: ursacomputing/crossbow @ actions-268559d178

Task	Status
test-conda-python-3.7-spark-v3.1.2
test-conda-python-3.8-spark-v3.2.0
test-conda-python-3.9-spark-master

[jorisvandenbossche]

Looking at the logs, I think the line "#7 7.578 /bin/bash: /arrow/ci/scripts/install_numpy.sh: Permission denied" is the problem (the file wasn't copied, so installing numpy using the nonexisting file fails)

However, I don't see any difference with what we already do in conda-python-pandas.dockerfile (that also does such a copy), so not directly sure why that is going wrong.

[AlenkaF]

Looking at the logs, I think the line "#7 7.578 /bin/bash: /arrow/ci/scripts/install_numpy.sh: Permission denied" is the problem (the file wasn't copied, so installing numpy using the nonexisting file fails)

However, I don't see any difference with what we already do in conda-python-pandas.dockerfile (that also does such a copy), so not directly sure why that is going wrong.

Yeah, I am trying out different things locally but none work 🤷‍♀️ Also asked Raul for help, if he has any ideas what could be the issue.

[AlenkaF]

@github-actions crossbow submit test-conda-python--spark-

[github-actions]

Revision: b1b3b99

Submitted crossbow builds: ursacomputing/crossbow @ actions-5f44f20302

Task	Status
test-conda-python-3.7-spark-v3.1.2
test-conda-python-3.8-spark-v3.2.0
test-conda-python-3.9-spark-master

[AlenkaF]

@github-actions crossbow submit test-conda-python--spark-

[github-actions]

Revision: d56f4b7

Submitted crossbow builds: ursacomputing/crossbow @ actions-cce4baca43

Task	Status
test-conda-python-3.7-spark-v3.1.2
test-conda-python-3.8-spark-v3.2.0
test-conda-python-3.9-spark-master

[AlenkaF]

@raulcd the fix is working now, thank you!
The CI failure is not related (maybe I need a rebase?).

[raulcd]

Thanks @AlenkaF I am going to trigger a final run just to validate and will merge once it finishes.

[raulcd]

@github-actions crossbow submit test-conda-python--spark-

[github-actions]

Revision: 1ebe276

Submitted crossbow builds: ursacomputing/crossbow @ actions-fd4f9f54ae

Task	Status
test-conda-python-3.7-spark-v3.1.2
test-conda-python-3.8-spark-v3.2.0
test-conda-python-3.9-spark-master

[ursabot]

Benchmark runs are scheduled for baseline = f9a1d19 and contender = 4c1448e. 4c1448e is a master commit associated with this PR. Results will be available as each benchmark for each run completes.
Conbench compare runs links:
[Finished ⬇️0.0% ⬆️0.0%] ec2-t3-xlarge-us-east-2
[Finished ⬇️0.15% ⬆️0.06%] test-mac-arm
[Finished ⬇️0.51% ⬆️0.0%] ursa-i9-9960x
[Finished ⬇️0.16% ⬆️0.03%] ursa-thinkcentre-m75q
Buildkite builds:
[Finished] 4c1448e8 ec2-t3-xlarge-us-east-2
[Finished] 4c1448e8 test-mac-arm
[Finished] 4c1448e8 ursa-i9-9960x
[Finished] 4c1448e8 ursa-thinkcentre-m75q
[Finished] f9a1d198 ec2-t3-xlarge-us-east-2
[Finished] f9a1d198 test-mac-arm
[Finished] f9a1d198 ursa-i9-9960x
[Finished] f9a1d198 ursa-thinkcentre-m75q
Supported benchmarks:
ec2-t3-xlarge-us-east-2: Supported benchmark langs: Python, R. Runs only benchmarks with cloud = True
test-mac-arm: Supported benchmark langs: C++, Python, R
ursa-i9-9960x: Supported benchmark langs: Python, R, JavaScript
ursa-thinkcentre-m75q: Supported benchmark langs: C++, Java