Fix CI: drop EOL Python 3.9 and update GitHub Actions

[rexminnis]

Summary

• CI is broken on all PRs — actions/setup-python@v2.3.4 fails with
  candidates is not iterable because Python 3.9 was removed from
  ubuntu-latest runners (EOL since Oct 2025)
• Replace Python 3.9 with 3.11 in the test matrix
• Update actions/checkout (v2 → v4), actions/setup-python (v2 → v5),
  actions/cache (v2 → v4)
• Remove stale dependency pins (numpy<1.24, pydantic<2.0, click<8.3.0)
  that were only needed for Python 3.9
• Update TensorFlow pin (==2.13.1 → >=2.16.1,<2.19) for Python 3.11
  compatibility
• Applies to both raydp.yml (PR CI) and ray_nightly_test.yml (nightly)

Test plan

• CI passes on this PR (self-validating)

[rexminnis]

@pang-wu can I get a review on this. It's needed to unblock #463 and few others.

[pang-wu]

we should do this as part of the spark 4.x support, otherwise the new tf will break older spark which required numpy 1.2x

[rexminnis]

Thanks for the review @pang-wu. I want to clarify — this PR doesn't upgrade TensorFlow or numpy. It actually pins them conservatively to protect older Spark:

• tensorflow-cpu>=2.13.1,<2.16 — stays on Keras 2 (same era as the old ==2.13.1)
• numpy<2 and pandas<2 — explicitly pinned for Spark 3.3/3.4 compatibility
• click<8.3.0 — pinned for Ray 2.37/2.40 compatibility

The core issue is that CI is broken on master right now — actions/setup-python@v2 can no longer find Python 3.9 (EOL'd and removed from GitHub runners). This blocks all PRs, including #450. The changes here are:

1. GitHub Actions v2 → v4/v5 (required — old versions broken)
2. Python 3.9 → 3.10/3.11 (required — 3.9 unavailable on runners)
3. pylint 2.8.3 → 3.2.7 (required — old version crashes on Python 3.11)
4. Dependency pins to keep older Spark working
5. pytest-rerunfailures for flaky cross-language serialization tests
6. fail-fast: false + matrix excludes for unsupported combos

All 13 matrix jobs pass (Spark 3.3.2/3.4.0/3.5.0 × Ray 2.37/2.40/2.50). Nothing here conflicts with or duplicates the Spark 4.x work.

[pang-wu]

master is still working -- I just retrigger another job in a PR i file yesterday and it pull python 3.9: https://github.com/ray-project/raydp/actions/runs/22073389233/job/63950790956?pr=462
Am I missing anything?

Another note -- we shouldn't deprecate Python 3.9 in CI until we have to for raydp 1.6.x. As they are built with that version, it is a guardrail to make sure there is not regression.

[rexminnis]

You're right — I can see Python 3.9 setup succeeded on your run. The failure I hit on PR #463 (candidates is not iterable from setup-python@v2.3.4) may have been transient. Happy to close this PR if you'd prefer to handle CI updates as part of the Spark 4.x work.

Separately, I noticed PR #462 rewrites getRDDPartition in RayDPExecutor.scala — that overlaps with my PR #463 which fixes a TaskContext leak in the same method. Your rewrite doesn't include TaskContext.unset() or releaseAllLocksForTask in a finally block, so the leak is still present. Would you be open to incorporating that fix into #462? Happy to add it as a suggestion on that PR if that's easier.