Skip to content

Conversation

@kyungjunleeme
Copy link
Contributor

@kyungjunleeme kyungjunleeme commented Oct 13, 2025

✨ Summary

This PR updates the Spark base image reference used in the Airflow Spark-on-Kubernetes example to align with the latest officially maintained Apache Spark images and Python version requirements.


🧩 Changes

Item Old New Python Version
Base Image gcr.io/spark-operator/spark-py:v3.1.1 apache/spark-py:v3.4.0 3.10.6
  • The legacy image (gcr.io/spark-operator/spark-py:v3.1.1) was built on an outdated Spark and Python base, and attempts to pull it from /artifacts/tags/spark-operator/us/gcr.io/spark-py/v3.1.1 failed due to registry access issues during migration.
  • The updated image (v3.4.0) is based on Ubuntu 22.04 (Jammy) and ships with Python 3.10.6.
  • This update ensures compatibility with the latest apache-airflow-providers-cncf-kubernetes, which now requires Python >= 3.10.

Verified with:

curl -s https://pypi.org/pypi/apache-airflow-providers-cncf-kubernetes/json \
  | jq -r '.info.requires_python'
# >=3.10

🧠 Rationale

  • The older gcr.io/spark-operator/spark-py:v3.1.1 image (Python 3.8.x) is deprecated and no longer maintained on GCR.
  • The official Apache image (apache/spark-py) on Docker Hub is now the actively maintained and recommended base for Spark 3.4+.
  • Verified the runtime environments manually:
docker run --rm apache/spark-py:v3.2.4 python3 -V
# Python 3.9.2

docker run --rm apache/spark-py:v3.4.0 python3 -V
# Python 3.10.6
image Screenshot 2025-10-14 at 12 05 35 AM image

https://hub.docker.com/r/apache/spark-py/tags

I checked all image tag

image image

^ Add meaningful description above
Read the Pull Request Guidelines for more information.
In case of fundamental code changes, an Airflow Improvement Proposal (AIP) is needed.
In case of a new dependency, check compliance with the ASF 3rd Party License Policy.
In case of backwards incompatible changes please leave a note in a newsfragment file, named {pr_number}.significant.rst or {issue_number}.significant.rst, in airflow-core/newsfragments.

@potiuk
Copy link
Member

potiuk commented Oct 13, 2025

Did you run the operator with the new image? can you post some output?

@potiuk potiuk merged commit fb601d8 into apache:main Oct 25, 2025
74 checks passed
@kyungjunleeme
Copy link
Contributor Author

Did you run the operator with the new image? can you post some output?

@potiuk sorry. I forgot this message. I will post image you mentioned! A.S.A.P

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants