Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[spark] Fix DefaultDatabricksRayOnSparkStartHook.on_spark_job_created #42178

Merged
merged 6 commits into from
Jan 8, 2024
Merged

[spark] Fix DefaultDatabricksRayOnSparkStartHook.on_spark_job_created #42178

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
6 commits
Select commit Hold shift + click to select a range
4d70546
init
WeichenXu123 Jan 4, 2024
b9f923b
update
WeichenXu123 Jan 4, 2024
04eafad
fix test
WeichenXu123 Jan 4, 2024
3bffec0
update
WeichenXu123 Jan 8, 2024
8a6c273
clean
WeichenXu123 Jan 8, 2024
c5a74a9
Merge branch 'master' into fix-DatabricksRayOnSparkStartHook
jjyao Jan 8, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 4 additions & 0 deletions python/ray/tests/spark/test_databricks_hook.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -23,6 +23,9 @@ def __init__(self):
def getIdleTimeMillisSinceLastNotebookExecution(self):
return (time.time() - self.created_time) * 1000

def registerBackgroundSparkJobGroup(self, job_group_id):
self.registered_job_groups.append(job_group_id)


class TestDatabricksHook:
@classmethod
Expand Down Expand Up @@ -62,6 +65,7 @@ def test_hook(self, monkeypatch):
head_node_options={"include_dashboard": False},
)
cluster = ray.util.spark.cluster_init._active_ray_cluster
assert db_api_entry.registered_job_groups == [cluster.spark_job_group_id]
assert not cluster.is_shutdown
time.sleep(35)
assert cluster.is_shutdown
Expand Down
4 changes: 2 additions & 2 deletions python/ray/util/spark/cluster_init.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1619,11 +1619,11 @@ def ray_cluster_job_mapper(_):
)
job_rdd = job_rdd.withResources(resource_profile)

job_rdd.mapPartitions(ray_cluster_job_mapper).collect()

hook_entry = _create_hook_entry(is_global=(ray_temp_dir is None))
hook_entry.on_spark_job_created(spark_job_group_id)
Copy link
Contributor Author

@WeichenXu123 WeichenXu123 Jan 4, 2024 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Move this line to front of job_rdd.mapPartitions(ray_cluster_job_mapper).collect() because this function runs inside background thread and job_rdd.mapPartitions(ray_cluster_job_mapper).collect() will block forever until cluster or the Ray worker node is terminated.


job_rdd.mapPartitions(ray_cluster_job_mapper).collect()


@PublicAPI(stability="alpha")
def shutdown_ray_cluster() -> None:
Expand Down
2 changes: 1 addition & 1 deletion python/ray/util/spark/databricks_hook.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -166,7 +166,7 @@ def auto_shutdown_watcher():

def on_spark_job_created(self, job_group_id):
db_api_entry = get_db_entry_point()
db_api_entry.registerBackgroundSparkJobGroup("job_group_id")
db_api_entry.registerBackgroundSparkJobGroup(job_group_id)

def custom_environment_variables(self):
"""Hardcode `GLOO_SOCKET_IFNAME` to `eth0` for Databricks runtime.
Expand Down
Loading