From d6c7eb97201c244dec4c98c376bf7bcfc4b46d7b Mon Sep 17 00:00:00 2001 From: Noah Date: Wed, 31 Jul 2024 10:15:51 -0400 Subject: [PATCH] Update README.md --- README.md | 16 ++++++++++++++++ 1 file changed, 16 insertions(+) diff --git a/README.md b/README.md index e204a95a..d6280b02 100644 --- a/README.md +++ b/README.md @@ -85,6 +85,22 @@ word_count.show() raydp.stop_spark() ``` +Note: to support a Ray RuntimeEnv with [pymodules](https://docs.ray.io/en/latest/ray-core/api/doc/ray.runtime_env.RuntimeEnv.html#ray.runtime_env.RuntimeEnv) you must set the `spark.executorEnv.PYTHONPATH` in you Spark config: +``` +import os +import ray +import raydp + +ray.init() +spark = raydp.init_spark(app_name="RayDP Example", + num_executors=2, + executor_cores=2, + executor_memory="4GB", + configs={ + "spark.executorEnv.PYTHONPATH": os.environ["PYTHONPATH"], + }) +``` + Spark features such as dynamic resource allocation, spark-submit script, etc are also supported. Please refer to [Spark on Ray](./doc/spark_on_ray.md) for more details.