Skip to content

[#552] docs: add more doc about spark.serializer requirement #556

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
advancedxy merged 2 commits into from
Feb 8, 2023
Merged

[#552] docs: add more doc about spark.serializer requirement #556

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
1c07582
chore: add more doc about spark.serializer requirement
advancedxy Feb 7, 2023
45c0b9a
fix style issue
advancedxy Feb 7, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 3 additions & 0 deletions README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -196,6 +196,9 @@ rss-xxx.tgz will be generated for deployment
2. Update Spark conf to enable Uniffle, e.g.,

```
# Uniffle transmits serialized shuffle data over network, therefore a serializer that supports relocation of
# serialized object should be used.
spark.serialier org.apache.spark.serializer.KryoSerializer # this could also be in the spark-defaults.conf
spark.shuffle.manager org.apache.spark.shuffle.RssShuffleManager
spark.rss.coordinator.quorum <coordinatorIp1>:19999,<coordinatorIp2>:19999
# Note: For Spark2, spark.sql.adaptive.enabled should be false because Spark2 doesn't support AQE.
Expand Down
8 changes: 8 additions & 0 deletions client-spark/spark3/src/main/java/org/apache/spark/shuffle/RssShuffleManager.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -20,6 +20,7 @@
import java.util.Collections;
import java.util.List;
import java.util.Map;
import java.util.Optional;
import java.util.Set;
import java.util.concurrent.Executors;
import java.util.concurrent.ScheduledExecutorService;
Expand Down Expand Up @@ -152,6 +153,13 @@ private synchronized void putBlockId(

public RssShuffleManager(SparkConf conf, boolean isDriver) {
this.sparkConf = conf;
boolean supportsRelocation = Optional.ofNullable(SparkEnv.get())
.map(env -> env.serializer().supportsRelocationOfSerializedObjects())
.orElse(true);
if (!supportsRelocation) {
LOG.warn("RSSShuffleManager requires a serializer which supports relocations of serialized object. Please set "
+ "spark.serializer to org.apache.spark.serializer.KryoSerializer instead");
}
this.user = sparkConf.get("spark.rss.quota.user", "user");
this.uuid = sparkConf.get("spark.rss.quota.uuid", Long.toString(System.currentTimeMillis()));
// set & check replica config
Expand Down
3 changes: 3 additions & 0 deletions docs/client_guide.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -38,6 +38,9 @@ This document will introduce how to deploy Uniffle client plugins with Spark and
2. Update Spark conf to enable Uniffle, eg,

```
# Uniffle transmits serialized shuffle data over network, therefore a serializer that supports relocation of
# serialized object should be used.
spark.serialier org.apache.spark.serializer.KryoSerializer # this could also be in the spark-defaults.conf
spark.shuffle.manager org.apache.spark.shuffle.RssShuffleManager
spark.rss.coordinator.quorum <coordinatorIp1>:19999,<coordinatorIp2>:19999
# Note: For Spark2, spark.sql.adaptive.enabled should be false because Spark2 doesn't support AQE.
Expand Down