Skip to content

[DOC] Replace Firestorm with Uniffle #60

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
roryqi merged 1 commit into from
Jul 18, 2022
Merged

[DOC] Replace Firestorm with Uniffle #60

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion docs/_config.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -15,7 +15,7 @@
# limitations under the License.
#

title: Firestorm homepage
title: Uniffle homepage
baseurl: ""
url: ""

Expand Down
26 changes: 13 additions & 13 deletions docs/client_guide.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,8 +1,8 @@
---
layout: page
displayTitle: Firestorm Shuffle Client Guide
title: Firestorm Shuffle Client Guide
description: Firestorm Shuffle Client Guide
displayTitle: Uniffle Shuffle Client Guide
title: Uniffle Shuffle Client Guide
description: Uniffle Shuffle Client Guide
license: |
Licensed to the Apache Software Foundation (ASF) under one or more
contributor license agreements. See the NOTICE file distributed with
Expand All @@ -19,13 +19,13 @@ license: |
See the License for the specific language governing permissions and
limitations under the License.
---
# Firestorm Shuffle Client Guide
# Uniffle Shuffle Client Guide

Firestorm is designed as a unified shuffle engine for multiple computing frameworks, including Apache Spark and Apache Hadoop.
Firestorm has provided pluggable client plugins to enable remote shuffle in Spark and MapReduce.
Uniffle is designed as a unified shuffle engine for multiple computing frameworks, including Apache Spark and Apache Hadoop.
Uniffle has provided pluggable client plugins to enable remote shuffle in Spark and MapReduce.

## Deploy
This document will introduce how to deploy Firestorm client plugins with Spark and MapReduce.
This document will introduce how to deploy Uniffle client plugins with Spark and MapReduce.

### Deploy Spark Client Plugin

Expand All @@ -35,7 +35,7 @@ This document will introduce how to deploy Firestorm client plugins with Spark a

The jar for Spark3 is located in <RSS_HOME>/jars/client/spark3/rss-client-XXXXX-shaded.jar

2. Update Spark conf to enable Firestorm, eg,
2. Update Spark conf to enable Uniffle, eg,

```
spark.shuffle.manager org.apache.spark.shuffle.RssShuffleManager
Expand All @@ -45,7 +45,7 @@ This document will introduce how to deploy Firestorm client plugins with Spark a

### Support Spark Dynamic Allocation

To support spark dynamic allocation with Firestorm, spark code should be updated.
To support spark dynamic allocation with Uniffle, spark code should be updated.
There are 2 patches for spark-2.4.6 and spark-3.1.2 in spark-patches folder for reference.

After apply the patch and rebuild spark, add following configuration in spark conf to enable dynamic allocation:
Expand All @@ -60,7 +60,7 @@ After apply the patch and rebuild spark, add following configuration in spark co

The jar for MapReduce is located in <RSS_HOME>/jars/client/mr/rss-client-mr-XXXXX-shaded.jar

2. Update MapReduce conf to enable Firestorm, eg,
2. Update MapReduce conf to enable Uniffle, eg,

```
-Dmapreduce.rss.coordinator.quorum=<coordinatorIp1>:19999,<coordinatorIp2>:19999
Expand Down Expand Up @@ -95,7 +95,7 @@ Notice:

### Adaptive Remote Shuffle Enabling

To select build-in shuffle or remote shuffle in a smart manner, Firestorm support adaptive enabling.
To select build-in shuffle or remote shuffle in a smart manner, Uniffle support adaptive enabling.
The client should use `DelegationRssShuffleManager` and provide its unique <access_id> so that the coordinator could distinguish whether it should enable remote shuffle.

```
Expand All @@ -110,12 +110,12 @@ Other configuration:

|Property Name|Default|Description|
|---|---|---|
|spark.rss.access.timeout.ms|10000|The timeout to access Firestorm coordinator|
|spark.rss.access.timeout.ms|10000|The timeout to access Uniffle coordinator|


### Client Quorum Setting

Firestorm supports client-side quorum protocol to tolerant shuffle server crash.
Uniffle supports client-side quorum protocol to tolerant shuffle server crash.
This feature is client-side behaviour, in which shuffle writer sends each block to multiple servers, and shuffle readers could fetch block data from one of server.
Since sending multiple replicas of blocks can reduce the shuffle performance and resource consumption, we designed it as an optional feature.

Expand Down
8 changes: 4 additions & 4 deletions docs/coordinator_guide.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,8 +1,8 @@
---
layout: page
displayTitle: Firestorm Coordinator Guide
title: Firestorm Coordinator Guide
description: Firestorm Coordinator Guide
displayTitle: Uniffle Coordinator Guide
title: Uniffle Coordinator Guide
description: Uniffle Coordinator Guide
license: |
Licensed to the Apache Software Foundation (ASF) under one or more
contributor license agreements. See the NOTICE file distributed with
Expand All @@ -20,4 +20,4 @@ license: |
limitations under the License.
---

# Firestorm Coordinator Guide
# Uniffle Coordinator Guide
16 changes: 8 additions & 8 deletions docs/index.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,8 +1,8 @@
---
layout: page
displayTitle: Firestorm Overview
displayTitle: Uniffle Overview
title: Overview
description: Firestorm documentation homepage
description: Uniffle documentation homepage
license: |
Licensed to the Apache Software Foundation (ASF) under one or more
contributor license agreements. See the NOTICE file distributed with
Expand All @@ -20,22 +20,22 @@ license: |
limitations under the License.
---

Firestorm is a Remote Shuffle Service, and provides the capability for Apache Spark applications
Uniffle is a Remote Shuffle Service, and provides the capability for Apache Spark applications
to store shuffle data on remote servers.


![Rss Architecture](asset/rss_architecture.png)


More advanced details for Firestorm users are available in the following:
More advanced details for Uniffle users are available in the following:

- [Firestorm Coordinator Guide](coordinator_guide.html)
- [Uniffle Coordinator Guide](coordinator_guide.html)

- [Firestorm Shuffle Server Guide](server_guide.html)
- [Uniffle Shuffle Server Guide](server_guide.html)

- [Firestorm Shuffle Client Guide](client_guide.html)
- [Uniffle Shuffle Client Guide](client_guide.html)


Here you can read API docs for Firestorm along with its submodules.
Here you can read API docs for Uniffle along with its submodules.

- [Java API (Javadoc)](apidocs/index.html)
8 changes: 4 additions & 4 deletions docs/server_guide.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,8 +1,8 @@
---
layout: page
displayTitle: Firestorm Shuffle Server Guide
title: Firestorm Shuffle Server Guide
description: Firestorm Shuffle Server Guide
displayTitle: Uniffle Shuffle Server Guide
title: Uniffle Shuffle Server Guide
description: Uniffle Shuffle Server Guide
license: |
Licensed to the Apache Software Foundation (ASF) under one or more
contributor license agreements. See the NOTICE file distributed with
Expand All @@ -19,4 +19,4 @@ license: |
See the License for the specific language governing permissions and
limitations under the License.
---
# Firestorm Shuffle Server Guide
# Uniffle Shuffle Server Guide