Skip to content

[SPARK-7693][Core] Remove "import scala.concurrent.ExecutionContext.Implicits.global" #6223

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Closed
zsxwing wants to merge 2 commits into from
Closed

[SPARK-7693][Core] Remove "import scala.concurrent.ExecutionContext.Implicits.global" #6223

zsxwing wants to merge 2 commits into from

Conversation

@zsxwing
Copy link
Member

@zsxwing zsxwing commented May 18, 2015

Learnt a lesson from SPARK-7655: Spark should avoid to use scala.concurrent.ExecutionContext.Implicits.global because the user may submit blocking actions to scala.concurrent.ExecutionContext.Implicits.global and exhaust all threads in it. This could crash Spark. So Spark should always use its own thread pools for safety.

This PR removes all usages of scala.concurrent.ExecutionContext.Implicits.global and uses proper thread pools to replace them.

@zsxwing
Copy link
Member Author

zsxwing commented May 18, 2015

cc @rxin

@AmplabJenkins
Copy link

AmplabJenkins commented May 18, 2015

Merged build triggered.

@AmplabJenkins
Copy link

AmplabJenkins commented May 18, 2015

Merged build started.

@SparkQA
Copy link

SparkQA commented May 18, 2015

Test build #32956 has started for PR 6223 at commit cf4b3fc.

rxin
rxin reviewed May 18, 2015
View reviewed changes
core/src/main/scala/org/apache/spark/rdd/AsyncRDDActions.scala Outdated
Copy link
Contributor

@rxin rxin May 18, 2015

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

is 1024 too high? In reality, I wonder if we are going to hit this high number of threads. (I know we also did it for the broadcast -- but I'm thinking about downgrading this to 128 might make sense)

Copy link
Member Author

@zsxwing zsxwing May 18, 2015

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I changed it to 128, including BroadcastHashJoin.

@AmplabJenkins
Copy link

AmplabJenkins commented May 18, 2015

Merged build triggered.

@AmplabJenkins
Copy link

AmplabJenkins commented May 18, 2015

Merged build started.

@AmplabJenkins
Copy link

AmplabJenkins commented May 18, 2015

Merged build triggered.

@AmplabJenkins
Copy link

AmplabJenkins commented May 18, 2015

Merged build started.

@SparkQA
Copy link

SparkQA commented May 18, 2015

Test build #32958 has started for PR 6223 at commit a33ff06.

@AmplabJenkins
Copy link

AmplabJenkins commented May 18, 2015

Merged build finished. Test FAILed.

@AmplabJenkins
Copy link

AmplabJenkins commented May 18, 2015

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/32957/
Test FAILed.

@SparkQA
Copy link

SparkQA commented May 18, 2015

Test build #32956 has finished for PR 6223 at commit cf4b3fc.

  • This patch passes all tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

@AmplabJenkins
Copy link

AmplabJenkins commented May 18, 2015

Merged build finished. Test PASSed.

@AmplabJenkins
Copy link

AmplabJenkins commented May 18, 2015

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/32956/
Test PASSed.

@SparkQA
Copy link

SparkQA commented May 18, 2015

Test build #32958 has finished for PR 6223 at commit a33ff06.

  • This patch passes all tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

@AmplabJenkins
Copy link

AmplabJenkins commented May 18, 2015

Merged build finished. Test PASSed.

@AmplabJenkins
Copy link

AmplabJenkins commented May 18, 2015

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/32958/
Test PASSed.

@rxin
Copy link
Contributor

rxin commented May 18, 2015

LGTM. Merging thanks.

@asfgit asfgit closed this in ff71d34 May 18, 2015
asfgit pushed a commit that referenced this pull request May 18, 2015
@zsxwing @rxin
[SPARK-7693][Core] Remove "import scala.concurrent.ExecutionContext.I…
2a42d2d 
…mplicits.global"

Learnt a lesson from SPARK-7655: Spark should avoid to use `scala.concurrent.ExecutionContext.Implicits.global` because the user may submit blocking actions to `scala.concurrent.ExecutionContext.Implicits.global` and exhaust all threads in it. This could crash Spark. So Spark should always use its own thread pools for safety.

This PR removes all usages of `scala.concurrent.ExecutionContext.Implicits.global` and uses proper thread pools to replace them.

Author: zsxwing <zsxwing@gmail.com>

Closes #6223 from zsxwing/SPARK-7693 and squashes the following commits:

a33ff06 [zsxwing] Decrease the max thread number from 1024 to 128
cf4b3fc [zsxwing] Remove "import scala.concurrent.ExecutionContext.Implicits.global"

(cherry picked from commit ff71d34)
Signed-off-by: Reynold Xin <rxin@databricks.com>
@zsxwing zsxwing deleted the SPARK-7693 branch May 18, 2015 04:10
jeanlyn pushed a commit to jeanlyn/spark that referenced this pull request May 28, 2015
@zsxwing @jeanlyn
[SPARK-7693][Core] Remove "import scala.concurrent.ExecutionContext.I…
a388e22 
…mplicits.global"

Learnt a lesson from SPARK-7655: Spark should avoid to use `scala.concurrent.ExecutionContext.Implicits.global` because the user may submit blocking actions to `scala.concurrent.ExecutionContext.Implicits.global` and exhaust all threads in it. This could crash Spark. So Spark should always use its own thread pools for safety.

This PR removes all usages of `scala.concurrent.ExecutionContext.Implicits.global` and uses proper thread pools to replace them.

Author: zsxwing <zsxwing@gmail.com>

Closes apache#6223 from zsxwing/SPARK-7693 and squashes the following commits:

a33ff06 [zsxwing] Decrease the max thread number from 1024 to 128
cf4b3fc [zsxwing] Remove "import scala.concurrent.ExecutionContext.Implicits.global"
jeanlyn pushed a commit to jeanlyn/spark that referenced this pull request Jun 12, 2015
@zsxwing @jeanlyn
[SPARK-7693][Core] Remove "import scala.concurrent.ExecutionContext.I…
c3718d3 
…mplicits.global"

Learnt a lesson from SPARK-7655: Spark should avoid to use `scala.concurrent.ExecutionContext.Implicits.global` because the user may submit blocking actions to `scala.concurrent.ExecutionContext.Implicits.global` and exhaust all threads in it. This could crash Spark. So Spark should always use its own thread pools for safety.

This PR removes all usages of `scala.concurrent.ExecutionContext.Implicits.global` and uses proper thread pools to replace them.

Author: zsxwing <zsxwing@gmail.com>

Closes apache#6223 from zsxwing/SPARK-7693 and squashes the following commits:

a33ff06 [zsxwing] Decrease the max thread number from 1024 to 128
cf4b3fc [zsxwing] Remove "import scala.concurrent.ExecutionContext.Implicits.global"
nemccarthy pushed a commit to nemccarthy/spark that referenced this pull request Jun 19, 2015
@zsxwing @nemccarthy
[SPARK-7693][Core] Remove "import scala.concurrent.ExecutionContext.I…
22a0b1d 
…mplicits.global"

Learnt a lesson from SPARK-7655: Spark should avoid to use `scala.concurrent.ExecutionContext.Implicits.global` because the user may submit blocking actions to `scala.concurrent.ExecutionContext.Implicits.global` and exhaust all threads in it. This could crash Spark. So Spark should always use its own thread pools for safety.

This PR removes all usages of `scala.concurrent.ExecutionContext.Implicits.global` and uses proper thread pools to replace them.

Author: zsxwing <zsxwing@gmail.com>

Closes apache#6223 from zsxwing/SPARK-7693 and squashes the following commits:

a33ff06 [zsxwing] Decrease the max thread number from 1024 to 128
cf4b3fc [zsxwing] Remove "import scala.concurrent.ExecutionContext.Implicits.global"
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

4 participants

@zsxwing @AmplabJenkins @SparkQA @rxin