[SPARK-6406] Launch Spark using assembly jar instead of a separate launcher jar #5085

nishkamravi2 · 2015-03-19T02:24:45Z

No description provided.

…extFiles The prefix "file:" is missing in the string inserted as key in HashMap

… HADOOP-10456)

…onsistent with rest of Spark

…nravi

…multiplier (redone to resolve merge conflicts)

…nravi Conflicts: yarn/common/src/main/scala/org/apache/spark/deploy/yarn/ClientBase.scala

…nravi Conflicts: yarn/common/src/main/scala/org/apache/spark/deploy/yarn/ClientArguments.scala yarn/common/src/main/scala/org/apache/spark/deploy/yarn/ClientBase.scala yarn/common/src/main/scala/org/apache/spark/deploy/yarn/YarnAllocator.scala

…n to check if maxResultSize > 0)

…nravi

…kManager.stop

nishkamravi2 · 2015-03-24T01:31:04Z

Addressed two of the three comments @vanzin. The third one is tricky, let's discuss tomorrow when you're around.

…path as an argument

…nravi Conflicts: launcher/src/main/java/org/apache/spark/launcher/AbstractCommandBuilder.java

nishkamravi2 · 2015-03-24T23:56:48Z

PR updated (instead of passing assembly jar on cmdline, we can find it dynamically using getLocation).

vanzin · 2015-03-25T02:28:50Z

bin/spark-class

You may need to add something like #4873 here.

vanzin · 2015-03-25T02:33:48Z

Can an admin trigger jenkins for this PR please?

@nishkamravi2 , I think you can also remove the line in make-distribution.sh that packages the launcher jar. Otherwise, some minor things, but looks good.

Can you also change the PR title since it's not accurate? Something like "Launch Spark using the assembly instead of a separate launcher jar".

andrewor14 · 2015-03-25T02:54:36Z

retest this please

SparkQA · 2015-03-25T03:49:26Z

Test build #29144 has finished for PR 5085 at commit 4a45d6a.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
- implicit class DslSymbol(sym: Symbol) extends ImplicitAttribute
- case class Alias(child: Expression, name: String)(
- class Column(protected[sql] val expr: Expression) extends Logging

vanzin · 2015-03-25T18:12:32Z

I can't comment on random code that you didn't change, but you'll need to change the code around this code in AbstractCommandBuilder:

 needsDataNucleus = assemblyJar.getEntry("org/apache/hadoop/hive/ql/exec/") != null;

You can just skip the check for Hive and add all datanucleus jars if they're present. That's the same thing Sean did in the PR I mentioned before. (Bonus points for getting rid of the annoying message printed to stderr.)

…nravi

nishkamravi2 · 2015-03-27T07:11:55Z

Addressed comments, fixed a couple of bugs. Can this be retested? Thanks.

srowen · 2015-03-27T08:31:54Z

Jenkins, retest this please.

SparkQA · 2015-03-27T09:59:43Z

Test build #29295 has finished for PR 5085 at commit bad4349.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

vanzin · 2015-03-27T17:04:53Z

LGTM.

srowen · 2015-03-28T07:04:15Z

Given @vanzin and @nishkamravi2 have thoroughly discussed and approved this, I'm inclined to merge it. The changes look sane and do simplify the launcher somewhat.

andrewor14 · 2015-04-14T00:31:51Z

@vanzin @nishkamravi2 When I try to run bin/spark-shell --master local-cluster[2,1,512] my executors keep coming alive and then failing immediately afterwards, complaining that the scala classes are not found. I dug into this a little and it turns out that the assembly jar path from this line is not what I expected: $SPARK_HOME/launcher/target/scala-2.10/classes/. Before this commit, the correct assembly jar path used to be on my executors' classpath in local cluster mode.

Is this a known issue?

andrewor14 · 2015-04-14T00:51:22Z

Looks like @davies filed a JIRA for this already: https://issues.apache.org/jira/browse/SPARK-6890

nishkamravi2 · 2015-04-14T08:35:57Z

Hey @andrewor14, is this a Mac/Windows issue? I can't seem to be able to reproduce it on Linux. Also, do you happen to have the path that was returned by getLocation().getPath() ?

andrewor14 · 2015-04-14T17:46:08Z

Yes, the path was $SPARK_HOME/launcher/target/scala-2.10/classes/. I'm on a mac.

nishkamravi2 and others added 30 commits June 3, 2014 15:28

Fix for SPARK-1758: failing test org.apache.spark.JavaAPISuite.wholeT…

681b36f

…extFiles The prefix "file:" is missing in the string inserted as key in HashMap

Fix in Spark for the Concurrent thread modification issue (SPARK-1097,…

5108700

… HADOOP-10456)

Undo the fix for SPARK-1758 (the problem is fixed)

6b840f0

Improved fix for ConcurrentModificationIssue (Spark-1097, Hadoop-10456)

df2aeb1

Merge branch 'master' of https://github.com/apache/spark

eb663ca

Merge branch 'master' of https://github.com/apache/spark

5423a03

Merge branch 'master' of https://github.com/apache/spark

3bf8fad

Accept memory input as "30g", "512M" instead of an int value, to be c…

2b630f9

…onsistent with rest of Spark

Merge branch 'master' of https://github.com/apache/spark

efd688a

Merge branch 'master' of https://github.com/apache/spark into master_…

2e69f11

…nravi

Modify default YARN memory_overhead-- from an additive constant to a …

ebcde10

…multiplier (redone to resolve merge conflicts)

Update YarnAllocator.scala

1cf2d1e

Improving logging for AM memoryOverhead

f00fa31

Merge branch 'master' of https://github.com/apache/spark into master_…

c726bd9

…nravi Conflicts: yarn/common/src/main/scala/org/apache/spark/deploy/yarn/ClientBase.scala

Additional changes for yarn memory overhead

362da5e

Additional changes for yarn memory overhead issue

42c2c3d

Additional documentation for yarn memory overhead issue

dac1047

Remove out

5ac2ec1

Slight change in the doc for yarn memory overhead

35daa64

Doc change for yarn memory overhead

8f76c8b

Update YarnAllocator.scala

636a9ff

Changes to maxResultSize code (improve error message and add conditio…

9f6583e

…n to check if maxResultSize > 0)

Modify test for maxResultSize

3e1b616

Update TaskSetManager.scala

535295a

Update TaskSetManagerSuite.scala

5c9a4cb

Merge branch 'master' of https://github.com/apache/spark into master_…

b446edc

…nravi

Merge branch 'master' of https://github.com/apache/spark into master_…

79ea8b4

…nravi

Workaround for IllegalStateException caused by recent changes to Bloc…

f0d12de

…kManager.stop

Update DiskBlockManager.scala

3c5ddba

nishkamravi2 added 2 commits March 24, 2015 16:42

Locate the jar using getLocation, obviates the need to pass assembly …

458af39

…path as an argument

Merge branch 'master' of https://github.com/apache/spark into master_…

4a45d6a

…nravi Conflicts: launcher/src/main/java/org/apache/spark/launcher/AbstractCommandBuilder.java

vanzin reviewed Mar 25, 2015
View reviewed changes

bin/spark-class Outdated

Copy link

Contributor

vanzin Mar 25, 2015

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You may need to add something like #4873 here.

nishkamravi2 and others added 3 commits March 26, 2015 22:38

Merge branch 'master' of https://github.com/apache/spark into master_…

b7f4ae7

…nravi

Minor changes and bug fixes

36a6f87

Update Main.java

bad4349

nishkamravi2 changed the title ~~[SPARK-6406] Launcher backward compatibility issues-- "hadoop" should not be mandatory; should not look for launcher jar~~ [SPARK-6406] Launch Spark using assembly jar instead of a separate launcher jar Mar 27, 2015

srowen mentioned this pull request Mar 27, 2015

[SPARK-6435] spark-shell --jars option does not add all jars to classpath #5227

Closed

asfgit closed this in e3eb393 Mar 29, 2015

vanzin mentioned this pull request Apr 2, 2015

[SPARK-6673] spark-shell.cmd can't start in Windows even when spark was built #5328

Closed

vanzin mentioned this pull request Apr 14, 2015

[SPARK-6890] [core] Fix launcher lib work with SPARK_PREPEND_CLASSES. #5504

Closed

[SPARK-6406] Launch Spark using assembly jar instead of a separate launcher jar #5085

[SPARK-6406] Launch Spark using assembly jar instead of a separate launcher jar #5085

Uh oh!

Conversation

nishkamravi2 commented Mar 19, 2015

Uh oh!

nishkamravi2 commented Mar 24, 2015

Uh oh!

nishkamravi2 commented Mar 24, 2015

Uh oh!

vanzin Mar 25, 2015

Choose a reason for hiding this comment

Uh oh!

vanzin commented Mar 25, 2015

Uh oh!

andrewor14 commented Mar 25, 2015

Uh oh!

SparkQA commented Mar 25, 2015

Uh oh!

vanzin commented Mar 25, 2015

Uh oh!

nishkamravi2 commented Mar 27, 2015

Uh oh!

srowen commented Mar 27, 2015

Uh oh!

SparkQA commented Mar 27, 2015

Uh oh!

vanzin commented Mar 27, 2015

Uh oh!

srowen commented Mar 28, 2015

Uh oh!

andrewor14 commented Apr 14, 2015

Uh oh!

andrewor14 commented Apr 14, 2015

Uh oh!

nishkamravi2 commented Apr 14, 2015

Uh oh!

andrewor14 commented Apr 14, 2015

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants