Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[SPARK-4924] Add a library for launching Spark jobs programmatically. #3916

Closed
wants to merge 65 commits into from
Closed

[SPARK-4924] Add a library for launching Spark jobs programmatically. #3916

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
65 commits
Select commit Hold shift + click to select a range
6f70eea
[SPARK-4924] Add a library for launching Spark jobs programatically.
Jan 6, 2015
27be98a
Modify Spark to use launcher lib.
Jan 6, 2015
25c5ae6
Centralize SparkSubmit command line parsing.
Dec 24, 2014
1b3f6e9
Call SparkSubmit from spark-class launcher for unknown classes.
Jan 5, 2015
7a01e4a
Fix pyspark on Yarn.
Jan 6, 2015
4d511e7
Fix tools search code.
Jan 6, 2015
656374e
Mima fixes.
Jan 6, 2015
a7936ef
Fix pyspark tests.
Jan 7, 2015
53faef1
Merge branch 'master' into SPARK-4924
Jan 8, 2015
bb5d324
Merge branch 'master' into SPARK-4924
Jan 8, 2015
799fc20
Merge branch 'master' into SPARK-4924
Jan 9, 2015
2f4e8b4
Changes needed to make this work with SPARK-4048.
Jan 9, 2015
f26556b
Fix a thread-safety issue in "local" mode.
Jan 10, 2015
fc6a3e2
Merge branch 'master' into SPARK-4924
Jan 10, 2015
c617539
Review feedback round 1.
Jan 13, 2015
e946a99
Merge PySparkLauncher into SparkSubmitCliLauncher.
Jan 14, 2015
8ac4e92
Minor test cleanup.
Jan 14, 2015
525ef5b
Rework Unix spark-class to handle argument with newlines.
Jan 14, 2015
e584fc3
Rework command building a little bit.
Jan 14, 2015
aae5897
Use launcher classes instead of jars in non-release mode.
Jan 14, 2015
61919df
Clean leftover debug statement.
Jan 14, 2015
54cd4fd
Merge branch 'master' into SPARK-4924
Jan 14, 2015
7ed8859
Some more feedback.
Jan 14, 2015
f7cacff
Remove "launch Spark in new thread" feature.
Jan 15, 2015
44cd5f7
Add package-info.java, clean up javadocs.
Jan 15, 2015
e570fb5
Merge branch 'master' into SPARK-4924
Jan 20, 2015
eae4d8e
Fix new unit tests on Windows.
Jan 20, 2015
7cff919
Javadoc updates.
Jan 21, 2015
23aa2a9
Read java-opts from conf dir, not spark home.
Jan 23, 2015
0b509d0
Merge branch 'master' into SPARK-4924
Jan 26, 2015
ad03c48
Revert "Fix a thread-safety issue in "local" mode."
Jan 26, 2015
b93692a
Merge branch 'master' into SPARK-4924
Feb 4, 2015
46d46da
Clean up a test and make it more future-proof.
Feb 4, 2015
2061967
Merge branch 'master' into SPARK-4924
Feb 10, 2015
86a87bf
Merge branch 'master' into SPARK-4924
Feb 10, 2015
de81da2
Fix CommandUtils.
Feb 10, 2015
e50dc5e
Merge branch 'master' into SPARK-4924
Feb 19, 2015
e4c80b6
Reorganize the code so that only SparkLauncher is public.
Feb 19, 2015
9cd5b44
Make all non-public APIs package-private.
Feb 19, 2015
62978e4
Minor cleanup of Windows code path.
Feb 19, 2015
72da7ec
Rename SparkClassLauncher.
Feb 19, 2015
95ddfa8
Fix handling of --help for spark-class command builder.
Feb 19, 2015
8ec0243
Add missing newline.
Feb 19, 2015
bb67b93
Remove unrelated Yarn change (that is also wrong).
Feb 20, 2015
304333a
Fix propagation of properties file arg.
Feb 20, 2015
28b1434
Add a comment.
Feb 23, 2015
b4d6912
Use spark-submit script in SparkLauncher.
Feb 23, 2015
43008a7
Don't make builder extend SparkLauncher.
Feb 23, 2015
e2d4d71
Simplify some code used to launch pyspark.
Feb 24, 2015
c12d84b
Review feedback. And fix spark-submit on Windows.
Feb 24, 2015
0031a8e
Review feedback.
Feb 25, 2015
7e66c18
Fix pyspark tests.
Feb 25, 2015
4c19196
Update comment.
Feb 26, 2015
6184c07
Rename field.
Feb 26, 2015
92a9cfb
Fix Win32 launcher, usage.
Feb 26, 2015
5f4ddcc
Better usage messages.
Feb 26, 2015
00505f9
Add blurb about new API in the programming guide.
Feb 26, 2015
b1d86b0
Merge branch 'master' into SPARK-4924
Feb 26, 2015
28cd35e
Remove stale comment.
Mar 2, 2015
e2367d2
Merge branch 'master' into SPARK-4924
Mar 3, 2015
897141f
Review feedback.
Mar 9, 2015
a1b8af1
Merge branch 'master' into SPARK-4924
Mar 9, 2015
3b28a75
Update new pom.
Mar 9, 2015
2ce741f
Add lots of quotes.
Mar 9, 2015
18c7e4d
Fix make-distribution.sh.
Mar 10, 2015
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions .gitignore
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -6,6 +6,7 @@
*.iml
*.iws
*.pyc
*.pyo
.idea/
.idea_modules/
build/*.jar
Expand Down
124 changes: 0 additions & 124 deletions bin/compute-classpath.cmd
View file
Open in desktop

This file was deleted.

161 changes: 0 additions & 161 deletions bin/compute-classpath.sh
View file
Open in desktop

This file was deleted.

8 changes: 4 additions & 4 deletions bin/load-spark-env.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -41,9 +41,9 @@ fi

if [ -z "$SPARK_SCALA_VERSION" ]; then

ASSEMBLY_DIR2="$FWDIR/assembly/target/scala-2.11"
ASSEMBLY_DIR1="$FWDIR/assembly/target/scala-2.10"
ASSEMBLY_DIR2="$SPARK_HOME/assembly/target/scala-2.11"
ASSEMBLY_DIR1="$SPARK_HOME/assembly/target/scala-2.10"

if [[ -d "$ASSEMBLY_DIR2" && -d "$ASSEMBLY_DIR1" ]]; then
echo -e "Presence of build for both scala versions(SCALA 2.10 and SCALA 2.11) detected." 1>&2
echo -e 'Either clean one of them or, export SPARK_SCALA_VERSION=2.11 in spark-env.sh.' 1>&2
Expand All @@ -54,5 +54,5 @@ if [ -z "$SPARK_SCALA_VERSION" ]; then
export SPARK_SCALA_VERSION="2.11"
else
export SPARK_SCALA_VERSION="2.10"
fi
fi
fi
Loading