Skip to content

[SPARK-13596] [BUILD] Move misc top-level build files into appropriate subdirs #11522

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
srowen wants to merge 5 commits into from
Closed

[SPARK-13596] [BUILD] Move misc top-level build files into appropriate subdirs #11522

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
5 commits
Select commit Hold shift + click to select a range
4e5ec1e
Move many top-level files in dev/ or other appropriate directory. In …
srowen Mar 4, 2016
fdf1cba
Move tox.ini to dev (and update pep8 while we're at it); fix newline …
srowen Mar 4, 2016
2c1981d
Put CONTRIBUTING.md back at the top level for visibility
srowen Mar 7, 2016
0b9bc68
Intentional style violation to check checkstyle
srowen Mar 7, 2016
8e96a7d
Revert "Intentional style violation to check checkstyle"
srowen Mar 7, 2016
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 0 additions & 3 deletions .gitignore
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -17,8 +17,6 @@ cache
work/
out/
.DS_Store
third_party/libmesos.so
third_party/libmesos.dylib
build/apache-maven*
build/zinc*
build/scala*
Expand Down Expand Up @@ -60,7 +58,6 @@ dev/create-release/*final
spark-*-bin-*.tgz
unit-tests.log
/lib/
rat-results.txt
scalastyle.txt
scalastyle-output.xml
R-unit-tests.log
Expand Down
0 .rat-excludes → dev/.rat-excludes
View file
Open in desktop
File renamed without changes.
7 changes: 4 additions & 3 deletions dev/check-license
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -58,7 +58,7 @@ else
declare java_cmd=java
fi

export RAT_VERSION=0.10
export RAT_VERSION=0.11
export rat_jar="$FWDIR"/lib/apache-rat-${RAT_VERSION}.jar
mkdir -p "$FWDIR"/lib

Expand All @@ -67,14 +67,15 @@ mkdir -p "$FWDIR"/lib
exit 1
}

$java_cmd -jar "$rat_jar" -E "$FWDIR"/.rat-excludes -d "$FWDIR" > rat-results.txt
mkdir target
$java_cmd -jar "$rat_jar" -E "$FWDIR"/dev/.rat-excludes -d "$FWDIR" > target/rat-results.txt

if [ $? -ne 0 ]; then
echo "RAT exited abnormally"
exit 1
fi

ERRORS="$(cat rat-results.txt | grep -e "??")"
ERRORS="$(cat target/rat-results.txt | grep -e "??")"

if test ! -z "$ERRORS"; then
echo "Could not find Apache license headers in the following files:"
Expand Down
0 checkstyle-suppressions.xml → dev/checkstyle-suppressions.xml
View file
Open in desktop
File renamed without changes.
0 checkstyle.xml → dev/checkstyle.xml
View file
Open in desktop
File renamed without changes.
2 changes: 1 addition & 1 deletion dev/create-release/release-build.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -165,7 +165,7 @@ if [[ "$1" == "package" ]]; then
# Get maven home set by MVN
MVN_HOME=`$MVN -version 2>&1 | grep 'Maven home' | awk '{print $NF}'`

./make-distribution.sh --name $NAME --mvn $MVN_HOME/bin/mvn --tgz $FLAGS \
./dev/make-distribution.sh --name $NAME --mvn $MVN_HOME/bin/mvn --tgz $FLAGS \
-DzincPort=$ZINC_PORT 2>&1 > ../binary-release-$NAME.log
cd ..
cp spark-$SPARK_VERSION-bin-$NAME/spark-$SPARK_VERSION-bin-$NAME.tgz .
Expand Down
6 changes: 3 additions & 3 deletions dev/lint-python
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -37,7 +37,7 @@ compile_status="${PIPESTATUS[0]}"
#+ See: https://github.com/apache/spark/pull/1744#issuecomment-50982162
#+ TODOs:
#+ - Download pep8 from PyPI. It's more "official".
PEP8_VERSION="1.6.2"
PEP8_VERSION="1.7.0"
PEP8_SCRIPT_PATH="$SPARK_ROOT_DIR/dev/pep8-$PEP8_VERSION.py"
PEP8_SCRIPT_REMOTE_PATH="https://raw.githubusercontent.com/jcrocholl/pep8/$PEP8_VERSION/pep8.py"

Expand Down Expand Up @@ -80,7 +80,7 @@ export "PATH=$PYTHONPATH:$PATH"
#+ first, but we do so so that the check status can
#+ be output before the report, like with the
#+ scalastyle and RAT checks.
python "$PEP8_SCRIPT_PATH" --ignore=E402,E731,E241,W503,E226 $PATHS_TO_CHECK >> "$PEP8_REPORT_PATH"
python "$PEP8_SCRIPT_PATH" --ignore=E402,E731,E241,W503,E226 --config=dev/tox.ini $PATHS_TO_CHECK >> "$PEP8_REPORT_PATH"
pep8_status="${PIPESTATUS[0]}"

if [ "$compile_status" -eq 0 -a "$pep8_status" -eq 0 ]; then
Expand Down Expand Up @@ -122,7 +122,7 @@ fi

# for to_be_checked in "$PATHS_TO_CHECK"
# do
# pylint --rcfile="$SPARK_ROOT_DIR/pylintrc" $to_be_checked >> "$PYLINT_REPORT_PATH"
# pylint --rcfile="$SPARK_ROOT_DIR/python/pylintrc" $to_be_checked >> "$PYLINT_REPORT_PATH"
# done

# if [ "${PIPESTATUS[0]}" -ne 0 ]; then
Expand Down
6 changes: 3 additions & 3 deletions make-distribution.sh → dev/make-distribution.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -29,7 +29,7 @@ set -e
set -x

# Figure out where the Spark framework is installed
SPARK_HOME="$(cd "`dirname "$0"`"; pwd)"
SPARK_HOME="$(cd "`dirname "$0"`/.."; pwd)"
Copy link
Contributor

@nchammas nchammas Mar 4, 2016

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Just a heads up: Moving this file is a breaking change for tools like spark-ec2 and Flintrock which automatically build Spark.

I think it's totally OK since 1) this is a developer feature, and 2) this is part of the 2.0 update.

Just wanted to call this out for the record.

Copy link
Member Author

@srowen srowen Mar 4, 2016

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah, I'm even happy to not do this one. It just bugged me that this file was in the top level. It could be symlinked but that doesn't save much. Yeah I suspect some tools would need to look for it in both locations.

Copy link
Contributor

@rxin rxin Mar 7, 2016

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can we update those tools to look for the file in both? It also really bugs me that this is a top level file ...

Copy link
Contributor

@nchammas nchammas Mar 7, 2016

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sure. I don't mind opening a PR against spark-ec2 to do this.

Copy link
Contributor

@nchammas nchammas Mar 7, 2016

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Oh never mind, spark-ec2 actually doesn't use make-distribution.sh, so this shouldn't be a problem.

DISTDIR="$SPARK_HOME/dist"

MAKE_TGZ=false
Expand All @@ -41,7 +41,7 @@ function exit_with_usage {
echo ""
echo "usage:"
cl_options="[--name] [--tgz] [--mvn <mvn-command>]"
echo "./make-distribution.sh $cl_options <maven build options>"
echo "make-distribution.sh $cl_options <maven build options>"
echo "See Spark's \"Building Spark\" doc for correct Maven options."
echo ""
exit 1
Expand Down Expand Up @@ -104,7 +104,7 @@ fi
if [ $(command -v git) ]; then
GITREV=$(git rev-parse --short HEAD 2>/dev/null || :)
if [ ! -z "$GITREV" ]; then
GITREVSTRING=" (git revision $GITREV)"
GITREVSTRING=" (git revision $GITREV)"
fi
unset GITREV
fi
Expand Down
0 tox.ini → dev/tox.ini
View file
Open in desktop
File renamed without changes.
6 changes: 3 additions & 3 deletions docs/building-spark.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -35,12 +35,12 @@ to the `sharedSettings` val. See also [this PR](https://github.com/apache/spark/

To create a Spark distribution like those distributed by the
[Spark Downloads](http://spark.apache.org/downloads.html) page, and that is laid out so as
to be runnable, use `make-distribution.sh` in the project root directory. It can be configured
to be runnable, use `./dev/make-distribution.sh` in the project root directory. It can be configured
with Maven profile settings and so on like the direct Maven build. Example:

./make-distribution.sh --name custom-spark --tgz -Psparkr -Phadoop-2.4 -Phive -Phive-thriftserver -Pyarn
./dev/make-distribution.sh --name custom-spark --tgz -Psparkr -Phadoop-2.4 -Phive -Phive-thriftserver -Pyarn

For more information on usage, run `./make-distribution.sh --help`
For more information on usage, run `./dev/make-distribution.sh --help`

# Setting up Maven's Memory Usage

Expand Down
4 changes: 2 additions & 2 deletions docs/running-on-mesos.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -98,10 +98,10 @@ To host on HDFS, use the Hadoop fs put command: `hadoop fs -put spark-{{site.SPA


Or if you are using a custom-compiled version of Spark, you will need to create a package using
the `make-distribution.sh` script included in a Spark source tarball/checkout.
the `dev/make-distribution.sh` script included in a Spark source tarball/checkout.

1. Download and build Spark using the instructions [here](index.html)
2. Create a binary package using `make-distribution.sh --tgz`.
2. Create a binary package using `./dev/make-distribution.sh --tgz`.
3. Upload archive to http/s3/hdfs


Expand Down
4 changes: 2 additions & 2 deletions pom.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2246,7 +2246,7 @@
<failOnWarning>false</failOnWarning>
<sourceDirectory>${basedir}/src/main/scala</sourceDirectory>
<testSourceDirectory>${basedir}/src/test/scala</testSourceDirectory>
<configLocation>scalastyle-config.xml</configLocation>
<configLocation>dev/scalastyle-config.xml</configLocation>
<outputFile>${basedir}/target/scalastyle-output.xml</outputFile>
<inputEncoding>${project.build.sourceEncoding}</inputEncoding>
<outputEncoding>${project.reporting.outputEncoding}</outputEncoding>
Expand All @@ -2270,7 +2270,7 @@
<failOnWarning>false</failOnWarning>
<sourceDirectory>${basedir}/src/main/java</sourceDirectory>
<testSourceDirectory>${basedir}/src/test/java</testSourceDirectory>
<configLocation>checkstyle.xml</configLocation>
<configLocation>dev/checkstyle.xml</configLocation>
<outputFile>${basedir}/target/checkstyle-output.xml</outputFile>
<inputEncoding>${project.build.sourceEncoding}</inputEncoding>
<outputEncoding>${project.reporting.outputEncoding}</outputEncoding>
Expand Down
0 pylintrc → python/pylintrc
View file
Open in desktop
File renamed without changes.
29 changes: 0 additions & 29 deletions sbt/sbt
View file
Open in desktop

This file was deleted.