[minor] [build] Use custom temp directory during build.

[vanzin]

Even with all the efforts to cleanup the temp directories created by
unit tests, Spark leaves a lot of garbage in /tmp after a test run.
This change overrides java.io.tmpdir to place those files under the
build directory instead.

After an sbt full unit test run, I was left with > 400 MB of temp
files. Since they're now under the build dir, it's much easier to
clean them up.

Also make a slight change to a unit test to make it not pollute the
source directory with test data.

[JoshRosen]

Nice catch. This plays nicely with some Jenkins tuning work that I'm doing, where we're experimenting with using ramdisks to store the Jenkins workspaces for the pull request builders.

[harishreedharan]

${project.build.directory}/target/tmp ?

[vanzin]

${project.build.directory} is already the target directory.

[harishreedharan]

ah, ok!

[harishreedharan]

+1

[SparkQA]

Test build #34212 has finished for PR 6653 at commit aa92944.

• This patch fails Spark unit tests.
• This patch merges cleanly.
• This patch adds no public classes.

[SparkQA]

Test build #883 has finished for PR 6653 at commit aa92944.

• This patch fails Spark unit tests.
• This patch merges cleanly.
• This patch adds no public classes.

[JoshRosen]

Update: ramdisk didn't make a difference, so we're not using it.

[SparkQA]

Test build #34219 has finished for PR 6653 at commit 31e2dd5.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds the following public classes (experimental):
  • public final class UnsafeRow extends BaseMutableRow
  • public abstract class BaseMutableRow extends BaseRow implements MutableRow
  • public abstract class BaseRow implements Row
  • protected class CodeGenContext
  • abstract class BaseMutableProjection extends MutableProjection
  • class SpecificProjection extends $
  • class BaseOrdering extends Ordering[Row]
  • class SpecificOrdering extends $
  • abstract class Predicate
  • class SpecificPredicate extends $
  • abstract class BaseProject extends Projection
  • class SpecificProjection extends $
  • final class SpecificRow extends $

[srowen]

Looks OK to me.

[JoshRosen]

We should probably merge this into branch-1.4 and 1.3 as well, since those are the more active backport branches; I think that Shane will appreciate the reduction in /tmp garbage on the Jenkins boxes :)

[JoshRosen]

Looks like this may have broken a SparkSubmitSuite test: https://amplab.cs.berkeley.edu/jenkins/job/Spark-Master-Maven-pre-YARN/2654/

We may need to ensure that the temp dir. exists before we run tests or something like that; haven't had time to investigate. Can someone look at this and either revert this patch or push a hotfix?

[vanzin]

Looking. Worked for me locally but may have been a test ordering thing.

[andrewor14]

I reverted this. Could we file a JIRA for this actually? This is a borderline minor change that would be good to track. The implications of the changes here are not super obvious from the diff.

[pwendell]

@vanzin @srowen this broke the Spark 1.3 builds, so I revert it in the Spark 1.3 branch.