[SPARK-14281][TESTS] Fix java8-tests and simplify their build

[JoshRosen]

This patch fixes a compilation / build break in Spark's java8-tests and refactors their POM to simplify the build. See individual commit messages for more details.

[JoshRosen]

AFAIK the idea behind this logic was to ensure that Java files would not be compiled by the Scala maven plugin, but disabling this has the side-effect of preventing this module's Scala test source file from being compiled and run in the Maven build.

Instead, I think it's okay to just modify the Scala Maven Plugin's javacArgs configuration.

[JoshRosen]

I have updated the pull request builder Jenkins job to use a Java 8 JDK, so these tests should now be run.

It just occurred to me that I may want to configure the dev/sparktestsupport/modules.py file to make sure that these tests are automatically run if only a subset of tests would otherwise be run.

[SparkQA]

Test build #54555 has finished for PR 12073 at commit 16b7823.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[vanzin]

Out of curiosity, isn't all this work mostly throw away if we're really dropping jdk7 support? At that point you could just move the source files to core/src/test, and get rid of all the build-related code.

[JoshRosen]

Yep, this would be irrelevant if we dropped JDK 7. However, I'm doing some other work which may have an impact on Java 8 lambda compatibility so I wanted to fix these tests to make my life a little easier.

[srowen]

+1. "Oops". Yeah we probably want to keep these tests in any event since they specifically test Java 8. Bigger question is just how far we want to go to update and streamline Java code and tests to use lambdas.

[JoshRosen]

Any objections to these build changes or is this now good to merge?

[vanzin]

Go ahead, it's a net improvement on the current state.

[JoshRosen]

Merging now; thanks everyone.