[SPARK-11636][SQL] Support classes defined in the REPL with Encoders

[marmbrus]

theScaryParts (i.e. changes to the repl, executor classloaders and codegen)...

[SparkQA]

Test build #46276 has finished for PR 9825 at commit 4428edd.

• This patch fails PySpark unit tests.
• This patch merges cleanly.
• This patch adds the following public classes (experimental):\n * public final class UnsafeSorterSpillReader extends UnsafeSorterIterator implements Closeable\n * class CountVectorizerModelWriter(instance: CountVectorizerModel) extends Writer\n * final class IDF(override val uid: String) extends Estimator[IDFModel] with IDFBase with Writable\n * class MinMaxScalerModelWriter(instance: MinMaxScalerModel) extends Writer\n * class StandardScalerModelWriter(instance: StandardScalerModel) extends Writer\n * class StringIndexModelWriter(instance: StringIndexerModel) extends Writer\n * |class $\n * trait ScalaReflection\n * case class Schema(dataType: DataType, nullable: Boolean)\n * s\"Unable to generate an encoder for inner class$\n *case class EncodeUsingSerializer(child: Expression, kryo: Boolean) extends UnaryExpression \n *case class DecodeUsingSerializer[T](child: Expression, tag: ClassTag[T], kryo: Boolean)`\n

[marmbrus]

test this please

[SparkQA]

Test build #46295 has finished for PR 9825 at commit 4428edd.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds the following public classes (experimental):\n * |class $\n

[SparkQA]

Test build #46304 has finished for PR 9825 at commit 57a3367.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds the following public classes (experimental):\n * |class $\n

[ScrapCodes]

I could not get this test to fail without this patch.

[marmbrus]

This test case only fails on partial applications of this patch. In particular with the classloader change here without the clasloader hack here.

[ScrapCodes]

Yes, I can see it failing now both in scala 2.10 and 2.11.

I will spend more time to understand, if it is possible to fix it in some other way.

[rxin]

I think we should include this in 1.6 -- otherwise there is no way for users to use datasets in the repl.

[yhuai]

+1

[SparkQA]

Test build #46438 has finished for PR 9825 at commit 671fe53.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds the following public classes (experimental):\n * |class $\n

[marmbrus]

Thanks, merging to master and 1.6