[SPARK-7478][SQL] Added SQLContext.getOrCreate

[tdas]

Having a SQLContext singleton would make it easier for applications to use a lazily instantiated single shared instance of SQLContext when needed. It would avoid problems like

1. In REPL/notebook environment, rerunning the line {{val sqlContext = new SQLContext}} multiple times created different contexts while overriding the reference to previous context, leading to issues like registered temp tables going missing.
2. In Streaming, creating SQLContext directly leads to serialization/deserialization issues when attempting to recover from DStream checkpoints. See [SPARK-6770]. Also to get around this problem I had to suggest creating a singleton instance - https://github.com/apache/spark/blob/master/examples/src/main/scala/org/apache/spark/examples/streaming/SqlNetworkWordCount.scala

This can be solved by {{SQLContext.getOrCreate}} which get or creates a new singleton instance of SQLContext using either a given SparkContext or a given SparkConf.

@rxin @marmbrus

[tdas]

@rxin Design question. Should contexts created directly (that is, not through getOrCreate) also be automatically considered for the singleton?

[SparkQA]

Test build #32220 has finished for PR 6006 at commit bc72868.

• This patch fails Spark unit tests.
• This patch merges cleanly.
• This patch adds no public classes.

[SparkQA]

Test build #32224 has finished for PR 6006 at commit f82ae81.

• This patch fails Spark unit tests.
• This patch merges cleanly.
• This patch adds no public classes.

[marmbrus]

What about HiveContext?

[tdas]

@marmbrus Good you raised the question. @rxin and I had half-a-discussion offline about it. Correct me if I am wrong but HiveContext is a superset of the SQLContext in terms of functionality. So the last HiveContext created should also be accessible through this interface. Since the SQLContext constructor register itself with singleton, the HiveContext will also do the same. This part made sense to me.

Going beyond that if you think there should also be a HiveContext.getOrCreate, which would return the last instantiated HiveContext (not SQLContext), I can add that too in this PR.

[marmbrus]

Yeah the first part seems reasonable. But I think you also need a way to force the getOrCreate command to create a hive context.

[tdas]

test this please.

[tdas]

So HiveContext.getOrCreate sounds good?

[SparkQA]

Test build #32264 has finished for PR 6006 at commit f82ae81.

• This patch fails Spark unit tests.
• This patch merges cleanly.
• This patch adds no public classes.

[SparkQA]

Test build #32880 has finished for PR 6006 at commit 83bc950.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[SparkQA]

Test build #32894 has finished for PR 6006 at commit b4e9721.

• This patch fails Spark unit tests.
• This patch merges cleanly.
• This patch adds no public classes.

[SparkQA]

Test build #32891 has finished for PR 6006 at commit d3ea8e4.

• This patch fails Spark unit tests.
• This patch does not merge cleanly.
• This patch adds no public classes.

[SparkQA]

Test build #32903 has finished for PR 6006 at commit dec5594.

• This patch fails Spark unit tests.
• This patch merges cleanly.
• This patch adds no public classes.

[SparkQA]

Test build #32904 has finished for PR 6006 at commit bf8cf50.

• This patch fails Spark unit tests.
• This patch merges cleanly.
• This patch adds no public classes.

[tdas]

@marmbrus Can you help me debug this? HiveContextSuite seems to be failing due to JDBC connection. What are the constraints of creating a HiveContext?

[scwf]

HiveContext use derby as the metastore database defaultly which only support single instance for one metastore path.

I think you need config the javax.jdo.option.ConnectionURL before creating the HiveContext
you can change getorcreate like this:

  def getOrCreate(sparkContext: SparkContext, config: Map[String, String] = Map.empty): HiveContext   = {
    INSTANTIATION_LOCK.synchronized {
      if (lastInstantiatedContext.get() == null) {
        new HiveContext(sparkContext) {
          override def configure(): Map[String, String] = config
        }
      }
    }
    lastInstantiatedContext.get()
  }

then when you create hivecontext in HiveContextSuite, you can configure it like this

    val hiveContext = HiveContext.getOrCreate(testSparkContext, HiveContext.newTemporaryConfiguration)

[SparkQA]

Test build #33199 has started for PR 6006 at commit 48adb14.

[tdas]

@marmbrus Could you take a look?

[SparkQA]

Test build #33201 has finished for PR 6006 at commit c66ca76.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[SparkQA]

Test build #33207 has finished for PR 6006 at commit 79fe069.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[marmbrus]

using the given SparkContext.

[tdas]

Good catch.

[marmbrus]

Minor comments, otherwise LGTM.

[SparkQA]

Test build #33264 has finished for PR 6006 at commit 25f4da9.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds the following public classes (experimental):
  • public class TaskMemoryManager

[tdas]

I addressed your comments and I am merging this. Thanks @marmbrus for reviewing!

[jelez]

Can you guys point me what is the proper way to use getOrCreate with HiveContext?

[mwws]

@jelez you can create an HiveContextSingleton to workaround it. Refer to example "SqlNetWorkWordCount"

@tdas Why you removed HiveContext.getOrCreate? I can't find obvious reasons from the conversation.