[SPARK-11042] [SQL] Add a mechanism to ban creating multiple root SQLContexts/HiveContexts in a JVM #9058
Conversation
|
Test build #43517 has finished for PR 9058 at commit
|
| @@ -1239,6 +1254,24 @@ object SQLContext { | |||
| instantiatedContext.compareAndSet(null, sqlContext) | |||
There was a problem hiding this comment.
@davies Seems when we create a sql context for a new session, instantiatedContext will be set to the context representing that session. Do you think if it makes sense to use instantiatedContext hold the root sql context (the one created directly from user-facing constructor instead of newSession)?
There was a problem hiding this comment.
We use compareAndSet, so the one created by newSession will NOT overwrite the original one.
|
Test build #43534 has finished for PR 9058 at commit
|
|
test this please |
|
Test build #43536 has finished for PR 9058 at commit
|
| if (!allowMultipleRootSQLContexts && isRootContext) { | ||
| getInstantiatedContextOption() match { | ||
| case Some(rootSQLContext) => | ||
| val errMsg = "Only one SparkContext/HiveContext may be running in this JVM." + |
|
LGTM, except some minor comments. |
|
Test build #43570 has finished for PR 9058 at commit
|
|
Test build #43571 has finished for PR 9058 at commit
|
|
Test build #43573 has finished for PR 9058 at commit
|
|
Merged into master, thanks! |
| private val allowMultipleContexts = | ||
| sparkContext.conf.getBoolean( | ||
| SQLConf.ALLOW_MULTIPLE_CONTEXTS.key, | ||
| SQLConf.ALLOW_MULTIPLE_CONTEXTS.defaultValue.get) |
There was a problem hiding this comment.
We can use conf.getConf(SQLConf.ALLOW_MULTIPLE_CONTEXTS) here.
There was a problem hiding this comment.
oh, at here, we have not populated SQLConf (conf you are referring at here is SQLConf, right?).
There was a problem hiding this comment.
ah yes, SQLConf has not been populated here. nvm.
https://issues.apache.org/jira/browse/SPARK-11042