[ADAM-1666] SQLContext creation fix for Spark 2.x #1777

akmorrow13 · 2017-10-24T17:37:12Z

Only for Spark 2.x. However, this breaks for Spark 1.x because SparkSession does not exist. Any ideas?

fnothaft · 2017-10-24T17:44:37Z

Hi @akmorrow13! Can you give a bit of background on why the fix necessitates moving to SparkSession from SparkContext?

akmorrow13 · 2017-10-24T17:48:26Z

Sure @fnothaft ! SQLContext is not backwards compatible in Spark 2.X, so trying to pass the sparkContext from pyspark to scala will not work. In this case, scala tries to create an SQLContext, which fails, because the metastore_db is already locked by the SQLContext created in pyspark (this throws an error). SparkSessions, however, can be passed back to scala. The SparkSession can then be used to activate scala's session using SparkSession.setActiveSession(ss). This way, scala will not try to re-instantiate an SQLContext.

fnothaft · 2017-10-24T17:53:09Z

I guess I'm a little curious as to why this happens in local mode, but doesn't occur in any of our unit tests. Any thoughts there?

AmplabJenkins · 2017-10-24T18:02:32Z

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/ADAM-prb/2444/

Build result: FAILURE

[...truncated 15 lines...] > /home/jenkins/git2/bin/git fetch --tags --progress https://github.com/bigdatagenomics/adam.git +refs/pull/:refs/remotes/origin/pr/ # timeout=15 > /home/jenkins/git2/bin/git rev-parse origin/pr/1777/merge^{commit} # timeout=10 > /home/jenkins/git2/bin/git branch -a -v --no-abbrev --contains d002a02ba71a9e174e61dc29c21759c2519e042c # timeout=10Checking out Revision d002a02ba71a9e174e61dc29c21759c2519e042c (origin/pr/1777/merge) > /home/jenkins/git2/bin/git config core.sparsecheckout # timeout=10 > /home/jenkins/git2/bin/git checkout -f d002a02ba71a9e174e61dc29c21759c2519e042cFirst time build. Skipping changelog.Triggering ADAM-prb ? 2.6.2,2.11,1.6.3,centosTriggering ADAM-prb ? 2.6.2,2.11,2.2.0,centosTriggering ADAM-prb ? 2.6.2,2.10,1.6.3,centosTriggering ADAM-prb ? 2.6.2,2.10,2.2.0,centosTriggering ADAM-prb ? 2.7.3,2.10,2.2.0,centosTriggering ADAM-prb ? 2.7.3,2.10,1.6.3,centosTriggering ADAM-prb ? 2.7.3,2.11,1.6.3,centosTriggering ADAM-prb ? 2.7.3,2.11,2.2.0,centosADAM-prb ? 2.6.2,2.11,1.6.3,centos completed with result FAILUREADAM-prb ? 2.6.2,2.11,2.2.0,centos completed with result SUCCESSADAM-prb ? 2.6.2,2.10,1.6.3,centos completed with result FAILUREADAM-prb ? 2.6.2,2.10,2.2.0,centos completed with result SUCCESSADAM-prb ? 2.7.3,2.10,2.2.0,centos completed with result SUCCESSADAM-prb ? 2.7.3,2.10,1.6.3,centos completed with result FAILUREADAM-prb ? 2.7.3,2.11,1.6.3,centos completed with result FAILUREADAM-prb ? 2.7.3,2.11,2.2.0,centos completed with result SUCCESSNotifying endpoint 'HTTP:https://webhooks.gitter.im/e/ac8bb6e9f53357bc8aa8'
Test FAILed.

akmorrow13 · 2017-10-24T18:08:59Z

I think the issue is that all the unit tests use SparkContext for the entry point, but the Dataset API uses SparkSession, which instantiates SQLContext. I can recreate (not quite the same) issue in the unit tests if I create a SparkSession in the unit tests:

    def setUp(self):
        self._old_sys_path = list(sys.path)
        class_name = self.__class__.__name__
        self.ss = SparkSession.builder.master('local[4]').appName(class_name).getOrCreate()
        print(self.ss)
        self.sc = self.ss.sparkContext

This throws the error:
Another SparkContext is being constructed (or threw an exception in its constructor). This may indicate an error, since only one SparkContext may be running in this JVM (see SPARK-2243).

However, the tests pass when you include these PR changes using a SparkSession.

fnothaft · 2017-10-24T18:33:17Z

Would it be possible to make a lower impact/backwards compatible change by using SparkSession.sparkContext? From the Scala API docs, I believe this solely a wrapper around SparkSession, and would avoid re-instantiating the SparkSession.

akmorrow13 · 2017-10-24T18:58:39Z

@fnothaft do you mean pass back ss.sparkContext._jsc.sc() to ADAMContext instead of SparkSession? This will cause the same issue, because SQLContext is not being communicated.

akmorrow13 · 2017-10-24T19:33:08Z

Ok I think I misunderstood you. I updated ADAMContextFromSparkSession() to get the SparkContext from the session. Let me know if this is what you meant @fnothaft .

AmplabJenkins · 2017-10-24T19:53:58Z

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/ADAM-prb/2446/

Build result: FAILURE

[...truncated 15 lines...] > /home/jenkins/git2/bin/git fetch --tags --progress https://github.com/bigdatagenomics/adam.git +refs/pull/:refs/remotes/origin/pr/ # timeout=15 > /home/jenkins/git2/bin/git rev-parse origin/pr/1777/merge^{commit} # timeout=10 > /home/jenkins/git2/bin/git branch -a -v --no-abbrev --contains 219b032 # timeout=10Checking out Revision 219b032 (origin/pr/1777/merge) > /home/jenkins/git2/bin/git config core.sparsecheckout # timeout=10 > /home/jenkins/git2/bin/git checkout -f 219b032db5b288f14ace55349290791e61def675First time build. Skipping changelog.Triggering ADAM-prb ? 2.6.2,2.11,1.6.3,centosTriggering ADAM-prb ? 2.6.2,2.11,2.2.0,centosTriggering ADAM-prb ? 2.6.2,2.10,1.6.3,centosTriggering ADAM-prb ? 2.6.2,2.10,2.2.0,centosTriggering ADAM-prb ? 2.7.3,2.10,2.2.0,centosTriggering ADAM-prb ? 2.7.3,2.10,1.6.3,centosTriggering ADAM-prb ? 2.7.3,2.11,1.6.3,centosTriggering ADAM-prb ? 2.7.3,2.11,2.2.0,centosADAM-prb ? 2.6.2,2.11,1.6.3,centos completed with result FAILUREADAM-prb ? 2.6.2,2.11,2.2.0,centos completed with result SUCCESSADAM-prb ? 2.6.2,2.10,1.6.3,centos completed with result FAILUREADAM-prb ? 2.6.2,2.10,2.2.0,centos completed with result SUCCESSADAM-prb ? 2.7.3,2.10,2.2.0,centos completed with result SUCCESSADAM-prb ? 2.7.3,2.10,1.6.3,centos completed with result FAILUREADAM-prb ? 2.7.3,2.11,1.6.3,centos completed with result FAILUREADAM-prb ? 2.7.3,2.11,2.2.0,centos completed with result SUCCESSNotifying endpoint 'HTTP:https://webhooks.gitter.im/e/ac8bb6e9f53357bc8aa8'
Test FAILed.

akmorrow13 · 2018-01-06T18:09:08Z

Jenkins, test this please.

AmplabJenkins · 2018-01-06T18:14:00Z

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/ADAM-prb/2550/

Build result: FAILURE

[...truncated 7 lines...] > /home/jenkins/git2/bin/git init /home/jenkins/workspace/ADAM-prb # timeout=10Fetching upstream changes from https://github.com/bigdatagenomics/adam.git > /home/jenkins/git2/bin/git --version # timeout=10 > /home/jenkins/git2/bin/git fetch --tags --progress https://github.com/bigdatagenomics/adam.git +refs/heads/:refs/remotes/origin/ # timeout=15 > /home/jenkins/git2/bin/git config remote.origin.url https://github.com/bigdatagenomics/adam.git # timeout=10 > /home/jenkins/git2/bin/git config --add remote.origin.fetch +refs/heads/:refs/remotes/origin/ # timeout=10 > /home/jenkins/git2/bin/git config remote.origin.url https://github.com/bigdatagenomics/adam.git # timeout=10Fetching upstream changes from https://github.com/bigdatagenomics/adam.git > /home/jenkins/git2/bin/git fetch --tags --progress https://github.com/bigdatagenomics/adam.git +refs/pull/:refs/remotes/origin/pr/ # timeout=15 > /home/jenkins/git2/bin/git rev-parse fa97028^{commit} # timeout=10 > /home/jenkins/git2/bin/git branch -a -v --no-abbrev --contains fa97028 # timeout=10Checking out Revision fa97028 (origin/pr/1777/merge, origin/pr/1777/head) > /home/jenkins/git2/bin/git config core.sparsecheckout # timeout=10 > /home/jenkins/git2/bin/git checkout -f fa970289aa997ec4884e16f61791649e8be8383eFirst time build. Skipping changelog.Triggering ADAM-prb ? 2.7.3,2.11,2.2.1,centosTriggering ADAM-prb ? 2.6.2,2.10,2.2.1,centosTriggering ADAM-prb ? 2.7.3,2.10,2.2.1,centosTriggering ADAM-prb ? 2.6.2,2.11,2.2.1,centosADAM-prb ? 2.7.3,2.11,2.2.1,centos completed with result FAILUREADAM-prb ? 2.6.2,2.10,2.2.1,centos completed with result FAILUREADAM-prb ? 2.7.3,2.10,2.2.1,centos completed with result FAILUREADAM-prb ? 2.6.2,2.11,2.2.1,centos completed with result FAILURENotifying endpoint 'HTTP:https://webhooks.gitter.im/e/ac8bb6e9f53357bc8aa8'
Test FAILed.

fnothaft · 2018-01-11T14:18:13Z

Jenkins, retest this please.

fnothaft · 2018-01-11T14:18:37Z

Actually, @akmorrow13 could you rebase this and clean conflicts?

AmplabJenkins · 2018-01-11T14:24:02Z

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/ADAM-prb/2562/

Build result: FAILURE

[...truncated 8 lines...]Fetching upstream changes from https://github.com/bigdatagenomics/adam.git > /home/jenkins/git2/bin/git --version # timeout=10 > /home/jenkins/git2/bin/git fetch --tags --progress https://github.com/bigdatagenomics/adam.git +refs/heads/:refs/remotes/origin/ # timeout=15 > /home/jenkins/git2/bin/git config remote.origin.url https://github.com/bigdatagenomics/adam.git # timeout=10 > /home/jenkins/git2/bin/git config --add remote.origin.fetch +refs/heads/:refs/remotes/origin/ # timeout=10 > /home/jenkins/git2/bin/git config remote.origin.url https://github.com/bigdatagenomics/adam.git # timeout=10Fetching upstream changes from https://github.com/bigdatagenomics/adam.git > /home/jenkins/git2/bin/git fetch --tags --progress https://github.com/bigdatagenomics/adam.git +refs/pull/:refs/remotes/origin/pr/ # timeout=15 > /home/jenkins/git2/bin/git rev-parse fa97028^{commit} # timeout=10 > /home/jenkins/git2/bin/git branch -a -v --no-abbrev --contains fa97028 # timeout=10Checking out Revision fa97028 (origin/pr/1777/head, origin/pr/1777/merge) > /home/jenkins/git2/bin/git config core.sparsecheckout # timeout=10 > /home/jenkins/git2/bin/git checkout -f fa97028 > /home/jenkins/git2/bin/git rev-list fa97028 # timeout=10 > /home/jenkins/git2/bin/git rev-list fa97028 # timeout=10Triggering ADAM-prb ? 2.7.3,2.11,2.2.1,centosTriggering ADAM-prb ? 2.6.2,2.10,2.2.1,centosTriggering ADAM-prb ? 2.7.3,2.10,2.2.1,centosTriggering ADAM-prb ? 2.6.2,2.11,2.2.1,centosADAM-prb ? 2.7.3,2.11,2.2.1,centos completed with result FAILUREADAM-prb ? 2.6.2,2.10,2.2.1,centos completed with result FAILUREADAM-prb ? 2.7.3,2.10,2.2.1,centos completed with result FAILUREADAM-prb ? 2.6.2,2.11,2.2.1,centos completed with result FAILURENotifying endpoint 'HTTP:https://webhooks.gitter.im/e/ac8bb6e9f53357bc8aa8'
Test FAILed.

AmplabJenkins · 2018-01-12T13:29:06Z

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/ADAM-prb/2564/

Build result: FAILURE

[...truncated 7 lines...] > /home/jenkins/git2/bin/git init /home/jenkins/workspace/ADAM-prb # timeout=10Fetching upstream changes from https://github.com/bigdatagenomics/adam.git > /home/jenkins/git2/bin/git --version # timeout=10 > /home/jenkins/git2/bin/git fetch --tags --progress https://github.com/bigdatagenomics/adam.git +refs/heads/:refs/remotes/origin/ # timeout=15 > /home/jenkins/git2/bin/git config remote.origin.url https://github.com/bigdatagenomics/adam.git # timeout=10 > /home/jenkins/git2/bin/git config --add remote.origin.fetch +refs/heads/:refs/remotes/origin/ # timeout=10 > /home/jenkins/git2/bin/git config remote.origin.url https://github.com/bigdatagenomics/adam.git # timeout=10Fetching upstream changes from https://github.com/bigdatagenomics/adam.git > /home/jenkins/git2/bin/git fetch --tags --progress https://github.com/bigdatagenomics/adam.git +refs/pull/:refs/remotes/origin/pr/ # timeout=15 > /home/jenkins/git2/bin/git rev-parse origin/pr/1777/merge^{commit} # timeout=10 > /home/jenkins/git2/bin/git branch -a -v --no-abbrev --contains 57341c7 # timeout=10Checking out Revision 57341c7 (origin/pr/1777/merge) > /home/jenkins/git2/bin/git config core.sparsecheckout # timeout=10 > /home/jenkins/git2/bin/git checkout -f 57341c7 > /home/jenkins/git2/bin/git rev-list fa97028 # timeout=10Triggering ADAM-prb ? 2.7.3,2.11,2.2.1,centosTriggering ADAM-prb ? 2.6.2,2.10,2.2.1,centosTriggering ADAM-prb ? 2.7.3,2.10,2.2.1,centosTriggering ADAM-prb ? 2.6.2,2.11,2.2.1,centosADAM-prb ? 2.7.3,2.11,2.2.1,centos completed with result FAILUREADAM-prb ? 2.6.2,2.10,2.2.1,centos completed with result FAILUREADAM-prb ? 2.7.3,2.10,2.2.1,centos completed with result FAILUREADAM-prb ? 2.6.2,2.11,2.2.1,centos completed with result FAILURENotifying endpoint 'HTTP:https://webhooks.gitter.im/e/ac8bb6e9f53357bc8aa8'
Test FAILed.

akmorrow13 · 2018-01-12T13:35:59Z

Jenkins, retest this please.

coveralls · 2018-01-12T13:49:25Z

Coverage increased (+0.04%) to 82.822% when pulling e25a12a on akmorrow13:sqlcontext-pyspark into 9adadef on bigdatagenomics:master.

AmplabJenkins · 2018-01-12T14:00:27Z

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/ADAM-prb/2565/

Build result: FAILURE

[...truncated 7 lines...] > /home/jenkins/git2/bin/git init /home/jenkins/workspace/ADAM-prb # timeout=10Fetching upstream changes from https://github.com/bigdatagenomics/adam.git > /home/jenkins/git2/bin/git --version # timeout=10 > /home/jenkins/git2/bin/git fetch --tags --progress https://github.com/bigdatagenomics/adam.git +refs/heads/:refs/remotes/origin/ # timeout=15 > /home/jenkins/git2/bin/git config remote.origin.url https://github.com/bigdatagenomics/adam.git # timeout=10 > /home/jenkins/git2/bin/git config --add remote.origin.fetch +refs/heads/:refs/remotes/origin/ # timeout=10 > /home/jenkins/git2/bin/git config remote.origin.url https://github.com/bigdatagenomics/adam.git # timeout=10Fetching upstream changes from https://github.com/bigdatagenomics/adam.git > /home/jenkins/git2/bin/git fetch --tags --progress https://github.com/bigdatagenomics/adam.git +refs/pull/:refs/remotes/origin/pr/ # timeout=15 > /home/jenkins/git2/bin/git rev-parse origin/pr/1777/merge^{commit} # timeout=10 > /home/jenkins/git2/bin/git branch -a -v --no-abbrev --contains 57341c7 # timeout=10Checking out Revision 57341c7 (origin/pr/1777/merge) > /home/jenkins/git2/bin/git config core.sparsecheckout # timeout=10 > /home/jenkins/git2/bin/git checkout -f 57341c7 > /home/jenkins/git2/bin/git rev-list 57341c7 # timeout=10Triggering ADAM-prb ? 2.7.3,2.11,2.2.1,centosTriggering ADAM-prb ? 2.6.2,2.10,2.2.1,centosTriggering ADAM-prb ? 2.7.3,2.10,2.2.1,centosTriggering ADAM-prb ? 2.6.2,2.11,2.2.1,centosADAM-prb ? 2.7.3,2.11,2.2.1,centos completed with result FAILUREADAM-prb ? 2.6.2,2.10,2.2.1,centos completed with result SUCCESSADAM-prb ? 2.7.3,2.10,2.2.1,centos completed with result SUCCESSADAM-prb ? 2.6.2,2.11,2.2.1,centos completed with result FAILURENotifying endpoint 'HTTP:https://webhooks.gitter.im/e/ac8bb6e9f53357bc8aa8'
Test FAILed.

coveralls · 2018-01-12T20:49:29Z

Coverage increased (+0.04%) to 82.822% when pulling 7365b28 on akmorrow13:sqlcontext-pyspark into 9adadef on bigdatagenomics:master.

AmplabJenkins · 2018-01-12T20:57:03Z

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/ADAM-prb/2566/

Build result: FAILURE

[...truncated 7 lines...] > /home/jenkins/git2/bin/git init /home/jenkins/workspace/ADAM-prb # timeout=10Fetching upstream changes from https://github.com/bigdatagenomics/adam.git > /home/jenkins/git2/bin/git --version # timeout=10 > /home/jenkins/git2/bin/git fetch --tags --progress https://github.com/bigdatagenomics/adam.git +refs/heads/:refs/remotes/origin/ # timeout=15 > /home/jenkins/git2/bin/git config remote.origin.url https://github.com/bigdatagenomics/adam.git # timeout=10 > /home/jenkins/git2/bin/git config --add remote.origin.fetch +refs/heads/:refs/remotes/origin/ # timeout=10 > /home/jenkins/git2/bin/git config remote.origin.url https://github.com/bigdatagenomics/adam.git # timeout=10Fetching upstream changes from https://github.com/bigdatagenomics/adam.git > /home/jenkins/git2/bin/git fetch --tags --progress https://github.com/bigdatagenomics/adam.git +refs/pull/:refs/remotes/origin/pr/ # timeout=15 > /home/jenkins/git2/bin/git rev-parse origin/pr/1777/merge^{commit} # timeout=10 > /home/jenkins/git2/bin/git branch -a -v --no-abbrev --contains 8be5dbe # timeout=10Checking out Revision 8be5dbe (origin/pr/1777/merge) > /home/jenkins/git2/bin/git config core.sparsecheckout # timeout=10 > /home/jenkins/git2/bin/git checkout -f 8be5dbe > /home/jenkins/git2/bin/git rev-list 57341c7 # timeout=10Triggering ADAM-prb ? 2.7.3,2.11,2.2.1,centosTriggering ADAM-prb ? 2.6.2,2.10,2.2.1,centosTriggering ADAM-prb ? 2.7.3,2.10,2.2.1,centosTriggering ADAM-prb ? 2.6.2,2.11,2.2.1,centosADAM-prb ? 2.7.3,2.11,2.2.1,centos completed with result FAILUREADAM-prb ? 2.6.2,2.10,2.2.1,centos completed with result SUCCESSADAM-prb ? 2.7.3,2.10,2.2.1,centos completed with result SUCCESSADAM-prb ? 2.6.2,2.11,2.2.1,centos completed with result FAILURENotifying endpoint 'HTTP:https://webhooks.gitter.im/e/ac8bb6e9f53357bc8aa8'
Test FAILed.

akmorrow13 · 2018-01-14T03:27:33Z

Jenkins, retest this please.

coveralls · 2018-01-14T03:39:57Z

Coverage increased (+0.04%) to 82.822% when pulling 7365b28 on akmorrow13:sqlcontext-pyspark into 9adadef on bigdatagenomics:master.

AmplabJenkins · 2018-01-14T03:47:31Z

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/ADAM-prb/2567/
Test PASSed.

fnothaft

LGTM! Can you fix the small typo, then I'll merge? (or, I can fix it and merge manually, LMK which you'd prefer)

fnothaft · 2018-01-14T15:32:37Z

adam-core/src/main/scala/org/bdgenomics/adam/rdd/ADAMContext.scala

@@ -999,6 +999,17 @@ object ADAMContext {
  // Add ADAM Spark context methods
  implicit def sparkContextToADAMContext(sc: SparkContext): ADAMContext = new ADAMContext(sc)

+  /**
+   * Creates an ADAMContext form SparkSession. Sets active session, including SQLContext, to input session.


form -> from

fnothaft · 2018-01-14T15:33:17Z

adam-core/src/main/scala/org/bdgenomics/adam/rdd/ADAMContext.scala

+   * @return ADAMContext
+   */
+  def ADAMContextFromSession(ss: SparkSession): ADAMContext = {
+    SparkSession.setActiveSession(ss)


OOC, why do you need the setActiveSession call?

This resets the sparkContext to read the session passed in. This fixes the issue if one sparkContext has already been started in the scala backend, then it will be replaced.

coveralls · 2018-01-14T15:55:46Z

Coverage increased (+0.04%) to 82.822% when pulling ab9338a on akmorrow13:sqlcontext-pyspark into 9adadef on bigdatagenomics:master.

AmplabJenkins · 2018-01-14T16:11:51Z

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/ADAM-prb/2571/
Test PASSed.

akmorrow13 · 2018-01-14T16:46:26Z

@fnothaft I haved fixed and squashed so we should be set

fnothaft · 2018-01-15T00:03:39Z

Thanks @akmorrow13! I added a comment explaining the setActiveSession and touched up the commit message and merge this manually as 4223f56.

heuermh changed the title ~~SQLContext creation fix for Spark 2.x~~ [ADAM-1774] SQLContext creation fix for Spark 2.x Oct 24, 2017

heuermh changed the title ~~[ADAM-1774] SQLContext creation fix for Spark 2.x~~ [ADAM-1666] SQLContext creation fix for Spark 2.x Oct 24, 2017

akmorrow13 mentioned this pull request Oct 25, 2017

mango-notebook: metastore_db error bigdatagenomics/mango#323

Closed

fnothaft added this to the 0.24.0 milestone Oct 25, 2017

akmorrow13 force-pushed the sqlcontext-pyspark branch from fa97028 to e25a12a Compare January 12, 2018 13:25

fnothaft approved these changes Jan 14, 2018

View reviewed changes

SQLContext creation fix for Spark 2.x

ab9338a

akmorrow13 force-pushed the sqlcontext-pyspark branch from 7365b28 to ab9338a Compare January 14, 2018 15:39

fnothaft closed this Jan 15, 2018

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[ADAM-1666] SQLContext creation fix for Spark 2.x #1777

[ADAM-1666] SQLContext creation fix for Spark 2.x #1777

akmorrow13 commented Oct 24, 2017 •

edited

Loading

fnothaft commented Oct 24, 2017

akmorrow13 commented Oct 24, 2017

fnothaft commented Oct 24, 2017

AmplabJenkins commented Oct 24, 2017

akmorrow13 commented Oct 24, 2017 •

edited

Loading

fnothaft commented Oct 24, 2017

akmorrow13 commented Oct 24, 2017

akmorrow13 commented Oct 24, 2017

AmplabJenkins commented Oct 24, 2017

akmorrow13 commented Jan 6, 2018

AmplabJenkins commented Jan 6, 2018

fnothaft commented Jan 11, 2018

fnothaft commented Jan 11, 2018

AmplabJenkins commented Jan 11, 2018

AmplabJenkins commented Jan 12, 2018

akmorrow13 commented Jan 12, 2018

coveralls commented Jan 12, 2018 •

edited

Loading

AmplabJenkins commented Jan 12, 2018

coveralls commented Jan 12, 2018 •

edited

Loading

AmplabJenkins commented Jan 12, 2018

akmorrow13 commented Jan 14, 2018

coveralls commented Jan 14, 2018 •

edited

Loading

AmplabJenkins commented Jan 14, 2018

fnothaft left a comment

fnothaft Jan 14, 2018

fnothaft Jan 14, 2018

akmorrow13 Jan 14, 2018

coveralls commented Jan 14, 2018 •

edited

Loading

AmplabJenkins commented Jan 14, 2018

akmorrow13 commented Jan 14, 2018

fnothaft commented Jan 15, 2018

[ADAM-1666] SQLContext creation fix for Spark 2.x #1777

[ADAM-1666] SQLContext creation fix for Spark 2.x #1777

Conversation

akmorrow13 commented Oct 24, 2017 • edited Loading

fnothaft commented Oct 24, 2017

akmorrow13 commented Oct 24, 2017

fnothaft commented Oct 24, 2017

AmplabJenkins commented Oct 24, 2017

Build result: FAILURE

akmorrow13 commented Oct 24, 2017 • edited Loading

fnothaft commented Oct 24, 2017

akmorrow13 commented Oct 24, 2017

akmorrow13 commented Oct 24, 2017

AmplabJenkins commented Oct 24, 2017

Build result: FAILURE

akmorrow13 commented Jan 6, 2018

AmplabJenkins commented Jan 6, 2018

Build result: FAILURE

fnothaft commented Jan 11, 2018

fnothaft commented Jan 11, 2018

AmplabJenkins commented Jan 11, 2018

Build result: FAILURE

AmplabJenkins commented Jan 12, 2018

Build result: FAILURE

akmorrow13 commented Jan 12, 2018

coveralls commented Jan 12, 2018 • edited Loading

AmplabJenkins commented Jan 12, 2018

Build result: FAILURE

coveralls commented Jan 12, 2018 • edited Loading

AmplabJenkins commented Jan 12, 2018

Build result: FAILURE

akmorrow13 commented Jan 14, 2018

coveralls commented Jan 14, 2018 • edited Loading

AmplabJenkins commented Jan 14, 2018

fnothaft left a comment

Choose a reason for hiding this comment

fnothaft Jan 14, 2018

Choose a reason for hiding this comment

fnothaft Jan 14, 2018

Choose a reason for hiding this comment

akmorrow13 Jan 14, 2018

Choose a reason for hiding this comment

coveralls commented Jan 14, 2018 • edited Loading

AmplabJenkins commented Jan 14, 2018

akmorrow13 commented Jan 14, 2018

fnothaft commented Jan 15, 2018

akmorrow13 commented Oct 24, 2017 •

edited

Loading

akmorrow13 commented Oct 24, 2017 •

edited

Loading

coveralls commented Jan 12, 2018 •

edited

Loading

coveralls commented Jan 12, 2018 •

edited

Loading

coveralls commented Jan 14, 2018 •

edited

Loading

coveralls commented Jan 14, 2018 •

edited

Loading