[SPARK-11699] TrackStateRDDSuite fails on Jenkins builds

[tedyu]

Here is the stack trace:

java.lang.NullPointerException
      at org.apache.spark.shuffle.sort.SortShuffleManager.registerShuffle(SortShuffleManager.scala:91)
      at org.apache.spark.ShuffleDependency.<init>(Dependency.scala:90)
      at org.apache.spark.rdd.ShuffledRDD.getDependencies(ShuffledRDD.scala:80)
      at org.apache.spark.rdd.RDD$$anonfun$dependencies$2.apply(RDD.scala:226)
      at org.apache.spark.rdd.RDD$$anonfun$dependencies$2.apply(RDD.scala:224)
      at scala.Option.getOrElse(Option.scala:120)
      at org.apache.spark.rdd.RDD.dependencies(RDD.scala:224)
      at org.apache.spark.scheduler.DAGScheduler.visit$1(DAGScheduler.scala:386)
      at org.apache.spark.scheduler.DAGScheduler.getParentStages(DAGScheduler.scala:398)
      at org.apache.spark.scheduler.DAGScheduler.getParentStagesAndId(DAGScheduler.scala:299)
      at org.apache.spark.scheduler.DAGScheduler.newResultStage(DAGScheduler.scala:334)
      at org.apache.spark.scheduler.DAGScheduler.handleJobSubmitted(DAGScheduler.scala:836)
      at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.doOnReceive(DAGScheduler.scala:1607)
      at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onReceive(DAGScheduler.scala:1599)
      at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onReceive(DAGScheduler.scala:1588)

[andrewor14]

@tdas @zsxwing

[zsxwing]

I don't think it's worth to fix it here. This failure is because TrackStateRDDSuite creates its SparkContext in the constructor. For maven build, it will create all suites at the beginning. So TrackStateRDDSuite's SparkEnv will be set to null when some test before TrackStateRDDSuite stops another SparkContext. See #9668

[tedyu]

I think both fixes should be kept.

[zsxwing]

If SparkEnv is null, I think it will crash somewhere as well. So why we need to fix it here?

[andrewor14]

I agree. It seems that this patch is a bandaid without addressing the real issue. Once we have #9668 we don't need this patch.

[SparkQA]

Test build #45763 has finished for PR 9672 at commit 0025e2b.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.