[SPARK-9924] [WEB UI] Don't schedule checkForLogs while some of them are already running.

[rohitagarwal003]

No description provided.

[andrewor14]

ok to test @vanzin

[squito]

you can avoid all mutability with

val tasks: Seq[Future[_]] = logInfos.grouped(20).map{ batch =>
  replayExecutor.submit(new Runnable {
    override def run(): Unit = mergeApplicationListing(batch)
  })
}

(I know the change to grouped is unrelated, but everytime I look at this code it confuses me for a second why we have overlapping sliding windows -- might as well as clean it up while you are messing around here)

[rohitagarwal003]

Thanks! I am a Scala noob - I did it the Java way. :-)
Your suggestion looks much cleaner - I have updated the PR.

[squito]

minor style comment, otherwise makes sense to me

[SparkQA]

Test build #40716 has finished for PR 8153 at commit bdb5955.

• This patch fails Spark unit tests.
• This patch merges cleanly.
• This patch adds the following public classes (experimental):
  • case class QRDecomposition[QType, RType](Q: QType, R: RType)

[vanzin]

Since we're talking about cleaning this up, you could do this also:

logInfos.grouped(20)
  .map { batch =>
    replayExecutor.submit(new Runnable {
      override def run(): Unit = mergeApplicationListing(batch)
    })
  }
  .foreach { task =>
    // Wait for all tasks to finish. This makes sure that checkForLogs is
    // not scheduled again while some tasks are already running in the
    // replayExecutor.
    try {
      task.get()
    } catch {
      case e: InterruptedException =>
        throw e
      case e: Exception =>
        logWarning("Error replaying logs.", e)
    }
  }

Note I added some missing exception handling, which would cause you to revert to the old behavior of piling up executions if an error happened.

[rohitagarwal003]

Thanks! I have updated the PR to add exception handling. I have modified the log message because errors while replaying event logs are already caught in the mergeApplicationListing method.

[vanzin]

In a way I think the underlying issue is more a problem with the aggressive default polling interval (10 seconds?). But this is a way to make it better.

I think in the (not so distant) future we should investigate using the recently added inotify-like API in HDFS, to see whether it helps us avoid polling altogether.

[SparkQA]

Test build #40782 has finished for PR 8153 at commit 249f4ef.

• This patch fails Spark unit tests.
• This patch merges cleanly.
• This patch adds no public classes.

[squito]

Jenkins, retest this please

[SparkQA]

Test build #40805 timed out for PR 8153 at commit 249f4ef after a configured wait of 175m.

[SparkQA]

Test build #40857 has finished for PR 8153 at commit 3e22b6c.

• This patch fails PySpark unit tests.
• This patch merges cleanly.
• This patch adds no public classes.

[squito]

Jenkins, retest this please

[vanzin]

LGTM.

[SparkQA]

Test build #40900 timed out for PR 8153 at commit cd1ef90 after a configured wait of 175m.

[vanzin]

retest this please

[SparkQA]

Test build #40933 has finished for PR 8153 at commit cd1ef90.

• This patch fails Spark unit tests.
• This patch merges cleanly.
• This patch adds no public classes.

[rohitagarwal003]

Can we retest this please?
The failures are unrelated.

[SparkQA]

Test build #1625 timed out for PR 8153 at commit cd1ef90 after a configured wait of 175m.

[vanzin]

retest this please

[SparkQA]

Test build #40975 timed out for PR 8153 at commit cd1ef90 after a configured wait of 175m.

[vanzin]

As usual, flaky tests in other unrelated modules. I'll just give up on jenkins and merge this Monday morning.

[vanzin]

Merged to master, thanks!