[SPARK-7889] [CORE] HistoryServer to refresh cache of incomplete applications#6935

Closed

steveloughran wants to merge 48 commits intoapache:masterfrom

steveloughran:stevel/patches/SPARK-7889-history-cache

Commits on Feb 4, 2016

SPARK-7889 cache with initial unit tests
steveloughran
committed
SPARK-7889 cache tests of refresh interval working.
steveloughran
committed
SPARK-7889 switch to explicit scan of SparkUI attempts to determine completion state; use mock spark UI
steveloughran
committed
SPARK-7889 use spark.history.cache.interval as config option; set default = 60; document
steveloughran
committed
SPARK-7889 resync with trunk & slightly improve code layout
steveloughran
committed
SPARK-7889 scalastyle
steveloughran
committed
SPARK-7889 scalastyle on ApplicationCacheSuite
steveloughran
committed
SPARK-7889 cleanup of comments and imports & the like
steveloughran
committed
SPARK-7889 intermin checkpoint on changes; about to move cache map from string to case class
steveloughran
committed
SPARK-7889 Cache designed to ask history server/provider for updates; metrics used to track load & time —and for testing
steveloughran
committed
SPARK-7889 playing losing battles with a test
steveloughran
committed
SPARK-7889 test working, verified other tests in package work, review comments & stylecheck
steveloughran
committed
SPARK-7889 - stylecheck on tests apparently skipped on mvn
steveloughran
committed
SPARK-7889 style and javadoc only
steveloughran
committed
SPARK-7889 pull out metrics into own class, make visible for testing
steveloughran
committed
history server web UI update test from @squito
steveloughran
committed
SPARK-7889 more on the cache suite tests: there is no evidence that eviction is taking place
steveloughran
committed
SPARK-7889: tests to make sure app eviction is taking place and that this triggers callbacs in the cache
steveloughran
committed
SPARK-7889 code style check
steveloughran
committed
SPARK-7889: adding return type of tests embedded inside a test method to keep scalastyle happy
steveloughran
committed
SPARK-7889 starting to add web filters
steveloughran
committed
SPARK-7889 ongoing test dev. Looks like the history provider isn't picking up any changes
steveloughran
committed
SPARK-7889: looks like the test is failing because a new history isn't being saved
steveloughran
committed
SPARK-7889 address scalastyle warnings
steveloughran
committed
SPARK-7889 : we aren't getting an updated log file on the second parallelize().count() call, so the FS history provider isn't seeing an update, etc, etc.
steveloughran
committed
SPARK-7889: filesize update time included in probe; update thread also scans through modified files to verify this takes.
steveloughran
committed
SPARK-7889: not all filesystems update modtime on a rename; the EventLoggingListener attempts to do so afterwards, swallowing exceptions raised
steveloughran
committed
SPARK-7889 cache update works in tests -unreliably. Traces imply that its a race condition between probe time and the scanner thread -if the initial load is after the file update but before the sca…
steveloughran
committed
SPARK-7889 still looking at a race condition in the test. This adds more time details, but I'm about to move the fshistory off time and into a generic "attempt version" counter which will be compar…
steveloughran
committed
SPARK-7889 moved off time differences to a simple generational counter and equality check
steveloughran
committed
SPARK-7889: this extends the test with
steveloughran
committed
SPARK-7889 remove obsolete method that had been commented out
steveloughran
committed
SPARK-7889 review work, plus update documentation for this change and other details (monitoring.md wasn't up to date with the incomplete apps feature)
steveloughran
committed
SPARK-7889 review changes
steveloughran
committed
SPARK-7889 review suggestions, especially removing all timestamp info from the loadedUI state and probes; makes for a leaner API
steveloughran
committed
[SPARK-7889] add a check of the REST API's app/jobs link too
steveloughran
committed
[SPARK-7889]: provider is required to return a probe to be invoked directly: no more cache and relay of state
steveloughran
committed
[SPARK-7889] review comments
steveloughran
committed
SPARK-7889 lookup with handling for NoSuchElement moved into ApplicationCache
steveloughran
committed
SPARK-7889 javadocs
steveloughran
committed
SPARK-7889 update probe is now a simple () => Boolean function
steveloughran
committed
[SPARK-7889] cleanup up merges after rebase. Test now failing
steveloughran
committed
[SPARK-7889] minor diagnostics enhancements; cause of test failures identified: SPARK-10873's switch to dynamic tables means scans of static HTML fails
steveloughran
committed
[SPARK-7889] working on tests
steveloughran
committed

Commits on Feb 5, 2016

Commits on Feb 8, 2016

[SPARK-7889] new scalastyle import check faiing
steveloughran
committed

Commits on Feb 9, 2016

SPARK-7889 rm unused functions in test
steveloughran
committed