[Umbrella] Add more detail stats to kyuubi engine page

[charlesy6]

1. Description

Kyuubi Query Engine shows basic stats currently, it will be more friendly after we add more details, like Session Statistics and SQL Statistics.

2. Describe the solution

• We need a KV store which supports clean up expired events.
• Watch operation and statement events, then update the KV store.
• Render the Session Statistics and SQL Statistics section at kyuubi engine page.

3. TaskList

• Let spark history logger handle events asynchronously #1043
• Add basic EngineStatusStore for events #1022
• Add Session stats on Kyuubi Query Engine Page #1159
• [FEATURE] Add Statement stats on Kyuubi Query Engine Page #1188
• [FEATURE] Add Kyuubi Query Sesstion Page #1284
• Support session && statements stats on SparkHistoryServer Page

[yaooqinn]

this is awesome!

[charlesy6]

Implementation

First of all, we need a store which can help us to save / index /query events, and supports remove expired events when reach a threshold. org.apache.spark.status.ElementTrackingStore meets our needs. Then the listener monitors the occurrence of various events and saves them in the store.

In addition, if spark.eventLog.enabled is enable, sparkhistoryserver will consume events and replay them.

Work Flow

[yaooqinn]

Most engine events are now in

1. Spark's event log - which are line-based JSON events in the same file of spark's events
2. Kyuubi's event JSON log - a JSON file/table-based system support querying by Kyuubi/spark directly
3. JDBC (not yet)
4. CUSTOM (not yet)

For the reason why we should use a private implementation - ElementTrackingStore, a detailed explanation is needed

[charlesy6]

Kyuubi's event JSON log - a JSON file/table-based system support querying by Kyuubi/spark directly

Hi @yaooqinn, when user open Kyuubi Engine Page, Session Statistics and SQL Statistics section are will generated dynamically. The ElementTrackingStore wrappers kvstore(InMemoryStore, LevelDB), it provides data very quickly.

[yaooqinn]

The ElementTrackingStore wrappers kvstore(InMemoryStore, LevelDB), it provides data very quickly.

So just for speed? I will bring a huge maintenance burden to Kyuubi? How about we provide another way faster?

[charlesy6]

So just for speed?

No, it also can support two more important feature, index and firing triggers when elements reach a certain threshold. They will very useful when clearing data.

I will bring a huge maintenance burden to Kyuubi? How about we provide another way faster?

Actually, it's very simple. ElementTrackingStore is designed for a live UI or an event log.

https://github.com/apache/spark/blob/master/core/src/main/scala/org/apache/spark/status/ElementTrackingStore.scala

/**
 * A KVStore wrapper that allows tracking the number of elements of specific types, and triggering
 * actions once they reach a threshold. This allows writers, for example, to control how much data
 * is stored by potentially deleting old data as new data is added.
 *
 * This store is used when populating data either from a live UI or an event log. On top of firing
 * triggers when elements reach a certain threshold, it provides two extra bits of functionality:
 *
 * - a generic worker thread that can be used to run expensive tasks asynchronously; the tasks can
 *   be configured to run on the calling thread when more determinism is desired (e.g. unit tests).
 * - a generic flush mechanism so that listeners can be notified about when they should flush
 *   internal state to the store (e.g. after the SHS finishes parsing an event log).
 *
 * The configured triggers are run on a separate thread by default; they can be forced to run on
 * the calling thread by setting the `ASYNC_TRACKING_ENABLED` configuration to `false`.
 */

[charlesy6]

I think ElementTrackingStore work as KyuubiStatementMonitor in kyuubi-spark-monitor. We can reuse ElementTrackingStore and remove KyuubiStatementMonitor later if possible.

[yaooqinn]

Actually, it's very simple.

This is not what I mean... I don't care about whether it is simple or not itself...

I mean it's a private package in Spark, they don't guarantee any compatibility in future versions. This is the maintenance burden.

Technically, we don't just say NO to these API calls, but in the meantime, we need to think twice -

1. Is there any public APIs there I can use?
2. Is this the only way to implement this feature?
3. How can I tell maintainers it is the right and only way to do it?
4. What is the cost if we introduce such APIs?
5. How can I reduce the private API calls?
6. And so on

[charlesy6]

Right, it's a private package in spark. We will just use its public API.

And, if want to support kyuubi page on the sparkhistoryserver, we need to implements a class which extends AppHistoryServerPlugin. it only transfer ElementTrackingStore in AppHistoryServerPlugin.

I think it's better to reuse ElementTrackingStore, it's also ok to rewrite a class which work as ElementTrackingStore.

[charlesy6]

It seems we don’t need to worry too much about the maintenance burden, ElementTrackingStore and KVStore were last updated in 2019 year. If we only supports kyuubi engine page on live server, we can implements the store like KyuubiStatementMonitor does, or enhance it in the current.

[yaooqinn]

If it is just a KV store, can we use a more generic one for Kyuubi events to hold in memory?

[charlesy6]

If it is just a KV store, can we use a more generic one for Kyuubi events to hold in memory?

Actually, it's seems not easy to implement the store, basically it needs to store different types of event and index / query them, and also update the state of execution statement, and so on. These which KyuubiStatementMonitor are doing

As mentioned above, if we suppots the page on historyserver later, we have to face the ElementTrackingStore.

Refer to:

https://github.com/apache/spark/blob/master/core/src/main/scala/org/apache/spark/status/AppHistoryServerPlugin.scala

https://github.com/apache/incubator-kyuubi/blob/master/externals/kyuubi-spark-monitor/src/main/scala/org/apache/kyuubi/engine/spark/monitor/KyuubiStatementMonitor.scala

[charlesy6]

I am also tring to look for whether there is a library which can meet these requirements, or write a store to decouple the spark project as much as possible.

[yaooqinn]

For the main code path, we don't need ElementTrackingStore or AppHistoryServerPlugin

For the history server, it is standalone, we can provide a plugin separately or somewhere else