Admin Dashboard - Global and Chart Level Filter Enhancement

[Shakthieshwari]

Hello Team ,

As part of 5.1 release, We have these https://project-sunbird.atlassian.net/browse/SB-30243 and https://project-sunbird.atlassian.net/browse/SB-29912 Story.

In the Admin Dashboard(HawkEye), Both Graphs and Tables have a Global and Chart Level Filters. Currently If we need to create any chart to include a filter, then we will have to add those columns in the groupBy of druid query, so that the json file which is an input to the Portal will have all those columns required for the filters.

For Ex :- if a report has four filters - Program, Project, District and Organisation, then, the big no. ‘Unique users who started project’ also shows unique users till the lowest drilled down level, which always give the wrong data.

This dependency should not be there between the data metrics and filters. Because If a user a select a filters, the count should be only at that level, For Ex :- Filters - Program,Project,District

1. If a user select only Program as the filter -> unique users count should be only at the Program
2. If a user select only Program and District as the filter -> unique users count should be both at the Program and District
3. If a user select all the Filters -> unique users count should at all the levels
4. If no filters is select ->uniquer users count should be overall

However, it is important to note that any filter added for report or chart should give filtered data in the big no. and charts according to the report viewer/program manager selects the filter.

So the requirement we have now is, Both the Global and Chart Level Filter should dynamically query in real-time as the filter is selected.

We also had a internal discussion within our team, But since hawkeye portal take the data from the json file and both data metric and filters are dependent on the same json file, it is not meeting our requirement and we could not find any proper solutions

And Also Filters Data should come up as and when the program & resource is uploaded dynamically.

Attaching the current screenshots of global and chart filter

Please help us out ... @rajeevsathish @rajeshkumaravel @SanthoshVasabhaktula @sowmya-dixit @anandp504 On how do we go about this enhancement ?

Cc- @aishwaryashikshalokam @Ashwiniev95 @Prateek-slokam @aks30 @kiranharidas187 @5Amogh @rakeshSgr @vijiurs

Thanks

[Shakthieshwari]

Team, Any update on this Please ?

Thanks

[SanthoshVasabhaktula]

@Shakthieshwari

Unique user counts are always tricky and generally don't give any valuable insights compared to other metrics like total users. For ex: If a user can enrol only once into a program, total count and unique count will always be some. In addition, we need to ask questions such as what insight or value does a "intersection" number across multiple projects provide? as it has some serious implications on infrastructure cost. I have gone through the stories and couldn't see the purpose of the metric. In the above case, to be able to compute unique users one has to create multiple combination of files to display the relevant metric.

From a tech perspective, Sunbird has capabilities to provide pre-computed files and run real-time queries. The real-time queries are not provided by default or integrated into the Admin dashboard for the following reasons:

1. The cost of executing a real-time querying of analytics store is very high, and it explodes if this capability is provided to a national scale system
2. As the data ingestion is done real-time, the data keeps on changing on every refresh. A metric needs to be consistent for a given day.
3. The infrastructure due to high ingestion loads may not necessarily be available fully to be queryable at every time of the day. The current design ensures that the infra is stable during the job runs.

An enhancement can definitely be made to the reporting framework to run real-time queries (I guess this is part of the backlog), but totally configurable at the instance level. Small instances with low volumes can enable this feature so as to not install the entire job management and execution feature, whereas large instances might go the route of pre-computed files. Please come up with an initial design of the enhancement and the community can help in reviewing the same.

However, please note that the enhancement might be completely switched off in large scale instances for the above mentioned reasons and one has to check with the instance engineering team on the enablement of this feature.

@rhwarrier