Measure and Visualize Total Measurement Size Evaluated by Spark-Evaluate Per Round

[pyropy]

Description:

We need to determine and visualize the total size of measurements evaluated by spark-evaluate for each round. This involves collecting data on the size of measurements processed during the evaluation of each round and aggregating this data to understand the total size per round. You can find an example of how the data is currently submitted to InfluxDB here

Task Breakdown:

1. Data Collection:

   • Capture the size of measurements evaluated by spark-evaluate for each round.
   • Ensure that the data collection process is efficient and does not impact the performance of the existing system.

2. Data Storage:

   • Set up an InfluxDB bucket to store the collected data.
   • Define the appropriate schema for storing measurement size data, including fields for total size and round number.
   • Implement a data pipeline to send the collected data to InfluxDB.

3. Data Aggregation:

   • Develop a mechanism to aggregate the sizes of all measurements associated with a single round in spark-evaluate.
   • Store the aggregated data in InfluxDB for further analysis.

4. Data Visualization:

   • Create a chart to display the total size of measurements for each round over time.
   • Create a chart to display the distribution of measurement sizes within each round.
   • Use a visualization tool like Grafana to create interactive and informative dashboards.

Acceptance Criteria:

• The system should accurately capture and store the size of measurements evaluated by spark-evaluate for each round.
• The data should be stored in InfluxDB with the appropriate schema and retention policies.
• The system should correctly aggregate the sizes of all measurements for each round in spark-evaluate.
• The visualization dashboard should provide real-time insights into the total measurement size per round and the distribution of measurement sizes.