Use AppInsigths to capture Cosmos Benchmark metrics

[M-Lipin]

Subtasks:

• Illustrate in the diagram how metrics are collected (detailed building blocks)
• Capture Realtime P90, P99, P999, P9999
• Explore other ways to capture metrics (AppMetrics and OpenTelemetry)
• Investigate integration with AppInsights
• Collect metrics with OpenTelemetry and send to AppInsights
• Collect metrics from multiple VMs and aggregate these in AppInsights
• Add dashboard using percentiles query
• Availability visualization and 10 seconds granularity for latencies

sequenceDiagram
    participant A as Azure VM
    participant B as Benchmark Tool
    participant C as CosmosDB
    participant D as OpenTelemetry
    participant E as Application Insights

    A->>B: Launch Benchmark Tool
        
        B->>C: Request
        activate C
        
        C->>B: Response 
        deactivate C
        B->>D: Put metrics on each operation completion
        activate D
        
        par Every 5 sec (flush)
            D->>E: Send successes and failures, latencies and RPS
            Note right of C: collect and aggregate raw data
        end
        deactivate D
        B->>B: Benchmark completed

Loading

[M-Lipin]

• Right way to capture.
• Need to fix the issue with point percentiles to Application Insights.
• Distinguish latencies by read latencies, write latencies, queries and so on.
• Fix the issue with window latencies (10 seconds or configured).

[M-Lipin]

• Need to collect metrics using OpenTelemetry
• Need to aggreagate metrics from multiple VMs
• Need to illustrate the process in details how metrics are collected and sent to AppInsights

[M-Lipin]

#3932 - Dashboard template

[philipthomas-MSFT]

Please add project type for associated PRs so that it appears on board.