From f9cbb931fb73b5cbba70a0311e0390d85d2c4e1f Mon Sep 17 00:00:00 2001 From: Jan Provaznik Date: Wed, 20 Nov 2024 21:34:56 +0100 Subject: [PATCH 1/5] telemetry onepager --- documentation/specs/proposed/telemetry.md | 63 +++++++++++++++++++++++ 1 file changed, 63 insertions(+) create mode 100644 documentation/specs/proposed/telemetry.md diff --git a/documentation/specs/proposed/telemetry.md b/documentation/specs/proposed/telemetry.md new file mode 100644 index 00000000000..45e49e87cce --- /dev/null +++ b/documentation/specs/proposed/telemetry.md @@ -0,0 +1,63 @@ +# Telemetry + +We want to implement telemetry collection for VS/MSBuild.exe scenarios where we are currently not collecting any data. VS OpenTelemetry initiative provides a good opportunity to use their infrastructure and library. +There is some data we collect via SDK which we want to make accessible. + +## Goals and Motivation + +We have no data about usage of MSBuild customers in VS. +This limits us in prioritization of features and scenarios to optimize performance for. +Implementing telemetry via VS OpenTelemetry initiative would give us comprehensive insight into how MSBuild is used in all scenarios. + +Goal: To have relevant data in that is actionable for decisions about development. Measuring real world performance impact of features (e.g. BuildCheck). Easily extensible telemetry infrastructure if we want to measure a new datapoint. + +## Impact +- Better planning of deployment of forces in MSBuild by product/team management. +- Customers can subscribe to telemetry locally to have data in standardized OpenTelemetry format + +## Stakeholders +- Jan (Krivanek|Provaznik) design and implementation of telemetry via VS OTel. Mariana - using data we already have from SDK. +- MSBuild Team+Management – want insights from builds in VS +- VS OpenTelemetry team – provide support for VS OpenTelemetry collector library, want successful adoption +- SourceBuild – consulting and approving usage of OpenTelemetry +- Chet – representing customers who want to monitor their builds locally + +### Successful handover +- Shipped to Visual Studio +- Data queryable in Kusto +- Dashboards +- Customers are able to monitor with OpenTelemetry collector of choice (can be cut) + +## Risks +- Performance regression risks - it's another thing MSBuild would do and if the perf hit would be too bad it would need mitigation effort. +- It introduces a closed source dependency for VS and MSBuild.exe distribution methods which requires workarounds to remain compatible with SourceBuild policy (conditional compilation/build). +- Using a new VS API - might have gaps +- Instrumenting data that would ultimately prove uninteresting. +- Potential additional costs and delays due to compliance with SourceBuild/VS data. + +## Cost +5 months of .5 developer's effort ~ 50 dev days (dd) + +20-30dd JanPro OTel design + implementation, 10-15dd JanK design + implementation, 5-10dd Mariana/someone getting available data in order/"data science"/dashboards + external documentation + +Uncertainties: +It’s an exploratory project for VS OpenTelemetry, we'll be their first OSS component, so there might come up issues. SourceBuild compliance could introduce delays. + +## Plan +- Prototyping the libraries/mechanism for collecting telemetry data (month 1) 10dd + +- Defining usful data points (month 1) 5dd + +- Design and approval of hooking VSTelemetry collectors and OTel collectors (month 2) 10dd + +- Formalizing, agreeing to sourcebuild and other external requirements (month 2) 2dd + +- Instrumenting MSBuild with defined datapoints (month 3) 10dd + +- Creating dashboards/insights (month 4) 5dd + +- Documenting for customers how to hook their own telemetry collection (month 4) 3dd + +- Buffer for discovered issues (VSData Platform, SourceBuild, OpenTelemetry) and more investments (month 5) 5dd + + \ No newline at end of file From 493265779f0ce99becfd0a1d32ee179289246734 Mon Sep 17 00:00:00 2001 From: Jan Provaznik Date: Thu, 21 Nov 2024 11:58:55 +0100 Subject: [PATCH 2/5] rename file --- .../specs/proposed/{telemetry.md => telemetry_onepager.md} | 0 1 file changed, 0 insertions(+), 0 deletions(-) rename documentation/specs/proposed/{telemetry.md => telemetry_onepager.md} (100%) diff --git a/documentation/specs/proposed/telemetry.md b/documentation/specs/proposed/telemetry_onepager.md similarity index 100% rename from documentation/specs/proposed/telemetry.md rename to documentation/specs/proposed/telemetry_onepager.md From 419a2ce9da31f0fe86b30ee00282578d2fa5fc1e Mon Sep 17 00:00:00 2001 From: Jan Provaznik Date: Wed, 27 Nov 2024 15:38:42 +0100 Subject: [PATCH 3/5] update with comments and scope section --- ...etry_onepager.md => telemetry-onepager.md} | 35 ++++++++++++------- 1 file changed, 23 insertions(+), 12 deletions(-) rename documentation/specs/proposed/{telemetry_onepager.md => telemetry-onepager.md} (62%) diff --git a/documentation/specs/proposed/telemetry_onepager.md b/documentation/specs/proposed/telemetry-onepager.md similarity index 62% rename from documentation/specs/proposed/telemetry_onepager.md rename to documentation/specs/proposed/telemetry-onepager.md index 45e49e87cce..4a2f0b9432b 100644 --- a/documentation/specs/proposed/telemetry_onepager.md +++ b/documentation/specs/proposed/telemetry-onepager.md @@ -1,13 +1,13 @@ # Telemetry -We want to implement telemetry collection for VS/MSBuild.exe scenarios where we are currently not collecting any data. VS OpenTelemetry initiative provides a good opportunity to use their infrastructure and library. +We want to implement telemetry collection for VS/MSBuild.exe scenarios where we are currently not collecting data. VS OpenTelemetry initiative provides a good opportunity to use their infrastructure and library. There is some data we collect via SDK which we want to make accessible. ## Goals and Motivation -We have no data about usage of MSBuild customers in VS. +We have limited data about usage of MSBuild by our customers in VS and no data about usage of standalone msbuild.exe. This limits us in prioritization of features and scenarios to optimize performance for. -Implementing telemetry via VS OpenTelemetry initiative would give us comprehensive insight into how MSBuild is used in all scenarios. +Over time we want to have comprehensive insight into how MSBuild is used in all scenarios. Collecting such a data without any constraints nor limitations would however be prohibitively expensive (from the data storage PoV and possibly as well from the client side performance impact PoV). Ability to sample / configure the collection is an important factor in deciding the instrumentation and collection tech stack. Implementing telemetry via VS OpenTelemetry initiative would give us this ability in the future. Goal: To have relevant data in that is actionable for decisions about development. Measuring real world performance impact of features (e.g. BuildCheck). Easily extensible telemetry infrastructure if we want to measure a new datapoint. @@ -16,13 +16,13 @@ Goal: To have relevant data in that is actionable for decisions about developmen - Customers can subscribe to telemetry locally to have data in standardized OpenTelemetry format ## Stakeholders -- Jan (Krivanek|Provaznik) design and implementation of telemetry via VS OTel. Mariana - using data we already have from SDK. +- @Jan(Krivanek|Provaznik) design and implementation of telemetry via VS OTel. @ - using data we already have from SDK. - MSBuild Team+Management – want insights from builds in VS - VS OpenTelemetry team – provide support for VS OpenTelemetry collector library, want successful adoption - SourceBuild – consulting and approving usage of OpenTelemetry -- Chet – representing customers who want to monitor their builds locally +- MSBuild PM @baronfel – representing customers who want to monitor their builds locally -### Successful handover +### Successful handover of V1 - Shipped to Visual Studio - Data queryable in Kusto - Dashboards @@ -32,7 +32,7 @@ Goal: To have relevant data in that is actionable for decisions about developmen - Performance regression risks - it's another thing MSBuild would do and if the perf hit would be too bad it would need mitigation effort. - It introduces a closed source dependency for VS and MSBuild.exe distribution methods which requires workarounds to remain compatible with SourceBuild policy (conditional compilation/build). - Using a new VS API - might have gaps -- Instrumenting data that would ultimately prove uninteresting. +- storage costs - Potential additional costs and delays due to compliance with SourceBuild/VS data. ## Cost @@ -44,20 +44,31 @@ Uncertainties: It’s an exploratory project for VS OpenTelemetry, we'll be their first OSS component, so there might come up issues. SourceBuild compliance could introduce delays. ## Plan +### V1 scope +- Collected data point definition +- Instrumented data points +- Telemetry sent to VS Telemetry in acceptable quantity +- Dashboards for collected data +- hooking of customer's telemetry collection + +#### Out of scope +- Unifying telemetry for SDK MSBuild and MSBuild.exe/VS MSBuild. +- using MSBuild server +- distributed tracing + +### Detailed cost - Prototyping the libraries/mechanism for collecting telemetry data (month 1) 10dd - Defining usful data points (month 1) 5dd - Design and approval of hooking VSTelemetry collectors and OTel collectors (month 2) 10dd -- Formalizing, agreeing to sourcebuild and other external requirements (month 2) 2dd +- Formalizing, agreeing to sourcebuild and other external requirements (month 2) 5dd -- Instrumenting MSBuild with defined datapoints (month 3) 10dd +- Instrumenting MSBuild with defined datapoints (month 3) 7dd - Creating dashboards/insights (month 4) 5dd - Documenting for customers how to hook their own telemetry collection (month 4) 3dd -- Buffer for discovered issues (VSData Platform, SourceBuild, OpenTelemetry) and more investments (month 5) 5dd - - \ No newline at end of file +- Buffer for discovered issues (VSData Platform, SourceBuild, OpenTelemetry) and more investments (month 5) 5dd \ No newline at end of file From 84850209b2b5a930470853e2752c0e979174fa8f Mon Sep 17 00:00:00 2001 From: Jan Provaznik Date: Wed, 27 Nov 2024 15:40:03 +0100 Subject: [PATCH 4/5] update --- documentation/specs/proposed/telemetry-onepager.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/documentation/specs/proposed/telemetry-onepager.md b/documentation/specs/proposed/telemetry-onepager.md index 4a2f0b9432b..e7c940cc29a 100644 --- a/documentation/specs/proposed/telemetry-onepager.md +++ b/documentation/specs/proposed/telemetry-onepager.md @@ -22,7 +22,7 @@ Goal: To have relevant data in that is actionable for decisions about developmen - SourceBuild – consulting and approving usage of OpenTelemetry - MSBuild PM @baronfel – representing customers who want to monitor their builds locally -### Successful handover of V1 +### V1 Successful handover - Shipped to Visual Studio - Data queryable in Kusto - Dashboards @@ -35,7 +35,7 @@ Goal: To have relevant data in that is actionable for decisions about developmen - storage costs - Potential additional costs and delays due to compliance with SourceBuild/VS data. -## Cost +## V1 Cost 5 months of .5 developer's effort ~ 50 dev days (dd) 20-30dd JanPro OTel design + implementation, 10-15dd JanK design + implementation, 5-10dd Mariana/someone getting available data in order/"data science"/dashboards + external documentation From 0e2f0def1d01bc7130c52dc6f31373a2422df4bd Mon Sep 17 00:00:00 2001 From: Jan Krivanek Date: Fri, 29 Nov 2024 12:36:32 +0100 Subject: [PATCH 5/5] Update telemetry-onepager.md --- .../specs/proposed/telemetry-onepager.md | 15 +++++++++------ 1 file changed, 9 insertions(+), 6 deletions(-) diff --git a/documentation/specs/proposed/telemetry-onepager.md b/documentation/specs/proposed/telemetry-onepager.md index e7c940cc29a..5bc8f22f9ce 100644 --- a/documentation/specs/proposed/telemetry-onepager.md +++ b/documentation/specs/proposed/telemetry-onepager.md @@ -17,6 +17,7 @@ Goal: To have relevant data in that is actionable for decisions about developmen ## Stakeholders - @Jan(Krivanek|Provaznik) design and implementation of telemetry via VS OTel. @ - using data we already have from SDK. +- @maridematte - documenting + dashboarding currently existing datapoints. - MSBuild Team+Management – want insights from builds in VS - VS OpenTelemetry team – provide support for VS OpenTelemetry collector library, want successful adoption - SourceBuild – consulting and approving usage of OpenTelemetry @@ -25,7 +26,7 @@ Goal: To have relevant data in that is actionable for decisions about developmen ### V1 Successful handover - Shipped to Visual Studio - Data queryable in Kusto -- Dashboards +- Dashboards (even for pre-existing data - not introduced by this work) - Customers are able to monitor with OpenTelemetry collector of choice (can be cut) ## Risks @@ -46,15 +47,17 @@ It’s an exploratory project for VS OpenTelemetry, we'll be their first OSS com ## Plan ### V1 scope - Collected data point definition -- Instrumented data points +- Instrumented data points (as an example how the instrumentation and collection works) - Telemetry sent to VS Telemetry in acceptable quantity - Dashboards for collected data -- hooking of customer's telemetry collection +- Hooking of customer's telemetry collection +- Documenting and leveraging pre-existing telemetry #### Out of scope - Unifying telemetry for SDK MSBuild and MSBuild.exe/VS MSBuild. -- using MSBuild server -- distributed tracing +- Thorough instrumentation of MSBuild +- Using MSBuild server +- Distributed tracing ### Detailed cost - Prototyping the libraries/mechanism for collecting telemetry data (month 1) 10dd @@ -71,4 +74,4 @@ It’s an exploratory project for VS OpenTelemetry, we'll be their first OSS com - Documenting for customers how to hook their own telemetry collection (month 4) 3dd -- Buffer for discovered issues (VSData Platform, SourceBuild, OpenTelemetry) and more investments (month 5) 5dd \ No newline at end of file +- Buffer for discovered issues (VSData Platform, SourceBuild, OpenTelemetry) and more investments (month 5) 5dd