Add definitions for a feature's enabled state, and the opening and closing of a module #835
Conversation
…osing of a module.
| }, | ||
| { | ||
| "name": "toolkit_featureState", | ||
| "description": "Represents the current enabled state of a feature. Used to track user journey through a feature. Emitted after feature-specific operations of interest in the Toolkit.", |
There was a problem hiding this comment.
how does this differ from aws_featureConfig which was recently added? https://github.com/aws/aws-toolkit-common/blob/940ac45eeb76a5a6bc6b92c8a66306a0d1f363b7/telemetry/definitions/commonDefinitions.json#L2701C22-L2703
aws_featureConfig mentions AB testing but i don't see why it needs to be restricted to that. @ivikash
There was a problem hiding this comment.
if this is truly different, would be helpful for the description to mention the differences
There was a problem hiding this comment.
I did consider using aws_featureConfig. My understanding of this metric is:
- it is a statement of immutable fact around a feature ("Toolkit has been told that for A/B purposes, feature X is A or B")
- it is likely not frequently emitted
The intent behind toolkit_featureState is:
- emitted often, representing the user's current state as they click or interact their way through a workflow
- the state is mutable (it is possible for User and/or Toolkit to affect the enabled state)
- indicates why state is being emitted ("User has specified that feature is enabled" or "User is entering this workflow, here's the starting value")
Based on the requirements I discussed with @dhruvigajjar , aws_featureConfig did not match the needs.
There was a problem hiding this comment.
representing the user's current state as they click or interact their way through a workflow
This "journey through a feature" sounds like it could live on the existing feature-related metrics. Seems like we're inverting things with toolkit_featureState: now the "feature" will live as a field on toolkit_featureState, even though each "feature" already has dedicated metrics (which could themselves have a featureId and related fields).
Feels like a denormalization
There was a problem hiding this comment.
I think that's okay. If we had done this in the past, we may have used it more. I'm not inclined to make metrics with "gettingStarted" in the name, that is how we continue to proliferate metrics definitions. Not that that is a bad thing, but fewer is easier to manage.
There was a problem hiding this comment.
featureConfig has lot of fields which are only specific to AB testing which might be confusing during interpretation.
Since VS's getting started page has purpose of giving user feature specific (AWS Toolkit or AmazonQ)experience I thought dedicated metric for user's experience is enabled or not would be helpful, thus toolkit_featureState.
I see this as equivalent of VSC/JB's auth_addConnection (as part of getting started page/auth page) that gives information about user's end-to-end experience through auth flow,toolkit_featureState will give information about user's journey through enabling a feature.
There was a problem hiding this comment.
This could be addressed by "container metrics": https://sim.amazon.com/issues/IDE-10178
oh well
This change is a follow-up to #835, to add a "credential type" field to the `toolkit_featureState` metric. This allows us to differentiate whether credentials profiles coming from shared credentials are sso based or regular access key/secret key profiles. ## License By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
Problem
We're revising the VS Toolkit's Getting Started experience to more cleanly enable and disable features. We wish to understand the user's journey through this experience, and identify when/why a feature gets enabled or disabled.
Solution
A generalized set of metrics that can be applied to any feature:
toolkit_featureState- intended to be emitted at various points in a workflow, to report whether or not a certain feature is enabled. Also emits "why" this state is being emitted (via the "source" field)toolkit_openModule- intended to be emitted when a feature/view/component of interest is opened/entered/displayed. "What" is being opened is represented by the "module" field, and "Why" it is being opened is represented in "source"toolkit_closeModule- intended to be emitted when a feature/view/component of interest is closed/exited. "What" is being opened is represented by the "module" field.License
By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.