Skip to content
This repository has been archived by the owner on Feb 20, 2023. It is now read-only.

Add 'useEmail' metrics event #9835

Merged
merged 1 commit into from
Apr 23, 2020
Merged

Conversation

vladikoff
Copy link
Contributor

@vladikoff vladikoff commented Apr 9, 2020

Fixes #9834

Fixes a metrics regression introduced in https://github.com/mozilla-mobile/fenix/pull/4931/files#diff-c0328a8e65981c7ed9bdbb1d366c2849R225 with the removal of the event.

Pull Request checklist

  • Tests: This PR includes thorough tests or an explanation of why it does not
    explanation: telemetry changes, didn't see any tests for this particular code path.
  • Screenshots: This PR includes screenshots or GIFs of the changes made or an explanation of why it does not
    explanation: no visual changes
  • Accessibility: The code in this PR follows accessibility best practices or does not include any user facing features. In addition, it includes a screenshot of a successful accessibility scan to ensure no new defects are added to the product.

After merge

  • Milestone: Make sure issues finished by this pull request are added to the milestone of the version currently in development.

To download an APK when reviewing a PR:

  1. click on Show All Checks,
  2. click Details next to "Taskcluster (pull_request)" after it appears and then finishes with a green checkmark,
  3. click on the "Fenix - assemble" task, then click "Run Artifacts".
  4. the APK links should be on the left side of the screen, named for each CPU architecture

app/metrics.yaml Outdated
bugs:
- https://github.com/mozilla-mobile/fenix/issues/9834
data_reviews:
- https://github.com/mozilla-mobile/fenix/issues/9835
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This needs to be a link to a data review response (once you get it).

app/metrics.yaml Outdated Show resolved Hide resolved
@codecov-io
Copy link

codecov-io commented Apr 9, 2020

Codecov Report

Merging #9835 into master will increase coverage by 0.07%.
The diff coverage is 0.00%.

Impacted file tree graph

@@             Coverage Diff              @@
##             master    #9835      +/-   ##
============================================
+ Coverage     19.07%   19.14%   +0.07%     
+ Complexity      533      521      -12     
============================================
  Files           343      337       -6     
  Lines         13946    13716     -230     
  Branches       1869     1840      -29     
============================================
- Hits           2660     2626      -34     
+ Misses        11046    10850     -196     
  Partials        240      240              
Impacted Files Coverage Δ Complexity Δ
...la/fenix/components/metrics/GleanMetricsService.kt 8.93% <0.00%> (+0.12%) 3.00 <0.00> (ø)
...va/org/mozilla/fenix/components/metrics/Metrics.kt 19.26% <0.00%> (+0.25%) 0.00 <0.00> (ø)
...a/fenix/settings/account/AccountProblemFragment.kt 0.00% <0.00%> (ø) 0.00 <0.00> (ø)
...zilla/fenix/settings/account/TurnOnSyncFragment.kt 0.00% <0.00%> (ø) 0.00 <0.00> (ø)
...ain/java/org/mozilla/fenix/perf/StartupTimeline.kt 0.00% <0.00%> (-62.50%) 0.00% <0.00%> (-2.00%)
...rc/main/java/org/mozilla/fenix/FenixApplication.kt 11.18% <0.00%> (-1.06%) 3.00% <0.00%> (-1.00%)
...in/java/org/mozilla/fenix/settings/SupportUtils.kt 62.79% <0.00%> (-0.85%) 9.00% <0.00%> (ø%)
...nix/components/toolbar/BrowserToolbarController.kt 65.11% <0.00%> (-0.80%) 0.00% <0.00%> (ø%)
.../java/org/mozilla/fenix/browser/BrowserAnimator.kt 7.40% <0.00%> (-0.44%) 0.00% <0.00%> (ø%)
...src/main/java/org/mozilla/fenix/components/Core.kt 23.15% <0.00%> (-0.25%) 1.00% <0.00%> (ø%)
... and 61 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 95f26f1...acb1b54. Read the comment docs.

@vladikoff
Copy link
Contributor Author

cc @boek @liuche as mobile data stewards.

Request for data collection review form

All questions are mandatory. You must receive review from a data steward peer on your responses to these questions before shipping new data collection.

  1. What questions will you answer with this data?

How users are interacting with FxA in Fenix - creating an account, login. Specifically when the users choose to use the email login over other means.

  1. Why does Mozilla need to answer these questions? Are there benefits for users? Do we need this information to address product or business requirements? Some example responses:

We need to provide our users with a seamless, robust and easy-to-use FxA experience, and these metrics help us understand if our implementation is working as expected, and which kinds of authentication flows users are following.

  1. What alternative methods did you consider to answer these questions? Why were they not sufficient?

An alternative could be to rely FxA metrics, but these do not allow us to complete a coherent picture of what's going on in a local authentication funnel, and some of the key local events (account recovery, auto-login) are not visible outside of the local client.

  1. Can current instrumentation answer these questions?

No. We tried to measure this and found this regression.

  1. List all proposed measurements and indicate the category of data collection for each measurement, using the Firefox data collection categories on the Mozilla wiki.

All data is Category 2.

  1. How long will this data be collected? Choose one of the following:

This is scoped to a time-limited experiment/project until date "2020-09-01", at which point it'll be reviewed for renewal.

  1. What populations will you measure?

All release, beta, and nightly users with telemetry enabled.

  1. If this data collection is default on, what is the opt-out mechanism for users?

This is part of events telemetry, so can be disabled via Data Collection settings.

  1. Please provide a general description of how you will analyze this data.

Dashboards in redash and amplitude from glean data.

  1. Where do you intend to share the results of your analysis?

Internally with the mobile teams, and on redash/aplitude.

  1. Is there a third-party tool (i.e. not Telemetry) that you are proposing to use for this data collection? If so:

N/A

@vladikoff vladikoff marked this pull request as ready for review April 13, 2020 15:40
@ekager ekager added the needs:data-review PR is awaiting a data review label Apr 14, 2020
@ekager ekager requested a review from boek April 14, 2020 21:29
app/metrics.yaml Outdated Show resolved Hide resolved
app/metrics.yaml Outdated
bugs:
- https://github.com/mozilla-mobile/fenix/issues/9834
data_reviews:
- https://github.com/mozilla-mobile/fenix/pull/9835#issuecomment-612951889
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This should be a link to the data steward's response, not the review form itself.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ideally, but sometimes we also just link to the PR with both of the forms 🤷‍♀️

Copy link
Contributor Author

@vladikoff vladikoff Apr 17, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Every time I've had a review from from glean folks, they've been pretty insistent on it :)

@vladikoff
Copy link
Contributor Author

Rebased and resolved the metrics doc comment 👍

Copy link
Contributor

@boek boek left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Data Review Form (to be filled by Data Stewards)

Instructions: Data Stewards will review a request for data collection and endorse responses to each question.

  1. Is there or will there be documentation that describes the schema for the ultimate data set in a public, complete, and accurate way?
    Yes, with metrics.yaml and metrics.md

  2. Is there a control mechanism that allows the user to turn the data collection on and off?
    Yes, in the Fenix settings

  3. If the request is for permanent data collection, is there someone who will monitor the data over time?
    Has expiry, Fenix team will monitor

  4. Using the category system of data types on the Mozilla wiki, what collection type of data do the requested measurements fall under?
    All data is category 2.

  5. Is the data collection request for default-on or default-off?
    Default on

  6. Does the instrumentation include the addition of any new identifiers (whether anonymous or otherwise; e.g., username, random IDs, etc. See the appendix for more details)?
    No

  7. Is the data collection covered by the existing Firefox privacy notice?
    yes

  8. Does there need to be a check-in in the future to determine whether to renew the data?
    Has expiry. Fenix team will monitor

  9. Does the data collection use a third-party collection tool?
    No

@vladikoff
Copy link
Contributor Author

Rebased and updated with the review link. Thank you Jeff!

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
needs:data-review PR is awaiting a data review
Projects
None yet
Development

Successfully merging this pull request may close these issues.

[Bug] It seems like we don't track the metrics of the "Use Email" button
5 participants