Skip to content
This repository has been archived by the owner on Feb 20, 2023. It is now read-only.

For #15369: Add synced tabs metrics. #16727

Merged
merged 2 commits into from
Dec 15, 2020
Merged

Conversation

mcarare
Copy link
Contributor

@mcarare mcarare commented Nov 24, 2020

Pull Request checklist

  • Tests: This PR includes thorough tests or an explanation of why it does not
  • Screenshots: This PR includes screenshots or GIFs of the changes made or an explanation of why it does not
  • Accessibility: The code in this PR follows accessibility best practices or does not include any user facing features. In addition, it includes a screenshot of a successful accessibility scan to ensure no new defects are added to the product.

To download an APK when reviewing a PR:

  1. click on Show All Checks,
  2. click Details next to "Taskcluster (pull_request)" after it appears and then finishes with a green checkmark,
  3. click on the "Fenix - assemble" task, then click "Run Artifacts".
  4. the APK links should be on the left side of the screen, named for each CPU architecture

@mcarare mcarare requested review from a team as code owners November 24, 2020 10:27
@mcarare mcarare marked this pull request as draft November 24, 2020 10:27
@codecov-io
Copy link

codecov-io commented Nov 24, 2020

Codecov Report

Merging #16727 (d283576) into master (d233229) will decrease coverage by 0.67%.
The diff coverage is 30.00%.

Impacted file tree graph

@@             Coverage Diff              @@
##             master   #16727      +/-   ##
============================================
- Coverage     30.78%   30.11%   -0.68%     
+ Complexity     1240     1206      -34     
============================================
  Files           455      454       -1     
  Lines         18604    18600       -4     
  Branches       2602     2565      -37     
============================================
- Hits           5728     5601     -127     
- Misses        12417    12557     +140     
+ Partials        459      442      -17     
Impacted Files Coverage Δ Complexity Δ
...la/fenix/components/metrics/GleanMetricsService.kt 12.42% <0.00%> (-0.10%) 5.00 <0.00> (ø)
...main/java/org/mozilla/fenix/tabtray/TabTrayView.kt 0.00% <0.00%> (ø) 0.00 <0.00> (ø)
...in/java/org/mozilla/fenix/sync/SyncedTabsLayout.kt 13.15% <33.33%> (+2.04%) 0.00 <0.00> (ø)
.../org/mozilla/fenix/tabtray/SyncedTabsController.kt 86.36% <50.00%> (ø) 3.00 <1.00> (ø)
...java/org/mozilla/fenix/components/metrics/Event.kt 34.21% <100.00%> (+0.44%) 0.00 <0.00> (ø)
...main/java/org/mozilla/fenix/components/Services.kt 0.00% <0.00%> (-69.24%) 0.00% <0.00%> (-1.00%)
...lla/fenix/components/toolbar/DefaultToolbarMenu.kt 0.00% <0.00%> (-42.74%) 0.00% <0.00%> (-3.00%)
...zilla/fenix/browser/OpenInAppOnboardingObserver.kt 0.00% <0.00%> (-30.44%) 0.00% <0.00%> (-8.00%)
...in/java/org/mozilla/fenix/AppRequestInterceptor.kt 65.62% <0.00%> (-27.40%) 6.00% <0.00%> (-8.00%)
...g/mozilla/fenix/addons/AddonsManagementFragment.kt 0.00% <0.00%> (-6.85%) 0.00% <0.00%> (-7.00%)
... and 65 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update d233229...73efc02. Read the comment docs.

@mcarare
Copy link
Contributor Author

mcarare commented Nov 24, 2020

Request for data collection review form

All questions are mandatory. You must receive review from a data steward peer on your responses to these questions before shipping new data collection.

1) What questions will you answer with this data?
How often users open a synced tab.

2) Why does Mozilla need to answer these questions? Are there benefits for users? Do we need this information to address product or business requirements?
Measure whether this feature is used by users and how often.

3) What alternative methods did you consider to answer these questions? Why were they not sufficient?
This is the only way.

4) Can current instrumentation answer these questions?
No. We need a new specific event for a specific app feature.

5) List all proposed measurements and indicate the category of data collection for each measurement, using the Firefox data collection categories found on the Mozilla wiki.

Note that the data steward reviewing your request will characterize your data collection based on the highest (and most sensitive) category.

Measurement Description Data Collection Category Tracking Bug #
Open synced tab events Category 2 https://github.com//issues/15369

6) Please provide a link to the documentation for this data collection which describes the ultimate data set in a public, complete, and accurate way.
This collection is documented in the metrics.md document document included in the project - https://github.com/mozilla-mobile/fenix/blob/master/docs/metrics.md

7) How long will this data be collected? Choose one of the following:
Until 2021-05-10.

8) What populations will you measure?
All release channels and locales.

9) If this data collection is default on, what is the opt-out mechanism for users?
Users can opt of of data collection by disabling Usage and technical data from Settings -> Data collection.

10) Please provide a general description of how you will analyze this data.
Glean / Amplitude

11) Where do you intend to share the results of your analysis?
Only on Glean, Amplitude and with mobile teams.

12) Is there a third-party tool (i.e. not Telemetry) that you are proposing to use for this data collection?
No.

@mcarare mcarare marked this pull request as ready for review November 24, 2020 11:35
@mcarare mcarare added the needs:data-review PR is awaiting a data review label Nov 24, 2020
@@ -659,6 +659,12 @@ private val Event.wrapper: EventWrapper<*>?
{ ProgressiveWebApp.backgroundKeys.valueOf(it) }
)

Event.SyncedTabOpened -> EventWrapper<NoExtraKeys>(
{ Metrics.syncedTabsOpenedCount.add(1) }
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

A counter that indicates how many synced tabs a user has opened

The description for this event reads like we want to count the number of times a synced tab item is clicked and not the synced tabs UI is opened. Is that right?

If so, we probably want to add this counter to where the tab was clicked.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We spoke offline and clarified how metrics are recorded.

Copy link
Contributor

@jonalmeida jonalmeida left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

My comments went as individual items by mistake, sorry about the notifications!

@mcarare
Copy link
Contributor Author

mcarare commented Dec 4, 2020

@jonalmeida Responding here, because I am not sure the right code was referenced in the comments.
I added the event tracking in TabTrayController handleSyncedTabClicked because this is triggered when clicking on a synced tab, it also contains the code to open that tab in the browser.
This is called by ListenerDelegate.onTabClicked but I did not want to trigger event tracking there because of the comment

    // Sanity-check: Remove this class when the feature flag is always enabled.

and I did not know how that would affect the code in ListenerDelegate.

@jonalmeida
Copy link
Contributor

I added the event tracking in TabTrayController handleSyncedTabClicked because this is triggered when clicking on a synced tab, it also contains the code to open that tab in the browser.

Ordinarily, this would be true. Although, Synced Tabs in the tabs tray is behind a flag in the "secret settings" so this metric would not be fired (unless a power user knows about this setting and enables it). The default location for the UI exists in the browser menu, so the data we collect from this metric would probably be skewed to those power users then. It might make more sense to track the metric in the BrowserToolbarMenuController instead. What do you think?

Another part I would like to understand is the synced_tabs_opened_count event in metrics.md, if I understand correctly, that is tracked with:

Event.SyncedTabOpened -> EventWrapper<NoExtraKeys>(
    { Metrics.syncedTabsOpenedCount.add(1) }
)

This seems to coupling two events together: opening the Synced Tabs UI and clicking on synced tab items. If I understand that correctly, then similar to the above situation, we may be better tracking this in the SyncedTabsLayout which is used in the fragment for the Synced Tabs fragment.

When we move Synced Tabs to a new place in the tabs tray, we should definitely fix the metrics into the re-written tabs tray.

Apologises if I've caused any confusion. 🙂

@mcarare mcarare force-pushed the 15369 branch 3 times, most recently from 7e3ae0b to ed10ae3 Compare December 7, 2020 09:42
Copy link
Contributor

@jonalmeida jonalmeida left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We spoke offline to clarify the questions. Thanks!

@Mugurell
Copy link
Contributor

Talked offline with Kate who confirmed with DS and Boek:

  • The newly added metrics should log events, not counters anymore
  • An an expiration date 6 months is preferred.
    Maybe if there is already related telemetry for that part of the screen / feature we can reuse the existing expiration date so that all related telemetry probes will be renewed at the same time.

@mcarare mcarare force-pushed the 15369 branch 2 times, most recently from 4b06923 to c486080 Compare December 10, 2020 10:27
@mcarare mcarare removed the request for review from kglazko December 10, 2020 10:27
Copy link
Contributor

@boek boek left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Data Review Form (to be filled by Data Stewards)

  1. Is there or will there be documentation that describes the schema for the ultimate data set in a public, complete, and accurate way?
    Yes, metrics.yaml and metrics.md

  2. Is there a control mechanism that allows the user to turn the data collection on and off?
    Yes, Fenix data collection settings.

  3. If the request is for permanent data collection, is there someone who will monitor the data over time?
    No, Fenix team will monitor

  4. Using the category system of data types on the Mozilla wiki, what collection type of data do the requested measurements fall under?
    Type 2

  5. Is the data collection request for default-on or default-off?
    Default on

  6. Does the instrumentation include the addition of any new identifiers (whether anonymous or otherwise; e.g., username, random IDs, etc. See the appendix for more details)?
    No

  7. Is the data collection covered by the existing Firefox privacy notice?
    Yes

  8. Does there need to be a check-in in the future to determine whether to renew the data?
    Has expiry

  9. Does the data collection use a third-party collection tool?
    No

@mergify
Copy link
Contributor

mergify bot commented Dec 15, 2020

This pull request has conflicts when rebasing. Could you fix it @mcarare? 🙏

@mcarare mcarare removed the needs:data-review PR is awaiting a data review label Dec 15, 2020
@mcarare mcarare added pr:needs-landing PRs that are ready to land [Will be merged by Mergify] and removed pr:do-not-land labels Dec 15, 2020
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
pr:needs-landing PRs that are ready to land [Will be merged by Mergify]
Projects
None yet
Development

Successfully merging this pull request may close these issues.

5 participants