[Telemetry] Opening a new tab from saved data

[vesta0]

Description & Product Manager / Data Scientist User Story

As a product manager, I need to know how our users interact with their saved sites so we can surface the most frequently accessed items on their homescreen.

What questions will you answer with this data?

• What % of users tap on a bookmarked site to open it in a new tab OR tap on a bookmarked site that shows up as a search suggestion? (We have bookmarks_management.open_in_new_tab which tracks the user interaction with the 3 dot menu "open in new tab" option but NOT the actual event of tapping on a bookmarked site)

• What % of users tap on a history item to open it in a new tab OR tap on a site from history that shows up as a search suggestion?? (We have history.opened_item but it is unclear whether that tracks actual event of tapping on a history item OR using the 3 dot menu "open in new tab" item. Needs confirmation.)

• What % of users tap on a synced tab to open it in a new tab OR tap on a synced site that shows up as a search suggestion?

Meta question for Data Science (can be broken out into a separate issue):

• Do we care equally about adoption and frequency? (e.g., 10% of users use a feature 100% of the time vs 100% of users use a feature 10% of the time)

Acceptance Criteria

• ENG files a DS JIRA request outlining their methodology.
• DS sign off on instrumentation methodology addressing product questions.
• Event pings can be queried via re:dash
• Event pings can be queried via amplitude
• We are sending telemetry events for the actions listed in the requirements
• We have documented the telemetry
• We have asked a data steward to review the telemetry

┆Issue is synchronized with this Jira Task

[yoasif]

I think that there is a real problem with these measurements in that there aren't many easy ways to open bookmarks, top sites, or history items in existing tabs, so you are really just measuring how many users are opening bookmarks, top sites and history.

See #13336 for an example where this is relevant.

I think that the only way to get history or bookmarks in an existing tab is to use the address bar suggestions based on typed data, whereas top sites are only accessible in new tabs.

If that is indeed what you want to measure, please feel free to ignore my feedback, but right now, my feeling is that you may eventually end up getting results based on flawed data built into the metric.

[vesta0]

Thank you @yoasif that is excellent feedback. I have tweaked the requirements to include the address bar suggestions.

I agree that this data will be skewed because according to user feedback, opening a new tab from bookmarks/history is not currently easy/intuitive. We want to fix this problem sooner rather than later. However, this usage data will help us measure whether whatever solution we decide on is successful.

[liuche]

Context: in other telemetry tix, have hopped on a call with Marissa to discuss whether something needs DS jira ticket.

[kglazko]

I created the Jira request here: https://jira.mozilla.com/browse/DO-329

[JohannVII]

In case you're not considering this (you might be, as you note the data will be skewed, but it's a bigger problem than being skewed because of specific contextual factors), you're using a data collection method that's going to bias your results if you're primarily relying on telemetry data at all: privacy-focused users, the user base to which Mozilla has been specifically marketing, are likelier than most users to disable telemetry (experiments, surveys, etc.), and the most-privacy-concerned users are most likely to disable them. This creates a pervasive and likely significant bias in the data set by excluding a subset of users who are likely to have similar preferences and behaviors. Telemetry data is useful if properly interpreted, of course, but it should not be your primary means of feedback for design decisions - you need to do UX research through means that don't amount to "spying" for privacy-concerned users, soliciting feedback through a channel other than something built in to the application, like focus groups or social media surveys or trawling through comments on Reddit and Google Play reviews.

[bifleming]

@kglazko Are we still waiting on something for this ticket? Vesta called this out as a priority for January.

[rocketsroger]

With all the latest changes. I've created a sheet that tracks the telemetry changes. https://docs.google.com/spreadsheets/d/1EB6Mme0LR3GRE00YkX7Vm1VJOuTSnhwrSc4MOoySX5A/edit?usp=sharing

[lobontiumira]

Hi all!
This ticket has been verified in multiple tickets:

• The awesome bar group was verified in here, and here.
• The History group was verified here.

The Bookmarks group was verified on the 3/16 Nightly build using a Google Pixel (Android 10):

• ping: 8542b09c-73bf-4b0f-ac15-b642e78fdf7a
• open bookmark:

{
          "category": "bookmarks_management",
          "name": "open",
          "timestamp": 4554
        }

• open bookmark in new tab:

{
          "category": "bookmarks_management",
          "name": "open_in_new_tab",
          "timestamp": 5488
        }

• open bookmarks in new tabs:

{
          "category": "bookmarks_management",
          "name": "open_in_new_tabs",
          "timestamp": 15655
        }

• open bookmark in private tab:

{
          "category": "bookmarks_management",
          "name": "open_in_private_tabs",
          "timestamp": 20899
        }

• open bookmarks in private tabs:

{
          "category": "bookmarks_management",
          "name": "open_in_private_tabs",
          "timestamp": 40635
        }

logcat bookmarks.txt
Glean dashboard - bookmarks

I will close this ticket.
LEt us know if there is anything else we should cover.