Telemetry for Collections

[bbinto]

Understand how useful collections are to the users and how they are using them.

Hypothesis

Our hypothesis is that collections are a useful to users

• They create collections (clicking), and this pattern increases over time
• They revisit existing collections (clicking), and this pattern increases over time

What questions will you answer with this data?

• How often do people delete collections
• How often new collections are being created
• How often do people revisit/open collections
• No. of collections shown per app collection
• How long do collections live on average
• What the life cycle of tabs is per collection

@bbinto updated, April 17

Probes

• track normal press on tab to open
• track long press on tab
• track tap on "+" to open new tab
• track tap on x to delete individual tab
  - track tap on "select tabs and save" button
  - when tab selection shows up, track saved tabs/open tabs
• track "Save to collection" from 3 dot menu and from home screen
• track tap on "save" button from the snackbar
• track saved into existing collection (vs saved as new collection)
• track "Add Tab" button
• track when a collection is deleted

Acceptance Criteria

• Pings can be queried via re:dash
• We are sending telemetry events for the actions listed in the requirements
• We have documented the telemetry
• Request Review

┆Issue is synchronized with this Jira Task

[bbinto]

@hmcgaw, @mheubusch please feel free to add your ideas based on UR findings/recommendation/questions.

We can use this data as soon Glean SDK event integration is done, even before/during Beta.

[boek]

I'm actually going to recommend we punt on this until we revisit sessions

[bbinto]

@mheubusch decided to wait until session research comes back, end of March 27 week.

[vesta0]

@bbinto and @mheubusch to update this for Collections.

[bbinto]

@mheubusch - please confirm when the Collections prototype is at a state where development can start - then I'll update this ticket.

[bbinto]

@vesta0 - this should def. go in IF we decide collections should go into Beta but I'd need to take another look at how we need to adjust the probes accordingly.

[vesta0]

@bbinto will update this to include Create new collection #1574 and Delete collections #1578

[bbinto]

@vesta0 updated!

[sblatz]

@vesta0

-Track tap on "save" button from the snackbar ❌

I want to push back against this one. That button simply transitions users to a new screen where they either "save to new collection" or "save to existing". I'm concerned adding an event for this would muddy our telemetry data (it's a save button, but the user hasn't actually saved any data yet once they've pressed it).

@AndiAJ

Track saved into existing collection (vs saved as new collection) ❌

I'm confused. This should be currently differentiated by tabs_added vs. saved. If a group of tabs are added to a currently existing collection the tabs_added event should fire, if a user created a new one, we fire a saved event.

[vesta0]

@sblatz you are right, we don't want to track "save" if we are tracking both "save to new collection" or "save to existing". Thanks for pushing back!

[AndiAJ]

Hi, I've re-tested the above on RC 1.2-rc2 using a Google Pixel 3a (Android 9) and there are still reproducible issue.

1. Track long press on a tab Ping 392c4f98-f5f2-4528-aa19-07a81ede8e84 ✅
2. Track tap on "Save to collection" button Ping 57342723-e170-4549-aa7f-cafc516dac38 ✅
3. Track tap on "Save to collection" button from the 3dot menu ❌

Collections section

1. Track share button Ping f24983c7-e615-4528-8cf7-56f2627d0284 ✅
2. Track normal press on tab Ping 2bc95e5c-ac7b-4d28-ad88-f1e6c2be3668 ✅
3. Track "Add tab" to existing collection from 3dot button Ping 5d22074e-3139-4e58-ac4c-66e08b6d569e ✅
4. Track Rename collection from 3dot button ❌
5. Track Open tabs from 3dot button 4f90f11d-674e-41e0-9a56-a7f3b79bce79 ✅
6. Track tap x to delete individual tab from 3dot button Ping 2991ef34-031a-4e45-b362-1bbf3554fd5e ✅
7. Track Delete collection from 3dot button Ping 92efd54d-5638-41f5-8fe8-04b0541a3766 ✅

Other events

1. Creation of new collections ❌
2. Track saved into existing collection ❌

@sblatz - I would kindly ask you to review the above, thanks! :)

[sblatz]

@AndiAJ

Other events
Creation of new collections ❌
Track saved into existing collection ❌

I'm confused by this section. These should be tracked with these events:

Is there something I'm missing here?

[AndiAJ]

Hi @sblatz , I've re-tested this on Fenix Nightly Build #12190111 from 7/8 using a Google Pixel 3a.

Open tabs section

1. Track long press on a tab Ping db18ce9c-7c50-49b8-ab3e-93bc8e8ac24f ✅
2. Track tap on "Save to collection" button Ping 2d33b751-8a4b-432c-8b87-b865a782ffd6✅
3. Track tap on "Save to collection" button from the 3dot menu Ping 1027eafc-062b-47e6-b661-b9e2f75eb0d1 ✅

Collections section
Track share button Ping 4f11b2de-d0a6-49fb-88c4-affe4088afcb ✅
Track normal press on tab Ping 2da9e998-6a70-451d-b9e0-39979b701a26 ✅
Track "Add tab" to existing collection from 3dot button Ping c0dceb38-7721-40ca-92e9-54cc0059dafb ✅
Track Rename collection from 3dot button 1ec6747e-6590-4759-896c-51fd8782489d ✅
Track Open tabs from 3dot button Ping 139b17d1-982b-4014-8ef2-00795c33b0ca ✅
Track tap x to delete individual tab from 3dot button Ping 641eca23-c56b-4d69-92a9-7f12559d89f9 ✅
Track Delete collection from 3dot button Ping 69459669-3dc3-4791-9af9-4206b669a4ea ✅

However, I would like to ask you a couple of questions.

1. Considering the fact that "Save to collection" button ("save_button") & "Add tab" to existing collection from 3dot button ("add_tab_button") are well defined, will any changes be made for 3. Track tap on "Save to collection" button from the 3dot menu ?

I'm asking because it generates 2 "tab_select_opened". One for the Select Tabs window and the other one for selecting a tab from the list.

2. Regarding the above misunderstanding, will the creation and addition to an existing collection be tracked only with "tab_select_opened" ?
   The extra_keys (tabs_open & tabs_selected) aren't displayed.

Logcat
Dashboard

[sblatz]

@AndiAJ This PR will address the "saved" and "tabsAdded" events. We missed adding those events: https://github.com/mozilla-mobile/fenix/pull/4592/files

Regarding the "Save to collection" from three dot menu, yes that is being merged here: https://github.com/mozilla-mobile/fenix/pull/4539/files#diff-dcd3c2fb5158fcd99c56aa5b26b6de5dR1093.

I'll put qa needed on this once those two PR's are merged. Sorry for the back and forth.

[sblatz]

@AndiAJ okay, this time we should be good to go. Can I get your review again?

[AndiAJ]

Hi, verified as fixed on Fenix Nightly Build #12210609 using a Google Pixel 3a (Android 9).

Open tabs section

1. Track long press on a tab Ping 852e28f5-5b50-44f6-9cb5-7e7192ec0bce ✅
2. Track tap on "Save to collection" button Ping 5e7e2b3a-41e9-4e44-90ab-b33049a592ed✅
3. Track tap on "Save to collection" button from the 3dot menu Ping 459947da-971d-4543-9d97-b79257ad5817 ✅

Collections section

1. Track share button Ping ae685666-a4c9-40f5-8588-47ec6959ca4f ✅
2. Track normal press on tab Ping 02ae0186-a1f3-4f85-93c2-ad8bac7f1253 ✅
3. Track "Add tab" to existing collection from 3dot button Ping ff8d431e-f149-48fc-9db3-13bc271dc883 ✅
4. Track Rename collection from 3dot button 134edacf-2826-45b7-b9ac-b07ac4d1ce2d ✅
5. Track Open tabs from 3dot button Ping ebfe2830-0939-4b0b-adbc-d33c50580912 ✅
6. Track tap x to delete individual tab from 3dot button Ping a831808a-a567-4da2-98b2-6f0bc0f35d90 ✅
7. Track Delete collection from 3dot button Ping 957519a3-8040-484d-bc41-6f34e8ce3043 ✅

Save to a new collection "saved" Ping 30311032-a887-4cec-b16b-b6f9dcadf1f0 ✅
Add to existing collection "tabsAdded" Ping ff8d431e-f149-48fc-9db3-13bc271dc883 ✅

@sblatz Great job! 👍