Privacy Preserving Product Analytics (P3A)

[bsclifton]

Description

The implementation of Privacy Preserving Product Analytics (P3A) landed with brave/brave-core#3242

More details on our blog here:
https://brave.com/privacy-preserving-product-analytics-p3a/

Detailed list of the questions here:
https://github.com/brave/brave-browser/wiki/P3A

Follow ups captured with:
#6284

Android side captured with:
#6176

Test plan

There is a bunch of command-line switches suitable to simplify the testing process (see https://github.com/brave/brave-core/blob/master/components/p3a/brave_p3a_switches.h), as well as detailed logs enabled as follows --vmodule=*/p3a/*=3 --enable-logging=stderr

The possible command-line for testing might look as follows:
brave --vmodule=*/p3a/*=3 --enable-logging=stderr --p3a-upload-interval-seconds=5 --p3a-rotation-interval-seconds=160 --p3a-upload-server-url="http://127.0.0.1/" --p3a-ignore-server-errors --p3a-do-not-randomize-upload-interval

This way, we decrease the interval between sending values, disable randomization of this interval, and decrease the interval between restarting the whole process from 1 week to 160 seconds. Note that there is an initial silent period of 60 seconds (probably we should also add a switch to disable it). Also, in case of unsuccessful upload there is a retry logic with backoff interval of 5 seconds multiplied by 2 after each upload failure.

Last recorded values are saved in local state and can be found by p3a key on brave://local-state

[bsclifton]

Fixed with brave/brave-core#3242

[LaurenWags]

Adding QA/Blocked label pending overview

[kjozwiak]

Removing QA/Blocked. Had a meeting with @bsclifton & @aekeus and should have enough information to go through P3A and ensure it's working as expected.

[GeetaSarvadnya]

@bsclifton Can we update the wiki to reflect the values for Q.5 as below?

1. 0-5
2. 6-20
3. 21-100
4. Over 100

[LaurenWags]

Proposed general P3A test plan:

• Verify new user messaging as per Initial implementation of P3A (Privacy-Preserving Product Analytics) brave-core#3242
• Verify upgraded user messaging as per Initial implementation of P3A (Privacy-Preserving Product Analytics) brave-core#3242
• Verify each question and answer shows in local-state file
• Verify answers to questions update in local-state (ex. if you open a Tor window for the first time, the response to that particular question should update. Similarly for all other questions)
• Verify answers send to stats
• Verify stats displays data for P3A

cc @bsclifton could you review the above and let me know if there any additions/changes you think should be made?

[bsclifton]

@GeetaSarvadnya yes! I just updated, let me know what you think:
https://github.com/brave/brave-browser/wiki/P3A

[bsclifton]

@LaurenWags LGTM! 👍

[GeetaSarvadnya]

Verification passed on

Brave	1.1.18 Chromium: 79.0.3945.56 (Official Build) (64-bit)
Revision	73cc6bf591f792b99f8fc7cdfb8addedbd084bf8-refs/branch-heads/3945@{#788}
OS	Windows 10 OS Version 1803 (Build 17134.1006)

• Verified the new user messaging as per Initial implementation of P3A (Privacy-Preserving Product Analytics) brave-core#3242
  • Verified that users are notified about P3A in the welcome screen in a clean profile
    
• Verified the upgraded user messaging as per Initial implementation of P3A (Privacy-Preserving Product Analytics) brave-core#3242
  • Verified that existing users will see P3A info bar brave upon upgrade and launch to 1.1.x
  • Verified that info bar remains until acknowledged (even after relaunching)
  • Verified that click Disable and will immediately disable the service and close the info bar
  • Verified that Hitting Got it or clicking the X will acknowledge this and close the info bar
  • Verified click on Learn more redirects to the blog post
    
• Logged P3A issues P3A - Wallet balance answer value isn't updated correctly in logs #7235, P3A- Tab stats along with it's value isn't displayed in logs #7230, P3A - Click on Skip welcome tour isn't giving correct response value #7262 and P3A - Extensions value isn't updated after installing extensions #7228
• Verified A-C retry and Monthly tip sent retry response value is reflected correctly in P3A logs after browser crash and restart
  (Setup A-C and monthly tips in staging env, before a-c time interval make the browser crash intentionally use steps from issue Brave crashes when select Sync option from Tor window #7225. After browser crash, relaunch the browser and wait for A-C to complete (ensured Q.13 - 4. Auto-contribute enabled, one successful contribution so far response value is displayed in logs correctly))
• Verified answers to questions update correctly in local-state in a clean profile 1.1.18

Q.1 `Brave.Uptime.BrowserOpenMinutes`
- Verified response value is displayed as expected when seven days have not passed since first open
- Verified response value is displayed as expected when browser utilise time is in the range 0 - 30min for last 7 days.
- Verified response value is displayed as expected when browser utilise time is in the range 30 mins to 5hr for last 7 days
- Verified response value is displayed as expected when browser utilise time is 5+hours for last 7 days.
Q.2 `DefaultBrowser.State`
- Verified response value is displayed as expected when brave is not set as default browser
- Verified response value is displayed as expected when brave is set as default browser
Q.3 `Brave.Welcome.InteractionStatus`
- Verified response value is displayed as expected when did not click within welcome screen
- Verified response value is displayed as expected when clicked once in welcome screen
- Verified response value is displayed as expected when clicked twice or more in welcome screen but did not make it to the end
- Verified response value is displayed as expected when Made it to end of on-boarding flow
Q.4 `Brave.Importer.ImporterSource`
- Verified response value is displayed as expected when settings and bookmarks never imported
- Verified response value is displayed as expected when settings and bookmarks are imported from brave
- Verified response value is displayed as expected when settings and bookmarks are imported from Chrome
- Verified response value is displayed as expected when settings and bookmarks are imported from Firefox
- Verified response value is displayed as expected when settings and bookmarks are imported from Bookmarks HTML
Q.5 `Brave.Core.BookmarksCountOnProfileLoad`
- Verified response value for BM’s count is displayed as expected in the range 0-5
- Verified response value for BM’s count is displayed as expected in the range 6-20
- Verified response value for BM’s count is displayed as expected in the range 21-100
- Verified response value for BM’s count is displayed as expected when BM's added are 100+
Q.6 `Brave.Core.WindowCount`
- Verified response value is displayed as expected when opened windows count is in 0-1 range
- Verified response value is displayed as expected when opened windows count is in 2-5 range
- Verified response value is displayed as expected when opened windows count is 5+
Q.7 `Brave.Core.TabCount`
- Verified response value is displayed as expected when the open tab count is in the range 0-1
- Verified response value is displayed as expected when the open tab count is in the range 2-5
- Verified response value is displayed as expected when the open tab count is in the range 6-10
- Verified response value is displayed as expected when the open tab count is in the range 11-50 - Logged #7230
- Verified response value is displayed as expected when the open tab count is 50+
Q.8 `Brave.Shields.UsageStatus`
- Verified response value is displayed as expected when shield is never clicked
- Verified response value is displayed as expected when clicked on shield, but never made a change
- Verified response value is displayed as expected when clicked on shield, shut off shields for one or more sites
- Verified response value is displayed as expected when clicked on it, changed Shields defaults or per-site shield settings in addition to (or in place of) shutting off shields on one or more sites
Q.9 `Brave.Core.LastTimeIncognitoUsed`
Set the system date behind and change the upload interval time to test 3,4,5 scenarios
- Verified response value is displayed as expected when Private window is never opened
- Verified response value is displayed as expected when Private window is used in last 24hr
- Verified response value is displayed as expected when Private window is used in last week but not 24h
- Verified response value is displayed as expected when Private window is used in last 28 days but not week
- Verified response value is displayed as expected when Private window is Ever used but not in last 28 days
Q.10
- Verified response value is displayed as expected when TOR window is never opened
- Verified response value is displayed as expected when TOR window is opened
Q.12 `Brave.Rewards.WalletBalance`
- Verified response value is displayed as expected when no wallet is created
- Verified response value is displayed as expected when balance is in the range 0-10 BAT, excluding grants
- Verified response value is displayed as expected when balance is in the range 10-50 BAT, excluding grants - logged #7235
- Verified response value is displayed as expected when balance is Over 50 BAT, excluding grants - #7235
Q.13 `Brave.Rewards.AutoContributionsState`
- Verified response value is displayed as expected when no wallet is created
- Verified response value is displayed as expected when Wallet created, Auto-Contribute off
- Verified response value is displayed as expected when Auto-contribute enabled, no contribution so far
- Verified response value is displayed as expected when Auto-contribute enabled, one successful contribution so far
- Verified response value is displayed as expected when Auto-contribute enabled, more than one successful contribution so far
Q.14 `Brave.Rewards.TipsState`
- Verified response value is displayed as expected when no wallet is created
- Verified response value is displayed as expected when Wallet created, no tips sent
- Verified response value is displayed as expected when Wallet created, one-time tip sent
- Verified response value is displayed as expected when Wallet created, one-time sent and at least one monthly tip queued up or sent
- Verified response value is displayed as expected when Wallet created, at least one monthly tip queued up or sent
Q.15 `Brave.Sync.Status`
- Verified response value is displayed when sync is not enabled
- Verified response value is displayed when sync is enabled with 2 devices
- Verified response value is displayed when sync is enabled with 3 devices
Q.16 `Brave.Core.NumberOfExtensions`
- Verified response value is displayed when no extensions are installed in brave
- Verified response value is displayed when no of extension installed is in the range 2-4 - Logged #7228
- Verified response value is displayed when no of extension installed are 5+ - Logged #7228
Q.17 `Brave.Rewards.AdsState
- Verified response value is displayed as expected when no wallet is created
- Verified response value is displayed as expected when Wallet created, ads enabled, then disabled, while Brave Rewards remains on 
- Verified response value is displayed as expected when Wallet created, ads enabled, then disabled, Brave Rewards disabled
- Verified response value is displayed as expected when Wallet created and ads enabled
- Verified response value is displayed as expected when Wallet created and ads NOT enabled
Q.18 `Brave.P3A.SentAnswersCount`
- Verified response value is displayed as expected when more than 10 questions were answered
-  Asked clarification to execute 1, 2 and 3 scenarios - https://bravesoftware.slack.com/archives/CDNJ9SVUL/p1575639169171900

• Verified answers/response value to questions displayed correctly in local-state in an upgraded profile 1.1.18 (executed above scenarios on upgraded profile)
• Logged issues which found in the upgraded profile testing P3A - Wallet balance answer value isn't updated correctly in logs #7235 P3A - Ads response value isn't updated correctly in upgraded profile #7253 P3A - Response value isn't displayed correctly for default browser set after upgrade #7259
• Verified that even after disabling the P3A notification, the questions are answered and response value is getting reflected as expected in upgraded profile 1.1.18
• Verified stats display for P3A
  

Verified passed with

Brave	1.1.18 Chromium: 79.0.3945.56 (Official Build) (64-bit)
Revision	73cc6bf591f792b99f8fc7cdfb8addedbd084bf8-refs/branch-heads/3945@{#788}
OS	macOS Version 10.13.6 (Build 17G5019)

• Verified new user experience as per Initial implementation of P3A (Privacy-Preserving Product Analytics) brave-core#3242.
  • Confirmed P3A mentioned on brave://welcome.
• Verified existing user experience (upgrade) per Initial implementation of P3A (Privacy-Preserving Product Analytics) brave-core#3242.
  • Confirmed info bar remains until acknowledged. It does only show on the tab which is active upon launch though.
  • Confirmed that when P3A is toggled off, data is not sent. Launched with parameters from description. Toggled P3A off. Confirmed no data shown as sent in the terminal. No data was marked as sent=true in local state. When P3A was toggled on, then data was shown as sent in the terminal and marked as sent=true in local-state.
  • Confirmed Learn More links to P3A blog and does not close the info bar.
  • Confirmed Disable from the info bar immediately disables P3A and sets toggle in brave://settings to off. This also closes the info bar. Info bar does not return on launch after selecting 'Disable'.
  • Confirmed toggle (on/off) remains after relaunch.
  • Confirmed selecting 'Got it' acknowledges the info bar and it does not return on relaunch.
• Verified questions and responses show in local state (see wiki for questions/responses or notes from Geeta above)
• Verified responses update except where noted in logged issues. Also confirmed stats shows various responses.
• Verified data is being sent to stats (in aggregate, unable to confirm my specific data in stats)
• Verified stats displays data for P3A
• Encountered/Logged/Related issues have been linked above.
• Utilized a combination of staging env and flags noted in description for testing P3A as well.

Verification passed on

Brave	1.1.19 Chromium: 79.0.3945.70 (Official Build) (64-bit)
Revision	65868db27916cebde51fc2465c9b34761e6383e2-refs/branch-heads/3945@{#869}
OS	Ubuntu 18.04 LTS

• Verified the new user messaging as per Initial implementation of P3A (Privacy-Preserving Product Analytics) brave-core#3242
  

• Verified the upgraded user messaging as per Initial implementation of P3A (Privacy-Preserving Product Analytics) brave-core#3242

  • Verified that existing users will see P3A info bar brave upon upgrade and launch to 1.1.x
  • Verified that info bar remains until acknowledged (even after relaunching)
  • Verified that click Disable and will immediately disable the service and close the info bar
  • Verified that Hitting Got it or clicking the X will acknowledge this and close the info bar
  • Verified click on Learn more redirects to the blog post
    

• Verified questions and responses show in local state on clean profile
  

• Verified responses update. Also confirmed stats shows various responses.

• Verified data is being sent to stats

• Verified stats displays data for P3A (last point is from 12-02-2019)