Skip to content
This repository has been archived by the owner on May 10, 2024. It is now read-only.

Add more P3A questions #6406

Closed
iccub opened this issue Nov 16, 2022 · 1 comment · Fixed by #6440
Closed

Add more P3A questions #6406

iccub opened this issue Nov 16, 2022 · 1 comment · Fixed by #6440
Assignees
@kylehickinson
Labels
enhancement Epic: Analytics QA Pass - iPhone QA/Yes
Milestone
1.46

Comments

@iccub
Copy link
Contributor

iccub commented Nov 16, 2022

Initially we integrated P3A here #6308
It came with only brave-core integrated questions like sync and wallet.

Porting over more questions is manual work and we plan to add more of them over time.
This is the first ticket that will add initial set of P3A questions outside of brave-core
@kylehickinson kylehickinson mentioned this issue Nov 18, 2022
Merged
11 tasks
kylehickinson added a commit that referenced this issue Nov 18, 2022
@kylehickinson
Merge pull request #6440 from brave/p3a-answers
d62863d 
Fix #6406: Add answers for basic P3A questions
@kylehickinson kylehickinson added this to the 1.46 milestone Nov 21, 2022
@GeetaSarvadnya
Copy link

GeetaSarvadnya commented Dec 5, 2022 •
edited
Loading

Verification PASSED on iPhone 8 iOS version 16.1.1 running Brave 1.46 (22.12.3.0)

Note:

  1. Q46 and Q47, verified Disabled and Standard ad blocking/fingerprinting settings as we don't have option to set "Aggressive" global shield settings in iOS.
  2. Q54, FP setting to be higher (block more) than the default?- cannot test as we don't have option to set "Aggressive" domain shield settings in iOS.
Brave.Core.TabCount_PASSED
Case 1: Brave.Core.TabCount = 0 
     1. Clean profile 1.46.133
     3. Don't open any new tabs
     4. Wait for the p3a-json.brave.com enpoint to show up in Charles
     5. `Brave.Core.TabCount` response value is `0` as expected
Screenshot 2022-12-05 at 11 22 37 AM
Case 2: Brave.Core.TabCount = 1 
     1. Clean profile 1.46.133
     2. Open 5 new tabs
     3. Wait for the p3a-json.brave.com enpoint to show up in Charles
     4. `Brave.Core.TabCount` response value is `1` as expected
Screenshot 2022-12-05 at 10 58 45 AM (1)
Case 3: Brave.Core.TabCount = 2 
     1. Clean profile 1.46.133
     2. Open 10 new tabs
     3. Wait for the p3a-json.brave.com enpoint to show up in Charles
     4. `Brave.Core.TabCount` response value is `2` as expected
Screenshot 2022-12-05 at 12 16 12 PM
Brave.Core.BookmarksCountOnProfileLoad.2_PASSED
Case 1: Brave.Core.BookmarksCountOnProfileLoad.2 = 0

Case 1.1

     1. Clean profile 1.46.133
     2. Don't bookmark any sites
     3. Open local state file via settings
     4. `Brave.Core.BookmarksCountOnProfileLoad.2` response value is `0` as expected

Case 1.2

     1. Clean profile 1.46.133
     2. Open 5 sites and bookmark them
     3. Open local state file via settings
     4. `Brave.Core.BookmarksCountOnProfileLoad.2` response value is `0` as expected

IMG_3235

Case 2: Brave.Core.BookmarksCountOnProfileLoad.2 = 1

Case 2.1

     1. Clean profile 1.46.133
     2. Open 6 sites and bookmark them
     3. Open local state file via settings
     4. `Brave.Core.BookmarksCountOnProfileLoad.2` response value is `1` as expected

Case 2.2

     1. Clean profile 1.46.133
     2. Open 20 sites and bookmark them
     3. Open local state file via settings
     4. `Brave.Core.BookmarksCountOnProfileLoad.2` response value is `1` as expected

IMG_3236

Brave.Shields.CookieListEnabled_PASSED
Brave.Shields.CookieListEnabled = 0 
      1. Clean profile Brave Dev 1.46 (22.12.6.19)
      2. Open Youtube.com
      3. Remove app from memory and restart the browser
      4. Cookies consent prompt is shown 
      6. Click "No Thanks" on the prompt
      7. Open local state file via settings
      8. `Brave.Shields.CookieListEnabled` response is `0` as expected

IMG_3252

Brave.Shields.CookieListEnabled = 1 
      1. Clean profile Brave Dev 1.46 (22.12.6.19)
      2. Open Youtube.com
      3. Remove app from memory and restart the browser
      4. Cookies consent prompt is shown 
      5. Click "Block cookie notices" on the prompt
      6. Open local state file via settings
      7. `Brave.Shields.CookieListEnabled` response is `1` as expected

IMG_3251

Brave.NTP.SponsoredImagesEnabled_PASSED

Case 1: NTP SI enabled

  • Clean profile 1.46.x
  • Wait for the p3a-json.brave.com enpoint to show up in Charles
  • Brave.NTP.SponsoredImagesEnabled response value is 1 as expected
Screenshot 2022-12-05 at 12 12 24 PM

Case 2: NTP SI disabled

  • Clean profile 1.46.x
  • Open settings navigate to New Tab Pages
  • Image Type-> Sponsored was default option, select default images option
  • Wait for the p3a-json.brave.com enpoint to shown up in Charles
  • Brave.NTP.SponsoredImagesEnabled response value is 0 as expected
Screenshot 2022-12-05 at 5 09 38 PM
Brave.Shields.CookieListPrompt_PASSED
Brave.Shields.CookieListPrompt = 0 
      1. Clean profile Brave Dev 1.46 (22.12.6.19)
      2. Open Youtube.com
      3. Open local state file via settings
      7. `Brave.Shields.CookieListPrompt` metrics is not shown in the local state file which is expected
Brave.Shields.CookieListPrompt = 1 
     1. Clean profile Brave Dev 1.46 (22.12.6.19)
      2. Open Youtube.com
      3. Remove app from memory and restart the browser
      4. Cookies consent prompt is shown 
      5. Don't take any action on the prompt
      6. Open local state file via settings
      7. `Brave.Shields.CookieListPrompt` response is `1` as expected

IMG_3254

Brave.Shields.CookieListPrompt = 2 
     1. Clean profile Brave Dev 1.46 (22.12.6.19)
      2. Open Youtube.com
      3. Remove app from memory and restart the browser
      4. Cookies consent prompt is shown 
      5. Click "No Thanks" on the prompt
      6. Open local state file via settings
      7. `Brave.Shields.CookieListPrompt` response is `2` as expected

IMG_3253

Brave.Shields.CookieListPrompt = 3 
         1. Clean profile 1.46.133
         2. Open Youtube.com
         3. Remove app from memory and restart the browser
         4. Cookies consent prompt is shown 
         5. Click "Block cookie notices" on the prompt
         6. Wait for the `p3a-json.brave.com` enpoint to show up in Charles
         7. `Brave.Shields.CookieListPrompt` response is `3` as expected
Screenshot 2022-12-05 at 8 14 37 PM
Brave.Search.DefaultEngine.4_PASSED
Case 1: Brave.Search.DefaultEngine.4 = 1 
     1. Clean profile Brave Dev1.46 (22.12.6.11)
     2. Set default SE to "Google"
     3. Open local state file via settings
     4. `Brave.Search.DefaultEngine.4` response value is `1` as expected

IMG_3238

Case 2: Brave.Search.DefaultEngine.4 = 2 
     1. Clean profile Brave Dev1.46 (22.12.6.11)
     2. Set default SE to "Duckduckgo"
     3. Open local state file via settings
     4. `Brave.Search.DefaultEngine.4` response value is `2` as expected

IMG_3239

Case 3: Brave.Search.DefaultEngine.4 = 3 
     1. Clean profile Brave Dev1.46 (22.12.6.11)
     2. Set default SE to "Startpage"
     3. Open local state file via settings
     4. `Brave.Search.DefaultEngine.4` response value is `3` as expected

IMG_3243

Case 4: Brave.Search.DefaultEngine.4 = 4 
     1. Clean profile Brave Dev1.46 (22.12.6.11)
     2. Set default SE to "Bing"
     3. Open local state file via settings
     4. `Brave.Search.DefaultEngine.4` response value is `4` as expected

IMG_3240

Case 5: Brave.Search.DefaultEngine.4 = 5 
     1. Clean profile Brave Dev1.46 (22.12.6.11)
     2. Set default SE to "Qwant"
     3. Open local state file via settings
     4. `Brave.Search.DefaultEngine.4` response value is `5` as expected

IMG_3242

Case 6: Brave.Search.DefaultEngine.4 = 7 
     1. Clean profile Brave Dev1.46 (22.12.6.11)
     2. Set default SE to "Ecosia"
     3. Open local state file via settings
     4. `Brave.Search.DefaultEngine.4` response value is `7` as expected

IMG_3241

Case 7: Brave.Search.DefaultEngine.4 = 8 
     1. Clean profile Brave Dev1.46 (22.12.6.11)
     2. Set default SE to "BraveSearch"
     3. Open local state file via settings
     4. `Brave.Search.DefaultEngine.4` response value is `8` as expected

IMG_3237

Brave.Shields.AdBlockSetting_PASSED
Case 1: Brave.Shields.AdBlockSetting = 0 
     1. Clean profile Brave Dev1.46 (22.12.6.11)
     2. Open settings->"Brave Shields Privacy", Turn off the switch for "Block cross-site trackers"
     3. Open local state file via settings
     4. `Brave.Shields.AdBlockSetting` response value is `0` as expected

IMG_3244

Case 2: Brave.Shields.AdBlockSetting = 1 
     1. Clean profile Brave Dev1.46 (22.12.6.11)
     2. Open settings->"Brave Shields Privacy", do not change the switch state for "Block cross-site trackers"
     3. Open local state file via settings
     4. `Brave.Shields.AdBlockSetting` response value is `1` as expected

IMG_3245

Brave.Shields.FingerprintBlockSetting_PASSED
Case 1: Brave.Shields.FingerprintBlockSetting = 0 
     1. Clean profile Brave Dev1.46 (22.12.6.11)
     2. Open settings->"Brave Shields Privacy", turn off the switch for "Block Fingerprinting"
     3. Open local state file via settings
     4. `Brave.Shields.FingerprintBlockSetting` response value is `0` as expected

IMG_3244

Case 2: Brave.Shields.FingerprintBlockSetting = 1 
     1. Clean profile Brave Dev1.46 (22.12.6.11)
     2. Open settings->"Brave Shields Privacy", do not change the switch state for "Block Fingerprinting"
     3. Open local state file via settings
     4. `Brave.Shields.FingerprintBlockSetting` response value is `1` as expected

IMG_3245

Brave.Shields.DomainAdsSettingsBelowGlobal_PASSED
Brave.Shields.DomainAdsSettingsBelowGlobal = 1_PASSED

Case 1: PASS

    1. Clean profile 1.46 (22.12.6.19)
    2. Open a site
    3. Open shield panel and disable "Block Cross-Site Trackers" switch via shields panel
    4. Open local state file via Settings 
    5. Remove app from memory and relaunch
    6. The metrics `Brave.Shields.DomainAdsSettingsBelowGlobal` response value is `1` as expected

Case 2: PASS

    1. Clean profile 1.46 (22.12.6.19)
    2. Open 5 sites
    3. Open shield panel and disable "Block Cross-Site Trackers" switch for all the 5 sites
    4. Open local state file via Settings 
    5. Remove app from memory and relaunch
    6. The metrics `Brave.Shields.DomainAdsSettingsBelowGlobal` response value is `1` as expected

IMG_3246

Brave.Shields.DomainAdsSettingsBelowGlobal = 0_FAILED

Case 1: FAIL #6592

    1. Clean profile 1.46 (22.12.6.19)
    2. Open a site
    3. Open local state file via Settings 
    4. Remove app from memory and relaunch
    5. The metrics `Brave.Shields.DomainAdsSettingsBelowGlobal` isn't shown in local state file

Case 2: PASS

    1. Clean profile 1.46 (22.12.6.19)
    2. Open a site
    6. Open shield panel and disable "Block Cross-Site Trackers" switch via shields panel
    7. Open local state file via Settings 
    9. Remove app from memory and relaunch
    10. The metrics `Brave.Shields.DomainAdsSettingsBelowGlobal` response value is `1` as expected
    11. Open shield panel and enable "Block Cross-Site Trackers" switch via shields panel
    12. Remove app from memory and relaunch
    13. The metrics `Brave.Shields.DomainAdsSettingsBelowGlobal` response value is `0` as expected

IMG_3248

Brave.Shields.DomainAdsSettingsBelowGlobal = 2_PASSED 
    1. Clean profile 1.46 (22.12.6.19)
    2. Open 6 sites
    3. Open shield panel and disable "Block Cross-Site Trackers" switch for all the 6 sites
    4. Open local state file via Settings 
    5. Remove app from memory and relaunch
    6. The metrics `Brave.Shields.DomainAdsSettingsBelowGlobal` response value is `2` as expected

IMG_3250

Brave.Shields.DomainAdsSettingsAboveGlobal_PASSED

Logged #6591

Brave.Shields.DomainFingerprintSettingsBelowGlobal_PASSED
Brave.Shields.DomainFingerprintSettingsBelowGlobal = 0 
    1. Clean profile 1.46 (22.12.7.14)
    2. Open local state file via Settings 
    3. The metrics `Brave.Shields.DomainAdsSettingsBelowGlobal` and `Brave.Shields.DomainFingerprintSettingsAboveGlobal` are not reported in local state file 
    7. Logged https://github.com/brave/brave-ios/issues/6596
Brave.Shields.DomainFingerprintSettingsBelowGlobal = 1

Case 1: PASS

    1. Clean profile 1.46 (22.12.7.14)
    2. Open a site
    3. Open shield panel and disable "Block Fingerprinting" switch for the site
    4. Open local state file via Settings 
    5. Remove app from memory and relaunch
    6. The metrics `Brave.Shields.DomainFingerprintSettingsBelowGlobal` response value is `1` as expected

Case 2: PASS

    1. Clean profile 1.46 (22.12.7.14)
    2. Open 5 sites
    3. Open shield panel and disable "Block Fingerprinting" switch for all the site
    4. Open local state file via Settings 
    5. Remove app from memory and relaunch
    6. The metrics `Brave.Shields.DomainFingerprintSettingsBelowGlobal` response value is `1` as expected

IMG_3255

Brave.Shields.DomainFingerprintSettingsBelowGlobal = 2 
    1. Clean profile 1.46 (22.12.7.14)
    2. Open 6 site
    3. Open shield panel and disable "Block Fingerprinting" switch for all the sites
    4. Open local state file via Settings 
    5. Remove app from memory and relaunch
    6. The metrics `Brave.Shields.DomainFingerprintSettingsBelowGlobal` response value is `2` as expected

IMG_3256

Brave.Shields.DomainFingerprintSettingsAboveGlobal_Q54_Cannot test_see note above
Brave.Today.DirectFeedsTotal_PASSED
Brave.Today.DirectFeedsTotal = 0 
   1. Clean profile 1.46 (22.12..22)
   2. Open local state file
   3. Ensured that the metrics `Brave.Today.DirectFeedsTotal` response value is `0`

IMG_0309

Brave.Today.DirectFeedsTotal = 1 
   1. Clean profile 1.46 (22.12..22)
   2. Add 1 external feed via `Settings->Brave News`
   3. Open local state file
   4. Ensured that the metrics `Brave.Today.DirectFeedsTotal` response value is `1`

IMG_0311

Brave.Today.DirectFeedsTotal = 2 
   1. Clean profile 1.46 (22.12..22)
   2. Add 2 external feed via `Settings->Brave News`
   3. Open local state file
   4. Ensured that the metrics `Brave.Today.DirectFeedsTotal` response value is `2`

IMG_0314

Brave.Today.DirectFeedsTotal = 3 
   1. Clean profile 1.46 (22.12..22)
   2. Add 3 external feed via `Settings->Brave News`
   3. Open local state file
   4. Ensured that the metrics `Brave.Today.DirectFeedsTotal` response value is `3`

IMG_0313

Brave.Today.DirectFeedsTotal = 4 
   1. Clean profile 1.46 (22.12..22)
   2. Add 4 external feed via `Settings->Brave News`
   3. Open local state file
   4. Ensured that the metrics `Brave.Today.DirectFeedsTotal` response value is `4`

IMG_0315

Additional testing

  • Confirmed that when P3A switch is disabled at client side then the P3A data will not be send to the server , the P3A host p3a-json.brave.com will not be shown over the network via Charles proxy
  • Confirmed that all the metrics data is sent over the network to the endpoint https://p3a-json.brave.com
  • Confirmed that the metrics data is sent in the readable json format
  • Confirmed that channel, version, metric_name, country_code, metric_value, platform details are shown correctly in the json format
  • Confirmed that the flag is set to true in local state file when the metrics response value is sent to the server
  • Confirmed that the timestam is recorded when the metrics response value is sent to the server
flag=true, and timestamp value
Example Example Example Example Example
IMG_3266 IMG_3269 IMG_3271 IMG_3273 IMG_3274
On profileload the default metrics values are sent to the server
Example Example Example Example Example
Screenshot 2022-12-08 at 9 45 17 PM Screenshot 2022-12-08 at 9 37 16 PM Screenshot 2022-12-08 at 9 36 44 PM Screenshot 2022-12-08 at 9 13 30 PM Screenshot 2022-12-08 at 9 12 56 PM

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Assignees

@kylehickinson kylehickinson

Labels
enhancement Epic: Analytics QA Pass - iPhone QA/Yes
Projects
None yet
Milestone
1.46
Development

Successfully merging a pull request may close this issue.

Fix #6406: Add answers for basic P3A questions brave/brave-ios
3 participants
@kylehickinson @iccub @GeetaSarvadnya