Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Round robin ads delivery #5945

Closed
jsecretan opened this issue Sep 9, 2019 · 3 comments · Fixed by brave/brave-core#3513
Closed

Round robin ads delivery #5945

jsecretan opened this issue Sep 9, 2019 · 3 comments · Fixed by brave/brave-core#3513

Comments

@jsecretan
Copy link

Description

Right now, the ad libraries will take a look at ads eligible to serve and will randomly select eligible ones. However, sometimes this results in seeing the same ad multiple times in a row. We should try to round robin deliver eligible ads instead so they are seen more evenly by the user.

So for all ads which have not yet reached frequency cap, we should first choose ads that have been seen the least over the observation window. The new algorithm would look as follows.

  1. Find all ads eligible to serve given daily, lifetime, and hourly limits.
  2. Among the ads in 1, count the number of times each has been seen over the observation window. Take that are tied for being seen the fewest number of times.
  3. Choose randomly among the set in 2.
@btlechowski
Copy link

@tmancey Is there a way of telling whether duplicate ad is shown after round robin? I contrast to simply showing a duplicate ad?

@btlechowski
Copy link

btlechowski commented Nov 27, 2019

@tmancey Thank you for the call, it was really helpful. Based on it I wrote a detailed test plan.

Setting up ads catalog

  1. Connect to US VPN
  2. Open https://ads-serve.brave.com/v1/catalog and copy the catalog
  3. Format the catalog with json formatter
  4. Search for Personal Finance set with at least 4 ads(or other creativeSet that has more than 4 ads)
  5. Change "perDay" to 30 for that set
    Change "totalMax" to 30
    Change "dailyCap" to 30
  6. Delete all the ads in the segment except the 4
  7. Change the payload.body and payload.title of the first ad to Test 1, do the same for other ads in the set but increment the number
  8. Save the catalog
    image

Setting Charles proxy

  1. Open Tools->Rewrite...
  2. Enable Enable Rewrite
  3. Click Add on the bottom left
  4. Change name to ads catalog
  5. Under Location click Add
    Protocol: https
    Host: ads-serve.brave.com
    Path: v1/catalog
  6. Under Action click Add
    Type : Body
    Where : Response
    Replace : paste the whole catalog
  7. Open https://ads-serve.brave.com/v1/catalog and check if your modified catalog is served
    image

Test Case 1 - one set

  1. Set up catalog according to above instructions
  2. If there are more than one Personal Finance segments then rename the rest to Technology & Computing and keep the segment that you changed the ads in.
  3. Set Charles proxy according to above instructions
  4. Clean install
  5. Run Brave with --enable-logging=stderr --vmodule=brave_ads=3
  6. Enable Rewards
  7. Change ads per hour to 5
  8. Check if proper ads catalog has been downloaded to Default/ads_service/catalog.json
  9. Open https://www.investopedia.com/articles/personal-finance/100516/setting-financial-goals/ or other page that classifies as Personal Finance
  10. Watch an ad every 15 minutes (important as there is limit on every ad to be shown once per hour)

Note: There are 4 ads in the set for Personal Finance segment. 4 ads total for Personal Finance segment

Expected behavior
Duplicate Ad is not shown till all other ads are not shown for the same segment type (Personal Finance)

Example for 4 ads in the segment:
no duplicates within first 4 ads
Test 4
Test 1
Test 2
Test 3

no duplicates within next 4 ads
Test 2
Test 4
Test 1
Test 3

no duplicates within next 4 ads
Test 1
Test 3
Test 4
Test 2

Test Case 2 - two sets of the same segment type

  1. Set up catalog according to above instructions. Use Personal Finance segment. The number of ads in the set must be 2.
  2. Set up catalog according to above instructions. Use a different Personal Finance segment. Rename the ads differently. The number of ads in the set must be 2.
  3. If there are more than one Personal Finance segments then rename the rest to Technology & Computing and keep the segment that you changed the ads in.
  4. Set Charles proxy according to above instructions
  5. Clean install
  6. Run Brave with --enable-logging=stderr --vmodule=brave_ads=3
  7. Enable Rewards
  8. Change ads per hour to 5
  9. Check if proper ads catalog has been downloaded to Default/ads_service/catalog.json
  10. Open https://www.investopedia.com/articles/personal-finance/100516/setting-financial-goals/ or other page that classifies as Personal Finance
  11. Watch an ad every 15 minutes (important as there is limit on every ad to be shown once per hour)

Note: There are 2 ads per set for Personal Finance segment. There are 2 sets, so 4 ads total for Personal Finance segment

Expected behavior
Duplicate Ad is not shown till all other ads are not shown for the same segment type (Personal Finance)

Example for 4 ads in the segment:
no duplicates within first 4 ads
Test 1
Test 1 Set 2
Test 2
Test 2 Set 2

no duplicates within next 4 ads
Test 2
Test 2 Set 2
Test 1
Test 1 Set 2

no duplicates within next 4 ads
Test 1
Test 1 Set 2
Test 2 Set 2
Test 2

Test Case 3 - one ad for the segment

  1. Set up catalog according to above instructions. Use Personal Finance segment. The number of ads in the set must be 1.
  2. If there are more than one Personal Finance segments then rename the rest to Technology & Computing and keep the segment that you changed the ads in.
  3. Set Charles proxy according to above instructions
  4. Clean install
  5. Run Brave with --enable-logging=stderr --vmodule=brave_ads=3
  6. Enable Rewards
  7. Change ads per hour to 5
  8. Check if proper ads catalog has been downloaded to Default/ads_service/catalog.json
  9. Open https://www.investopedia.com/articles/personal-finance/100516/setting-financial-goals/ or other page that classifies as Personal Finance
  10. Watch an ad every 60 minutes (important as there is limit on every ad to be shown once per hour)

Note: There is an ad in the set for Personal Finance segment. 1 ad total for Personal Finance segment

Expected behavior
The ad is shown every hour.

Test Case 4 - two sets of different segment types

Note: the point of this TC is to check if round robins from different segments interfere with each other

  1. Set up catalog according to above instructions. Use Personal Finance segment. The number of ads in the set must be 2. Name them PF T1 and PF T2
  2. Set up catalog according to above instructions. Use a different Arts & Entertainment segment. Rename the ads differently. The number of ads in the set must be 2. Name them A&E T1 and A&E T2
  3. If there are more than one Personal Finance/Arts & Entertainment segments then rename the rest to Technology & Computing and keep the segment that you changed the ads in.
  4. Set Charles proxy according to above instructions
  5. Clean install
  6. Run Brave with --enable-logging=stderr --vmodule=brave_ads=3
  7. Enable Rewards
  8. Change ads per hour to 5
  9. Check if proper ads catalog has been downloaded to Default/ads_service/catalog.json
  10. Open https://www.investopedia.com/articles/personal-finance/100516/setting-financial-goals/ or other page that classifies as Personal Finance
  11. Watch an ad after 15 minutes -> Expected to get an ad from Personal Finance segment
  12. Watch an ad after 15 minutes -> Expected to get the other ad from Personal Finance segment
  13. Open https://www.imdb.com or other page that classifies as Arts & Entertainment, make sure that Arts & Entertainment wins over Personal Finance
  14. Watch an ad after 15 minutes -> Expected to get an ad from Arts & Entertainment segment
  15. Open https://www.investopedia.com/articles/personal-finance/100516/setting-financial-goals/ or other page that classifies as Personal Finance
  16. Watch an ad after 15 minutes -> Expected to get an ad from Personal Finance segment
  17. Open https://www.imdb.com or other page that classifies as Arts & Entertainment
  18. Watch an ad after 15 minutes -> Expected to get the other ad from Arts & Entertainment segment
  19. Open https://www.investopedia.com/articles/personal-finance/100516/setting-financial-goals/ or other page that classifies as Personal Finance
  20. Watch an ad after 15 minutes -> Expected to get the other ad from Personal Finance segment
  21. Watch an ad after 15 minutes -> Expected to get an ad from Personal Finance segment
  22. Open https://www.imdb.com or other page that classifies as Arts & Entertainment
  23. Watch an ad after 15 minutes -> Expected to get an ad from Arts & Entertainment segment
  24. Watch an ad after 15 minutes -> Expected to get the other ad from Arts & Entertainment segment

Expected result:
round robins from different segments do not interfere with each other

Example:
PF T2
PF T1
A&E T2
PF T1
A&E T1
PF T2
PF T1
A&E T2
A&E T1

Now divide by segments and search for duplicates within pairs:
Personal Finance
PF T2
PF T1

PF T1
PF T2

PF T1

Arts & Entertainment
A&E T2
A&E T1

A&E T2
A&E T1

@btlechowski
Copy link

btlechowski commented Nov 27, 2019

Verification passed on

Brave 1.1.11 Chromium: 78.0.3904.108 (Official Build) beta (64-bit)
Revision 4b26898a39ee037623a72fcfb77279fce0e7d648-refs/branch-heads/3904@{#889}
OS Ubuntu 18.04 LTS

Verified test plan from #5945 (comment)

Verification passed on

Brave 1.1.19 Chromium: 79.0.3945.70 (Official Build) (64-bit)
Revision 65868db27916cebde51fc2465c9b34761e6383e2-refs/branch-heads/3945@{#869}
OS Windows 7 Service Pack 1 (Build 7601.24530)

Verified test plan from #5945 (comment)

Verification PASSED on macOS 10.15.1 x64 using the following build:

Brave 1.1.19 Chromium: 79.0.3945.70 (Official Build) (64-bit)
Revision 65868db27916cebde51fc2465c9b34761e6383e2-refs/branch-heads/3945@{#869}
OS macOS Version 10.15.1 (Build 19B88)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging a pull request may close this issue.

5 participants