Domains not displayed on my ledger shows up in payment receipt as payment recipients

[da2x]

Test Plan

1. run with LEDGER_VERBOSE=true
2. visit some sites, etc.
3. go to about:preferences#payments
4. some time later you'll see something that looks like the output below (sorry, it doesn't look "nice" in a variable-width font).
5. verify that things look "right" according to https://github.com/brave/browser-laptop/blob/issue-6531/app/ledger.js#L22 et. seq.

    ledger client run: clientP=true delayTime=582412
              publisher  blockedP   stickyP  verified  excluded eligibleP  visibleP  contribP  duration    visits
            wikihow.com     false      true      true     false      true      true      true    117448        28
            archive.org     false     false      true     false      true      true      true     22729         8
          wikipedia.org     false     false     false     false      true      true      true     13709        10
           coindesk.com     false      true      true     false      true      true      true       841        19
           usesthis.com     false     false      true     false      true      true      true       172        10
       scotthelme.co.uk     false     false      true     false      true      true      true       130         7
             github.com     false     false     false      true      true     false     false       565         3
           facebook.com     false     false     false      true      true     false     false       546         1
             google.com     false     false     false      true      true     false     false        97         2

Original issue description

I screenshoted my edger five minutes before the payment was processed. My ledger is also not cleared out immediately because of bug #5371, so I was able to capture a screenshot from about a minute after payment had been processed (list was identical to the previous screenshots, as expected).

Upon more careful inspection of my payment history recipt, I found six domains that don’t show up in on the ledger in about:config#payments! These snuck in on the list uninvited, as far as I’m concerned.

Every other domain on the payment receipt matches domains shown in the top positions on the about:config#payments page

These are the six domains (in case the specific domain names matter):

• blackducksoftware.com
• browsix.org
• netonnet.se
• prisjakt.no
• redd.it
• suckless.org

I have visited these websites, probably the week before Christmas.

I have captured the ledger-publisher.json, ledger-scores.json, ledger-state.json, and ledger-synopsis.json files for analysis. I won’t upload them here, but can send them to a @brave.com email address upon request. (I guess the files grant access to my wallet? Anyhow, I don’t want to make them publicly available.)

Brave 0.12.15 on Fedora Linux.

cc/@mrose17

[da2x]

I just noticed the opposite problem as well! My #3 site as reported by about:config#payments (19 views, 32m 17s) is nowhere to be seen in the payment receipt. Four out of my top 10 sites are actually missing.

[mrose17]

what is in ledger-synopsis.json ? that is the file that records the focus-% for your browsing? are those sites there?

[da2x]

@mrose17, yes. However, they only show as recorded having one visit. I thought the absolute minimum was two visits? For example, blackducksoftware.com is recorded in ledger-synopsis.json as following:

"blackducksoftware.com": {
  "visits": 1,
  "duration": 40423,
  "scores": {
    "concave": 2.3326962852564166,
    "visits": 1
  },
  "window": [
    {
      "timestamp": 1481484765415,
      "visits": 1,
      "duration": 40423,
      "scores": {
        "concave": 2.3326962852564166,
        "visits": 1
} } ] }

[mrose17]

@da2x - i've tried to reproduce this without luck. if you were to look at your synopsis now, what are you seeing that looks odd? maybe we can come at this from a different angle... thanks!

[da2x]

I have no idea what triggered this. I’ve used Brave as my daily driver so I haven’t got the faintest.

Shouldn’t the list in about:preferences#payments have reset by now? I’m still seeing the sites I visited last month. I thought he list reset after each payment to start counting the new month? The list updates as I visit new sites, but the old stuff is still there.

If the list should have been reset at the end of the payment cycle (which I believe is how it’s suppose to work), then figuring out why that didn’t happen this month should solve the mystery of the invisible websites as well on that very same list.

[mrose17]

the list shows you the last 30 days of browsing attention, regardless of payments. so if you go a site on day 1 and never go back, it will show up on days 1-30, and on day 31 it will go away. perhaps there's a bug in that, or some caching interaction...

if you do

grep timestamp ledger-synopsis.json | sort

do you see anything less than 1481838692025 ?

i see a couple of entries in my synopsis and restarting the browser purges them automatically. let me try to address that more frequently.

[da2x]

the list shows you the last 30 days of browsing attention, regardless of payments. so if you go a site on day 1 and never go back, it will show up on days 1-30, and on day 31 it will go away. perhaps there's a bug in that, or some caching interaction...

Yeah, that isn’t happening. The list reset three days ago. I use the browser daily and visit a bunch of sites. Yet, the list only contained about two–three sites; but has since started growing again. I’m fairly sure this has always been how it has behaved.

[mrose17]

by "reset" you mean it went empty? fascinating. i'm not sure how to reconcile this with the behavior i'm seeing. i have a "test browser" that contributes every 3 days (instead of 30) and in watching the ledger-synopsis.json file, entries disappear 30 days as expected. i've never seen everything disappear (except when messing with the system clock)...

[da2x]

by "reset" you mean it went empty?

Indeed. This has happen every single month since the initial beta release. It happens within a few days after payment is processed. My list drops from several hundred domains to like 5 with only 2–5 visits and a few seconds recorded on each. So it appears to me to have been reset. Given my use, it should contain a few hundred domains constantly.

(except when messing with the system clock)...

I doubt that is the issue for me.

[mrose17]

well that is just fascinating. if you have the patience, let's try this: first, click on "Advanced Settings..." in about:preferences#payments and see what the min page time and min publisher visit numbers are. i'm interested in finding out what you have there (maybe that's a cause)

next, if you could save a copy of your ledger-synopsis.json file just before the payment and just after (hopefully with a lot of entries and then a little), we can do a little analysis (i'll just ask you to grep a few things).

will that work?

thanks!

[da2x]

"Advanced Settings..."

See #6241.

"Contribution due date is ⁨in 10 days⁩."

I’ve set a reminder to do this manually and a cron script to capture it automatically. I guess I’ll have the relevant data in ~two weeks time.

[mrose17]

many thanks!

[da2x]

Hm. I saved states before and after payment was processed. (However, it’s been two hours since I got the notification but I still don’t have a receipt in the payment history tab.)

What usually happens is that the list will be emptied within the next couple of days. I’ll let you know as soon as it happens, @mrose17. What commands do you want me to run at that time?

[da2x]

The reciept just showed up.

5 of 40 domains on the payment reciept don’t show up on about:preferences#payments. This time the domains all have one thing in common: They’re domains with only one page. I’m unlikely to have visited them more than once as there is very little content or neeed to see these pages. I beleive that about:preferences#payments respects my “Minimum visits for publisher relevancy” option of at least 2 visits, but that option isn’t respected when payments are due.

Here is an example entry for one of those 5 domains from my ledger-synopsis.json

"example.com": {
  "visits": 1,
  "duration": 52774,
  "scores": {
    "concave": 2.7870155406452214,
    "visits": 1
  },
  "window": [
    {
      "timestamp": 1484786688148,
      "visits": 1,
      "duration": 52774,
      "scores": {
        "concave": 2.7870155406452214,
        "visits": 1
      }
    }
  ],
  "protocol": "http:",
  "options": {}
}

[mrose17]

thanks! that is very, very helpful... i think i can figure out what's going on now!

[mrose17]

@da2x - this code is now in the monster branch, along with some other fixes (that should address your issue). please let me know what you think going forward... thanks!