Sync : missing bookmarks from import on other machine

[alexwykoff]

• Did you search for similar issues before submitting this one?
  Yes

• Describe the issue you encountered:
  When importing a reasonably large number of bookmarks (>6000) the other machine which receives the sync does not have many of the bookmarks even after waiting for 20 minutes or so.

• Platform (Win7, 8, 10? macOS? Linux distro?):
  macOS, Windows 10

• Brave Version (revision SHA):
  0.17.3 (02a7878)

• Steps to reproduce:

  1. with clean profiles, sync two laptops with 0.17.3
  2. import a large number of bookmarks with nested upon nested folders
  3. check bookmarks in nested folders on device 2, notice they are missing in many cases

• Actual result:
  Bookmark order is scrambled in places an many are missing.
  Numerous (thousands) of similar errors appeared on the console indicating some sort of failure with AWS.

• Expected result:
  the structure and contents of the first laptop should be matched by the second laptop

• Will the steps above reproduce in a fresh profile? If not what other info can be added?
  Yes

• Is this an issue in the currently released version?
  Unsure due to prior hierarchy bug.

• Can this issue be consistently reproduced?
  Yes

• Extra QA steps:
  1.
  2.
  3.

• Screenshot if needed:

• Any related issues:

[ayumi]

Was able to repro:

• Mac OS
• 07166c7973563be8c86a1204a80f508f28245cc5
• npm run start
• sync staging
• 10K bookmarks

[ayumi]

This was fixed by #122. To track the pause/resume uploads issue I've opened #125.

[alexwykoff]

still noticing this on 0.17.16

[bsclifton]

@ayumi can you check this out?

[ayumi]

I found that:

1. if both pyramids are online concurrently, there might be a delay in what pyramid 1 sees from the data store (S3). this can cause bookmarks to fail to download.
2. if pyramid 0 syncs, then you wait 10 minutes, then pyramid 1 syncs, there is much less loss of bookmarks.

MacOS, browser-laptop 0.17.x

AWS S3 "eventual consistency"

[ayumi]

Ideas to make this better:

1. Browser-clients should reupload pending sync records after time t (currently reupload happens on browser restart).
2. Browser-clients concurrently online should detect S3 inconsistency (maybe by periodically measuring total record count) and redownload all records when that happens.

[bsclifton]

Original issue (as described) has been solved- I'm going to mark as closed. Steps that should always work:

1. clean profiles, sync 2 devices (pyramid 0, pyramid 1)
2. turn pyramid 1 off
3. import into pyramid 0
4. wait 15 mins
5. turn pyramid 1 on
6. wait 1-2 minutes
7. compare exported bookmark files

Above comments captured in a new issue #139