Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Optimize XidtoUID map used by live and bulk loader #2998

Merged
merged 11 commits into from
Feb 11, 2019
Merged

Optimize XidtoUID map used by live and bulk loader #2998

merged 11 commits into from
Feb 11, 2019

Commits on Feb 9, 2019

  1. Work on optimizing XidToUid map.

    @manishrjain
    manishrjain committed Feb 9, 2019
    Configuration menu
    Copy the full SHA
    a3ecbb0 View commit details
    Browse the repository at this point in the history

  2. Add the test and benchmark for xid to uid map

    @manishrjain
    manishrjain committed Feb 9, 2019
    Configuration menu
    Copy the full SHA
    febc181 View commit details
    Browse the repository at this point in the history

Commits on Feb 10, 2019

  1. Working code with decreased memory usage. Includes a new BumpUp API.

    @manishrjain
    manishrjain committed Feb 10, 2019
    Configuration menu
    Copy the full SHA
    d175657 View commit details
    Browse the repository at this point in the history

  2. Working live loader, which can optionally just keep all the mapping i… 

    …n memory.
    @manishrjain
    manishrjain committed Feb 10, 2019
    Configuration menu
    Copy the full SHA
    b85b81d View commit details
    Browse the repository at this point in the history

  3. Changes found while running

    @manishrjain
    manishrjain committed Feb 10, 2019
    Configuration menu
    Copy the full SHA
    90d7154 View commit details
    Browse the repository at this point in the history

  4. Adding shards back to XidMap speed up operations by a huge factor. Be… 

    …nchmark shows each allocation is 300ns.
    @manishrjain
    manishrjain committed Feb 10, 2019
    Configuration menu
    Copy the full SHA
    af49090 View commit details
    Browse the repository at this point in the history

  5. Make BumpTo much faster by calling Zero directly, instead of looping … 

    …through the newRanges channel.
    @manishrjain
    manishrjain committed Feb 10, 2019
    Configuration menu
    Copy the full SHA
    a698ef5 View commit details
    Browse the repository at this point in the history

  6. Reassign all shards after bumping up to a UID.

    @manishrjain
    manishrjain committed Feb 10, 2019
    Configuration menu
    Copy the full SHA
    21feaa2 View commit details
    Browse the repository at this point in the history

  7. Improve how BumpTo() happens by using a maxSeenUid variable.

    @manishrjain
    manishrjain committed Feb 10, 2019
    Configuration menu
    Copy the full SHA
    0c3a233 View commit details
    Browse the repository at this point in the history

Commits on Feb 11, 2019

  1. Martin's review

    @manishrjain
    manishrjain committed Feb 11, 2019
    Configuration menu
    Copy the full SHA
    071826d View commit details
    Browse the repository at this point in the history

  2. Martin review

    @manishrjain
    manishrjain committed Feb 11, 2019
    Configuration menu
    Copy the full SHA
    7825041 View commit details
    Browse the repository at this point in the history