Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fine-grained snapshot diffs #129

Merged
merged 8 commits into from
Jul 15, 2021
Merged

Fine-grained snapshot diffs #129

merged 8 commits into from
Jul 15, 2021

Conversation

Shillaker
Copy link
Collaborator

@Shillaker Shillaker commented Jul 14, 2021

Previously snapshot diffs were only down to a page-level granularity, which meant that distributed threads modifying a shared array would trample on each other (as they would all have their own modified version of the same page).

This PR adds a couple of steps to the existing snapshot diff process:

  • Push the snapshot to worker hosts before thread execution
  • Map thread memory from the snapshot
  • Reset dirty page checks
  • Execute the thread
  • Get dirty pages
  • Perform a bytewise comparison of the thread's memory with the original snapshot
  • Push the bytewise diffs back to the original host (instead of the page-wise diffs)

Changes:

  • Add logic for working out bytewise diffs based on dirty pages
  • Encapsulate all the batch-level variables in an executor inside an ExeutorTask class (instead of a mix of a std::pair and class members on the Executor)
  • Move logic related to applying diffs into the SnapshotData class
  • Share test code via SnapshotTestFixture
  • Check for zero-size snapshots (previously missing)

@Shillaker Shillaker self-assigned this Jul 14, 2021
@Shillaker Shillaker marked this pull request as ready for review July 14, 2021 15:32
Copy link
Collaborator

@csegarragonz csegarragonz left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM, very minor couple things.

include/faabric/scheduler/Scheduler.h Outdated Show resolved Hide resolved
src/scheduler/Executor.cpp Show resolved Hide resolved
src/snapshot/SnapshotRegistry.cpp Show resolved Hide resolved
@Shillaker Shillaker merged commit 7872298 into master Jul 15, 2021
@Shillaker Shillaker deleted the bitwise-snaps branch July 15, 2021 09:16
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants