Fasterlog lowmem #178
Merged
Fasterlog lowmem #178
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Contributor
badrishc
commented
Sep 26, 2019
badrishc
commented
- Adding support for low memory footprint (4 pages)
- Added support for odd-sized payloads in presence of holes in log
- Fixed concurrency issue that occurs with low num of pages
- Improved max throughput by eliminating a 10ms sleep in BlockAllocate
- Misc cleanup of logic to track flush and close addresses in log
badrishc
added a commit
that referenced
this pull request
Oct 30, 2019
* Initial checkin * Cleaned up epochs, improved fine grain scalability. * Added commit and recovery support. * Added TryAppend so users can implement log throttling. * Fasterlog lowmem (#178) Adding support for low memory footprint (4 pages) Added support for odd-sized payloads in presence of holes in log Fixed concurrency issue that occurs with low num of pages Improved max throughput by eliminating a 10ms sleep in BlockAllocate Misc cleanup of logic to track flush and close addresses in log * Fasterlog TryAppend (#179) Adding truly non-blocking TryAppend functionality. See sample for how this is used. * Fasterlog async (#180) * Added support for TryAppend. Removed List-based batch support. * Added non-blocking TryAppend * Added span variant * Fix definition of SecondChanceFraction for read cache, to be 1 - MutableFraction of the log. * Added async FlushAndCommit * Added batched version by separating out in-memory append and wait for commit - gives better perf as the first operation is usually sync * Tweak async sample to get back to 2GB/sec * Other updates: 1) Allocations can handle thousands of parallel tasks 2) Removed concept of negative address - allocations are always over available pages 3) Improved scan interface to allow user memory pooling 4) Exposed commit task 5) Cleaned up sample * Added check for entry fitting on single page * Added batch interface (sync and async) to log append. * Added tailing iterator WaitAsync to wait for iteration to proceed. * Convert Span to ReadOnlySpan for appends * Added MemoryPool/IMemoryOwner variant of iterator * Updated way to pin pooled memory * Update azure-pipelines.yml * Support minimum buffer size of just 1 page! * Actually checking in support for 1 page in memory, added initial draft of disposing task * Added a test * Improved sample, changed GetMemory to use byte[] instead of Span<byte> * Update next address of iterator if GetNext fails early. * Added random read functionality (ReadAsync) for FasterLog. Moved GetMemory to FasterLogSettings instead of Scan. Speed up TruncateUntil. Updated nuspec. * Ensure begin addresses commit if needed, even when tail addresses do not change. Added CommittedBeginAddress metric. * changed test project target * Updated random read example * Use TrySetResult instead of SetResult, since log closure moves the task to completed state. * Added simple version/checksum to commit info. * Added opt-in support for per-entry 8-byte checksum (xor) in header of entry. * Fixing issue with async enqueue. * Fixed testcase since thread abort not supported on some platforms. * Fixing concurrency issue with contiguous partial flush requests. Removed spin-wait for adjacent flush completion. * Fasterlog exceptions (#189) * Added storage exception handling, connecting to tasks. * Cleanup of error handling, control when exception is bubbled up to user. * Added yield in NeedToWait * Improved iterator support in case of exception * Added async iterator support * Added support for persistent/recoverable named iterators.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.