[C#] FASTER v2 release #563
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
* various changes from indexing work - Add BeginAddress and EndAddress to scan iterators - Allow null deltaFileDevice in HybridLogCheckpointInfo.Recover - Rename FoldOverSnapshot to useFoldOverCheckpoint - Make TakeHybridLogCheckpoint overload call through to overload to avoid duplication - Add default values to new scanDelta and recoverTo arguments - Add missing checkpointManager.OnRecovery call to IntenralRecoverAsync - Fix some comments * Fix shortened name in test to work on Linux
* Add support for record-expiration management to RMW and I(Advanced)Functions * Update Remote to RMW changes for expiration support * Update SpanByteFunctionsForServer.cs * Update SpanByteFunctionsForServer.cs * Update ServerKVFunctions.cs * Added SpanByteAdvancedFunctions * moving remote to IAdvFun * LogDir semantics - minor update Co-authored-by: Badrish Chandramouli <badrishc@microsoft.com>
* Add support for record-expiration management to RMW and I(Advanced)Functions * Update Remote to RMW changes for expiration support * Update SpanByteFunctionsForServer.cs * Update SpanByteFunctionsForServer.cs * Update ServerKVFunctions.cs * Added SpanByteAdvancedFunctions * moving remote to IAdvFun * LogDir semantics - minor update * Added PCU * fixed pcu logic * Fixing testcase so that RMW does not expect old address. * Fix PCU bugs * Extend PostCopyUpdate to IFunctions; fix a few comments in remote Co-authored-by: TedHartMS <15467143+TedHartMS@users.noreply.github.com>
* Merge IAdvancedFunctions into IFunctions * Add SupportsPostOperations and Post(IU|CU|SD); expand "ref RecordInfo recordInfo, long address" on Functions * Add RecordMetadata to RMWCompletionCallback params; improve ReadAddressTests * Add tests of Upsert with Input
* RecordInfo v2 * fix version jump in CPR. * Updated misc stuff * nit * misc updates to SpanByte * Remove IntExclLocker * Add Try APIs to record locking * cleanup
* Add overloads of Upsert that take "ref Output" and one that takes "out RecordMetadata"; Fix RMW setting pendingContext.RecordInfo/LA; add locking of PostCopyUpdater, PostInitalUpdater, PostSingleWriter; * nits * Reorganize ClientSession IntenralFasterSession to group functions for a given area (Read, Upsert, RMW, etc.) together * Fix locking on Post* calls * Fixes for Post* locking * remove net461 and netcoreapp2.1; add net6.0 in <Choose> to work on later in VS2022 * Update to new RecordInfo locking calls * Update to new IFunctions Locking signatures and RecordInfo implementation * Add post operations to Faster.benchmark
* Removed WAIT_PENDING * * Use only 1 bit for newVersion in RecordInfo, for CPR. Semantics are that sessions in v+1 will create records in fuzzy region with this bit set. Recovery only scans fuzzy region to elide such records. * Add 1 Filler bit for future use with fillers in records * Increase read lock bits to 6 (64 parallel readers) * Use dirty bit for incremental checkpoints * fix ycsb to not delete data file if we are using --recover * nits * Fix epoch re-entry violation * Move to long versions, for KV checkpoints. * Shuffle around stuff for better cache alignment in benchmark. * Updates based on review
* Checkpoint work * Add notion of manual commit * fix * Recovery of commit cookie * alternative design of log-cookie * Implementation improvement * code review comments * Cleanup + testing * Add management utility for LogCommitManager * Handle race on manual commit code path where not the latest tail is written out * Rough sketch of new commit code path * save work * Prototype now works * fix to truncate log * Add condition to only scan log in fastCommitMode * Rough draft to allow for 1. elided commit record when fastCommit turned off and 2. "loose" commits * checkpoint changes * fix bug with readonly log allocator initialization * disable fast commit temporarily for CI * fix MLSD and mem device fast commit behavior * fix AzureStorageDevice exception type and remove self-termination on failure * test case on concurrent inserts * nits and test fixes * Re-enable epoch protection on CommitInternal * Try to identify issue on CI * fix Linux bug? * Remove debugging * Ensure that full flush list error callbacks do not break FlushedUntilAddress tracking * Fix unclean shutdown of stress test * update device log test to scan until tail instead * change error code used for full flush list to better distinguish from others * nits * Minor updates. Co-authored-by: Badrish Chandramouli <badrishc@microsoft.com>
* Checkin misc changes, removed overWrite commits * simplify OnRecovery logic
… outside table loop.
…ful probe per thread. Adjust table size based on number of machine processors.
* Use GC.AllocateArray where possible * update benchmark as well.
…onditionals from all .csprojs now that we've moved to VS 2022 (#687) - add Status.Expired - update FASTER.client's copies of Status(Code).cs - Replace a TODO with an explanatory comment
* Add hook for failed CAS during SingleWriter. Still need to support other failed CAS scenarios. * Fix remote sln * Implement Dispose() functions - (MemoryLogCompactionTests fails at this time) * Fix MemoryLogCompactionTests; move DisposeDeserializedFromDisk up to ICPRFC * fix remote build Co-authored-by: TedHartMS <15467143+TedHartMS@users.noreply.github.com>
badrishc
changed the title
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
FASTER C# will soon be releasing a v2, that will introduce a number of improvements and features. Applications will have some breaking changes, but should not be too much if you are not using
IFunctionsdirectly (we recommend using base classes for functions, see samples for details).List of major changes
Changes to
IFunctionsSingleWriterprovides reason for writing to tail. AddedPostSingleWriterfor work done after copy to tail, during various operations: [C#] Add WriteReason parameter to SingleWriter #630IAdvancedFunctions-- merge capability intoIFunctionsIFunctions, record-level locking is now a core FASTER functionality. It can still be disabled at store level.Advanced operations
Recovery
Log compaction
BeginAddressvariable in memoryShiftBeginAddressbehavior: only shifts the (logical)BeginAddressvariable in memory, and does not actually truncate physical log on disk, unless an option is specified (destructive operation)store.Log.TruncateAPI for physical log truncation until currentBeginAddress(destructive operation).BeginAddressafter taking a checkpoint. This is the only safe way to truncate the log on disk without data loss.FasterLog v2
Other changes
FasterKV<K,V,I,O,C>Features planned for v2.1
Secondary Indexing (SI) Framework for FASTER
Hash Value Index as example SI
{ age > 30 })