[C#] FasterLog v2: Fix FasterLog missed final commit #597
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
tli2
approved these changes
Nov 18, 2021
badrishc
added a commit
that referenced
this pull request
Apr 13, 2022
* various changes from indexing work (#562) * various changes from indexing work - Add BeginAddress and EndAddress to scan iterators - Allow null deltaFileDevice in HybridLogCheckpointInfo.Recover - Rename FoldOverSnapshot to useFoldOverCheckpoint - Make TakeHybridLogCheckpoint overload call through to overload to avoid duplication - Add default values to new scanDelta and recoverTo arguments - Add missing checkpointManager.OnRecovery call to IntenralRecoverAsync - Fix some comments * Fix shortened name in test to work on Linux * [C#] Record-expiration support in RMW (#545) * Add support for record-expiration management to RMW and I(Advanced)Functions * Update Remote to RMW changes for expiration support * moving remote to IAdvFun * LogDir semantics - minor update Co-authored-by: Badrish Chandramouli <badrishc@microsoft.com> * [C#] Added PostCopyUpdater to IAdvancedFunctions (#561) * Add support for record-expiration management to RMW and I(Advanced)Functions * Update Remote to RMW changes for expiration support * Added PCU * Fixing testcase so that RMW does not expect old address. * Fix PCU bugs * Extend PostCopyUpdate to IFunctions; fix a few comments in remote Co-authored-by: TedHartMS <15467143+TedHartMS@users.noreply.github.com> * [C#] Merge IAdvancedFunctions into IFunctions (#565) * Merge IAdvancedFunctions into IFunctions * Add SupportsPostOperations and Post(IU|CU|SD); expand "ref RecordInfo recordInfo, long address" on Functions * Add RecordMetadata to RMWCompletionCallback params; improve ReadAddressTests * Add tests of Upsert with Input * [C#] RecordInfo v2 (#577) * RecordInfo v2 * fix version jump in CPR. * Updated misc stuff * misc updates to SpanByte * Remove IntExclLocker * Add Try APIs to record locking * Remove FASTERLegacy (#581) * Add Output parameter to Upsert (#574) * Add overloads of Upsert that take "ref Output" and one that takes "out RecordMetadata"; Fix RMW setting pendingContext.RecordInfo/LA; add locking of PostCopyUpdater, PostInitalUpdater, PostSingleWriter; * Reorganize ClientSession IntenralFasterSession to group functions for a given area (Read, Upsert, RMW, etc.) together * Fix locking on Post* calls * Fixes for Post* locking * remove net461 and netcoreapp2.1; add net6.0 in <Choose> to work on later in VS2022 * Update to new RecordInfo locking calls * Update to new IFunctions Locking signatures and RecordInfo implementation * Add post operations to Faster.benchmark * [C#] Checkpoint, Recovery, RecordInfo updates (#588) * Removed WAIT_PENDING * * Use only 1 bit for newVersion in RecordInfo, for CPR. Semantics are that sessions in v+1 will create records in fuzzy region with this bit set. Recovery only scans fuzzy region to elide such records. * Add 1 Filler bit for future use with fillers in records * Increase read lock bits to 6 (64 parallel readers) * Use dirty bit for incremental checkpoints * fix ycsb to not delete data file if we are using --recover * Fix epoch re-entry violation * Move to long versions, for KV checkpoints. * Shuffle around stuff for better cache alignment in benchmark. * Updates based on review * [C#] FasterLog Commit Optimization (v2) (#587) * Checkpoint work * Add notion of manual commit * Recovery of commit cookie * alternative design of log-cookie * Implementation improvement * code review comments * Cleanup + testing * Add management utility for LogCommitManager * Handle race on manual commit code path where not the latest tail is written out * Add condition to only scan log in fastCommitMode * Rough draft to allow for 1. elided commit record when fastCommit turned off and 2. "loose" commits * checkpoint changes * fix bug with readonly log allocator initialization * disable fast commit temporarily for CI * fix MLSD and mem device fast commit behavior * fix AzureStorageDevice exception type and remove self-termination on failure * test case on concurrent inserts * Re-enable epoch protection on CommitInternal * Ensure that full flush list error callbacks do not break FlushedUntilAddress tracking * Fix unclean shutdown of stress test * update device log test to scan until tail instead * change error code used for full flush list to better distinguish from others * Minor updates. Co-authored-by: Badrish Chandramouli <badrishc@microsoft.com> * Remove legacy local log commit manager * [C#] FasterLog v2 - misc changes, removed OverWrite commits (#593) * Checkin misc changes, removed overWrite commits * simplify OnRecovery logic * Fix build after merge * fix alignment of device commit manager * [C#] v2 pending flush list (#596) * Make PendingFlushList never error out in stressful situations. Current implementation uses lock for ~5-10% perf degradation over prior structure. * [C#] FasterLog v2: Fix FasterLog missed final commit (#597) * Fix FasterLog missed final commit. * [C#] Log compaction v2 (#598) * initial checkin * New ShiftBeginAddress behavior -- only shifts BeginAddress variable and does not truncate physical log unless option is specified. Also added Log.Truncate API for physical log truncation. Truncate log until persisted BeginAddress after taking a checkpoint. * Fix epoch protection orderings. * minor improvement to TestUtils to avoid race exception. * update Program.cs * updates from master. * [C#] V2 Custom Commit Strategy (#599) * Add FasterLogCommitStrategy * Use Bakery algorithm for latch * Documentation + tests + improved threading performance * Fix initialization error * Revert to v2 synchronization scheme * Optimization around commit strategies * Refactor CommitPolicy only for normal commits * fix missing spin wait * [C#] Fix v2 CommitPolicy Warnings (#607) * fix warnings * fix broken test * update for code consistency * Merged PR 59053: Network Stack Refactoring into FASTER This pull request contains the necessary changes for the updated network stack interface used by FASTER. It adds INetworkSender.cs and NetworkSenderBase.cs that can be used to implement custom communication protocols. Currently only TcpNetworkSender is supported. * Create .dockerignore * Added TentativeHeadAddress - we first shift this, and when all threads agree, we can do some pre-closing work such as eviction. Then we can shift the actual HeadAddress, closing pages when all threads agree on this. (#608) * Cleanup and remove dead code + reuse base provider. * Code cleanup * [C#] minor cleanup of fasterlog v2 (#611) * minor cleanup of fasterlog * [C#] Remove CheckpointType from CheckpointSettings (v2) (#614) * Remove CheckpointType from CheckpointSettings. Users have to specify checkpoint type on a per checkpoint basis. * fix build break * Fix CTT to call PostSingleWriter. * Avoid double init of FasterLog * [C#] Fixing warnings (#615) * Fixing CI warnings * [C#] Updating dependent package versions in v2 (#616) * Updating dependent package versions * nit * [C#] Improved FasterKV and FasterLog settings API for v2 (#618) * Simpler config API for v2. Move FasterLog to its own folder. * Remove purgeEarlierCommits from reader. Instead: 1. writer retries file deletes (in device commit manager) if access denied, in a loop, until timeout 2. reader (user code) retries recover async if it fails due to file deleted, until timeout * [C#] FasterLog V2 Device error handling (#612) * prototype device * prototype failure handling * Add proposed error handling code * Basic error handling tasks * Fault tolerant devices * Change to failure handler based approach instead of just throwing an exception * Remove brittle error handling code * make sure to read pages from flaky device on recovery in test * fix warnings * code review comments * See if we can enforce clean shutdown * Add extensions to enqueue SpanByte to FasterLog, and unsafely iterate without memory copy. (#621) * [C#] FasterLog v2 Add ability to explicitly terminate a log (#620) * Add ability to explicitly terminate a log * remove special 0-length return on eof * Add LogCompleted in iterator * Change comment and test * fix async iteration logic * fixes Co-authored-by: Badrish Chandramouli <badrishc@microsoft.com> * Maybe fix SVS issue with atomic reads (#622) * [C#] FasterLog v2 iteration fix (#623) * First pass of fixes * undo local change * [C#] Updated docs for FasterLog v2 (#624) * some updates * Add missing documentation * Update 40-fasterlog-basics.md Co-authored-by: Badrish Chandramouli <badrish@gmail.com> * Fix logic for directory delete timeouts. * Speed up testcases * [C#] Lockable Unsafe Context (LUC) in FasterKV v2 (WIP) (#605) * WIP on ManualFasterOperations * fixes for v2 merge * Updates to locking (still in progress) - Remove User-defined locks - Move SupportsLocking up to FasterKV level - Skip Stub and Sealed in iterators - Change PendingContext.OperationFlags from byte to ushort - Add ReadFlags.SkipCopyToTail - Add IsLocked*, TranferLocks methods to RecordInfo - Add UnsafeCompletePending* to ManualOps - Track lock counts in ManualOps and throw if Dispose() is called with locks or epoch held - Add LockInfo for ManualOps.*Lock - Implementation and testing of Faster operations with ManualOps locks - Fix ReadAddressTests to reflect improved consistency of not returning readcache addresses * Locking docs * Update 30-fasterkv-manual-locking.md * Updated doc to GH review and discussions * WIP for LockableRawContext * Existing tests run * Rename to LockableRawContext, IFasterContext * Perf workaround for LockTable.IsActive * WIP on ReadCacheEvict and MemoryPageLockEvictionScan * Add ReadCache and LockTable-transfer tests; move ReadCache tests out to their own sub-hierarchy * Test MemoryPageLockEvictionScan * Add testing for two-phase upsert/copytotail * Remove manual locking bits in Recovery; minor refactorings in Recovery.cs to reduce duplication * Add manual locking to FASTER.benchmark * More LockTable testing and fixes, mostly around ocking nonexistent key while Upsert/RMW/Deleting that key * Add UnsafeContext and make it the default for FASTER.Benchmark * Updates from code review: - IsLocked returns shared lock count (added test) - Fix location of IsIntermediate checks in RecordInfo.Lock/Seal - Remove unused IEqualityComparer in LockTableEntry * Rename Take(Full|Index|HybridLog)Checkpoint to TryInitiate(Full|Index|HybridLog)Checkpoint, so it is obvious they need to complete it * - Remove SupportsPostOperations; this is now always done - Add SingleDeleter for symmetry with Upsert - Add DisposeKey and DisposeValue * Add CopyWriter, to distinguish "maintentance" copying (CopyToTail or copy to ReadCache) from API-parameter copies * merge fixes * Remove obsolete Functions-level locking and postOps specifications * Fix Remote build Co-authored-by: Badrish Chandramouli <badrishc@microsoft.com> * fix semantic merge conflict in MemoryPageScanIterator (#632) * [C#] Add WriteReason parameter to SingleWriter (#630) * Add Functions.PostCopyWriter; update MemOnlyCache to use Post(Single|Copy)Writer * Remove (Post)CopyWriter and replace with WriteReason parameter to (Post)SingleWriter * Move SingleWriter "reason" parameter * Move ReadFlags and WriteReason to their own files; rename SupportsLocking -> DisableLocking with default 'false' * Update Remote to disableLocking name and default change * [C#] Various Fixes (#635) * Fixes: - Fixed WriteReason when CTT overrides ReadCache - make DisableLocking a readonly member - Call SetTentativeAtomic after Post* - Make readcache and comparer private with internal accessors for test - FIx intermittent failure of BasicHighLatencyDeviceTest * back out changes to move from internal -> private for FKV.readcache and .comparer * [C#] Remove StrictCPR (#636) * Removed WAIT_PENDING * StrictCPR removal WIP * Fix bug in state machine test, as it was not using unsafe context for fine grained control over the state machine execution. * Remove PendingContext.HeldLatch and heldOperation local * Revise FASTER.Benchmark to replace --noaff with --safectx and integrate this into Ycsb and SpanbyteYcsb * check epoch protection for complete pending * *UnsafeContext: AnyInstanceProtected -> ThisInstanceProtected Co-authored-by: TedHartMS <15467143+TedHartMS@users.noreply.github.com> * [C#] Support int session ID for FasterKV (#641) * Support int session ID for FasterKV * fix remote solution, minor name updates * update FasterLog API for TryEnqueue of custom header * [C#] IEntry interface for FasterLog (#639) * enqueue with entries prototype * Add IEntry tests and method variants * Address code review comments * Rename + read entries * Add async versions of consumer * Make interfaces top-level * Correct header offset in FasterLog new API * [C#] Recover returns version we recovered to (#648) * add to FasterLog SpanByte API * fix breaking tests * Fix dispose loop * Update FasterServerBase.cs * do not remove ioPendingRequests in RMWAsync if there was no disk request (#653) * [C#] [WIP] Status return code v2 (#638) * Prototype of status as struct * improving status a bit * fix build break, expose underlying (raw) status code to user * misc * Update StatusCode * Update Remote to new Status(Code); fix test cases; fix StatusCode layout; fine-tune Status(Code) names; make StatusCode internal * Update StatusCode and Status to final spec * Update Remote to new Status spec * unrelated nit Co-authored-by: TedHartMS <15467143+TedHartMS@users.noreply.github.com> * [C#] Change IFunctions 'address' parameter to UpdateInfo or ReadInfo (#652) * Add UpdateInfo and ReadInfo to IFunctions calls * Update Remote IFunctions implementations to UpdateInfo/ReadInfo changes Co-authored-by: Badrish Chandramouli <badrishc@microsoft.com> * Wire status into remote v2 (#657) Add NotFound API for status * Status naming changes and UpdateInfo structure split for v2 preview (#658) * [C#] Add Status.Record (#659) * Add Status.Record; update OperationStatus with all Advanced Status codes * Update Remote for status.Pending => .IsPending * Update nuspecs * more updates * Update azure-pipelines.yml * Update azure-pipelines.yml * revert windows-2022 in yml * Rename StatusCode.OK to Found (#662) * nit * update SpanByte with API to get pointer including metadata * Fixing ranges in OnPagesReadyToClose * [C#] Websocket bug-fixes (#663) * Fixed nits in websockets: * Clean disconnection * Support correction for large packets * Small nit for correcting message length * changed status to new FASTER status codes * Modified status codes Co-authored-by: Rohan Kadekodi <kadekodirohan@gmail.com> * Revert THA * nits * Improving cache tracker * fix NET5_0 to NET5_0_OR_GREATER * [C#] Add CompletePending[WithOutputs] to IFasterContext (#664) * Move CompletePending(WithOutputs) into IFasterContext * Skip LockEvictionObserver entirely if no LUCs active * Make Decrement interlocked; inline one accessor; revert CMakeLists.txt * [C#] Upgrade blob tests to Azurite, enable blob tests on Linux, use windows-2022 for CI (#666) * install azurite * Change testcase to make sure commit completed before dispose, it is up to user to ensure no ongoing commits at time of dispose. * [C#] UnsealPhysicalAddress fix (#665) * Fix unsealPA going below HA; return SUCCESS instead of NOTFOUND from normal reads that find an obsolete record in ITCTT * Ensure InternalLock calls require epoch protection * WIP: Spin wait on Pending Reads/RMWs until ClosedUntilAddress catches up. Still needs to handle partial page evictions. * Ensure partial ClosedUntilAddress shifts occur correctly. * [C#] Cancellation, ReadOptions, ReadFlags refactor (#668) * Add DeleteRecord and CancelOperation to *Info structures and IFunctions * Added ReadOptions * Move IFunctions methods' "ref RecordInfo" parameter to an *Info property, as it does not need to be updated * Update ReadFlags * ReadFlags fixes * Change XxxInfo.DeleteRecord/.CancelAction to XxxAction enum * Fixing Expiration tests * Add Phase to ExpirationTests * Add ReadOnly region to expiration tests * Delete and Upsert search down to HeadAddress if active LUC * [C#] Misc v2 fixes (#675) Fix misc bugs in v2 * update spanbyte * Yield thread when unable to reserve epoch entry, move iteration count outside table loop. * remove exception from overloaded lightepoch * Fix benchmark to run on .net 6 and linux. * fix another place * [C#] Improvements to LightEpoch * Use AllocateArray in LightEpoch * LightEpoch: Use two hash functions + linear probe, cache last successful probe per thread. Adjust table size based on number of machine processors. * [C#] Use GC.AllocateArray (#683) * Use GC.AllocateArray where possible * update benchmark as well. * Update buffer pool to use GC.AllocateArray * fix debug assert * Make DisposeSession protected in FasterServerBase. * Added SetBytesRead to IServerSession * - update sample .csprojs to target net6 and remove platform-related conditionals from all .csprojs now that we've moved to VS 2022 (#687) - add Status.Expired - update FASTER.client's copies of Status(Code).cs - Replace a TODO with an explanatory comment * Slight refactor of network stack. * Use GC.AllocateArray in SeaaBuffer * Another GC.AllocateArray substitution * [C#] [WIP] Wire disposal into IFunctions v2 (#685) * Add hook for failed CAS during SingleWriter. Still need to support other failed CAS scenarios. * Implement Dispose() functions - (MemoryLogCompactionTests fails at this time) * Fix MemoryLogCompactionTests; move DisposeDeserializedFromDisk up to ICPRFC Co-authored-by: TedHartMS <15467143+TedHartMS@users.noreply.github.com> Co-authored-by: Ted Hart <15467143+TedHartMS@users.noreply.github.com> Co-authored-by: Tianyu Li <litianyu@mit.edu> Co-authored-by: Vasileios Zois <vazois@microsoft.com> Co-authored-by: Sebastian Burckhardt <sburckha@microsoft.com> Co-authored-by: rohankadekodi-msr <69916400+rohankadekodi-msr@users.noreply.github.com> Co-authored-by: Rohan Kadekodi <kadekodirohan@gmail.com>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
No description provided.