[C#] Added store.Log.FixedRecordSize API #399
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
…fixed-size part) on the main log.
badrishc
changed the title
darrenge
pushed a commit
that referenced
this pull request
Feb 10, 2021
…fixed-size part) on the main log. (#399)
badrishc
pushed a commit
that referenced
this pull request
Mar 7, 2021
…y covered (#420) * Just minor addition to test where it isn't taking default but acutally passing in CancellationToken * Some new tests in progress. Not fully finished. * Update 40-fasterlog-basics.md * [C#] Add IAdvancedFunctions documentation (#386) * Add IAdvancedFunctions documentation * Fix links * Finished RecoverReadOnlyBasicTest * Added three types Enqueue tests and RecoverReadOnly tests * Moving LogReadAsync tests to its own file * Fixed a bug with SpanBatch test and added better verification code * Added LogReadAsync test group that has various tests for different parameter combinations * Removed an assert so doesn't cause test failure * [C#] Add FasterLog.CreateAsync and RecoverReadOnlyAsync (#384) * First pass on RecoverAsync * Change to semaphore * FIx stale tokens; make test names consistent * add readonly * Add CancellationToken to RecoverAsync; increase CancellationToken usage in a couple other places * Add clarifying comments and expected behavior in FasterLogPubSub. Add FasterLog.CreateAsync (which calls RecoverAsync) and RecoverReadOnlyAsync. Replace obsolete DeleteDirectory functions with Directory.Delete(path, true) * Remove some debugging code * FasterLogPubSub: catch OperationCanceledException, add using/Dispose() * Updated FasterLog test names and added a couple tests * Updated Enqueue Span Batch test as was passing locally but failing in CI * Update EnqueueTests.cs * Categorized tests by into FasterLog, FasterKV and Smoke * Finished basic test for the 2nd and the 3rd parameter of Log.ReadAsync * Removed a couple tests that were failing so CI passes .. will get those tests passing before checking them. * Added basic TryEnqueue test that tests all parameters of TryEnqueue * Added test to cover parameters in CreateLogDevice and FasterLog that were not already covered by other tests * Added basic test for EnqueueAndWaitForCommit - handles all three overloads of Method * Added WaitForCommit test and fixed timing error in LogReadAsync * Had issue with DeviceAndLogConfig that failed on Linux C# so fixed that. * Reduced size of FasterLog for Enqueue tests as was taking 40+ mins to run * Reduced number of log entries in the LogScan tests as it was taking too long in CIs * Changed faster tests from default of 1,000,000 entries to 100,000 entries in log file to help reduce the execution time for entire test run * Categorized the rest of the tests that weren't categorized * Added a Timeout to 75 mins to pipeline yml instead of default 60 mins. * Removing the timeout from yaml file as it causes other issues * Added ScanUncommittedTest and ScanBufferingModeNoBufferingTest. Also, removed log and device test to see if that is what is causing issue in CI runs. * Enabled LogAndDeviceConfigTests test, added log.dispose in FasterLog file tear down. Reduced TryEnqueueVariousConfig log size to help with perf of test run * Updated CommitFalse test to have been fail messages as having timing issues. Also something not right with TryEnqueueVariousConfigs test as takes 9 mins per to run on release. Taking it out to see if that is the issue. * Brought back TryEnqueue2 and working on timing of CommitFalse * Simplified CommitFalse - trying figure out why failing on release in CI * Commented out a few tests that take 10+ minutes to run on release to try to narrow down cause of release running so long * Removed comments so all tests are running again ... * Fixed a timing issue where TearDown was delaying a long time under release builds * Removing EnqueueAndWaitForCommit test and also changing TearDown to see if that helps issue where Release is timing out * Had a bug in clean up code that caused fails, but looks like timing issue could potentially be fixed * Added Try \ Catch to clean up code * Benchmark: Pin loading thread if it is not used for checkpointing * Added tests to handle all the overload calls to .Read. Also added a test of sample code from docs. * Added a couple tests that specifically test ReadFlags options for .Read() * Added tests for ReadAsync, RMW, Upsert and ReadAtAddress * Changed the name of some tests * Added ReadyToCompletePendingAsyncTest * Names with Generic were hard to figure out what tests were so changed name to be a bit more descriptiive. Also added a couple tests * Added EnqueueAndWaitForCommitAsync tests * Added EnqueueAndWaitForCommitBasicTest where using task.run instead of launching threads. Also combined EnqueueAndWaitForCommitAsyncEntry into one test instead of three separate * Setting EnqueueAndWaitForCommitBasicTest to run for debug only for now as for some reason when this is run for release it causes tests to bog down where they take minutes instead of a sec or two * Running EnqueueAndWaitForCommit on Release with reduced test data size to see if that was problem on Release test run * Set EnqueueAndWaitForCommitBasicTest to only run on Debug builds ... also other checkins for Check point test * Trying Release run of Enqueue And Wait test. Was a long test name so reduced that. Also, set pipeline timeoutInMinutes: 120 * Shortening test name .. have seen issues with Azure where long names caused weird failures * Fixed EnqueueWaitCommitBasicTest - wasn't waiting for tasks to finish. * Cleaned up WaitForCommitBasicTest to use tasks instead of threads. * Couple clean up things in WaitForCommitBasicTest * Added a timeout on WaitForCommitBasicTest * Put some extra verification in WaitForCommit and EnqueueAndWaitForCommit as seeing issues with tests running on Release * Set commit to true * Update 20-fasterkv-basics.md * Added EnqueueAsyncBasicTest and made WaitForCommit and EnqueueAndWaitForCommit to be Debug only * Added CommitAsyncPrevTask test. Also, added a Sync / Async test for WaitForCommitBasicTest. * Added RefreshUncommittedAsyncTest * Added changes from Ted regarding EnqueueWaitCommitBasicTest and enhanced ReadAtAddress * Added if Debug to EnqueueAndWaitCommitBasicTest because the fix is not in master yet, so need this to not break the test queueu * Update README.md * Just comments * Removed if DEBUG because bug has been fixed * Removed #DEBUG from test since bug was fixed and no longer needed * Added ManagedLocalStorageTests and added a bit more verification to other tests * Fixed GrowIndex to work correctly (#395) * Fixed GrowIndex to work correctly, and return only after index size doubling is complete. * Reset overflowBucketsAllocator during resize. * Add API to get number of overflow buckets. * Record locking (#394) * Add RecordInfo.SpinLock * Pass kInvalidAddress to SingleReader for readcache * IntExclusiveLocker does not require unsafe * Assert logicalAddress >= ReadOnlyAddress in RecordAccessor.SpinLock Co-authored-by: Badrish Chandramouli <badrishc@microsoft.com> * [C#] Sample as a memory-only cache (#396) * Initial sample + minor mods to support use case * cleanup * added zipf distribution to example. moved to samples folder. * updates * Change CopyReadsToTail from bool to enum. * Add support for CopyReadsToTail to copy to tail when read from ReadOnly region. * updated sample, use container to store value being reinserted. * Added support for SubscribeEvictions * updated sample to ignore tombstones * Added store.Log.FixedRecordSize API to get the size per record (only fixed-size part) on the main log. (#399) * fix(iterator): null conditional dispose (#401) Co-authored-by: Stephan Kempkes <stephan.kempkes@esrlabs.com> * Fixed some conflicts with these tests * More merge changes * Removed a couple tests that passed on their own but affected other tests. Will put them back when it gets fixed. * Fixed RecoverReadOnlyAsyncBasicTest so it won't affect other tests. * Mid stage of getting ScanUncommittedTest working * Finished ScanUncommittedTest * Cleaned up ScanUncommittedTest a bit
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
var size = store.Log.FixedRecordSize