[C#] LibDPR API improvements + Subscription #589
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Contributor
tli2
commented
Nov 3, 2021
tli2
commented
- Bring LibDPR branch up-to-date with master
- Changes to LibDPR API surrounding memory management with Span
- Support for subscription batches
* added sample * cleaned up async stress * Fix memory leak + add serialized wrapper sample * added testcases for concurrency within session in async. * clean up LMD
* Initial checkin of test suite for remote * updated configs. * fix sln * Update azure-pipelines.yml * Update azure-pipelines.yml * Update 50-remote-basics.md * Update 50-remote-basics.md
* Initial checkin of test suite for remote * updated configs. * fix sln * Update azure-pipelines.yml * Update azure-pipelines.yml * Ensure correct disposals of server and client. * rename testcase * Update 50-remote-basics.md * Update 50-remote-basics.md
* [C#] Fix double dispose in CompletePendingWithOutputs * Update MultiReadSpanByteKey.cs * Cleaned up testcase, merged with other SpanByte test. Co-authored-by: Tim Coleman <timothy.coleman@gmail.com>
…pulating) (microsoft#455) * don't pre-create 120 read and 120 write file handles. * fix asyncpool semaphore sizing. allow control over LocalStorageNamedDeviceFactory's Device.ThrottleLimit * dispose off the semaphore in AsyncPool.Dispose() * minor fixes to AsyncPool. * Speed up async pool. * Fix ordering of GetOrAdd and wait. Co-authored-by: Badrish Chandramouli <badrishc@microsoft.com>
…t#462) * Updated SpanByte sample with RMW for ASCII number sums * Sample refactoring as class per sub-sample
* [C#] Pub/sub functionality in remote FasterKV (microsoft#514) * Add client side support for SubscribeKV * basic working impl * Separate span byte serializer into client and server * Added support in Subscriptions. TODO: prefix matching for subscriptions, publish deletes to keys (microsoft#499) * Added support in Subscriptions. TODO: prefix matching for subscriptions, publish deletes to keys * Added prefix matching, but similar to redis and unoptimized * Added null checking for subscriptions on removeSubscriptions * Made changes to correct concurrency and type checking * Optimized and corrected the Start() method and Publish() method in publish call * Added prefix subscriptions. Added a new client serializer call, need to discuss * Corrected few nits * Cleanup of code, and resolving NIE * Nit fix of null checking * Fixed small nit in MemoryparamSerializer to return Memory<byte> of correct length, and in publish call corner case * Fixed small nit in MemoryparamSerializer to return Memory<byte> of correct length, and in publish call corner case (microsoft#511) * Resolving merge conflicts * Removing the subscriptions belonging to a session on disconnection * Added unit tests and VarLenClient test * nit fix * Made broker associated with FasterServer and not FasterKVProvider * fixed a bug for large values in pub-sub * Changed the SubscribeKVBroker to be spawned by the user and passed to provider * fixed nit * Merged with recent master code * Checked fixed len client * Fixed bug in Write of key in server serializer * Added null check for broker Co-authored-by: Badrish Chandramouli <badrishc@microsoft.com> * Improved byte array comparer, moved to FASTER.server * refactor tests * [C#] Remote pubsub (microsoft#531) * Add client side support for SubscribeKV * basic working impl * Separate span byte serializer into client and server * Added support in Subscriptions. TODO: prefix matching for subscriptions, publish deletes to keys (microsoft#499) * Added support in Subscriptions. TODO: prefix matching for subscriptions, publish deletes to keys * Added prefix matching, but similar to redis and unoptimized * Added null checking for subscriptions on removeSubscriptions * Made changes to correct concurrency and type checking * Optimized and corrected the Start() method and Publish() method in publish call * Added prefix subscriptions. Added a new client serializer call, need to discuss * Corrected few nits * Cleanup of code, and resolving NIE * Nit fix of null checking * Fixed small nit in MemoryparamSerializer to return Memory<byte> of correct length, and in publish call corner case * Fixed small nit in MemoryparamSerializer to return Memory<byte> of correct length, and in publish call corner case (microsoft#511) * Resolving merge conflicts * Removing the subscriptions belonging to a session on disconnection * Added unit tests and VarLenClient test * nit fix * Made broker associated with FasterServer and not FasterKVProvider * fixed a bug for large values in pub-sub * Changed the SubscribeKVBroker to be spawned by the user and passed to provider * fixed nit * Merged with recent master code * Checked fixed len client * Fixed bug in Write of key in server serializer * Working with non-kv pub-sub. Based off remote-subkv PR * Fixed some nits, making tests work * Changed the subscriptions and prefixSubscriptions dictionary to use sid as key Co-authored-by: Badrish Chandramouli <badrishc@microsoft.com> * [C#] Remote subkv fixes (microsoft#542) * Add client side support for SubscribeKV * basic working impl * Separate span byte serializer into client and server * Added support in Subscriptions. TODO: prefix matching for subscriptions, publish deletes to keys (microsoft#499) * Added support in Subscriptions. TODO: prefix matching for subscriptions, publish deletes to keys * Added prefix matching, but similar to redis and unoptimized * Added null checking for subscriptions on removeSubscriptions * Made changes to correct concurrency and type checking * Optimized and corrected the Start() method and Publish() method in publish call * Added prefix subscriptions. Added a new client serializer call, need to discuss * Corrected few nits * Cleanup of code, and resolving NIE * Nit fix of null checking * Fixed small nit in MemoryparamSerializer to return Memory<byte> of correct length, and in publish call corner case * Fixed small nit in MemoryparamSerializer to return Memory<byte> of correct length, and in publish call corner case (microsoft#511) * Resolving merge conflicts * Removing the subscriptions belonging to a session on disconnection * Added unit tests and VarLenClient test * nit fix * Made broker associated with FasterServer and not FasterKVProvider * fixed a bug for large values in pub-sub * Changed the SubscribeKVBroker to be spawned by the user and passed to provider * fixed nit * Merged with recent master code * Checked fixed len client * Fixed bug in Write of key in server serializer * Added null check for broker * Fixed a nit: subscriptions dictionary should contain key as sid and val as serversession * fixed nit * Removed old ByteArrayComparer Co-authored-by: Badrish Chandramouli <badrishc@microsoft.com> * [C#] Remote websocket (microsoft#530) * Add client side support for SubscribeKV * basic working impl * Separate span byte serializer into client and server * Added support in Subscriptions. TODO: prefix matching for subscriptions, publish deletes to keys (microsoft#499) * Added support in Subscriptions. TODO: prefix matching for subscriptions, publish deletes to keys * Added prefix matching, but similar to redis and unoptimized * Added null checking for subscriptions on removeSubscriptions * Made changes to correct concurrency and type checking * Optimized and corrected the Start() method and Publish() method in publish call * Added prefix subscriptions. Added a new client serializer call, need to discuss * Corrected few nits * Cleanup of code, and resolving NIE * Nit fix of null checking * Fixed small nit in MemoryparamSerializer to return Memory<byte> of correct length, and in publish call corner case * Fixed small nit in MemoryparamSerializer to return Memory<byte> of correct length, and in publish call corner case (microsoft#511) * Resolving merge conflicts * Removing the subscriptions belonging to a session on disconnection * Added unit tests and VarLenClient test * nit fix * Made broker associated with FasterServer and not FasterKVProvider * fixed a bug for large values in pub-sub * Changed the SubscribeKVBroker to be spawned by the user and passed to provider * fixed nit * Merged with recent master code * Checked fixed len client * Working on JS client * Added batching in websocket handling. Next step: finish HandlePending() * Modified websocket server. Fixed bug in Write key of server serializer * duplicate file from faster.common * Added sample for WebClient * fixed small nit * Fixed some nits in js clientsession * fixed nit * removed old ByteArrayComparer * Fixed nits, added non-kv PubSub to WebsocketServerSession Co-authored-by: Badrish Chandramouli <badrishc@microsoft.com> * Cleanup and updates. * fix break. * [C#] Modifications to pubsub (microsoft#547) * Added FASTER Log for SubscribeKV and Subscribe * Added input in subscribeKV * Added SubscribeCallback and SubscribeKV sample in html * set enablePubSub: false for non-pubsub tests Co-authored-by: rohankadekodi-msr <69916400+rohankadekodi-msr@users.noreply.github.com>
* Cleanup of server * More cleanup * Proper dispose of broker * updates
* Fix bug in AcquireLatchX Check version of traced record rather than of a bucket's tail record while deciding to update in place or COW * Separate out InVersionNew overloads Co-authored-by: Badrish Chandramouli <badrishc@microsoft.com>
`(seq % 1L << 20)` = 0 for any `seq`, should be either `seq % (1L << 20)` or `seq % fht.IndexSize`.
* Seeing Out of Memory failures on Release \ Any CPU test runs. These are caused by log allocation sizes being too big. I reduced several tests to try to stabilize the tests in all configs. * Stabilized a few tests that were sporadically failing on Release Any CPU config. Getting out of memory errors so reduced the size of the KV Log. * RecoveryTestByAllocatorType was failing on Debug x64 due to the log size was not valid. So putting size back to what it was * Change Assert.IsTrue -> Assert.AreEqual (and other more-precise calls). Remove Console.WriteLine in tests and replace with Debug.WriteLine in LocalMemoryDevice. Fix RecoverTests Segment and Page SizeBits to match reduced LMD capacity argument * Restore RecoveryTests Segment and Page SizeBits per Darren's push * Updated a couple Device Faster Log Tests to use "using var" to make sure device and logs are cleaned up properly when the test is done. Co-authored-by: TedHartMS <15467143+TedHartMS@users.noreply.github.com> Co-authored-by: Badrish Chandramouli <badrishc@microsoft.com>
* Adding comments to structures that were made public (Phase, SystemState, DeltaLogEntryType * updates * misc minor cleanup Co-authored-by: TedHartMS <15467143+TedHartMS@users.noreply.github.com>
* MemoryPool-based override for FasterLog.ReadAsync() * FasterLogIterator.CompleteUntilRecordAtAsync() added. Co-authored-by: Badrish Chandramouli <badrishc@microsoft.com>
* Add uring IO handler. * minor update. * Make uring an option. Co-authored-by: Badrish Chandramouli <badrishc@microsoft.com>
* Misc remote cleanup * nit
* Removed multiple enqueues for SubscribeBroker during Publish() and replaced with just one. Disadvantage is that there are multiple small allocations happening * Fixed nit that was causing valPtr to send wrong values when there were multiple subscribers * Fixed couple of nits.
# Conflicts: # .gitignore # azure-pipelines.yml # cs/playground/AsyncStress/SerializedFasterWrapper.cs # cs/remote/FASTER.remote.sln # cs/remote/samples/FixedLenClient/Functions.cs # cs/remote/samples/FixedLenClient/Program.cs # cs/remote/samples/FixedLenServer/FixedLenServer.csproj # cs/remote/samples/FixedLenServer/Program.cs # cs/remote/samples/FixedLenServer/Types.cs # cs/remote/samples/VarLenClient/CustomTypeSamples.cs # cs/remote/samples/VarLenServer/Program.cs # cs/remote/src/FASTER.client/ClientSession.cs # cs/remote/src/FASTER.common/ReusableObject.cs # cs/remote/src/FASTER.common/SimpleObjectPool.cs # cs/remote/src/FASTER.common/WireFormat.cs # cs/remote/src/FASTER.server/BinaryServerSession.cs # cs/remote/src/FASTER.server/ConnectionArgs.cs # cs/remote/src/FASTER.server/FASTER.server.csproj # cs/remote/src/FASTER.server/FasterKVProvider.cs # cs/remote/src/FASTER.server/FasterKVServerSessionBase.cs # cs/remote/src/FASTER.server/FasterServer.cs # cs/remote/src/FASTER.server/ISessionProvider.cs # cs/remote/src/FASTER.server/ServerKVFunctions.cs # cs/remote/src/FASTER.server/ServerSessionBase.cs # cs/remote/src/FASTER.server/SpanByteClientSerializer.cs # cs/remote/src/FASTER.server/SpanByteFunctionsForServer.cs # cs/remote/src/FASTER.server/SpanByteServerSerializer.cs # cs/remote/test/FASTER.remote.test/FASTER.remote.test.csproj # cs/remote/test/FASTER.remote.test/FixedLenBinaryTests.cs # cs/remote/test/FASTER.remote.test/FixedLenClient.cs # cs/remote/test/FASTER.remote.test/VarLenBinaryTests.cs # cs/remote/test/FASTER.remote.test/VarLenMemoryClient.cs # cs/src/core/Async/ReadAsync.cs # cs/src/core/Index/Common/Contexts.cs # cs/src/core/Index/FASTER/FASTER.cs # cs/src/core/Index/Recovery/Recovery.cs # cs/src/devices/AzureStorageDevice/AzureStorageDevice.cs # cs/test/BasicFASTERTests.cs # cs/test/CompletePendingTests.cs # cs/test/FasterLogResumeTests.cs # cs/test/FasterLogTests.cs # cs/test/LowMemAsyncTests.cs # cs/test/NativeReadCacheTests.cs # cs/test/RecoveryChecks.cs # cs/test/ReproReadCacheTest.cs # cs/test/SimpleAsyncTests.cs # cs/test/SpanByteTests.cs # cs/test/VariableLengthStructFASTERTests.cs # docs/_docs/25-fasterkv-recovery.md # docs/_pages/home.md
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.