Sorted collections #65

vihanb · 2021-07-21T17:00:55Z

This introduces two new sorted collections within a SortedCollections subpackage as initially
discussed in #1.

These sorted collections maintain their order by internally maintaining in-memory B-trees which allows them to balance the performance benefits of contiguous buffers and trees.

Currently, this contains an implementation of SortedSet and SortedDictionary which are designed to mimic the API of the standard library’s Set and Dictionary as closely as possible and provide additional methods and properties specific to sorted collection types.

Checklist

I've read the Contribution Guidelines
My contributions are licensed under the Swift license.
I've followed the coding style of the rest of the project.
I've added tests covering all new code paths my change adds to the project (if appropriate).
I've added benchmarks covering new functionality (if appropriate).
I've verified that my change does not break any existing tests or introduce unexplained benchmark regressions.
I've updated the documentation if necessary.

ishantheperson · 2021-07-22T21:24:53Z

Sources/SortedCollections/BTree/_BTree.PackedList.swift

+extension _BTree {
+  /// A stack-allocated list of some values.
+  /// 
+  /// - Warning: This may hold strong references to objects after they


After they what?

Signed-off-by: Vihan Bhargava <vihan_bhargava@apple.com>

Key Changes: - Implement deficiency correction methods - Remove inlining from _Node.ensureUnique to better inline closures - Add new computed properties to _Node.UnsafeHandle Minor changes - Added missing mutability checks to certain _Node.UnsafeHandle methods - Remove unneeded _Node.UnsafeHandle.findValue(key:) method - Allow _Node.UnsafeHandle.removeElement(at:) to be called on non-leaves - Expand doc comments and fix typos

Key Changes: - Implement removal through SortedDictionary subscript - Implement _BTree.contains for efficient existence checking - Add tests - _Node.UnsafeHandle.rotation/collapse tests - _BTree.removeAny tests - Expand benchmarks - Add C++ reference lookup benchmarks - Correct old benchmarks Minor Changes: - Remove B-Tree benchmarks, correct SortedDictionary benchmarks - Adjust debug B-Tree printer to show both numElements and numTotalElements - Add total count synchronization invariant check - Implement CustomReflectable for B-Tree

…Set. Key Changes: - SortedSet - Implement BidirectionalCollection - SortedDictionary - Implement Codable - Implement Partial RangeReplaceableCollection - Includes remove*() operations - init(grouping:by:) - var keys: Keys (aka SortedSet<Key>) - func (compact)mapValues(_:) Minor Changes: - Correct time documented complexities of various methods - Implement more variations of removal methods.

Using an array was very heavy, now using a stack-allocated 64-byte wide type which mimics an array.

@Freeze

Key Changes: - SortedSet - Implement Partial RangeReplaceableCollection - Implement ExpressibleByArrayLiteral - Implement init<S: Sequence>(_: S) - Implement Set methods: - update(with:) / insert(_:) / contains(_:) - SortedDictionary - Implement SortedDictionary.Values: BidirectionalCollection - Implement SortedDictionary.Keys: BidirectionalCollection - _BTree - Implement Partial RangeReplaceableCollection - Implement Sequence - Add high performance linear iterator Minor Changes: - Benchmarks - Clean up benchmarks - _Node.UnsafeHandle - @Freeze enum InsertionResult - _Node.Storage - Use UnsafeMutablePointer instead of ManagedBuffer Refactors: - Various refactors for method signatures and documentation.

Key Changes: - _BTree.init() dynamically computes capacity from stride - _BTree.UnsafePath uses Unmanaged +0 references. Minor Changes: - Fix _BTree.Iterator bug on single-leaf trees.

Key Changes: - SortedSet - Implement SetAlgebra - Implement Hashable - Implement Equatable - Implement Custom(Debug)StringConvertible - Implement CustomReflectable - Implement Codable - Implement subscript(range: Range<Element>) - Implement subscript(offset:) - Implement remove(atOffset:) Minor Changes: - Fix inefficiency on SortedDictionary.customMirror - Optimize to O(1) _BTree.UnsafePath comparator - Optimize away ARC traffic on findAnyPath(forKey:) Various additional refactors and renames.

Key Changes: - Implement _BTree.Cursor for performing efficient mutations on a tree Minor Changes: - New implementation of _BTree.Index and iteration

Major Changes: - Implement Sequence for SortedSet Minor Changes: - Fix bug within _Node.UnsafeHandle.pointerToChild(atSlot:) - Cleanup cursor implementation and fix bugs. - Fix various bugs in _BTree.FixedSizeArray

@inline

Key Changes: - Add custom SubSequences - Custom SortedDictionary.(Keys|Values).Sequence iterator - Add SortedDictionary.modifyValue(forKey:default:_:) Minor Changes: - Update documentation - Various API optimizations - Fix bug where offsetBy:limitedBy: calls did not validate indices - Remove unique initializers with documented duplicate handling behavior. - Implement _Node.UnsafeHandle.append*() methods - SortedSet no longer allocates a 'values' buffer (_Node.Header.values is now optional) Documentation: - Complexity guarantees - BidirectionalCollection conformances - Sequence/IteratorProtocol methods Optimizations: - Add optimized dictionary value mapping (through Node.init(mappingFrom:_:)) - Custom _BTree.forEach - Greater explicit @inline(__always) annotations throughout. - Simplify _BTree: Sequence implementation - Move methods to using _BTree.Builder - _BTree.FixedSizeArray now uses `withUnsafePointer` to prevent branching.

Dante-Broggi · 2021-08-11T05:04:59Z

Sources/SortedCollections/BTree/_BTree+Partial RangeReplaceableCollection.swift

+
+  /// Removes the first element of a tree, if it exists.
+  ///
+  /// - Returns: The moved first element of the tree.


Suggested change

/// - Returns: The moved first element of the tree.

/// - Returns: The moved last element of the tree.

Key Changes: - Implement Equatable, Hashable for SubSequences - Implement SortedDictionary initializers - init(keysWithValues:) and init(sortedKeysAndValues:) with named and unnamed variants. - Expand benchmark suite for SortedDictionary - Commit _BTree.Builder Minor Changes: - Update documentation - Expand test suits - Fix bug where SubSequences did not define subscript(bounds: Range<Index>) - _Node - Implement `depth` property for `_Node` - Packs both `depth`, `count`, and `capacity` within a single UInt64. - Implement `init(leftChild:seperator:rightChild:capacity:)` for `_Node` - Implement `concatenateWith(node:seperatedBy:)` to `_Node.UnsafeHandle` - Remove bad invariant check on `_Node.UnsafeHandle` deletion subroutines - _BTree - Add invariant checks as `_BTree.checkInvariants()` - Separate `_BTree.*capacity` into `*size` and `*capacity` - Make _FixedSizeArray top-level - Remove deque behavior. Documentation: - Update complexity documentation style from 'n' to `self.count` Testing: - Expand tests for SortedDictionary - Add tests for SortedDictionary.Keys - Add tests for _Node.join(...) routines - Add tests for _BTree.Builder - Restore checkBidirectionalCollection tests for _BTree

vihanb · 2021-08-18T23:41:18Z

Our initial benchmark numbers are promising, in most cases we come close to, when not exceeding, the performance at large collection sizes compared to std::map

Key Changes: - Add duplicate handling functionality to _BTree.Builder - Remove transient allocation in SortedDictionary.values _modify - Add missing SortedDictionary.Values.subscript _modify - Add SortedSet(sortedElements:) - Add optimized SetAlgebra implementations - Remove remaining high-level offset APIs - Various optimizations - Expand internal documentation Minor Changes: - Add ability to create dummy node - _BTree.Builder.append(key: Key) for sets. - Fix bug where findAnyIndex(forKey:) didn't calculate the offset. - Remove old, unused files - findAnyCursor -> takeCursor (to emphasize consuming nature) - Add takeCursor(at: Index) - Optimize init(grouping:by:) to use modifyValue to reduce CoW Optimizations: - Use dummy node for transient _modify state rather than allocating new buffer

Key Changes: - Add tests and benchmarks for SortedSet - Fix bug during _Node.UnsafeHandle.popFirstElement() rebalancing - Enable _Node.UnsafeHandle.subscript(elementAt:) to work on value-less nodes. Minor Changes: - Optimize Codable to use _BTree.Builder and forEach - Codable now checks for order, not strictly uniqueness - Add smaller node size for debug builds to easily catch errors

Signed-off-by: Vihan Bhargava <vihan_bhargava@apple.com>

tamc · 2021-10-18T09:56:49Z

Anything I can do to help get this over the line?

lorentey · 2021-11-22T23:10:49Z

Merging to feature branch feature/SortedCollections.

mdznr · 2022-12-03T09:18:48Z

Sources/SortedCollections/SortedSet/SortedSet.swift

+  @usableFromInline
+  internal var _root: _Tree
+
+  //// Creates an empty set.


Suggested change

//// Creates an empty set.

/// Creates an empty set.

vihanb requested a review from lorentey as a code owner July 21, 2021 17:00

ishantheperson reviewed Jul 22, 2021

View reviewed changes

vihanb force-pushed the sorted-collections branch from 68c737a to 45a989d Compare August 10, 2021 02:05

vihanb added 11 commits August 9, 2021 19:13

Initial commit for SortedCollections

9cf2cda

Signed-off-by: Vihan Bhargava <vihan_bhargava@apple.com>

Implement and use PackedOffsetList for UnsafePath

761950d

Using an array was very heavy, now using a stack-allocated 64-byte wide type which mimics an array.

Continue applying refactors and optimizations.

98f2785

Key Changes: - _BTree.init() dynamically computes capacity from stride - _BTree.UnsafePath uses Unmanaged +0 references. Minor Changes: - Fix _BTree.Iterator bug on single-leaf trees.

Add cursor type for imperative tree modification.

5690056

Key Changes: - Implement _BTree.Cursor for performing efficient mutations on a tree Minor Changes: - New implementation of _BTree.Index and iteration

Add cursor tests and set iterator.

21d9bde

Major Changes: - Implement Sequence for SortedSet Minor Changes: - Fix bug within _Node.UnsafeHandle.pointerToChild(atSlot:) - Cleanup cursor implementation and fix bugs. - Fix various bugs in _BTree.FixedSizeArray

vihanb force-pushed the sorted-collections branch from 45a989d to 0d0d66f Compare August 10, 2021 02:14

Dante-Broggi reviewed Aug 11, 2021

View reviewed changes

vihanb added 3 commits August 19, 2021 00:02

Update swift-collections-benchmark scheme.

478d3cc

Signed-off-by: Vihan Bhargava <vihan_bhargava@apple.com>

vihanb force-pushed the sorted-collections branch from 2a509cb to 478d3cc Compare August 21, 2021 08:47

lorentey changed the base branch from main to feature/SortedCollections November 22, 2021 23:10

lorentey merged commit b472a6f into apple:feature/SortedCollections Nov 22, 2021

mdznr reviewed Dec 3, 2022

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Sorted collections #65

Sorted collections #65

vihanb commented Jul 21, 2021

ishantheperson Jul 22, 2021

Dante-Broggi Aug 11, 2021

vihanb commented Aug 18, 2021

tamc commented Oct 18, 2021

lorentey commented Nov 22, 2021

mdznr Dec 3, 2022

	/// - Returns: The moved first element of the tree.
	/// - Returns: The moved last element of the tree.

Sorted collections #65

Sorted collections #65

Conversation

vihanb commented Jul 21, 2021

Checklist

ishantheperson Jul 22, 2021

Choose a reason for hiding this comment

Dante-Broggi Aug 11, 2021

Choose a reason for hiding this comment

vihanb commented Aug 18, 2021

tamc commented Oct 18, 2021

lorentey commented Nov 22, 2021

mdznr Dec 3, 2022

Choose a reason for hiding this comment