Loader optimization: Fix incorrect in-place sort of chunks by partiti…

…on (#116) * Fix incorrect in-place sort * Update CHANGELOG
stac-utils · May 20, 2022 · 3baae42 · 3baae42
1 parent 78e7ad0
commit 3baae42
Show file tree

Hide file tree

Showing 2 changed files with 6 additions and 1 deletion.
diff --git a/CHANGELOG.md b/CHANGELOG.md
@@ -1,5 +1,9 @@
 # Changelog
 
+## Unreleased
+
+- Fixed issue loader grouping an unordered iterable by partition, speeding up loads of items with mixed partitions [#116](https://github.com/stac-utils/pgstac/pull/116)
+
 ## [v0.6.3]
 
 ### Fixed

diff --git a/pypgstac/pypgstac/load.py b/pypgstac/pypgstac/load.py
@@ -552,7 +552,8 @@ def load_items(
             items = self.read_hydrated(file)
 
         for chunk in chunked_iterable(items, chunksize):
-            list(chunk).sort(key=lambda x: x["partition"])
+            chunk = list(chunk)
+            chunk.sort(key=lambda x: x["partition"])
             for k, g in itertools.groupby(chunk, lambda x: x["partition"]):
                 self.load_partition(self._partition_cache[k], g, insert_mode)