Query index improvements (#6376)

* Move shared index query code to a IndexEvaluator This enables StringIndex::find_all_no_copy in more places which should speed up equals queries with many results for indexed Int, ObjectId, UUID and Mixed types. This also fixed a bug on indexed Mixed equality queries that would return case insensitive results. * use an index if available for timestamp and bool equals queries * more readable serialization for combined lists * Fix and optimize Mixed case insensitive queries optimize EqualIns for mixed properties * changelog changelog * review feedback * changelog * changelog
realm · Mar 23, 2023 · 8f27206 · 8f27206
1 parent b8aad6b
commit 8f27206
Show file tree

Hide file tree

Showing 12 changed files with 678 additions and 322 deletions.
diff --git a/CHANGELOG.md b/CHANGELOG.md
@@ -1,11 +1,19 @@
 # NEXT RELEASE
 
 ### Enhancements
-* <New feature description> (PR [#????](https://github.com/realm/realm-core/pull/????))
+* Performance improvement for the following queries ([6376](https://github.com/realm/realm-core/issues/6376)):
+    * Significant (~75%) improvement when counting (`Query::count()`) the number of exact matches (with no other query conditions) on a string/int/UUID/ObjectID property that has an index. This improvement will be especially noticiable if there are a large number of results returned (duplicate values).
+    * Significant (~99%) improvement when querying for an exact match on a Timestamp property that has an index.
+    * Significant (~99%) improvement when querying for a case insensitive match on a Mixed property that has an index.
+    * Moderate (~25%) improvement when querying for an exact match on a Boolean property that has an index.
+    * Small (~5%) improvement when querying for a case insensitive match on a Mixed property that does not have an index.
 * Expose `Results::is_valid()` in the C API, in order to prevent to use an invalid Results. (PR [#6407](https://github.com/realm/realm-core/pull/6407))
 * Restore `SyncManager::set_logger_factory()` to enable setting different loggers for different SyncManagers ([PR #6400](https://github.com/realm/realm-core/pull/6400)).
 
 ### Fixed
+* Fixed a crash when querying a mixed property with a string operator (contains/like/beginswith/endswith) or with case insensitivity. ([6376](https://github.com/realm/realm-core/issues/6376) since introduction of Mixed)
+* Querying for equality of a string on an indexed mixed property was returning case insensitive matches. For example querying for `myIndexedMixed == "Foo"` would incorrectly match on values of "foo" or "FOO" etc. ([6376](https://github.com/realm/realm-core/issues/6376) since introduction of Mixed)
+* Adding an index to a Mixed property on a non-empty table would crash with an assertion. ([6376](https://github.com/realm/realm-core/issues/6376) since introduction of Mixed)
 * `SyncSession::pause()` could hold a reference to the database open after shutting down the sync session, preventing users from being able to delete the realm. ([#6372](https://github.com/realm/realm-core/issues/6372), since v13.3.0)
 * `Logger::set_default_logger()` did not perform any locking, resulting in data races if it was called while the default logger was being read on another thread ([PR #6398](https://github.com/realm/realm-core/pull/6398), since v13.7.0).
 

diff --git a/src/realm/index_string.hpp b/src/realm/index_string.hpp
@@ -507,6 +507,11 @@ size_t StringIndex::count(T value) const
     return m_array->index_string_count(Mixed(value), m_target_column);
 }
 
+inline Allocator& StringIndex::get_alloc() const noexcept
+{
+    return m_array->get_alloc();
+}
+
 inline void StringIndex::destroy() noexcept
 {
     return m_array->destroy_deep();

diff --git a/src/realm/query.cpp b/src/realm/query.cpp
@@ -992,12 +992,14 @@ void Query::aggregate(QueryStateBase& st, ColKey column_key) const
             auto node = pn->m_children[best];
             if (node->has_search_index()) {
                 auto keys = node->index_based_keys();
+                REALM_ASSERT(keys);
                 // The node having the search index can be removed from the query as we know that
                 // all the objects will match this condition
                 pn->m_children[best] = pn->m_children.back();
                 pn->m_children.pop_back();
-                for (auto key : keys) {
-                    auto obj = m_table->get_object(key);
+                const size_t num_keys = keys->size();
+                for (size_t i = 0; i < num_keys; ++i) {
+                    auto obj = m_table->get_object(keys->get(i));
                     if (pn->m_children.empty() || eval_object(obj)) {
                         st.m_key_offset = obj.get_key().value;
                         st.match(realm::npos, obj.get<T>(column_key));
@@ -1312,15 +1314,18 @@ void Query::do_find_all(TableView& ret, size_t limit) const
             auto best = find_best_node(pn);
             auto node = pn->m_children[best];
             if (node->has_search_index()) {
+                auto keys = node->index_based_keys();
+                REALM_ASSERT(keys);
                 KeyColumn& refs = ret.m_key_values;
 
                 // The node having the search index can be removed from the query as we know that
                 // all the objects will match this condition
                 pn->m_children[best] = pn->m_children.back();
                 pn->m_children.pop_back();
 
-                auto keys = node->index_based_keys();
-                for (auto key : keys) {
+                const size_t num_keys = keys->size();
+                for (size_t i = 0; i < num_keys; ++i) {
+                    ObjKey key = keys->get(i);
                     if (limit == 0)
                         break;
                     if (pn->m_children.empty()) {
@@ -1433,13 +1438,15 @@ size_t Query::do_count(size_t limit) const
         auto node = pn->m_children[best];
         if (node->has_search_index()) {
             auto keys = node->index_based_keys();
+            REALM_ASSERT(keys);
             if (pn->m_children.size() > 1) {
                 // The node having the search index can be removed from the query as we know that
                 // all the objects will match this condition
                 pn->m_children[best] = pn->m_children.back();
                 pn->m_children.pop_back();
-                for (auto key : keys) {
-                    auto obj = m_table->get_object(key);
+                const size_t num_keys = keys->size();
+                for (size_t i = 0; i < num_keys; ++i) {
+                    auto obj = m_table->get_object(keys->get(i));
                     if (eval_object(obj)) {
                         ++cnt;
                         if (cnt == limit)
@@ -1449,7 +1456,7 @@ size_t Query::do_count(size_t limit) const
             }
             else {
                 // The node having the search index is the only node
-                auto sz = keys.size();
+                auto sz = keys->size();
                 cnt = std::min(limit, sz);
             }
         }

diff --git a/src/realm/query_conditions.hpp b/src/realm/query_conditions.hpp
@@ -72,7 +72,7 @@ struct Contains : public HackClass {
     {
         if (m1.is_null())
             return !m2.is_null();
-        if (Mixed::types_are_comparable(m1, m2)) {
+        if (m1.is_type(type_String, type_Binary) && Mixed::types_are_comparable(m1, m2)) {
             BinaryData b1 = m1.get_binary();
             BinaryData b2 = m2.get_binary();
             return operator()(b1, b2, false, false);
@@ -133,7 +133,7 @@ struct Like : public HackClass {
     {
         if (m1.is_null() && m2.is_null())
             return true;
-        if (Mixed::types_are_comparable(m1, m2)) {
+        if (m1.is_type(type_String, type_Binary) && Mixed::types_are_comparable(m1, m2)) {
             BinaryData b1 = m1.get_binary();
             BinaryData b2 = m2.get_binary();
             return operator()(b1, b2, false, false);
@@ -186,7 +186,7 @@ struct BeginsWith : public HackClass {
 
     bool operator()(const QueryValue& m1, const QueryValue& m2) const
     {
-        if (Mixed::types_are_comparable(m1, m2)) {
+        if (m1.is_type(type_String, type_Binary) && Mixed::types_are_comparable(m1, m2)) {
             BinaryData b1 = m1.get_binary();
             BinaryData b2 = m2.get_binary();
             return b2.begins_with(b1);
@@ -232,7 +232,7 @@ struct EndsWith : public HackClass {
 
     bool operator()(const QueryValue& m1, const QueryValue& m2) const
     {
-        if (Mixed::types_are_comparable(m1, m2)) {
+        if (m1.is_type(type_String, type_Binary) && Mixed::types_are_comparable(m1, m2)) {
             BinaryData b1 = m1.get_binary();
             BinaryData b2 = m2.get_binary();
             return operator()(b1, b2, false, false);
@@ -395,7 +395,7 @@ struct ContainsIns : public HackClass {
     {
         if (m1.is_null())
             return !m2.is_null();
-        if (Mixed::types_are_comparable(m1, m2)) {
+        if (m1.is_type(type_String, type_Binary) && Mixed::types_are_comparable(m1, m2)) {
             BinaryData b1 = m1.get_binary();
             BinaryData b2 = m2.get_binary();
             return operator()(b1, b2, false, false);
@@ -479,7 +479,7 @@ struct LikeIns : public HackClass {
     {
         if (m1.is_null() && m2.is_null())
             return true;
-        if (Mixed::types_are_comparable(m1, m2)) {
+        if (m1.is_type(type_String, type_Binary) && Mixed::types_are_comparable(m1, m2)) {
             BinaryData b1 = m1.get_binary();
             BinaryData b2 = m2.get_binary();
             return operator()(b1, b2, false, false);
@@ -544,7 +544,7 @@ struct BeginsWithIns : public HackClass {
 
     bool operator()(const QueryValue& m1, const QueryValue& m2) const
     {
-        if (Mixed::types_are_comparable(m1, m2)) {
+        if (m1.is_type(type_String, type_Binary) && Mixed::types_are_comparable(m1, m2)) {
             BinaryData b1 = m1.get_binary();
             BinaryData b2 = m2.get_binary();
             return operator()(b1, b2, false, false);
@@ -610,7 +610,7 @@ struct EndsWithIns : public HackClass {
 
     bool operator()(const QueryValue& m1, const QueryValue& m2) const
     {
-        if (Mixed::types_are_comparable(m1, m2)) {
+        if (m1.is_type(type_String, type_Binary) && Mixed::types_are_comparable(m1, m2)) {
             BinaryData b1 = m1.get_binary();
             BinaryData b2 = m2.get_binary();
             return operator()(b1, b2, false, false);
@@ -675,8 +675,18 @@ struct EqualIns : public HackClass {
 
     bool operator()(const QueryValue& m1, const QueryValue& m2) const
     {
-        return (m1.is_null() && m2.is_null()) ||
-               (Mixed::types_are_comparable(m1, m2) && operator()(m1.get_binary(), m2.get_binary(), false, false));
+        if (m1.is_null() && m2.is_null()) {
+            return true;
+        }
+        else if (Mixed::types_are_comparable(m1, m2)) {
+            if (m1.is_type(type_String, type_Binary)) {
+                return operator()(m1.get_binary(), m2.get_binary(), false, false);
+            }
+            else {
+                return m1 == m2;
+            }
+        }
+        return false;
     }
 
     template <class A, class B>