Merge remote-tracking branch 'origin/master' into feature/exception-u…

…nification * origin/master: (22 commits) Fix a race condition in error reporting for async open (#5968) Updated release notes version bump to 12.11.0 Fix a deadlock with simultaneous sync and nonsync commit notifications while tearing down RealmCoordinator (#5948) Track the last line seen and report it when there's an uncaught exception Verify that the same key is used when opening multiple DBs for a file Add stricter checks for valid reads on encrypted files Run core tests with encryption enabled on windows (#5967) Fix assertion failures after calling SectionedResults::reset_section_callback() (#5965) Add sync integration test for in-memory mode (#5955) Added realm core version to app login request (#5961) The client sending too much data is now reported as a ProtocolError::limits_exceeded (#5956) Updated release notes release core v12.10.0 Fix C-API for realm_object_get_parent (#5960) fixed a bug in recovery when copying embedded lists with a single link property prefix in the path (#5957) Js/sectioned notification (#5926) Use named apikey callbacks Fix IN query with TypedLink as left operator Fix a use-after-free in client reset when the original RealmCoordinator is gone (#5949) ...
realm · Oct 26, 2022 · f2b799a · f2b799a
2 parents 2836692 + 7ee0764
commit f2b799a
Show file tree

Hide file tree

Showing 51 changed files with 768 additions and 350 deletions.
diff --git a/CHANGELOG.md b/CHANGELOG.md
@@ -2,18 +2,62 @@
 
 ### Enhancements
 * <New feature description> (PR [#????](https://github.com/realm/realm-core/pull/????))
-* Improve performance of client reset with automatic recovery and converting top-level tables into embedded tables (PR [#5897](https://github.com/realm/realm-core/pull/5897)).
-* Adding `realm_query_parse_for_set` in the C API ([#5935](https://github.com/realm/realm-core/pull/5935)).
+* None.
 
 ### Fixed
 * <How do the end-user experience this issue? what was the impact?> ([#????](https://github.com/realm/realm-core/issues/????), since v?.?.?)
+* Fix a race condition which could result in "operation cancelled" errors being delivered to async open callbacks rather than the actual sync error which caused things to fail ([PR #5968](https://github.com/realm/realm-core/pull/5968), since the introduction of async open).
 * The C API type `realm_sync_error_code_t` did not include a textural representation of the underlying category. ([#5399](https://github.com/realm/realm-core/issues/5399)),
-* Calling Results::sum/min/max on a collection field throws an exception rather than aggregating on the underlying memory addresses. ([#5137](https://github.com/realm/realm-core/issues/5137))
+
+### Breaking changes
+* All exceptions thrown out of Core are now of type 'Exception'. All use of std::runtime_error and std::logical_error etc. has stopped and the specialized error classes that beforehand were based on these are now based on Exception.
+
+### Compatibility
+* Fileformat: Generates files with format v22. Reads and automatically upgrade from fileformat v5.
+
+-----------
+
+### Internals
+* None.
+
+----------------------------------------------
+
+# 12.11.0 Release notes
+
+### Fixed
+* Calling `SectionedResults::reset_section_callback()` on a `SectionedResults` which had been evaluated would result in an assertion failure the next time the sections are evaluated ([PR #5965](https://github.com/realm/realm-core/pull/5965), since v12.10.0).
+* Opening an unencrypted file with an encryption key would sometimes report a misleading error message that indicated that the problem was something other than a decryption failure ([PR #5915](https://github.com/realm/realm-core/pull/5915), since 0.86.1).
+* Fix a rare deadlock which could occur when closing a synchronized Realm immediately after committing a write transaction when the sync worker thread has also just finished processing a changeset from the server ([PR #5948](https://github.com/realm/realm-core/pull/5948)).
+
+### Breaking changes
+* Websocket errors caused by the client sending a websocket message that is too large (i.e. greater than 16MB) now get reported as a `ProtocolError::limits_exceeded` error with a `ClientReset` requested by the server ([#5209](https://github.com/realm/realm-core/issues/5209)).
+
+### Compatibility
+* Fileformat: Generates files with format v22. Reads and automatically upgrade from fileformat v5.
+
+-----------
+
+### Internals
+* Added integration test for opening synchronized realms as in-memory realms ([#5955](https://github.com/realm/realm-core/pull/5955)).
+* Added realm core version to the app login request ([#5959](https://github.com/realm/realm-core/issues/5959))
+
+----------------------------------------------
+
+# 12.10.0 Release notes
+
+### Enhancements
+* Improve performance of client reset with automatic recovery and converting top-level tables into embedded tables (PR [#5897](https://github.com/realm/realm-core/pull/5897)).
+* Adding `realm_query_parse_for_set` in the C API ([#5935](https://github.com/realm/realm-core/pull/5935)).
+
+### Fixed
+* Fixed an assertion failure when observing change notifications on a sectioned result, if the first modification was to a linked property that did not cause the state of the sections to change. ([#5912](https://github.com/realm/realm-core/issues/5912), since the introduction of sectioned results in v12.3.0)
 * CompensatingWriteErrorInfo reported string primary keys as boolean values instead ([PR #5938](https://github.com/realm/realm-core/pull/5938), since the introduction of CompensatingWriteErrorInfo in 12.1.0).
+* Fix a use-after-free if the last external reference to an encrypted Realm was closed between when a client reset error was received and when the download of the new Realm began. ([PR #5949](https://github.com/realm/realm-core/pull/5949), since 12.4.0).
+* Fixed an assertion failure during client reset with recovery when recovering a list operation on an embedded object that has a link column in the path prefix to the list from the top level object. ([PR #5957](https://github.com/realm/realm-core/issues/5957), since introduction of automatic recovery in v11.16.0).
 
 ### Breaking changes
-* Rename RealmConfig::automatic_handle_backlicks_in_migrations to RealmConfig::automatically_handle_backlinks_in_migrations  ([PR #5897](https://github.com/realm/realm-core/pull/5897)).
-* All exceptions thrown out of Core are now of type 'Exception'. All use of std::runtime_error and std::logical_error etc. has stopped and the specialized error classes that beforehand were based on these are now based on Exception.
+* Rename RealmConfig::automatic_handle_backlicks_in_migrations to RealmConfig::automatically_handle_backlinks_in_migrations ([PR #5897](https://github.com/realm/realm-core/pull/5897)).
+* Introduced new callback type realm_return_apikey_list_func_t and realm_return_apikey_func_t in the C-API ([PR #5945](https://github.com/realm/realm-core/pull/5945)).
 
 ### Compatibility
 * Fileformat: Generates files with format v22. Reads and automatically upgrade from fileformat v5.
@@ -41,6 +85,7 @@
 * Changed the behaviour of creating a collection of non-nullable Mixed type to throw a descriptive error message rather than asserting in debug mode. ([#5894](https://github.com/realm/realm-core/issues/5894), since the introduction of the Mixed type).
 * Fix a use-after-free when a sync session is closed and the app is destroyed at the same time ([#5752](https://github.com/realm/realm-core/issues/5752), since v11.5.2).
 * Fix Http transport doesn't correctly preserve the request body ([#5890](https://github.com/realm/realm-core/issues/5890), since 12.8.0).
+* IN Query fails if left operator is a TypedLink ([5946](https://github.com/realm/realm-core/issues/5946), since v10.5.2)
 
 ### Breaking changes
 * Removed breaking callback changes for `GenericNetworkTransport::send_request_to_server()` ([PR #5898](https://github.com/realm/realm-core/pull/5898)).

diff --git a/Package.swift b/Package.swift
@@ -3,7 +3,7 @@
 import PackageDescription
 import Foundation
 
-let versionStr = "12.9.0"
+let versionStr = "12.11.0"
 let versionPieces = versionStr.split(separator: "-")
 let versionCompontents = versionPieces[0].split(separator: ".")
 let versionExtra = versionPieces.count > 1 ? versionPieces[1] : ""

diff --git a/dependencies.list b/dependencies.list
@@ -1,4 +1,4 @@
 PACKAGE_NAME=realm-core
-VERSION=12.9.0
+VERSION=12.11.0
 OPENSSL_VERSION=1.1.1n
 MDBREALM_TEST_SERVER_TAG=2022-10-10
diff --git a/evergreen/config.yml b/evergreen/config.yml
@@ -529,6 +529,8 @@ tasks:
   exec_timeout_secs: 1800
   commands:
   - func: "compile"
+    vars:
+      target_to_build: "CoreTests"
   - func: "run tests"
     vars:
       test_filter: CoreTests
@@ -696,8 +698,7 @@ task_groups:
   - compile
   - .test_suite
 
-- name: compile_core_tests
-  max_hosts: 1
+- name: core_tests_group
   setup_group_can_fail_task: true
   setup_group:
   - func: "fetch source"
@@ -707,7 +708,6 @@ task_groups:
   timeout:
   - func: "run hang analyzer"
   tasks:
-  - compile
   - core-tests
 
 - name: benchmarks
@@ -771,7 +771,7 @@ buildvariants:
     run_with_encryption: On
     enable_tsan: On
   tasks:
-  - name: compile_core_tests
+  - name: core_tests_group
     distros:
     - ubuntu2004-large
 
@@ -790,7 +790,7 @@ buildvariants:
     run_with_encryption: On
     enable_asan: On
   tasks:
-  - name: compile_core_tests
+  - name: core_tests_group
     distros:
     - ubuntu2004-large
 
@@ -948,7 +948,7 @@ buildvariants:
     extra_flags: -DCMAKE_SYSTEM_NAME=Darwin -DCMAKE_OSX_ARCHITECTURES=x86_64
     run_with_encryption: On
   tasks:
-  - name: compile_core_tests
+  - name: core_tests_group
 
 - name: macos-release
   display_name: "MacOS 11.0 x86_64 (Release build)"
@@ -1009,7 +1009,7 @@ buildvariants:
 
 - name: windows-64-vs2019
   display_name: "Windows x86_64 (VS 2019)"
-  run_on: windows-vsCurrent-small
+  run_on: windows-vsCurrent-large
   expansions:
     cmake_url: "https://s3.amazonaws.com/static.realm.io/evergreen-assets/cmake-3.20.3-windows-x86_64.zip"
     cmake_bindir: "./cmake-3.20.3-windows-x86_64/bin"
@@ -1021,13 +1021,27 @@ buildvariants:
     python3: "/cygdrive/c/python/python37/python.exe"
   tasks:
   - name: compile_test_and_package
-    distros:
-    - windows-vsCurrent-large
   - name: long-running-tests
 
+- name: windows-64-encryption
+  display_name: "Windows x86_64 (Encryption enabled)"
+  run_on: windows-vsCurrent-large
+  expansions:
+    cmake_url: "https://s3.amazonaws.com/static.realm.io/evergreen-assets/cmake-3.20.3-windows-x86_64.zip"
+    cmake_bindir: "./cmake-3.20.3-windows-x86_64/bin"
+    cmake_generator: "Visual Studio 16 2019"
+    extra_flags: "-A x64"
+    max_jobs: $(($(grep -c proc /proc/cpuinfo) / 2))
+    fetch_missing_dependencies: On
+    build_zlib: On
+    python3: "/cygdrive/c/python/python37/python.exe"
+    run_with_encryption: On
+  tasks:
+  - name: core_tests_group
+
 - name: windows-64-vs2019-release
   display_name: "Windows x86_64 (VS 2019 Release build)"
-  run_on: windows-vsCurrent-small
+  run_on: windows-vsCurrent-large
   expansions:
     cmake_url: "https://s3.amazonaws.com/static.realm.io/evergreen-assets/cmake-3.20.3-windows-x86_64.zip"
     cmake_bindir: "./cmake-3.20.3-windows-x86_64/bin"
@@ -1040,5 +1054,3 @@ buildvariants:
     python3: "/cygdrive/c/python/python37/python.exe"
   tasks:
   - name: compile_test
-    distros:
-    - windows-vsCurrent-large
diff --git a/src/realm.h b/src/realm.h
@@ -1371,7 +1371,8 @@ RLM_API realm_object_t* realm_get_object(const realm_t*, realm_class_key_t class
  * Get the parent object for the object passed as argument. Only works for embedded objects.
  * @return true, if no errors occurred.
  */
-RLM_API bool realm_object_get_parent(const realm_object_t* object, realm_object_t* parent);
+RLM_API bool realm_object_get_parent(const realm_object_t* object, realm_object_t** parent,
+                                     realm_class_key_t* class_key);
 
 /**
  * Find an object with a particular primary key value.
@@ -2678,6 +2679,11 @@ typedef struct realm_user_identity {
     realm_auth_provider_e provider_type;
 } realm_user_identity_t;
 
+typedef void (*realm_return_apikey_func_t)(realm_userdata_t userdata, realm_app_user_apikey_t*,
+                                           const realm_app_error_t*);
+typedef void (*realm_return_apikey_list_func_t)(realm_userdata_t userdata, realm_app_user_apikey_t[], size_t count,
+                                                realm_app_error_t*);
+
 /**
  * Generic completion callback for asynchronous Realm App operations.
  *
@@ -2976,36 +2982,38 @@ RLM_API bool realm_app_email_password_provider_client_call_reset_password_functi
  * Creates a user API key that can be used to authenticate as the current user.
  * @return True if no error was recorded. False otherwise
  */
-RLM_API bool realm_app_user_apikey_provider_client_create_apikey(
-    const realm_app_t*, const realm_user_t*, const char* name,
-    void (*)(realm_userdata_t userdata, realm_app_user_apikey_t*, const realm_app_error_t*),
-    realm_userdata_t userdata, realm_free_userdata_func_t userdata_free);
+RLM_API bool realm_app_user_apikey_provider_client_create_apikey(const realm_app_t*, const realm_user_t*,
+                                                                 const char* name,
+                                                                 realm_return_apikey_func_t callback,
+                                                                 realm_userdata_t userdata,
+                                                                 realm_free_userdata_func_t userdata_free);
 
 /**
  * Fetches a user API key associated with the current user.
  * @return True if no error was recorded. False otherwise
  */
-RLM_API bool realm_app_user_apikey_provider_client_fetch_apikey(
-    const realm_app_t*, const realm_user_t*, realm_object_id_t id,
-    void (*)(realm_userdata_t userdata, realm_app_user_apikey_t*, const realm_app_error_t*),
-    realm_userdata_t userdata, realm_free_userdata_func_t userdata_free);
+RLM_API bool realm_app_user_apikey_provider_client_fetch_apikey(const realm_app_t*, const realm_user_t*,
+                                                                realm_object_id_t id,
+                                                                realm_return_apikey_func_t callback,
+                                                                realm_userdata_t userdata,
+                                                                realm_free_userdata_func_t userdata_free);
 
 /**
  * Fetches the user API keys associated with the current user.
  * @return True if no error was recorded. False otherwise
  */
-RLM_API bool realm_app_user_apikey_provider_client_fetch_apikeys(
-    const realm_app_t*, const realm_user_t*,
-    void (*)(realm_userdata_t userdata, realm_app_user_apikey_t[], size_t count, realm_app_error_t*),
-    realm_userdata_t userdata, realm_free_userdata_func_t userdata_free);
+RLM_API bool realm_app_user_apikey_provider_client_fetch_apikeys(const realm_app_t*, const realm_user_t*,
+                                                                 realm_return_apikey_list_func_t callback,
+                                                                 realm_userdata_t userdata,
+                                                                 realm_free_userdata_func_t userdata_free);
 
 /**
  * Deletes a user API key associated with the current user.
  * @return True if no error was recorded. False otherwise
  */
 RLM_API bool realm_app_user_apikey_provider_client_delete_apikey(const realm_app_t*, const realm_user_t*,
                                                                  realm_object_id_t id,
-                                                                 realm_app_void_completion_func_t,
+                                                                 realm_app_void_completion_func_t callback,
                                                                  realm_userdata_t userdata,
                                                                  realm_free_userdata_func_t userdata_free);
 
@@ -3015,7 +3023,7 @@ RLM_API bool realm_app_user_apikey_provider_client_delete_apikey(const realm_app
  */
 RLM_API bool realm_app_user_apikey_provider_client_enable_apikey(const realm_app_t*, const realm_user_t*,
                                                                  realm_object_id_t id,
-                                                                 realm_app_void_completion_func_t,
+                                                                 realm_app_void_completion_func_t callback,
                                                                  realm_userdata_t userdata,
                                                                  realm_free_userdata_func_t userdata_free);
 
@@ -3025,7 +3033,7 @@ RLM_API bool realm_app_user_apikey_provider_client_enable_apikey(const realm_app
  */
 RLM_API bool realm_app_user_apikey_provider_client_disable_apikey(const realm_app_t*, const realm_user_t*,
                                                                   realm_object_id_t id,
-                                                                  realm_app_void_completion_func_t,
+                                                                  realm_app_void_completion_func_t callback,
                                                                   realm_userdata_t userdata,
                                                                   realm_free_userdata_func_t userdata_free);
 

diff --git a/src/realm/alloc_slab.cpp b/src/realm/alloc_slab.cpp
@@ -732,7 +732,7 @@ ref_type SlabAlloc::attach_file(const std::string& file_path, Config& cfg)
     if (cfg.encryption_key && size == 0 && physical_file_size != 0) {
         // The opened file holds data, but is so small it cannot have
         // been created with encryption
-        throw RuntimeError(ErrorCodes::DecryptionFailed, "Attempt to open unencrypted file with encryption key");
+        throw InvalidDatabase("Attempt to open unencrypted file with encryption key", path);
     }
     if (size == 0 || cfg.clear_file) {
         if (REALM_UNLIKELY(cfg.read_only))
@@ -757,16 +757,21 @@ ref_type SlabAlloc::attach_file(const std::string& file_path, Config& cfg)
         note_reader_start(this);
         // we'll read header and (potentially) footer
         File::Map<char> map_header(m_file, File::access_ReadOnly, sizeof(Header));
-        // if file is too small we'll catch it in validate_header - but we need to prevent an invalid mapping first
-        size_t footer_ref = size < (sizeof(StreamingFooter) + sizeof(Header)) ? 0 : (size - sizeof(StreamingFooter));
-        size_t footer_page_base = footer_ref & ~(page_size() - 1);
-        size_t footer_offset = footer_ref - footer_page_base;
-        File::Map<char> map_footer(m_file, footer_page_base, File::access_ReadOnly,
-                                   sizeof(StreamingFooter) + footer_offset, 0);
         realm::util::encryption_read_barrier(map_header, 0, sizeof(Header));
-        realm::util::encryption_read_barrier(map_footer, footer_offset, sizeof(StreamingFooter));
         auto header = reinterpret_cast<const Header*>(map_header.get_addr());
-        auto footer = reinterpret_cast<const StreamingFooter*>(map_footer.get_addr() + footer_offset);
+
+        File::Map<char> map_footer;
+        const StreamingFooter* footer = nullptr;
+        if (is_file_on_streaming_form(*header) && size >= sizeof(StreamingFooter) + sizeof(Header)) {
+            size_t footer_ref = size - sizeof(StreamingFooter);
+            size_t footer_page_base = footer_ref & ~(page_size() - 1);
+            size_t footer_offset = footer_ref - footer_page_base;
+            map_footer = File::Map<char>(m_file, footer_page_base, File::access_ReadOnly,
+                                         sizeof(StreamingFooter) + footer_offset, 0);
+            realm::util::encryption_read_barrier(map_footer, footer_offset, sizeof(StreamingFooter));
+            footer = reinterpret_cast<const StreamingFooter*>(map_footer.get_addr() + footer_offset);
+        }
+
         top_ref = validate_header(header, footer, size, path); // Throws
         m_attach_mode = cfg.is_shared ? attach_SharedFile : attach_UnsharedFile;
         m_data = map_header.get_addr(); // <-- needed below
@@ -1015,6 +1020,7 @@ ref_type SlabAlloc::validate_header(const Header* header, const StreamingFooter*
             std::string msg = "Invalid streaming format size (" + util::to_string(size) + ")";
             throw InvalidDatabase(msg, path);
         }
+        REALM_ASSERT(footer);
         top_ref = footer->m_top_ref;
         if (REALM_UNLIKELY(footer->m_magic_cookie != footer_magic_cookie)) {
             std::string msg = "Invalid streaming format cookie (" + util::to_string(footer->m_magic_cookie) + ")";