RCORE-1973 Add role/permissions tests for new bootstrap feature #7675

michael-wb · 2024-05-03T16:53:46Z

What, How & Why?

Moved the role change bootstrap tests to their own file and reduced the number of times the App was created and destroyed during the tests.
Additional tests for the server initiated bootstraps; specifically for role change bootstraps:

"Pending changes are lost if not allowed after role change" - Verify pending local changes are either recovered or lost during role change depending on whether changes are still in view after role change.
"Role changes during bootstrap complete successfully" - Verify role change during initial schema and subscriptions bootstraps complete successfully and have the correct data.

Fixes #7327

☑️ ToDos

📝 Changelog update
🚦 Tests (or not relevant)
~~[ ] C-API, if public C++ API changed~~
~~[ ] bindgen/spec.yml, if public C++ API changed~~

coveralls-official · 2024-05-03T18:11:34Z

Pull Request Test Coverage Report for Build michael.wilkersonbarker_1094

Details

157 of 157 (100.0%) changed or added relevant lines in 2 files are covered.
47 unchanged lines in 9 files lost coverage.
Overall coverage increased (+0.3%) to 90.761%

Files with Coverage Reduction	New Missed Lines	%
src/realm/sort_descriptor.cpp	1	94.06%
test/test_table.cpp	1	99.51%
src/realm/table_view.cpp	2	92.99%
src/realm/sync/noinst/server/server_history.cpp	3	63.51%
src/realm/table.cpp	3	90.08%
src/realm/bplustree.cpp	6	72.55%
src/realm/sync/noinst/client_impl_base.cpp	7	82.36%
src/realm/sync/noinst/server/server.cpp	7	73.89%
test/fuzz_group.cpp	17	55.02%

Totals
Change from base Build michael.wilkersonbarker_1088:	0.3%
Covered Lines:	212810
Relevant Lines:	234473

💛 - Coveralls

…iption store during server initiated boostrap

…o mwb/user-role-change-tests

… change

…o mwb/user-role-change-tests

src/realm/sync/client.cpp

danieltabacaru · 2024-06-19T10:41:30Z

src/realm/sync/client.cpp

@@ -1208,10 +1212,24 @@ void SessionWrapper::on_flx_sync_progress(int64_t new_version, DownloadBatchStat
        return;
    }
    REALM_ASSERT(!m_finalized);
-    REALM_ASSERT(new_version >= m_flx_last_seen_version);


shouldn't new_version always be either m_flx_last_seen_version, m_flx_active_version or m_flx_active_version+1?

If a query bootstrap is in progress when a server initiated bootstrap takes place (which cancels the current query bootstrap), m_flx_last_seen_version will be m_flx_active_version + 1 while new_version will be m_flx_active_version and this assertion will fail.

it's still one of the cases I mentioned. I was proposing we update the assert, not keep the deleted one.

the m_flx_last_seen_version can actually be greater than m_flx_active_version + 1 in the case of a QUERY_ERROR response and it always starts out at 0 when the session is started until the first download message is received. Since the current validation and checking for server-initiated bootstrap logic is valid, I am going to leave this as-is.

It seems that m_flx_last_seen_version should be updated when we get a query error then. m_flx_active_version is initialized though when the SessionWrapper is actualized. Either way, your call.

I would update it, but this is a change that might affect progress notifications, which uses this value. I created RCORE-2173 to handle updating this separately if we want to update it.

test/object-store/sync/flx_role_change.cpp

…o mwb/user-role-change-tests

… check for role change bootstrap as first validation step

…o mwb/user-role-change-tests

jbreams · 2024-06-27T20:16:28Z

test/object-store/sync/flx_role_change.cpp

+    if (!harness) {
+        return; // nothing to do
+    }
+    harness->app()->sync_manager()->wait_for_sessions_to_terminate();


Why do we need to explicitly wait for sessions to terminate?

I don't think we need to - this was copied over from another test case that reused the harness across its sections. This entire function was removed.

jbreams · 2024-06-27T20:16:56Z

test/object-store/sync/flx_role_change.cpp

+        return; // nothing to do
+    }
+    harness->app()->sync_manager()->wait_for_sessions_to_terminate();
+    harness.reset();


Do we re-use the harness unique_ptr? Why not just take the unique_ptr by value and let it go out of scope? Why have this function at all?

A single harness is re-used across the sections within a test_case. I ended up removing this function and just call harness.reset() in the last section.

jbreams · 2024-06-27T20:18:42Z

test/object-store/sync/flx_role_change.cpp

+                    case Event::ErrorMessageReceived:
+                        REQUIRE(cur_state == TestState::start);
+                        REQUIRE(data.error_info);
+                        REQUIRE(data.error_info->raw_error_code == 200);


should we compare the error_code rather than the raw_error_code?

This is a ProtocolErrorInfo struct, which doesn't have error_code; only raw_error_code is available.

jbreams · 2024-06-27T20:22:26Z

test/object-store/sync/flx_role_change.cpp

+        state_machina.transition_with([&](TestState cur_state) {
+            switch (expected.bootstrap) {
+                case BootstrapMode::NoReconnect:
+                    // Confirm that the session did receive an error and a bootstrap did not occur


Where do we confirm that we received an error? Should the name of this enum value be NoBootstrapNoError or something? Not sure what NoReconnect actually means here.

Renamed to NoErrorNoBootstrap and updated comment

jbreams · 2024-06-27T20:23:47Z

test/object-store/sync/flx_role_change.cpp

+                    REQUIRE_FALSE(role_change_bootstrap);
+                    break;
+                case BootstrapMode::None:
+                    // Confirm that a bootstrap nor a client reset did not occur


So this is something like NoBootstrapReceivedError? "Confirm that neither a bootstrap nor a client rest did not occur" seems like a double negative?

Renamed state to GotErrorNoBootstrap and updated comment

jbreams · 2024-06-27T20:26:19Z

test/object-store/sync/flx_role_change.cpp

+            update_perms_and_verify(*harness, realm_1, test_rules,
+                                    {BootstrapMode::Any, params.num_emps, params.num_mgrs, 0});
+
+            // Realm 2 data should not change (and there shouldn't be any bootstrap messages)


If there shouldn't be any bootstrap messages, why do we pass BootstrapMode::Any to update_perms_and_verify above?

I added a comment for this - realm_1 should receive a role change bootstrap, but realm_2 should not (and there is a on_sync_client_event_hook fcn for realm_2 that verifies this).

jbreams · 2024-06-27T20:26:44Z

test/object-store/sync/flx_role_change.cpp

+            update_perms_and_verify(*harness, realm_1, test_rules,
+                                    {BootstrapMode::Any, params.num_emps, params.num_mgrs, params.num_dirs});
+
+            // Realm 2 data should not change (and there shouldn't be any bootstrap messages)


If there shouldn't be any bootstrap messages, why do we pass BootstrapMode::Any to update_perms_and_verify above? Or am I mis-understanding the meaning of BootstrapMode::Any?

BootstrapMode::Any means that a bootstrap is expected, but we don't care what type of bootstrap (single message, multi message, etc.) occurs.

jbreams · 2024-06-27T20:28:11Z

test/object-store/sync/flx_role_change.cpp

+                if (cur_state == BootstrapTestState::not_ready)
+                    return std::nullopt;
+
+                std::optional<BootstrapTestState> new_state = std::nullopt;


this will default initialize to std::nullopt.

jbreams · 2024-06-27T20:28:52Z

test/object-store/sync/flx_role_change.cpp

+
+                    case Event::ErrorMessageReceived:
+                        REQUIRE(data.error_info);
+                        if (data.error_info->raw_error_code == 200) {


are there other errors we could receive here that we don't want to track?

Not really - changed the if to a REQUIRE()

jbreams · 2024-06-27T20:29:22Z

test/object-store/sync/flx_role_change.cpp

+                        break;
+
+                    case Event::ErrorMessageReceived:
+                        REQUIRE(data.error_info);


I think we can just assume the sync client will do the right thing here.

danieltabacaru · 2024-06-26T11:43:23Z

test/object-store/sync/flx_role_change.cpp

+    return harness;
+}
+
+void teardown_harness(std::unique_ptr<FLXSyncTestHarness>& harness)


I think this should not be a reference and instead pass the ownership because it's not expected to use the harness after calling this function.

I actually removed this function.

…o mwb/user-role-change-tests

… bootstrap is interrupted by role change (#7831) * Updated pending bootstrap store to be processed (applied or cleared) when the session or connection is restarted without restarting the Sync Session

danieltabacaru · 2024-07-02T12:34:30Z

test/object-store/sync/flx_role_change.cpp

+    // 10 objects before flush
+    // 1536 download soft max bytes
+    TestParams params{};
+    params.max_download_bytes = 1536;


what's the meaning of this value?

it is sent to the server to configure the "qbs_download_changeset_soft_max_byte_size" parameter that limits the number of bytes of changeset data in a download message. This is used to break up the bootstrap into multiple messages to test the multi-part bootstrap download and to "slow down" the bootstrap processing a bit.

I know that, but why this specific value?

It is totally arbitrary - I wanted a value more than 1K and less than 2K, so I chose 1.5 KB (1024 * 1.5) = 1536

danieltabacaru · 2024-07-02T12:55:17Z

test/object-store/sync/flx_role_change.cpp

+    // ----------------------------------------------------------------
+    // Add new sections before this one
+    // ----------------------------------------------------------------
+    SECTION("Pending changes are lost if not allowed after role change") {


do we have a test where pending changes are allowed?

That test actually validates pending changes that are both allowed and not allowed after the role change.
I can update the name of the test to clarify that

…o mwb/user-role-change-tests

jbreams · 2024-07-01T17:32:44Z

CHANGELOG.md

@@ -2,7 +2,7 @@

 ### Enhancements
 * <New feature description> (PR [#????](https://github.com/realm/realm-core/pull/????))
-* None.
+* Add support for server initiated bootstraps. ([PR #7440](https://github.com/realm/realm-core/pull/7440))


Wasn't this already done in a separate PR? Should this be under Internals? What is a user or SDK developer supposed to take away from this changelog entry? How will this effect them?

I moved this comment to the internal section and added a new comment to Enhancements:
Role and permissions changes no longer require a client reset to update the local realm.

jbreams · 2024-07-01T17:33:07Z

CHANGELOG.md

@@ -20,6 +20,8 @@

 ### Internals
 * Fixed `Table::remove_object_recursive` which wouldn't recursively follow links through a single `Mixed` property. This feature is exposed publicly on `Table` but no SDK currently uses it, so this is considered internal. ([#7829](https://github.com/realm/realm-core/issues/7829), likely since the introduction of Mixed)
+* Protocol version has been updated to v14 to support server intiated bootstraps and role change updates without a client reset. ([PR #7440](https://github.com/realm/realm-core/pull/7440))
+* Create additional role change tests to verify role change during initial schema and subscription bootstraps. ([PR #7675](https://github.com/realm/realm-core/pull/7675))


I don't think we need a changelog entry for just adding new tests.

jbreams · 2024-07-01T17:34:14Z

CHANGELOG.md

@@ -52,7 +54,6 @@
 * It is no longer an error to set a base url for an App with a trailing slash - for example, `https://services.cloud.mongodb.com/` instead of `https://services.cloud.mongodb.com` - before this change that would result in a 404 error from the server ([PR #7791](https://github.com/realm/realm-core/pull/7791)).
 * Performance has been improved for range queries on integers and timestamps. Requires that you use the "BETWEEN" operation in MQL or the Query::between() method when you build the query. (PR [#7785](https://github.com/realm/realm-core/pull/7785))
 * Expose `Obj::add_int()` in the bindgen spec. ([PR #7797](https://github.com/realm/realm-core/pull/7797)).
-* Add support for server initiated bootstraps. ([PR #7440](https://github.com/realm/realm-core/pull/7440))


Ah, I see, you're handling merge conflicts so these are going to jump during each PR remaining for this project ☹️

jbreams · 2024-07-03T19:18:26Z

src/realm/sync/noinst/client_impl_base.cpp

@@ -1527,6 +1527,16 @@ void Session::cancel_resumption_delay()
    if (unbind_process_complete())
        initiate_rebind(); // Throws

+    try {


How is this different from #7831 ?

It probably wasn't the best (or correct) way to do it, but PR #7831 was branched off of this PR branch, since it needed tests from this PR in order to validate the changes...

…o mwb/user-role-change-tests

* RCORE-1872 Sync client should allow server bootstrapping at any time (#7440) * First round of changes for server-initiated bootstraps * Added test for role change bootstraps * Updated test for handle role bootstraps * Updated baas/baasaas to use branch with fixes * Updated test to verify bootstrap actually occurred * Fixed tsan warning * Updates from review; added comments to clarify bootstrap detection logic * Reverted baas branch to master and protocol version to 12 * Added comments to changes needed when merging to master; update baas version to not use master * Pulled over changes from other branch and tweaking download params * Refactored tests to validate different bootstrap types * Updated tests to get passing using the server params * Updated to support new batch_state protocol changes; updated tests * Updated role change tests and merged test from separate PR * Fixed issue with flx query verion 0 not being treated as a bootstrap * Cleaned up the tests a bit and reworked query version 0 handling * Updates from review; updated batch_state for schema bootstraps * Removed extra mutex in favor of state machine's mutex * Increased timeout when waiting for app initial sync to complete * Updated role change test to use test commands * Update resume and ident message handling * Updated future waits for the pause/resume test command * Added session connected event for when session multiplexing is disabled * Added wait_until() to state machine to wait for callback; updated role change test * RCORE-1973 Add role/permissions tests for new bootstrap feature (#7675) * Moved role change tests to separate test file * Fixed building of new flx_role_change.cpp file * Added local changes w/role bootstrap test - fixed exception in subscription store during server initiated boostrap * Updated local change test to include valid offline writes during role change * Added role change test during initial schema bootstrap * Wrapped up role change during bootstrap tests * Removed debug statments to fix thread sanitizer * Updated sub state comments and reverted a minor change * Refactored role change tests and broke out into 2 separate test cases * Moved harness from a global to a static var in each test case * Reverted resetting the bootstrapping subscription state back to Pending * Updated baas to use protocol v14 and removed the feature flag for role change bootstraps * Updated baasaas version to be a cached version * Updated baasaas githash and reordered role change during bootstrap to check for role change bootstrap as first validation step * Minor updates to reuse the verify_records() fcn * RCORE-2174 Bootstrap store is not being reset if initial subscription bootstrap is interrupted by role change (#7831) * Updated pending bootstrap store to be processed (applied or cleared) when the session or connection is restarted without restarting the Sync Session * RCORE-1974 Add tests for role/permissions changed during client reset (#7840) * re-applied changes after base branch was merged to feature branch * Updates to address test failures * Disable role change check during fresh realm download * Updated comments for clarity * Updates from review; added a bunch of comments to test * Updates to role change tests per review comments * removed ostream support for SyncClientHookEvent

michael-wb self-assigned this May 3, 2024

cla-bot bot added the cla: yes label May 3, 2024

michael-wb linked an issue May 3, 2024 that may be closed by this pull request

Add role/permissions tests for new bootstrap feature #7327

Closed

Base automatically changed from mwb/role-change-bootstraps to feature/role-change June 11, 2024 17:13

Michael Wilkerson-Barker added 8 commits June 11, 2024 14:00

Moved role change tests to separate test file

67691ce

Fixed building of new flx_role_change.cpp file

2d304d5

Added local changes w/role bootstrap test - fixed exception in subscr…

8d91a08

…iption store during server initiated boostrap

Merge branch 'feature/role-change' of github.com:realm/realm-core int…

e8924dc

…o mwb/user-role-change-tests

Updated local change test to include valid offline writes during role…

ac79b1f

… change

Added role change test during initial schema bootstrap

b78283e

Merge branch 'feature/role-change' of github.com:realm/realm-core int…

d350217

…o mwb/user-role-change-tests

Updated changelog

f9aec9b

michael-wb force-pushed the mwb/user-role-change-tests branch from eee7a17 to f9aec9b Compare June 18, 2024 13:45

Michael Wilkerson-Barker added 3 commits June 18, 2024 23:05

Wrapped up role change during bootstrap tests

2d908be

Merge branch 'feature/role-change' of github.com:realm/realm-core int…

1878811

…o mwb/user-role-change-tests

Removed debug statments to fix thread sanitizer

89b2c39

michael-wb marked this pull request as ready for review June 19, 2024 04:14

michael-wb requested review from jbreams and danieltabacaru June 19, 2024 04:14

Updated sub state comments and reverted a minor change

363581c

danieltabacaru reviewed Jun 19, 2024

View reviewed changes

src/realm/sync/client.cpp Outdated Show resolved Hide resolved

danieltabacaru reviewed Jun 19, 2024

View reviewed changes

test/object-store/sync/flx_role_change.cpp Outdated Show resolved Hide resolved

Michael Wilkerson-Barker added 2 commits June 19, 2024 11:34

updates from review

990da9e

Merge branch 'feature/role-change' of github.com:realm/realm-core int…

fb91292

…o mwb/user-role-change-tests

michael-wb requested a review from danieltabacaru June 19, 2024 15:41

Michael Wilkerson-Barker added 2 commits June 19, 2024 15:10

Refactored role change tests and broke out into 2 separate test cases

5ad76e9

Moved harness from a global to a static var in each test case

2eb382e

sync-by-unito bot mentioned this pull request Jun 20, 2024

Update m_flx_active_version when an QUERY_ERROR occurs #7825

Closed

Michael Wilkerson-Barker added 3 commits June 20, 2024 16:45

Updated baasaas githash and reordered role change during bootstrap to…

6994a85

… check for role change bootstrap as first validation step

Merge branch 'feature/role-change' of github.com:realm/realm-core int…

579c50d

…o mwb/user-role-change-tests

Minor updates to reuse the verify_records() fcn

7983f9d

michael-wb mentioned this pull request Jun 26, 2024

RCORE-1974 Add tests for role/permissions changed during client reset #7840

Merged

1 task

jbreams reviewed Jun 27, 2024

View reviewed changes

danieltabacaru reviewed Jun 27, 2024

View reviewed changes

Michael Wilkerson-Barker added 2 commits June 27, 2024 21:14

Merge branch 'feature/role-change' of github.com:realm/realm-core int…

3f17ce6

…o mwb/user-role-change-tests

Updates from review

524261a

michael-wb requested review from jbreams and danieltabacaru June 28, 2024 15:23

Michael Wilkerson-Barker added 6 commits June 30, 2024 20:57

Merge branch 'feature/role-change' of github.com:realm/realm-core int…

48497b1

…o mwb/user-role-change-tests

Updated changelog after release

2d67e5d

Fixed one more comment in changelog

2d27215

Merge branch 'feature/role-change' of github.com:realm/realm-core int…

7ee0ad0

…o mwb/user-role-change-tests

Merge branch 'feature/role-change' of github.com:realm/realm-core int…

75c9654

…o mwb/user-role-change-tests

RCORE-2174 Bootstrap store is not being reset if initial subscription…

321b889

… bootstrap is interrupted by role change (#7831) * Updated pending bootstrap store to be processed (applied or cleared) when the session or connection is restarted without restarting the Sync Session

danieltabacaru reviewed Jul 2, 2024

View reviewed changes

danieltabacaru approved these changes Jul 2, 2024

View reviewed changes

danieltabacaru reviewed Jul 2, 2024

View reviewed changes

Merge branch 'feature/role-change' of github.com:realm/realm-core int…

9f4b041

…o mwb/user-role-change-tests

jbreams reviewed Jul 3, 2024

View reviewed changes

jbreams approved these changes Jul 3, 2024

View reviewed changes

jbreams reviewed Jul 3, 2024

View reviewed changes

Michael Wilkerson-Barker added 2 commits July 3, 2024 16:33

Updated changelog comments

f869865

Merge branch 'feature/role-change' of github.com:realm/realm-core int…

2e5a010

…o mwb/user-role-change-tests

michael-wb merged commit 03305a4 into feature/role-change Jul 4, 2024
40 checks passed

michael-wb deleted the mwb/user-role-change-tests branch July 4, 2024 04:14

github-actions bot mentioned this pull request Jul 19, 2024

Prepare for 14.11.0 #7901

Merged

github-actions bot locked as resolved and limited conversation to collaborators Aug 3, 2024

RCORE-1973 Add role/permissions tests for new bootstrap feature #7675

RCORE-1973 Add role/permissions tests for new bootstrap feature #7675

Conversation

michael-wb commented May 3, 2024 • edited Loading

What, How & Why?

☑️ ToDos

coveralls-official bot commented May 3, 2024 • edited Loading

Pull Request Test Coverage Report for Build michael.wilkersonbarker_1094

Details

💛 - Coveralls

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

danieltabacaru Jun 19, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

michael-wb Jun 28, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

michael-wb commented May 3, 2024 •

edited

Loading

coveralls-official bot commented May 3, 2024 •

edited

Loading

danieltabacaru Jun 19, 2024 •

edited

Loading

michael-wb Jun 28, 2024 •

edited

Loading