Cets/prevent crash on upgrade #4256
Conversation
When performing a rolling upgrade e.g. from 6.2.0 to the upcoming 6.2.1, CETS would crash on all nodes due to version incompatibility. This can be prevented by putting cets 'Major.Minor' version in the cluster name, making the two cluster components disjoint during upgrade. The only thing to keep in mind is to bump the Major or Minor version number when there is any incompatibility in the CETS inter-node API.
|
elasticsearch_and_cassandra_26 / elasticsearch_and_cassandra_mnesia / e4ef412 small_tests_25 / small_tests / e4ef412 small_tests_26 / small_tests / e4ef412 small_tests_26_arm64 / small_tests / e4ef412 ldap_mnesia_25 / ldap_mnesia / e4ef412 pubsub_SUITE:dag+basic:request_all_items_test{error,{{badmatch,false},
[{pubsub_tools,check_response,2,
[{file,"/home/circleci/project/big_tests/tests/pubsub_tools.erl"},
{line,444}]},
{pubsub_tools,receive_response,3,
[{file,"/home/circleci/project/big_tests/tests/pubsub_tools.erl"},
{line,434}]},
{pubsub_tools,receive_and_check_response,4,
[{file,"/home/circleci/project/big_tests/tests/pubsub_tools.erl"},
{line,424}]},
{escalus_story,story,4,
[{file,"/home/circleci/project/big_tests/_build/default/lib/escalus/src/escalus_story.erl"},
{line,72}]},
{test_server,ts_tc,3,[{file,"test_server.erl"},{line,1782}]},
{test_server,run_test_case_eval1,6,
[{file,"test_server.erl"},{line,1291}]},
{test_server,run_test_case_eval,9,
[{file,"test_server.erl"},{line,1223}]}]}}dynamic_domains_pgsql_mnesia_25 / pgsql_mnesia / e4ef412 dynamic_domains_pgsql_mnesia_26 / pgsql_mnesia / e4ef412 ldap_mnesia_26 / ldap_mnesia / e4ef412 dynamic_domains_mysql_redis_26 / mysql_redis / e4ef412 pgsql_cets_26 / pgsql_cets / e4ef412 graphql_cets_SUITE:init_per_suite{'EXIT',{{timeout,true,[{times,50,false}],ok},
[{mongoose_helper,do_wait_until,2,
[{file,"/home/circleci/project/big_tests/../test/common/mongoose_helper.erl"},
{line,359}]},
{graphql_cets_SUITE,init_per_suite,1,
[{file,"/home/circleci/project/big_tests/tests/graphql_cets_SUITE.erl"},
{line,51}]},
{test_server,ts_tc,3,[{file,"test_server.erl"},{line,1793}]},
{test_server,run_test_case_eval1,6,
[{file,"test_server.erl"},{line,1390}]},
{test_server,run_test_case_eval,9,
[{file,"test_server.erl"},{line,1234}]}]}}internal_mnesia_26 / internal_mnesia / e4ef412 dynamic_domains_mssql_mnesia_26 / odbc_mssql_mnesia / e4ef412 mysql_redis_26 / mysql_redis / e4ef412 pgsql_mnesia_25 / pgsql_mnesia / e4ef412 pgsql_mnesia_26 / pgsql_mnesia / e4ef412 mssql_mnesia_26 / odbc_mssql_mnesia / e4ef412 |
Codecov ReportAll modified and coverable lines are covered by tests ✅
Additional details and impacted files@@ Coverage Diff @@
## master #4256 +/- ##
==========================================
- Coverage 84.43% 84.41% -0.03%
==========================================
Files 552 552
Lines 33547 33551 +4
==========================================
- Hits 28327 28322 -5
- Misses 5220 5229 +9 ☔ View full report in Codecov by Sentry. |
|
elasticsearch_and_cassandra_26 / elasticsearch_and_cassandra_mnesia / 082f1e0 small_tests_25 / small_tests / 082f1e0 small_tests_26 / small_tests / 082f1e0 small_tests_26_arm64 / small_tests / 082f1e0 dynamic_domains_pgsql_mnesia_25 / pgsql_mnesia / 082f1e0 graphql_SUITE:tls_enabled:tls_connect_admin_unknown_certificate{error,{{assertMatch,[{module,graphql_SUITE},
{line,253},
{expression,"Result"},
{pattern,"{ error , { tls_alert , { unknown_ca , _ } } }"},
{value,{error,connection_closed}}]},
[{graphql_SUITE,tls_connect_admin_unknown_certificate,1,
[{file,"/home/circleci/project/big_tests/tests/graphql_SUITE.erl"},
{line,253}]},
{test_server,ts_tc,3,[{file,"test_server.erl"},{line,1782}]},
{test_server,run_test_case_eval1,6,
[{file,"test_server.erl"},{line,1291}]},
{test_server,run_test_case_eval,9,
[{file,"test_server.erl"},{line,1223}]}]}}ldap_mnesia_25 / ldap_mnesia / 082f1e0 ldap_mnesia_26 / ldap_mnesia / 082f1e0 dynamic_domains_mysql_redis_26 / mysql_redis / 082f1e0 dynamic_domains_pgsql_mnesia_26 / pgsql_mnesia / 082f1e0 internal_mnesia_26 / internal_mnesia / 082f1e0 dynamic_domains_mssql_mnesia_26 / odbc_mssql_mnesia / 082f1e0 pgsql_cets_26 / pgsql_cets / 082f1e0 pgsql_mnesia_25 / pgsql_mnesia / 082f1e0 pgsql_mnesia_26 / pgsql_mnesia / 082f1e0 mysql_redis_26 / mysql_redis / 082f1e0 mssql_mnesia_26 / odbc_mssql_mnesia / 082f1e0 mod_blocking_SUITE:manage:remove_user_from_blocklist{error,{{assertion_failed,assert_many,false,
[is_iq_result,#Fun<mod_blocking_SUITE.32.129816315>],
[],[]},
[{escalus_new_assert,assert_true,2,
[{file,"/home/circleci/project/big_tests/_build/default/lib/escalus/src/escalus_new_assert.erl"},
{line,84}]},
{mod_blocking_SUITE,'-remove_user_from_blocklist/1-fun-0-',2,
[{file,"/home/circleci/project/big_tests/tests/mod_blocking_SUITE.erl"},
{line,196}]},
{escalus_story,story,4,
[{file,"/home/circleci/project/big_tests/_build/default/lib/escalus/src/escalus_story.erl"},
{line,72}]},
{test_server,ts_tc,3,[{file,"test_server.erl"},{line,1793}]},
{test_server,run_test_case_eval1,6,
[{file,"test_server.erl"},{line,1302}]},
{test_server,run_test_case_eval,9,
[{file,"test_server.erl"},{line,1234}]}]}}mod_blocking_SUITE:manage:add_another_user_to_blocklist{error,{{assertion_failed,assert_many,false,
[is_iq_result,#Fun<mod_blocking_SUITE.32.129816315>],
[],[]},
[{escalus_new_assert,assert_true,2,
[{file,"/home/circleci/project/big_tests/_build/default/lib/escalus/src/escalus_new_assert.erl"},
{line,84}]},
{mod_blocking_SUITE,'-add_another_user_to_blocklist/1-fun-0-',2,
[{file,"/home/circleci/project/big_tests/tests/mod_blocking_SUITE.erl"},
{line,176}]},
{escalus_story,story,4,
[{file,"/home/circleci/project/big_tests/_build/default/lib/escalus/src/escalus_story.erl"},
{line,72}]},
{test_server,ts_tc,3,[{file,"test_server.erl"},{line,1793}]},
{test_server,run_test_case_eval1,6,
[{file,"test_server.erl"},{line,1302}]},
{test_server,run_test_case_eval,9,
[{file,"test_server.erl"},{line,1234}]}]}}mod_blocking_SUITE:manage:add_many_users_to_blocklist{error,{{badmatch,0},
[{escalus_story,drop_presences,2,
[{file,"/home/circleci/project/big_tests/_build/default/lib/escalus/src/escalus_story.erl"},
{line,191}]},
{escalus_story,'-start_ready_clients/2-fun-0-',3,
[{file,"/home/circleci/project/big_tests/_build/default/lib/escalus/src/escalus_story.erl"},
{line,135}]},
{lists,foldl,3,[{file,"lists.erl"},{line,1594}]},
{escalus_story,start_ready_clients,2,
[{file,"/home/circleci/project/big_tests/_build/default/lib/escalus/src/escalus_story.erl"},
{line,128}]},
{lists,flatmap_1,2,[{file,"lists.erl"},{line,1579}]},
{lists,flatmap_1,2,[{file,"lists.erl"},{line,1579}]},
{escalus_story,story,4,
[{file,"/home/circleci/project/big_tests/_build/default/lib/escalus/src/escalus_story.erl"},
{line,69}]},
{test_server,ts_tc,3,[{file,"test_server.erl"},{line,1793}]}]}}mod_blocking_SUITE:manage:remove_many_user_from_blocklist{error,
{{badmatch,
{error,
{connection_step_failed,
{{escalus_session,session},
{client,
<<"kate_unnamed_2625_remove_many_user_from_blocklist_2635@localhost/res1">>,
escalus_tcp,<0.64064.0>,undefined,
[{event_client,
[{event_manager,<0.64036.0>},
{server,<<"localhost">>},
{username,
<<"kate_unnamed_2625_remove_many_user_from_blocklist_2635">>},
{resource,<<"res1">>}]},
{resource,<<"res1">>},
{username,
<<"kate_unnamed_2625_remove_many_user_from_blocklist_2635">>},
{server,<<"localhost">>},
{host,<<"localhost">>},
{port,5222},
{auth,{escalus_auth,auth_plain}},
{wspath,undefined},
{username,
<<"kate_unnamed_2625_remove_many_user_from_blocklist_2635">>},
{server,<<"localhost">>},
{password,<<"makrowe;p">>},
{stream_id,<<"65f6807913ba58f2">>}]},
[{compression,false},
{starttls,true},
{stream_management,true},
{advanced_message_processing,true},
{client_state_indication,false},
{sasl_mechanisms,[<<"SCRAM-SHA-256">>,<<"PLAIN">>]},
{caps,undefined}]},
{timeout,session_reply}}}},
[{escalus_story,'-start_ready_clients/2-fun-0-',3,
[{file,
"/home/circleci/project/big_tests/_...mod_blocking_SUITE:manage:clear_blocklist{error,{{assertion_failed,assert_many,false,
[is_iq_result,#Fun<mod_blocking_SUITE.32.129816315>],
[],[]},
[{escalus_new_assert,assert_true,2,
[{file,"/home/circleci/project/big_tests/_build/default/lib/escalus/src/escalus_new_assert.erl"},
{line,84}]},
{mod_blocking_SUITE,'-clear_blocklist/1-fun-0-',3,
[{file,"/home/circleci/project/big_tests/tests/mod_blocking_SUITE.erl"},
{line,217}]},
{escalus_story,story,4,
[{file,"/home/circleci/project/big_tests/_build/default/lib/escalus/src/escalus_story.erl"},
{line,72}]},
{test_server,ts_tc,3,[{file,"test_server.erl"},{line,1793}]},
{test_server,run_test_case_eval1,6,
[{file,"test_server.erl"},{line,1302}]},
{test_server,run_test_case_eval,9,
[{file,"test_server.erl"},{line,1234}]}]}}dynamic_domains_pgsql_mnesia_25 / pgsql_mnesia / 082f1e0 mssql_mnesia_26 / odbc_mssql_mnesia / 082f1e0 |
Add CETS
major.minorversion as a suffix to cluster name in thediscovery_nodestable, preventing CETS from crashing if a rolling upgrade is performed, and the CETS versions are incompatible.This would occur when upgrading MIM from
6.2.0to the upcoming6.2.1, which translates to CETS upgrade from0.1.0to0.2.0. If only the patch version of CETS is changed, e.g.0.2.0to0.2.1, they are still considered compatible. From now on, we only need to ensure that CETS versions are marked accordingly before MIM releases.This change was tested manually by upgrading from
6.2.0to this PR. The test was performed twice:test-helper,helm upgrade.Note: The version is not a separate DB column, because the nodes running version 6.2.0 would still try to connect to the upgraded nodes. Extending the name seems like a sane option, and similar solutions are used in well-known tools and platforms like k8s.