Fix slow getaddrs call in global distribution #4086

arcusfelis · 2023-08-08T11:24:35Z

This PR addresses "MIM-1990".
Fixes flaky mod_global_distrib_SUITE:test_host_refreshing testcase

There was an error happening from time to time (once every 10 CI jobs). Report

mod_global_distrib_SUITE:hosts_refresher:test_host_refreshing
{error,
  {{trees_for_connections_present,true,[{times,50,false}],ok},
   [{mongoose_helper,do_wait_until,2,
      [{file,"/home/circleci/project/big_tests/tests/mongoose_helper.erl"},
       {line,358}]},
    {mod_global_distrib_SUITE,test_host_refreshing,1,
      [{file,
         "/home/circleci/project/big_tests/tests/mod_global_distrib_SUITE.erl"},
       {line,384}]},
    {test_server,ts_tc,3,[{file,"test_server.erl"},{line,1782}]},
    {test_server,run_test_case_eval1,6,
      [{file,"test_server.erl"},{line,1291}]},
    {test_server,run_test_case_eval,9,
      [{file,"test_server.erl"},{line,1223}]}]}}

The error does not contain a lot of information, but we know that the previous test group was advertised_endpoints, where we try to do something with endpoints.

The trees_for_connections_present checks that mod_global_distrib_server_mgr creates a supervisor for each of global-distribution domains.

I.e. logic is:

%% In mod_global_distrib_hosts_refresher.erl
refresh(LocalHost) ->
    Hosts = mod_global_distrib_mapping:hosts(),
    ?LOG_DEBUG(#{what => gd_refresher_fetched_hosts,
                 hosts => Hosts, local_host => LocalHost}),
    lists:foreach(fun mod_global_distrib_outgoing_conns_sup:ensure_server_started/1,
                  lists:delete(LocalHost, Hosts)).

While ensure_server_started looks like:

-spec ensure_server_started(Server :: jid:lserver()) -> ok | {error, any()}.
ensure_server_started(Server) ->
    case mod_global_distrib_server_sup:is_available(Server) of
        false -> add_server(Server);
        true -> ok
    end.

The initial testcase had very little info. I.e. it only checked that one of the Erlang nodes had issues. Let's improve logging:

{error,
  {{trees_for_connections_present,[],
     [{times,1,[europe_node1,europe_node2,asia_node]},
      {times,1,[europe_node2,asia_node]},
      {times,98,[europe_node2]}],
     ok},
   [{mongoose_helper,do_wait_until,2,
      [{file,"/home/circleci/project/big_tests/tests/mongoose_helper.erl"},
       {line,357}]},
    {mod_global_distrib_SUITE,test_host_refreshing,1,
      [{file,
         "/home/circleci/project/big_tests/tests/mod_global_distrib_SUITE.erl"},
       {line,404}]},
    {test_server,ts_tc,3,[{file,"test_server.erl"},{line,1783}]},
    {test_server,run_test_case_eval1,6,
      [{file,"test_server.erl"},{line,1292}]},
    {test_server,run_test_case_eval,9,
      [{file,"test_server.erl"},{line,1224}]}]}}

europe_node2 is mim2 node.
So, we know that there is not enough outgoing connections on mim2.
But the only host we need to create connections is reg1. (i.e. refresh function only calls mod_global_distrib_outgoing_conns_sup:ensure_server_started(<<"reg1">>)).

ensure_server_started(Server) ->
    case mod_global_distrib_server_sup:is_available(Server) of %% This call returns false, because this call fails with timeout

What is is_available(Server) call? it is a gen_server:call:

%% In mod_global_distrib_server_mgr.erl
-spec ping_proc(Server :: jid:lserver()) -> pong | pang.
ping_proc(Server) ->
    case catch do_call(Server, ping_proc) of
        pong -> pong;
        _Error -> pang
    end.

So, the server process is busy.
It is actually could be blocked up to 16 seconds and it is doing inet:getaddrs:

when=2023-08-08T09:18:12.474654+00:00 level=info what=long_task_progress pid=<0.4902.0> at=mongoose_long:monitor_loop/4:83 stacktrace="{current_stacktrace,[{inet_gethost_native,getit,2,[{file,\"inet_gethost_native.erl\"},{line,506}]},{inet,gethostbyname_tm_native,4,[{file,\"inet.erl\"},{line,1571}]},{inet,getaddrs_tm,3,[{file,\"inet.erl\"},{line,1522}]},{inet,getaddrs,3,[{file,\"inet.erl\"},{line,845}]},{mod_global_distrib_utils,getaddrs,2,[{file,\"/home/circleci/project/src/global_distrib/mod_global_distrib_utils.erl\"},{line,267}]},{mod_global_distrib_utils,to_ip_tuples,1,[{file,\"/home/circleci/project/src/global_distrib/mod_global_distrib_utils.erl\"},{line,226}]},{mod_global_distrib_utils,resolve_endpoint,1,[{file,\"/home/circleci/project/src/global_distrib/mod_global_distrib_utils.erl\"},{line,130}]},{lists,flatmap_1,2,[{file,\"lists.erl\"},{line,1335}]}]}" time_ms=15017 call=refresh_in_init

First of all, we see that it is sitting in refresh_in_init, which means it is in init/1 callback of a gen_server.
Oh, it is also crashing after 16 seconds in init function:

when=2023-08-08T08:43:18.139426+00:00 level=error pid=<0.7867.0> at=proc_lib:crash_report/4:539 report="[[{initial_call,{mod_global_distrib_server_mgr,init,['Argument__1']}},{pid,<0.7867.0>},{registered_name,[]},{error_info,{error,{domain_not_resolved,{\
"somefakedomain.com\",nxdomain,nxdomain}},[{mod_global_distrib_utils,resolve_endpoint,1,[{file,\"/home/circleci/project/src/global_distrib/mod_global_distrib_utils.erl\"},{line,143}]},{lists,flatmap_1,2,[{file,\"lists.erl\"},{line,1335}]},{mongoose_long,run_tracked,2,[{file,\"/home/circleci/project/src/mongoose_long.erl\"},{line,50}]},{mod_global_distrib_server_mgr,refresh_connections,1,[{file,\"/home/circleci/project/src/global_distrib/mod_global_distrib_server_mgr.erl\"},{line,389}]},{mongoose_long,ru
n_tracked,2,[{file,\"/home/circleci/project/src/mongoose_long.erl\"},{line,50}]},{mod_global_distrib_server_mgr,init,1,[{file,\"/home/circleci/project/src/global_distrib/mod_global_distrib_server_mgr.erl\"},{line,143}]},{gen_server,init_it,2,[{file,\"gen_server.erl\"},{line,851}]},{gen_server,init_it,6,[{file,\"gen_server.erl\"},{line,814}]}]}},{ancestors,[mod_global_distrib_server_sup_reg1,mod_global_distrib_outgoing_conns_sup,ejabberd_sup,<0.3060.0>]},{message_queue_len,7},{messages,[{'$gen_call',{<0.7936.0>,[alias|#Ref<0.3796509223.2464219140.210594>]},ping_proc},{'$gen_cast',{call_timeout,<0.7936.0>,ping_proc}},{'$gen_call',{<0.7936.0>,[alias|#Ref<0.3796509223.2464219140.210620>]},ping_proc},{'$gen_call',{<0.8161.0>,[alias|#Ref<0.3796509223.2464481281.11871>]},ping_proc},{'$gen_cast',{call_timeout,<0.7936.0>,ping_proc}},{'$gen_call',{<0.7936.0>,[alias|#Ref<0.3796509223.2464481281.11886>]},ping_proc},{'EXIT',<0.7869.0>,normal}]},{links,[<0.7868.0>,<0.7866.0>]},{dictionary,[]},{trap_exit,true},{stat
us,running},{heap_size,4185},{stack_size,28},{reductions,68727}],[]]" label={proc_lib,crash}

So, this means it blocks the supervisor for 16 seconds, which is pretty bad (i.e. we cannot for example, init connections in parallel, because supervisor is busy). So, improvement - we could do delayed initialisation.

But what about our nxdomain?
Apparently, a single init:getaddrs call could take 2 seconds or more:

when=2023-08-08T10:52:06.659501+00:00 level=info what=long_task_finished pid=<0.24064.1> at=mongoose_long:run_tracked/2:63 version=inet time_ms=2352 call=gd_getaddrs address=somefakedomain.com

Also, non-existing domain is set in the previous testcase advertised_endpoints.

%% A fake address we don't try to connect to.
%% Used in test_advertised_endpoints_override_endpoints testcase.
advertised_endpoints() ->
    [
     {fake_domain(), get_port(reg, gd_endpoint_port)}
    ].

fake_domain() ->
    "somefakedomain.com".

(investigating why a testcase is affected by a testcase, executed before is outside of the bug investigation. Eventually, GD would retry creating an outgoing connection with the proper endpoints. I would assume, that the mod_global_distrib is not fully restarted between test groups). I.e.:

% This is from the previous test
when=2023-08-08T10:52:47.290249+00:00 level=info what=gd_get_endpoints pid=<0.24979.1> at=mod_global_distrib_server_mgr:get_endpoints/1:426 server=reg1 endpoints="[{\"somefakedomain.com\",7777}]"

wait a lot here...
crash of mod_global_distrib_server_mgr...

% Finally, something that we actually expect in our testcase
when=2023-08-08T10:52:47.767358+00:00 level=info what=gd_get_endpoints pid=<0.25055.1> at=mod_global_distrib_server_mgr:get_endpoints/1:426 server=reg1 endpoints=[{{127,0,0,1},7777}]

So, we could run getaddrs in parallel using pmap and limit maximum execution time to 5 seconds.
It fixes the testcase (verified with {repeat_until_any_fail, 60} for the compile PR which was able to consistently reproduce the bug without the fix).

Proposed changes include:

Delayed initial_refresh in src/global_distrib/mod_global_distrib_server_mgr.erl
Do getaddrs call in parallel and limit how long it can take to 5 seconds.

Stuff we've learned:

inet functions could be slow, really slow.
trap_exits are bad, please try not to use them (especially with code which does unsafe calls or blocks for 16 seconds).
gen_server:init/1 should be fast and not fail, ideally.
Adding logging for slow tasks is pretty useful (i.e. to report time of a function, if it takes more than 1 second).
Though, I am not trying to redesign global distribution in this PR, just fix a flaky test :)

Debug code for this fix could be found in #4084 (just for the reference, if someone wants to debug something similar again).

Fixes flaky mod_global_distrib_SUITE:test_host_refreshing testcase

mongoose-im · 2023-08-08T11:32:26Z

small_tests_24 / small_tests / 6b3daf7
Reports root / small

elasticsearch_and_cassandra_25 / elasticsearch_and_cassandra_mnesia / 6b3daf7
Reports root/ big
OK: 369 / Failed: 0 / User-skipped: 38 / Auto-skipped: 0

small_tests_25 / small_tests / 6b3daf7
Reports root / small

small_tests_25_arm64 / small_tests / 6b3daf7
Reports root / small

dynamic_domains_pgsql_mnesia_24 / pgsql_mnesia / 6b3daf7
Reports root/ big
OK: 4221 / Failed: 0 / User-skipped: 82 / Auto-skipped: 0

ldap_mnesia_24 / ldap_mnesia / 6b3daf7
Reports root/ big
OK: 2258 / Failed: 0 / User-skipped: 827 / Auto-skipped: 0

ldap_mnesia_25 / ldap_mnesia / 6b3daf7
Reports root/ big
OK: 2258 / Failed: 0 / User-skipped: 827 / Auto-skipped: 0

dynamic_domains_mysql_redis_25 / mysql_redis / 6b3daf7
Reports root/ big
OK: 4187 / Failed: 5 / User-skipped: 108 / Auto-skipped: 3

metrics_session_SUITE:session:login_one

{error,
  {{badmatch,{value,6}},
   [{metrics_helper,assert_counter,3,
      [{file,"/home/circleci/project/big_tests/tests/metrics_helper.erl"},
       {line,36}]},
    {metrics_session_SUITE,'-login_one/1-fun-0-',3,
      [{file,
         "/home/circleci/project/big_tests/tests/metrics_session_SUITE.erl"},
       {line,79}]},
    {escalus_story,story,4,
      [{file,
         "/home/circleci/project/big_tests/_build/default/lib/escalus/src/escalus_story.erl"},
       {line,72}]},
    {test_server,ts_tc,3,[{file,"test_server.erl"},{line,1782}]},
    {test_server,run_test_case_eval1,6,
      [{file,"test_server.erl"},{line,1291}]},
    {test_server,run_test_case_eval,9,
      [{file,"test_server.erl"},{line,1223}]}]}}

Report log

metrics_session_SUITE:session_global:session_unique

{error,
  {{uniqueSessionCount,
     {value,1},
     [{times,25,
        {error,
          {badmatch,{value,6}},
          [{metrics_helper,assert_counter,3,
             [{file,
              "/home/circleci/project/big_tests/tests/metrics_helper.erl"},
            {line,36}]},
           {mongoose_helper,do_wait_until,2,
             [{file,
              "/home/circleci/project/big_tests/tests/mongoose_helper.erl"},
            {line,360}]},
           {metrics_session_SUITE,'-session_unique/1-fun-0-',2,
             [{file,
              "/home/circleci/project/big_tests/tests/metrics_session_SUITE.erl"},
            {line,120}]},
           {escalus_story,story,4,
             [{file,
              "/home/circleci/project/big_tests/_build/default/lib/escalus/src/escalus_story.erl"},
            {line,72}]},
           {test_server,ts_tc,3,
             [{file,"test_server.erl"},{line,1782}]},
           {test_server,run_test_case_eval1,6,
             [{file,"test_server.erl"},{line,1291}]},
           {test_server,run_test_case_eval,9,
             [{file,"test_server.erl"},{line,1223}]}]}}],
     ok},
   [{mongoose_helper,do_wait_until,2,
      [{file,"/home/circleci/project/big_tests/tests/mongoose_helper.erl"},
       {line,357}]},
    {metrics_session_SUITE,'-session_unique/1-fun-0-',2,
      [{file,
         "/home/circleci/project/big_tests/tests/metrics_session_SUITE.erl"},
       {line,120}]},
    {escalus_story,story,4,
      [{file,
         "/home/circleci/pr...

Report log

metrics_api_SUITE:global:session_counters

{error,{{assertEqual,[{module,metrics_api_SUITE},
            {line,231},
            {expression,"fetch_global_gauge_value ( uniqueSessionCount , Config )"},
            {expected,2},
            {value,7}]},
    [{metrics_api_SUITE,'-session_counters/1-fun-3-',4,
              [{file,"/home/circleci/project/big_tests/tests/metrics_api_SUITE.erl"},
               {line,231}]},
     {escalus_story,story,4,
            [{file,"/home/circleci/project/big_tests/_build/default/lib/escalus/src/escalus_story.erl"},
             {line,72}]},
     {test_server,ts_tc,3,[{file,"test_server.erl"},{line,1782}]},
     {test_server,run_test_case_eval1,6,
            [{file,"test_server.erl"},{line,1291}]},
     {test_server,run_test_case_eval,9,
            [{file,"test_server.erl"},{line,1223}]}]}}

Report log

mongooseimctl_SUITE:sessions:sessions_info

{error,
  {test_case_failed,
    #{user_info =>
        ["alice@domain.example.com/res1\tc2s\t127.0.0.1\t42140\t0\tmongooseim@localhost\t0",
         "kate@domain.example.com/res1\tc2s\t127.0.0.1\t42154\t0\tmongooseim@localhost\t0",
         "neustradamus_log_one_scram_sha1_plus_1620@domain.example.com/res1\tc2s_tls\t127.0.0.1\t47280\t0\tmongooseim@localhost\t169",
         "bob@domain.example.com/res1\tc2s\t127.0.0.1\t42148\t0\tmongooseim@localhost\t0",
         "neustradamus_log_one_scram_sha512_plus_1619@domain.example.com/res1\tc2s_tls\t127.0.0.1\t47262\t0\tmongooseim@localhost\t169",
         "neustradamus_log_one_scram_sha256_plus_1617@domain.example.com/res1\tc2s_tls\t127.0.0.1\t47272\t0\tmongooseim@localhost\t169",
         "neustradamus_log_one_scram_sha384_plus_1621@domain.example.com/res1\tc2s_tls\t127.0.0.1\t47308\t0\tmongooseim@localhost\t169",
         "neustradamus_log_one_scram_sha224_plus_1618@domain.example.com/res1\tc2s_tls\t127.0.0.1\t47256\t0\tmongooseim@localhost\t169"],
      users =>
        [{client,<<"alice@domain.example.com/res1">>,escalus_tcp,
           <0.31303.1>,
           [{event_manager,<0.31302.1>},
          {server,<<"domain.example.com">>},
          {username,<<"alicE">>},
          {resource,<<"res1">>}],
           [{event_client,
            [{event_manager,<0.31302.1>},
             {server,<<"domain.example.com">>},
             {username,<<"alicE">>},
             {resource,<<"res1">>}]},
          {resource,<<"res1">>},
        ...

Report log

mongooseimctl_SUITE:stats:stats_host

{error,{{badmatch,{"7\n",0}},
    [{mongooseimctl_SUITE,'-stats_host/1-fun-0-',3,
                [{file,"/home/circleci/project/big_tests/tests/mongooseimctl_SUITE.erl"},
                 {line,1139}]},
     {escalus_story,story,4,
            [{file,"/home/circleci/project/big_tests/_build/default/lib/escalus/src/escalus_story.erl"},
             {line,72}]},
     {test_server,ts_tc,3,[{file,"test_server.erl"},{line,1782}]},
     {test_server,run_test_case_eval1,6,
            [{file,"test_server.erl"},{line,1291}]},
     {test_server,run_test_case_eval,9,
            [{file,"test_server.erl"},{line,1223}]}]}}

Report log

dynamic_domains_mssql_mnesia_25 / odbc_mssql_mnesia / 6b3daf7
Reports root/ big
OK: 4218 / Failed: 0 / User-skipped: 85 / Auto-skipped: 0

dynamic_domains_pgsql_mnesia_25 / pgsql_mnesia / 6b3daf7
Reports root/ big
OK: 4221 / Failed: 0 / User-skipped: 82 / Auto-skipped: 0

pgsql_mnesia_24 / pgsql_mnesia / 6b3daf7
Reports root/ big
OK: 4604 / Failed: 0 / User-skipped: 89 / Auto-skipped: 0

internal_mnesia_25 / internal_mnesia / 6b3daf7
Reports root/ big
OK: 2404 / Failed: 0 / User-skipped: 681 / Auto-skipped: 0

pgsql_mnesia_25 / pgsql_mnesia / 6b3daf7
Reports root/ big
OK: 4604 / Failed: 0 / User-skipped: 89 / Auto-skipped: 0

mysql_redis_25 / mysql_redis / 6b3daf7
Reports root/ big
OK: 4590 / Failed: 0 / User-skipped: 103 / Auto-skipped: 0

mssql_mnesia_25 / odbc_mssql_mnesia / 6b3daf7
Reports root/ big
OK: 4601 / Failed: 0 / User-skipped: 92 / Auto-skipped: 0

codecov · 2023-08-08T11:34:02Z

Codecov Report

Patch coverage: 88.88% and project coverage change: -0.01% ⚠️

Comparison is base (4433414) 83.90% compared to head (46915d4) 83.90%.
Report is 2 commits behind head on master.

Additional details and impacted files

@@            Coverage Diff             @@
##           master    #4086      +/-   ##
==========================================
- Coverage   83.90%   83.90%   -0.01%     
==========================================
  Files         527      527              
  Lines       33158    33164       +6     
==========================================
+ Hits        27822    27826       +4     
- Misses       5336     5338       +2

Files Changed	Coverage Δ
src/global_distrib/mod_global_distrib_utils.erl	`83.72% <83.33%> (-0.23%)`	⬇️
...c/global_distrib/mod_global_distrib_server_mgr.erl	`77.14% <100.00%> (+0.13%)`	⬆️

... and 8 files with indirect coverage changes

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

…_server_mgr

mongoose-im · 2023-08-08T12:03:35Z

small_tests_24 / small_tests / e845a71
Reports root / small

elasticsearch_and_cassandra_25 / elasticsearch_and_cassandra_mnesia / e845a71
Reports root/ big
OK: 369 / Failed: 0 / User-skipped: 38 / Auto-skipped: 0

small_tests_25 / small_tests / e845a71
Reports root / small

small_tests_25_arm64 / small_tests / e845a71
Reports root / small

dynamic_domains_pgsql_mnesia_24 / pgsql_mnesia / e845a71
Reports root/ big
OK: 4221 / Failed: 0 / User-skipped: 82 / Auto-skipped: 0

ldap_mnesia_24 / ldap_mnesia / e845a71
Reports root/ big
OK: 2258 / Failed: 0 / User-skipped: 827 / Auto-skipped: 0

ldap_mnesia_25 / ldap_mnesia / e845a71
Reports root/ big
OK: 2258 / Failed: 0 / User-skipped: 827 / Auto-skipped: 0

dynamic_domains_pgsql_mnesia_25 / pgsql_mnesia / e845a71
Reports root/ big
OK: 4221 / Failed: 0 / User-skipped: 82 / Auto-skipped: 0

dynamic_domains_mysql_redis_25 / mysql_redis / e845a71
Reports root/ big
OK: 4195 / Failed: 0 / User-skipped: 108 / Auto-skipped: 0

dynamic_domains_mssql_mnesia_25 / odbc_mssql_mnesia / e845a71
Reports root/ big
OK: 4218 / Failed: 0 / User-skipped: 85 / Auto-skipped: 0

pgsql_mnesia_24 / pgsql_mnesia / e845a71
Reports root/ big
OK: 4604 / Failed: 0 / User-skipped: 89 / Auto-skipped: 0

mysql_redis_25 / mysql_redis / e845a71
Reports root/ big
OK: 4590 / Failed: 0 / User-skipped: 103 / Auto-skipped: 0

pgsql_mnesia_25 / pgsql_mnesia / e845a71
Reports root/ big
OK: 4604 / Failed: 0 / User-skipped: 89 / Auto-skipped: 0

internal_mnesia_25 / internal_mnesia / e845a71
Reports root/ big
OK: 2404 / Failed: 0 / User-skipped: 681 / Auto-skipped: 0

mssql_mnesia_25 / odbc_mssql_mnesia / e845a71
Reports root/ big
OK: 4601 / Failed: 0 / User-skipped: 92 / Auto-skipped: 0

mongoose-im · 2023-08-08T12:39:00Z

small_tests_24 / small_tests / 7ad6c5d
Reports root / small

elasticsearch_and_cassandra_25 / elasticsearch_and_cassandra_mnesia / 7ad6c5d
Reports root/ big
OK: 369 / Failed: 0 / User-skipped: 38 / Auto-skipped: 0

small_tests_25 / small_tests / 7ad6c5d
Reports root / small

small_tests_25_arm64 / small_tests / 7ad6c5d
Reports root / small

ldap_mnesia_24 / ldap_mnesia / 7ad6c5d
Reports root/ big
OK: 2258 / Failed: 0 / User-skipped: 827 / Auto-skipped: 0

dynamic_domains_pgsql_mnesia_24 / pgsql_mnesia / 7ad6c5d
Reports root/ big
OK: 4221 / Failed: 0 / User-skipped: 82 / Auto-skipped: 0

ldap_mnesia_25 / ldap_mnesia / 7ad6c5d
Reports root/ big
OK: 2258 / Failed: 0 / User-skipped: 827 / Auto-skipped: 0

dynamic_domains_pgsql_mnesia_25 / pgsql_mnesia / 7ad6c5d
Reports root/ big
OK: 4221 / Failed: 0 / User-skipped: 82 / Auto-skipped: 0

dynamic_domains_mysql_redis_25 / mysql_redis / 7ad6c5d
Reports root/ big
OK: 4195 / Failed: 0 / User-skipped: 108 / Auto-skipped: 0

internal_mnesia_25 / internal_mnesia / 7ad6c5d
Reports root/ big
OK: 2404 / Failed: 0 / User-skipped: 681 / Auto-skipped: 0

dynamic_domains_mssql_mnesia_25 / odbc_mssql_mnesia / 7ad6c5d
Reports root/ big
OK: 4218 / Failed: 0 / User-skipped: 85 / Auto-skipped: 0

pgsql_mnesia_24 / pgsql_mnesia / 7ad6c5d
Reports root/ big
OK: 4604 / Failed: 0 / User-skipped: 89 / Auto-skipped: 0

mam_SUITE:rdbms_async_cache_muc_all:muc_configurable_archiveid:muc_no_elements

{failed,
  {mam_SUITE,end_per_testcase,
    {'EXIT',
      {{room_archive_size,0,[{times,200,1}],ok},
       [{mongoose_helper,do_wait_until,2,
          [{file,
             "/home/circleci/project/big_tests/tests/mongoose_helper.erl"},
           {line,357}]},
        {mam_helper,wait_for_room_archive_size,3,
          [{file,
             "/home/circleci/project/big_tests/tests/mam_helper.erl"},
           {line,789}]},
        {mam_helper,clean_room_archive,1,
          [{file,
             "/home/circleci/project/big_tests/tests/mam_helper.erl"},
           {line,733}]},
        {mam_helper,destroy_room,1,
          [{file,
             "/home/circleci/project/big_tests/tests/mam_helper.erl"},
           {line,726}]},
        {mam_SUITE,end_per_testcase,2,
          [{file,
             "/home/circleci/project/big_tests/tests/mam_SUITE.erl"},
           {line,962}]},
        {test_server,do_end_per_testcase,4,
          [{file,"test_server.erl"},{line,1627}]},
        {test_server,run_test_case_eval1,6,
          [{file,"test_server.erl"},{line,1335}]},
        {test_server,run_test_case_eval,9,
          [{file,"test_server.erl"},{line,1224}]}]}}}}

Report log

mam_SUITE:rdbms_async_cache_muc_all:muc_configurable_archiveid:muc_only_stanzaid

{failed,
  {mam_SUITE,end_per_testcase,
    {'EXIT',
      {{room_archive_size,0,[{times,200,1}],ok},
       [{mongoose_helper,do_wait_until,2,
          [{file,
             "/home/circleci/project/big_tests/tests/mongoose_helper.erl"},
           {line,357}]},
        {mam_helper,wait_for_room_archive_size,3,
          [{file,
             "/home/circleci/project/big_tests/tests/mam_helper.erl"},
           {line,789}]},
        {mam_helper,clean_room_archive,1,
          [{file,
             "/home/circleci/project/big_tests/tests/mam_helper.erl"},
           {line,733}]},
        {mam_helper,destroy_room,1,
          [{file,
             "/home/circleci/project/big_tests/tests/mam_helper.erl"},
           {line,726}]},
        {mam_SUITE,end_per_testcase,2,
          [{file,
             "/home/circleci/project/big_tests/tests/mam_SUITE.erl"},
           {line,965}]},
        {test_server,do_end_per_testcase,4,
          [{file,"test_server.erl"},{line,1627}]},
        {test_server,run_test_case_eval1,6,
          [{file,"test_server.erl"},{line,1335}]},
        {test_server,run_test_case_eval,9,
          [{file,"test_server.erl"},{line,1224}]}]}}}}

Report log

pgsql_mnesia_25 / pgsql_mnesia / 7ad6c5d
Reports root/ big
OK: 4604 / Failed: 0 / User-skipped: 89 / Auto-skipped: 0

mysql_redis_25 / mysql_redis / 7ad6c5d
Reports root/ big
OK: 4590 / Failed: 0 / User-skipped: 103 / Auto-skipped: 0

mssql_mnesia_25 / odbc_mssql_mnesia / 7ad6c5d
Reports root/ big
OK: 4601 / Failed: 0 / User-skipped: 92 / Auto-skipped: 0

mongoose-im · 2023-08-08T13:56:42Z

small_tests_24 / small_tests / 46915d4
Reports root / small

elasticsearch_and_cassandra_25 / elasticsearch_and_cassandra_mnesia / 46915d4
Reports root/ big
OK: 369 / Failed: 0 / User-skipped: 38 / Auto-skipped: 0

small_tests_25 / small_tests / 46915d4
Reports root / small

small_tests_25_arm64 / small_tests / 46915d4
Reports root / small

ldap_mnesia_24 / ldap_mnesia / 46915d4
Reports root/ big
OK: 2258 / Failed: 0 / User-skipped: 827 / Auto-skipped: 0

dynamic_domains_pgsql_mnesia_24 / pgsql_mnesia / 46915d4
Reports root/ big
OK: 4220 / Failed: 1 / User-skipped: 82 / Auto-skipped: 0

service_mongoose_system_metrics_SUITE:module_opts:rdbms_module_opts_are_reported

{error,
  {{assertEqual,
     [{module,service_mongoose_system_metrics_SUITE},
      {line,370},
      {expression,
        "is_module_opt_reported ( atom_to_binary ( Module ) , Key , Value )"},
      {expected,true},
      {value,false}]},
   [{service_mongoose_system_metrics_SUITE,check_module_opt,3,
      [{file,
         "/home/circleci/project/big_tests/tests/service_mongoose_system_metrics_SUITE.erl"},
       {line,370}]},
    {service_mongoose_system_metrics_SUITE,rdbms_module_opts_are_reported,
      1,
      [{file,
         "/home/circleci/project/big_tests/tests/service_mongoose_system_metrics_SUITE.erl"},
       {line,236}]},
    {test_server,ts_tc,3,[{file,"test_server.erl"},{line,1783}]},
    {test_server,run_test_case_eval1,6,
      [{file,"test_server.erl"},{line,1292}]},
    {test_server,run_test_case_eval,9,
      [{file,"test_server.erl"},{line,1224}]}]}}

Report log

dynamic_domains_pgsql_mnesia_25 / pgsql_mnesia / 46915d4
Reports root/ big
OK: 4221 / Failed: 0 / User-skipped: 82 / Auto-skipped: 0

dynamic_domains_mysql_redis_25 / mysql_redis / 46915d4
Reports root/ big
OK: 4195 / Failed: 0 / User-skipped: 108 / Auto-skipped: 0

pgsql_mnesia_24 / pgsql_mnesia / 46915d4
Reports root/ big
OK: 4604 / Failed: 0 / User-skipped: 89 / Auto-skipped: 0

ldap_mnesia_25 / ldap_mnesia / 46915d4
Reports root/ big
OK: 2258 / Failed: 0 / User-skipped: 827 / Auto-skipped: 0

internal_mnesia_25 / internal_mnesia / 46915d4
Reports root/ big
OK: 2404 / Failed: 0 / User-skipped: 681 / Auto-skipped: 0

dynamic_domains_mssql_mnesia_25 / odbc_mssql_mnesia / 46915d4
Reports root/ big
OK: 4218 / Failed: 0 / User-skipped: 85 / Auto-skipped: 0

mysql_redis_25 / mysql_redis / 46915d4
Reports root/ big
OK: 4590 / Failed: 0 / User-skipped: 103 / Auto-skipped: 0

pgsql_mnesia_25 / pgsql_mnesia / 46915d4
Reports root/ big
OK: 4604 / Failed: 0 / User-skipped: 89 / Auto-skipped: 0

mssql_mnesia_25 / odbc_mssql_mnesia / 46915d4
Reports root/ big
OK: 4601 / Failed: 0 / User-skipped: 92 / Auto-skipped: 0

dynamic_domains_pgsql_mnesia_24 / pgsql_mnesia / 46915d4
Reports root/ big
OK: 4221 / Failed: 0 / User-skipped: 82 / Auto-skipped: 0

NelsonVides

Wow. That's like the best PR description github has ever seen. Very good debug, nice conclusions, cool fix. Thanks! 😃

Fix slow getaddrs call in global distribution

6b3daf7

Fixes flaky mod_global_distrib_SUITE:test_host_refreshing testcase

Move initial_refresh outside of init/1 function in mod_global_distrib…

e845a71

…_server_mgr

Return v6 IPs first from get_addrs_in_parallel

7ad6c5d

arcusfelis marked this pull request as ready for review August 8, 2023 13:31

Use handle_continue to init mod_global_distrib_server_mgr

46915d4

arcusfelis added the waiting-for-review label Aug 8, 2023

DenysGonchar approved these changes Aug 9, 2023

View reviewed changes

NelsonVides approved these changes Aug 9, 2023

View reviewed changes

NelsonVides merged commit 5854ea9 into master Aug 9, 2023

NelsonVides deleted the fix-slow-getaddr-in-global-distrib branch August 9, 2023 15:32

chrzaszcz added this to the 6.2.0 milestone Dec 11, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Fix slow getaddrs call in global distribution #4086

Fix slow getaddrs call in global distribution #4086

arcusfelis commented Aug 8, 2023 •

edited

Loading

mongoose-im commented Aug 8, 2023 •

edited

Loading

codecov bot commented Aug 8, 2023 •

edited

Loading

mongoose-im commented Aug 8, 2023 •

edited

Loading

mongoose-im commented Aug 8, 2023 •

edited

Loading

mongoose-im commented Aug 8, 2023 •

edited

Loading

NelsonVides left a comment

Fix slow getaddrs call in global distribution #4086

Fix slow getaddrs call in global distribution #4086

Conversation

arcusfelis commented Aug 8, 2023 • edited Loading

mongoose-im commented Aug 8, 2023 • edited Loading

codecov bot commented Aug 8, 2023 • edited Loading

Codecov Report

mongoose-im commented Aug 8, 2023 • edited Loading

mongoose-im commented Aug 8, 2023 • edited Loading

mongoose-im commented Aug 8, 2023 • edited Loading

NelsonVides left a comment

Choose a reason for hiding this comment

arcusfelis commented Aug 8, 2023 •

edited

Loading

mongoose-im commented Aug 8, 2023 •

edited

Loading

codecov bot commented Aug 8, 2023 •

edited

Loading

mongoose-im commented Aug 8, 2023 •

edited

Loading

mongoose-im commented Aug 8, 2023 •

edited

Loading

mongoose-im commented Aug 8, 2023 •

edited

Loading