Instrument: global probes and counters #4317

chrzaszcz · 2024-07-05T15:11:27Z

Put system-wide instrumentation probes in mongoose_system_probes.

The affected probes include:

system_up_time, system_process_queue_lengths, system_info, system_memory, system_dist_data. They are moved from mongoose_metrics.
- system_memory contains all stats from erlang:memory().
- system_process_queue_lengths used to take into consideration the p1_fsm_old queues, which are not used anymore. Although one could collect internal gen_statem event queue data, it would be a complex task, and it might decrease system performance.
additionally, cets_info and mnesia_info are set up with mongoose_system_probes, although their configuration and implementation is in mongoose_internal_databases.
- In case of cets_info, it is further delegated to mongoose_instrument_probe_cets. I didn't do the same for Mnesia, because the Mnesia probe is very simple.
- In case of mnesia_info, db_nodes is added to running_db_nodes (which used to be called clusterSize).

Other changes:

Initialization of Mnesia in mongoose_internal_databases is refactored without any functional changes.
Fixed carbon_cache_server. This Graphite mock expected packets preceded by their length, while the Graphite text protocol uses text delimited by newlines. As a result, mongooseim_metrics_SUITE could take up to a minute to
execute, because the first bytes were interpreted as packet length, and the code was just waiting for more and more metrics.
Replaced metrics tests from test/mongooseim_metrics_SUITE.erl with event checks in big_tests/system_probes_SUITE.erl.
Updated metrics used in tests and docs.
Remove complex GraphQL types, that are no longer needed, because the probes use gauges instead.

codecov · 2024-07-08T06:47:25Z

Codecov Report

Attention: Patch coverage is 95.83333% with 2 lines in your changes missing coverage. Please review.

Project coverage is 76.16%. Comparing base (31ad729) to head (5c7ea1c).
Report is 5 commits behind head on feature/instrument.

Files	Patch %	Lines
src/mongoose_system_probes.erl	94.28%	2 Missing ⚠️

❗ There is a different number of reports uploaded between BASE (31ad729) and HEAD (5c7ea1c). Click for more details.

HEAD has 13 uploads less than BASE

Flag BASE (31ad729) HEAD (5c7ea1c)

15 2

Additional details and impacted files

@@                  Coverage Diff                   @@
##           feature/instrument    #4317      +/-   ##
======================================================
- Coverage               84.39%   76.16%   -8.24%     
======================================================
  Files                     553      554       +1     
  Lines                   33874    33856      -18     
======================================================
- Hits                    28587    25785    -2802     
- Misses                   5287     8071    +2784

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

mongoose-im · 2024-07-15T11:15:45Z

elasticsearch_and_cassandra_26 / elasticsearch_and_cassandra_mnesia / 6b6dc1a
Reports root/ big
OK: 457 / Failed: 0 / User-skipped: 45 / Auto-skipped: 0

small_tests_25 / small_tests / 6b6dc1a
Reports root / small

small_tests_26 / small_tests / 6b6dc1a
Reports root / small

small_tests_26_arm64 / small_tests / 6b6dc1a
Reports root / small

ldap_mnesia_26 / ldap_mnesia / 6b6dc1a
Reports root/ big
OK: 2293 / Failed: 0 / User-skipped: 910 / Auto-skipped: 0

internal_mnesia_26 / internal_mnesia / 6b6dc1a
Reports root/ big
OK: 2435 / Failed: 0 / User-skipped: 768 / Auto-skipped: 0

dynamic_domains_pgsql_mnesia_26 / pgsql_mnesia / 6b6dc1a
Reports root/ big
OK: 4640 / Failed: 0 / User-skipped: 106 / Auto-skipped: 0

dynamic_domains_mysql_redis_26 / mysql_redis / 6b6dc1a
Reports root/ big
OK: 4607 / Failed: 0 / User-skipped: 139 / Auto-skipped: 0

dynamic_domains_pgsql_mnesia_25 / pgsql_mnesia / 6b6dc1a
Reports root/ big
OK: 4640 / Failed: 0 / User-skipped: 106 / Auto-skipped: 0

pgsql_cets_26 / pgsql_cets / 6b6dc1a
Reports root/ big
OK: 4534 / Failed: 0 / User-skipped: 177 / Auto-skipped: 0

pgsql_mnesia_25 / pgsql_mnesia / 6b6dc1a
Reports root/ big
OK: 5032 / Failed: 0 / User-skipped: 115 / Auto-skipped: 0

ldap_mnesia_25 / ldap_mnesia / 6b6dc1a
Reports root/ big
OK: 2293 / Failed: 0 / User-skipped: 910 / Auto-skipped: 0

mysql_redis_26 / mysql_redis / 6b6dc1a
Reports root/ big
OK: 5013 / Failed: 0 / User-skipped: 134 / Auto-skipped: 0

mssql_mnesia_26 / odbc_mssql_mnesia / 6b6dc1a
Reports root/ big
OK: 5029 / Failed: 0 / User-skipped: 118 / Auto-skipped: 0

pgsql_mnesia_26 / pgsql_mnesia / 6b6dc1a
Reports root/ big
OK: 5032 / Failed: 0 / User-skipped: 115 / Auto-skipped: 0

dynamic_domains_mssql_mnesia_26 / odbc_mssql_mnesia / 6b6dc1a
Reports root/ big
OK: 4637 / Failed: 0 / User-skipped: 109 / Auto-skipped: 0

Included probes: - Mnesia and system - moved from mongoose_metrics

- Add start/stop calls - Start/stop CETS and Mnesia probes in mongoose_internal_databases

It is started by mongoose_system_probes now

This Graphite mock expected packets preceded by their length, while the Graphite text protocol uses text delimited by newlines. As a result, mongooseim_metrics_SUITE could take up to a minute to execute, because the first bytes were interpreted as packet length.

Remove metric tests, which are now replaced by events checked in big tests.

Not all possible gagues were checked, because they will be tested in system_probes_SUITE.

- Remove instrument_cets_SUITE

mongoose-im · 2024-07-15T13:20:14Z

elasticsearch_and_cassandra_26 / elasticsearch_and_cassandra_mnesia / 5c7ea1c
Reports root/ big
OK: 457 / Failed: 0 / User-skipped: 45 / Auto-skipped: 0

small_tests_25 / small_tests / 5c7ea1c
Reports root / small

small_tests_26 / small_tests / 5c7ea1c
Reports root / small

small_tests_26_arm64 / small_tests / 5c7ea1c
Reports root / small

ldap_mnesia_25 / ldap_mnesia / 5c7ea1c
Reports root/ big
OK: 2293 / Failed: 0 / User-skipped: 910 / Auto-skipped: 0

dynamic_domains_mysql_redis_26 / mysql_redis / 5c7ea1c
Reports root/ big
OK: 4607 / Failed: 0 / User-skipped: 139 / Auto-skipped: 0

ldap_mnesia_26 / ldap_mnesia / 5c7ea1c
Reports root/ big
OK: 2293 / Failed: 0 / User-skipped: 910 / Auto-skipped: 0

dynamic_domains_pgsql_mnesia_25 / pgsql_mnesia / 5c7ea1c
Reports root/ big
OK: 4640 / Failed: 0 / User-skipped: 106 / Auto-skipped: 0

dynamic_domains_pgsql_mnesia_26 / pgsql_mnesia / 5c7ea1c
Reports root/ big
OK: 4640 / Failed: 0 / User-skipped: 106 / Auto-skipped: 0

internal_mnesia_26 / internal_mnesia / 5c7ea1c
Reports root/ big
OK: 2434 / Failed: 1 / User-skipped: 768 / Auto-skipped: 0

pubsub_SUITE:dag+node_config:send_last_published_item_test

{error,{{badmatch,false},
    [{pubsub_tools,check_response,2,
             [{file,"/home/circleci/project/big_tests/tests/pubsub_tools.erl"},
            {line,444}]},
     {pubsub_tools,receive_response,3,
             [{file,"/home/circleci/project/big_tests/tests/pubsub_tools.erl"},
            {line,434}]},
     {pubsub_tools,receive_and_check_response,4,
             [{file,"/home/circleci/project/big_tests/tests/pubsub_tools.erl"},
            {line,424}]},
     {escalus_story,story,4,
            [{file,"/home/circleci/project/big_tests/_build/default/lib/escalus/src/escalus_story.erl"},
             {line,72}]},
     {test_server,ts_tc,3,[{file,"test_server.erl"},{line,1793}]},
     {test_server,run_test_case_eval1,6,
            [{file,"test_server.erl"},{line,1302}]},
     {test_server,run_test_case_eval,9,
            [{file,"test_server.erl"},{line,1234}]}]}}

Report log

pgsql_cets_26 / pgsql_cets / 5c7ea1c
Reports root/ big
OK: 4533 / Failed: 1 / User-skipped: 177 / Auto-skipped: 0

connect_SUITE:just_tls:starttls_required:starttls_required_parallel:cannot_connect_with_proxy_header

{error,
  {thrown,
    {{timeout,stream_end},
     [{escalus_connection,get_stream_end,2,
        [{file,
           "/home/circleci/project/big_tests/_build/default/lib/escalus/src/escalus_connection.erl"},
         {line,369}]},
      {escalus_connection,end_stream,1,
        [{file,
           "/home/circleci/project/big_tests/_build/default/lib/escalus/src/escalus_connection.erl"},
         {line,507}]},
      {escalus_connection,stop,1,
        [{file,
           "/home/circleci/project/big_tests/_build/default/lib/escalus/src/escalus_connection.erl"},
         {line,407}]},
      {test_server,ts_tc,3,[{file,"test_server.erl"},{line,1793}]},
      {test_server,run_test_case_eval1,6,
        [{file,"test_server.erl"},{line,1302}]},
      {test_server,run_test_case_eval,9,
        [{file,"test_server.erl"},{line,1234}]}]}}}

Report log

dynamic_domains_mssql_mnesia_26 / odbc_mssql_mnesia / 5c7ea1c
Reports root/ big
OK: 4637 / Failed: 0 / User-skipped: 109 / Auto-skipped: 0

pgsql_mnesia_25 / pgsql_mnesia / 5c7ea1c
Reports root/ big
OK: 5032 / Failed: 0 / User-skipped: 115 / Auto-skipped: 0

mysql_redis_26 / mysql_redis / 5c7ea1c
Reports root/ big
OK: 5013 / Failed: 0 / User-skipped: 134 / Auto-skipped: 0

pgsql_mnesia_26 / pgsql_mnesia / 5c7ea1c
Reports root/ big
OK: 5032 / Failed: 0 / User-skipped: 115 / Auto-skipped: 0

mssql_mnesia_26 / odbc_mssql_mnesia / 5c7ea1c
Reports root/ big
OK: 5029 / Failed: 0 / User-skipped: 118 / Auto-skipped: 0

internal_mnesia_26 / internal_mnesia / 5c7ea1c
Reports root/ big
OK: 2435 / Failed: 0 / User-skipped: 768 / Auto-skipped: 0

pgsql_cets_26 / pgsql_cets / 5c7ea1c
Reports root/ big
OK: 4534 / Failed: 0 / User-skipped: 177 / Auto-skipped: 0

jacekwegr

Thanks for the changes! The new tests look excellent, and I appreciate the simplification of the GraphQL types. Everything looks good to me 👍

chrzaszcz changed the title ~~WIP~~ Instrument: global probes and counters Jul 5, 2024