Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Instrument: global probes and counters #4317

Merged
merged 12 commits into from
Jul 16, 2024
Merged

Instrument: global probes and counters #4317

merged 12 commits into from
Jul 16, 2024

Conversation

chrzaszcz
Copy link
Member

@chrzaszcz chrzaszcz commented Jul 5, 2024 •
edited
Loading

Put system-wide instrumentation probes in mongoose_system_probes.

The affected probes include:

  • system_up_time, system_process_queue_lengths, system_info, system_memory, system_dist_data. They are moved from mongoose_metrics.
    • system_memory contains all stats from erlang:memory().
    • system_process_queue_lengths used to take into consideration the p1_fsm_old queues, which are not used anymore. Although one could collect internal gen_statem event queue data, it would be a complex task, and it might decrease system performance.
  • additionally, cets_info and mnesia_info are set up with mongoose_system_probes, although their configuration and implementation is in mongoose_internal_databases.
    • In case of cets_info, it is further delegated to mongoose_instrument_probe_cets. I didn't do the same for Mnesia, because the Mnesia probe is very simple.
    • In case of mnesia_info, db_nodes is added to running_db_nodes (which used to be called clusterSize).

Other changes:

  • Initialization of Mnesia in mongoose_internal_databases is refactored without any functional changes.
  • Fixed carbon_cache_server. This Graphite mock expected packets preceded by their length, while the Graphite text protocol uses text delimited by newlines. As a result, mongooseim_metrics_SUITE could take up to a minute to
    execute, because the first bytes were interpreted as packet length, and the code was just waiting for more and more metrics.
  • Replaced metrics tests from test/mongooseim_metrics_SUITE.erl with event checks in big_tests/system_probes_SUITE.erl.
  • Updated metrics used in tests and docs.
  • Remove complex GraphQL types, that are no longer needed, because the probes use gauges instead.

@chrzaszcz chrzaszcz changed the title WIP Instrument: global probes and counters Jul 5, 2024
@mongoose-im

This comment was marked as outdated.

Sign in to view
@mongoose-im

This comment was marked as outdated.

Sign in to view
@codecov Codecov
Copy link

codecov bot commented Jul 8, 2024 •
edited
Loading

Codecov Report

Attention: Patch coverage is 95.83333% with 2 lines in your changes missing coverage. Please review.

Project coverage is 76.16%. Comparing base (31ad729) to head (5c7ea1c).
Report is 5 commits behind head on feature/instrument.

Files Patch % Lines
src/mongoose_system_probes.erl 94.28% 2 Missing ⚠️

❗ There is a different number of reports uploaded between BASE (31ad729) and HEAD (5c7ea1c). Click for more details.

HEAD has 13 uploads less than BASE
Flag BASE (31ad729) HEAD (5c7ea1c)
15 2
Additional details and impacted files 
@@                  Coverage Diff                   @@
##           feature/instrument    #4317      +/-   ##
======================================================
- Coverage               84.39%   76.16%   -8.24%     
======================================================
  Files                     553      554       +1     
  Lines                   33874    33856      -18     
======================================================
- Hits                    28587    25785    -2802     
- Misses                   5287     8071    +2784

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

@mongoose-im

This comment was marked as outdated.

Sign in to view
@mongoose-im

This comment was marked as outdated.

Sign in to view
@mongoose-im

This comment was marked as outdated.

Sign in to view
@mongoose-im

This comment was marked as outdated.

Sign in to view
@mongoose-im

This comment was marked as outdated.

Sign in to view
@mongoose-im

This comment was marked as outdated.

Sign in to view
@mongoose-im

This comment was marked as outdated.

Sign in to view
@mongoose-im

This comment was marked as outdated.

Sign in to view
@mongoose-im

This comment was marked as outdated.

Sign in to view
@mongoose-im

This comment was marked as outdated.

Sign in to view
@mongoose-im

This comment was marked as outdated.

Sign in to view
@mongoose-im

This comment was marked as outdated.

Sign in to view
@mongoose-im

This comment was marked as outdated.

Sign in to view
@mongoose-im

This comment was marked as outdated.

Sign in to view
@mongoose-im
Copy link
Collaborator

mongoose-im commented Jul 15, 2024 •
edited
Loading

elasticsearch_and_cassandra_26 / elasticsearch_and_cassandra_mnesia / 6b6dc1a
Reports root/ big
OK: 457 / Failed: 0 / User-skipped: 45 / Auto-skipped: 0

small_tests_25 / small_tests / 6b6dc1a
Reports root / small

small_tests_26 / small_tests / 6b6dc1a
Reports root / small

small_tests_26_arm64 / small_tests / 6b6dc1a
Reports root / small

ldap_mnesia_26 / ldap_mnesia / 6b6dc1a
Reports root/ big
OK: 2293 / Failed: 0 / User-skipped: 910 / Auto-skipped: 0

internal_mnesia_26 / internal_mnesia / 6b6dc1a
Reports root/ big
OK: 2435 / Failed: 0 / User-skipped: 768 / Auto-skipped: 0

dynamic_domains_pgsql_mnesia_26 / pgsql_mnesia / 6b6dc1a
Reports root/ big
OK: 4640 / Failed: 0 / User-skipped: 106 / Auto-skipped: 0

dynamic_domains_mysql_redis_26 / mysql_redis / 6b6dc1a
Reports root/ big
OK: 4607 / Failed: 0 / User-skipped: 139 / Auto-skipped: 0

dynamic_domains_pgsql_mnesia_25 / pgsql_mnesia / 6b6dc1a
Reports root/ big
OK: 4640 / Failed: 0 / User-skipped: 106 / Auto-skipped: 0

pgsql_cets_26 / pgsql_cets / 6b6dc1a
Reports root/ big
OK: 4534 / Failed: 0 / User-skipped: 177 / Auto-skipped: 0

pgsql_mnesia_25 / pgsql_mnesia / 6b6dc1a
Reports root/ big
OK: 5032 / Failed: 0 / User-skipped: 115 / Auto-skipped: 0

ldap_mnesia_25 / ldap_mnesia / 6b6dc1a
Reports root/ big
OK: 2293 / Failed: 0 / User-skipped: 910 / Auto-skipped: 0

mysql_redis_26 / mysql_redis / 6b6dc1a
Reports root/ big
OK: 5013 / Failed: 0 / User-skipped: 134 / Auto-skipped: 0

mssql_mnesia_26 / odbc_mssql_mnesia / 6b6dc1a
Reports root/ big
OK: 5029 / Failed: 0 / User-skipped: 118 / Auto-skipped: 0

pgsql_mnesia_26 / pgsql_mnesia / 6b6dc1a
Reports root/ big
OK: 5032 / Failed: 0 / User-skipped: 115 / Auto-skipped: 0

dynamic_domains_mssql_mnesia_26 / odbc_mssql_mnesia / 6b6dc1a
Reports root/ big
OK: 4637 / Failed: 0 / User-skipped: 109 / Auto-skipped: 0

@chrzaszcz
Put system-wide instrumentation probes in mongoose_system_probes
35f7e51 
Included probes:
- Mnesia and system - moved from mongoose_metrics
@chrzaszcz
Start and stop mongoose_system_probes
9f4c738 
- Add start/stop calls
- Start/stop CETS and Mnesia probes in mongoose_internal_databases
@chrzaszcz
Don't start CETS probe in tests
ddba60b 
It is started by mongoose_system_probes now
@chrzaszcz
Clean up removed metrics
933d553
@chrzaszcz
Fix and update carbon_cache_server
1cca9c8 
This Graphite mock expected packets preceded by their length, while
the Graphite text protocol uses text delimited by newlines.

As a result, mongooseim_metrics_SUITE could take up to a minute to
execute, because the first bytes were interpreted as packet length.
@chrzaszcz
Clean up mongooseim_metrics_SUITE
c1acffc 
Remove metric tests, which are now replaced by events checked in big tests.
@chrzaszcz
Replace metric checks with instrumentation events in big tests
89431bc
@chrzaszcz
Update graphql_metric_SUITE with the new gauges
696df68 
Not all possible gagues were checked, because they will be tested
in system_probes_SUITE.
@chrzaszcz
Update metrics tested in metrics_api_SUITE
9608c19
@chrzaszcz
Test system probes in system_probes_SUITE
f1796e3 
- Remove instrument_cets_SUITE
@chrzaszcz
Remove unused types (there are gauges now instead)
b7123ff
@chrzaszcz
Update docs with the new metric names
5c7ea1c
@mongoose-im
Copy link
Collaborator

mongoose-im commented Jul 15, 2024 •
edited
Loading

elasticsearch_and_cassandra_26 / elasticsearch_and_cassandra_mnesia / 5c7ea1c
Reports root/ big
OK: 457 / Failed: 0 / User-skipped: 45 / Auto-skipped: 0

small_tests_25 / small_tests / 5c7ea1c
Reports root / small

small_tests_26 / small_tests / 5c7ea1c
Reports root / small

small_tests_26_arm64 / small_tests / 5c7ea1c
Reports root / small

ldap_mnesia_25 / ldap_mnesia / 5c7ea1c
Reports root/ big
OK: 2293 / Failed: 0 / User-skipped: 910 / Auto-skipped: 0

dynamic_domains_mysql_redis_26 / mysql_redis / 5c7ea1c
Reports root/ big
OK: 4607 / Failed: 0 / User-skipped: 139 / Auto-skipped: 0

ldap_mnesia_26 / ldap_mnesia / 5c7ea1c
Reports root/ big
OK: 2293 / Failed: 0 / User-skipped: 910 / Auto-skipped: 0

dynamic_domains_pgsql_mnesia_25 / pgsql_mnesia / 5c7ea1c
Reports root/ big
OK: 4640 / Failed: 0 / User-skipped: 106 / Auto-skipped: 0

dynamic_domains_pgsql_mnesia_26 / pgsql_mnesia / 5c7ea1c
Reports root/ big
OK: 4640 / Failed: 0 / User-skipped: 106 / Auto-skipped: 0

internal_mnesia_26 / internal_mnesia / 5c7ea1c
Reports root/ big
OK: 2434 / Failed: 1 / User-skipped: 768 / Auto-skipped: 0

pubsub_SUITE:dag+node_config:send_last_published_item_test 
{error,{{badmatch,false},
    [{pubsub_tools,check_response,2,
             [{file,"/home/circleci/project/big_tests/tests/pubsub_tools.erl"},
            {line,444}]},
     {pubsub_tools,receive_response,3,
             [{file,"/home/circleci/project/big_tests/tests/pubsub_tools.erl"},
            {line,434}]},
     {pubsub_tools,receive_and_check_response,4,
             [{file,"/home/circleci/project/big_tests/tests/pubsub_tools.erl"},
            {line,424}]},
     {escalus_story,story,4,
            [{file,"/home/circleci/project/big_tests/_build/default/lib/escalus/src/escalus_story.erl"},
             {line,72}]},
     {test_server,ts_tc,3,[{file,"test_server.erl"},{line,1793}]},
     {test_server,run_test_case_eval1,6,
            [{file,"test_server.erl"},{line,1302}]},
     {test_server,run_test_case_eval,9,
            [{file,"test_server.erl"},{line,1234}]}]}}

Report log

pgsql_cets_26 / pgsql_cets / 5c7ea1c
Reports root/ big
OK: 4533 / Failed: 1 / User-skipped: 177 / Auto-skipped: 0

connect_SUITE:just_tls:starttls_required:starttls_required_parallel:cannot_connect_with_proxy_header 
{error,
  {thrown,
    {{timeout,stream_end},
     [{escalus_connection,get_stream_end,2,
        [{file,
           "/home/circleci/project/big_tests/_build/default/lib/escalus/src/escalus_connection.erl"},
         {line,369}]},
      {escalus_connection,end_stream,1,
        [{file,
           "/home/circleci/project/big_tests/_build/default/lib/escalus/src/escalus_connection.erl"},
         {line,507}]},
      {escalus_connection,stop,1,
        [{file,
           "/home/circleci/project/big_tests/_build/default/lib/escalus/src/escalus_connection.erl"},
         {line,407}]},
      {test_server,ts_tc,3,[{file,"test_server.erl"},{line,1793}]},
      {test_server,run_test_case_eval1,6,
        [{file,"test_server.erl"},{line,1302}]},
      {test_server,run_test_case_eval,9,
        [{file,"test_server.erl"},{line,1234}]}]}}}

Report log

dynamic_domains_mssql_mnesia_26 / odbc_mssql_mnesia / 5c7ea1c
Reports root/ big
OK: 4637 / Failed: 0 / User-skipped: 109 / Auto-skipped: 0

pgsql_mnesia_25 / pgsql_mnesia / 5c7ea1c
Reports root/ big
OK: 5032 / Failed: 0 / User-skipped: 115 / Auto-skipped: 0

mysql_redis_26 / mysql_redis / 5c7ea1c
Reports root/ big
OK: 5013 / Failed: 0 / User-skipped: 134 / Auto-skipped: 0

pgsql_mnesia_26 / pgsql_mnesia / 5c7ea1c
Reports root/ big
OK: 5032 / Failed: 0 / User-skipped: 115 / Auto-skipped: 0

mssql_mnesia_26 / odbc_mssql_mnesia / 5c7ea1c
Reports root/ big
OK: 5029 / Failed: 0 / User-skipped: 118 / Auto-skipped: 0

internal_mnesia_26 / internal_mnesia / 5c7ea1c
Reports root/ big
OK: 2435 / Failed: 0 / User-skipped: 768 / Auto-skipped: 0

pgsql_cets_26 / pgsql_cets / 5c7ea1c
Reports root/ big
OK: 4534 / Failed: 0 / User-skipped: 177 / Auto-skipped: 0

@chrzaszcz chrzaszcz marked this pull request as ready for review July 15, 2024 14:11
jacekwegr
jacekwegr approved these changes Jul 16, 2024
View reviewed changes
Copy link
Contributor

@jacekwegr jacekwegr left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the changes! The new tests look excellent, and I appreciate the simplification of the GraphQL types. Everything looks good to me 👍

@jacekwegr jacekwegr merged commit 783dbea into feature/instrument Jul 16, 2024
3 of 4 checks passed
@jacekwegr jacekwegr deleted the instrument/global-metrics branch July 16, 2024 07:23
@jacekwegr jacekwegr added this to the 6.3.0 milestone Oct 3, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@jacekwegr jacekwegr jacekwegr approved these changes

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
6.3.0
Development

Successfully merging this pull request may close these issues.
3 participants
@chrzaszcz @mongoose-im @jacekwegr