Add a method to provide statistics about CRUD operations

[ligurio]

Description

This is a raw idea, but I hope it would be useful for CRUD users.

CRUD perform operations on storage servers, and sometimes we need to know how many operations were done on each storage (in tests for #222 and #166). Probably (it's a hypothesis now) our users need such information too. However, we cannot use box.stat on storage servers because CRUD is not a single who uses storage servers, there is at least vshard that can move buckets between servers in background.

In this ticket we need to understand

• a priority of task
• a set of counters interested in CRUD users
• integration with metrics module?

Stat counters in Tarantool

CRUD implements the same interface as in Tarantool (create, replace, update, delete etc) and probably we can inherit stat interface from Tarantool too. See module box.stat:

tarantool> box.stat() -- return 11 tables
---
- DELETE:
    total: 1873949
    rps: 123
  SELECT:
    total: 1237723
    rps: 4099
  INSERT:
    total: 0
    rps: 0
  EVAL:
    total: 0
    rps: 0
  CALL:
    total: 0
    rps: 0
  REPLACE:
    total: 1239123
    rps: 7849
  UPSERT:
    total: 0
    rps: 0
  AUTH:
    total: 0
    rps: 0
  ERROR:
    total: 0
    rps: 0
  EXECUTE:
    total: 0
    rps: 0
  UPDATE:
    total: 0
    rps: 0
...

Stat counters in graphql

Also graphql.0 has a method to provide statistics:

function statistics.new(opts)
    local opts = opts or {}
    local resulting_object_cnt_max = opts.resulting_object_cnt_max
    local fetched_object_cnt_max = opts.fetched_object_cnt_max

    return setmetatable({
        resulting_object_cnt = 0,          -- retire
        fetches_cnt = 0,                   -- fetch
        fetched_object_cnt = 0,            -- fetch
        full_scan_cnt = 0,                 -- fetch
        index_lookup_cnt = 0,              -- fetch
        cache_hits_cnt = 0,                -- cache lookup
        cache_hit_objects_cnt = 0,         -- cache lookup
        limits = {
            resulting_object_cnt_max = resulting_object_cnt_max, -- retire limit
            fetched_object_cnt_max = fetched_object_cnt_max,     -- fetch limit
        }
    }, {
        __index = {
            objects_fetched = objects_fetched,
            objects_retired = objects_retired,
            cache_lookup = cache_lookup,
        }
    })
end

[Totktonada]

NB: Whatever we'll implement, add it into default metrics.

[Totktonada]

Raw thoughts about implementation.

We can decouple statistics implementation from main crud code for readability reasons. We should provide ability to set a callback on storage and on router, which is called before/after execution a function that serves insert/delete/select/etc (or one callback for all of them with a types passed to the function). A return value of 'before' callback should be passed to 'after' one, so we can store startup time and calculate a request duration. Or a context table should be created per request and passed to 'before' and 'after' callbacks both.

After this we can place all statistics code in its own module within crud or even implement it entirely in metrics. I would prefer the former, because it may be useful to inspect crud's statistics interactively from tarantool console (say, if monitoring is not configured for the application).

Whether we need to pass statistics information from storage to router together with a response? If all nodes are in monitoring, it does not look required.

[DifferentialOrange]

One of the main questions is how crud statistics will be related to metrics module.

There are two possible options:

1. crud rock depends on metrics,
2. crud rock not depends on metrics.

The plan is to support crud statistics, such as requests count and requests latency. If requests count can be manually organized with a plain number variable, it is much more complicated for a latency statistics. To aggregate information about time of execution to something useful, we need a tool like histogram or summary with quantiles. It would be unwise to reimplement it here because we already have it in metrics, especially since there is a high-load summary collector with aging based on ffi and non-trivial algorithms, which was tested in production. Drawback of "crud rock depends on metrics" approach is described by itself -- it is an excessive dependency tarantool packages, both of which are used widely. There are many projects that depends on both crud and metrics now, and if we add this dependency, it became complicated. "Pessimistic" or "greater or equal" metrics version dependency in rockspec (~> or >=) may help, but since current metrics version policy is 0.Y.Z with new non-breaking 0.Y+1.0 releasing every couple of months, it may not help.

So I think the answer is

• crud rock depends on metrics.

[DifferentialOrange]

The other one is how to enable/disable metrics for crud. I think it is a reasonable requirement to have such a handle.

metrics since 0.10.0 (tarantool/metrics#222) supports enable/disable for groups of metrics. It is a mechanism that works both with vanilla Tarantool and have convenient Cartridge integration through configuration. Currently there are no ability to add custom groups of default metrics from the outside. I see two solutions:

1. add a feature to support external groups of default metrics,
2. add a group of crud metrics to list of default metrics in metrics.

Since there are already "optional" groups of default metrics (like cartridge metrics or luajit metrics) that works if they can work (there is cartridge/Tarantool have required version), I don't think there will be any problems with second one. Both solutions should work fine, but both integrations should be discussed with metrics maintainers.

Considering the handles, we can describe them in crud code and then run them with metrics enable/disable process. So we will be able to enable them even without metrics, and at the same time integrate them into common pipeline. But it will have not much sense if we will use metrics collectors and, thus, depend on metrics package.

[DifferentialOrange]

metrics collector values can be seen without any monitoring systems attached in console, but it may be not very human-friendly (but still readable). But if we will use metrics collectors, there will be no in-built way to see current rps like in box.* response, only summary details (quantiles with aging will show current situation, but total requests may be not very useful).

Considering sending info between storages and routers, I don't think it is necessary. We can measure full time of request execution on router without getting info from storage, since each request starts and finishes on a router. I see the following model:

• router collects info on requests and their latency on its side,
• storage collects info on requests and their latency on its side.

This info then could be visualized separately. So we may analyze

• how requests are distributed between storages,
• are requests stuck on a storage or on a router/on a way between router and storage (the latter two will be inseparatable).