stats: add statistics for CRUD router operations

Add statistics module for collecting metrics of CRUD operations on
router. Wrap all CRUD operation calls in the statistics collector.
Statistics must be enabled manually with `crud.cfg`. They can be
disabled, restarted or re-enabled later.

This patch introduces `crud.cfg`. `crud.cfg` is a tool to set module
configuration. It is similar to Tarantool `box.cfg`, although we don't
need to call it to bootstrap the module -- it is used only to change
configuration. `crud.cfg` is a callable table. To change configuration,
call it: `crud.cfg{ stats = true }`. You can check table contents as
with ordinary table, but do not change them directly -- use call
instead. Table contents is immutable and use proxy approach
(see [1, 2]). Iterating through `crud.cfg` with pairs is not supported
yet, refer to #265.

`crud.stats()` returns

---
- spaces:
    my_space:
      insert:
        ok:
          latency: 0.002
          count: 19800
          time: 39.6
        error:
          latency: 0.000001
          count: 4
          time: 0.000004
...

`spaces` section contains statistics for each observed space.
If operation has never been called for a space, the corresponding
field will be empty. If no requests has been called for a
space, it will not be represented. Space data is based on
client requests rather than storages schema, so requests
for non-existing spaces are also collected.

Possible statistics operation labels are
`insert` (for `insert` and `insert_object` calls),
`get`, `replace` (for `replace` and `replace_object` calls), `update`,
`upsert` (for `upsert` and `upsert_object` calls), `delete`,
`select` (for `select` and `pairs` calls), `truncate`, `len`, `count`
and `borders` (for `min` and `max` calls).

Each operation section consists of different collectors
for success calls and error (both error throw and `nil, err`)
returns. `count` is the total requests count since instance start
or stats restart. `latency` is the average time of requests execution,
`time` is the total time of requests execution.

Since `pairs` request behavior differs from any other crud request, its
statistics collection also has specific behavior. Statistics (`select`
section) are updated after `pairs` cycle is finished: you
either have iterated through all records or an error was thrown.
If your pairs cycle was interrupted with `break`, statistics will
be collected when pairs objects are cleaned up with Lua garbage
collector.

Statistics are preserved between package reloads. Statistics are
preserved between Tarantool Cartridge role reloads [3] if CRUD Cartridge
roles are used.

1. http://lua-users.org/wiki/ReadOnlyTables
2. tarantool/tarantool#2867
3. https://www.tarantool.io/en/doc/latest/book/cartridge/cartridge_api/modules/cartridge.roles/#reload

Part of #224

CHANGELOG.md

@@ -8,6 +8,7 @@ and this project adheres to [Semantic Versioning](http://semver.org/spec/v2.0.0.
 ## [Unreleased]
 
 ### Added
+* Statistics for CRUD operations on router (#224).
 
 ### Changed
 

README.md

@@ -674,6 +674,97 @@ Combinations of `mode`, `prefer_replica` and `balance` options lead to:
   * prefer_replica, balance -
     [vshard call `callbre`](https://www.tarantool.io/en/doc/latest/reference/reference_rock/vshard/vshard_api/#router-api-callbre)
 
+### Statistics
+
+`crud` routers can provide statistics on called operations.
+```lua
+-- Enable statistics collect.
+crud.cfg{ stats = true }
+
+-- Returns table with statistics information.
+crud.stats()
+
+-- Returns table with statistics information for specific space.
+crud.stats('my_space')
+
+-- Disables statistics collect and destroys all collectors.
+crud.cfg{ stats = false }
+
+-- Destroys all statistics collectors and creates them again.
+crud.reset_stats()
+```
+
+You can use `crud.cfg` to check current stats state.
+```lua
+crud.cfg
+---
+- stats: true
+...
+```
+Beware that iterating through `crud.cfg` with pairs is not supported yet,
+refer to [tarantool/crud#265](https://github.com/tarantool/crud/issues/265).
+
+Format is as follows.
+```lua
+crud.stats()
+---
+- spaces:
+    my_space:
+      insert:
+        ok:
+          latency: 0.002
+          count: 19800
+          time: 39.6
+        error:
+          latency: 0.000001
+          count: 4
+          time: 0.000004
+...
+crud.stats('my_space')
+---
+- insert:
+    ok:
+      latency: 0.002
+      count: 19800
+      time: 39.6
+    error:
+      latency: 0.000001
+      count: 4
+      time: 0.000004
+...
+```
+`spaces` section contains statistics for each observed space.
+If operation has never been called for a space, the corresponding
+field will be empty. If no requests has been called for a
+space, it will not be represented. Space data is based on
+client requests rather than storages schema, so requests
+for non-existing spaces are also collected.
+
+Possible statistics operation labels are
+`insert` (for `insert` and `insert_object` calls),
+`get`, `replace` (for `replace` and `replace_object` calls), `update`,
+`upsert` (for `upsert` and `upsert_object` calls), `delete`,
+`select` (for `select` and `pairs` calls), `truncate`, `len`, `count`
+and `borders` (for `min` and `max` calls).
+
+Each operation section contains of different collectors
+for success calls and error (both error throw and `nil, err`)
+returns. `count` is total requests count since instance start
+or stats restart. `latency` is average time of requests execution,
+`time` is the total time of requests execution.
+
+Since `pairs` request behavior differs from any other crud request, its
+statistics collection also has specific behavior. Statistics (`select`
+section) are updated after `pairs` cycle is finished: you
+either have iterated through all records or an error was thrown.
+If your pairs cycle was interrupted with `break`, statistics will
+be collected when pairs objects are cleaned up with Lua garbage
+collector.
+
+Statistics are preserved between package reloads. Statistics are preserved
+between [Tarantool Cartridge role reloads](https://www.tarantool.io/en/doc/latest/book/cartridge/cartridge_api/modules/cartridge.roles/#reload)
+if you use CRUD Cartridge roles.
+
 ## Cartridge roles
 
 `cartridge.roles.crud-storage` is a Tarantool Cartridge role that depends on the

cartridge/roles/crud-router.lua

@@ -1,8 +1,10 @@
 local crud = require('crud')
+local stash = require('crud.common.stash')
 
 -- removes routes that changed in config and adds new routes
 local function init()
     crud.init_router()
+    stash.setup_cartridge_reload()
 end
 
 local function stop()

cartridge/roles/crud-storage.lua

@@ -1,7 +1,9 @@
 local crud = require('crud')
+local stash = require('crud.common.stash')
 
 local function init()
     crud.init_storage()
+    stash.setup_cartridge_reload()
 end
 
 local function stop()

crud.lua

@@ -2,6 +2,7 @@
 --
 -- @module crud
 
+local cfg = require('crud.cfg')
 local insert = require('crud.insert')
 local replace = require('crud.replace')
 local get = require('crud.get')
@@ -15,6 +16,7 @@ local count = require('crud.count')
 local borders = require('crud.borders')
 local sharding_metadata = require('crud.common.sharding.sharding_metadata')
 local utils = require('crud.common.utils')
+local stats = require('crud.stats')
 
 local crud = {}
 
@@ -23,71 +25,71 @@ local crud = {}
 
 -- @refer insert.tuple
 -- @function insert
-crud.insert = insert.tuple
+crud.insert = stats.wrap(insert.tuple, stats.op.INSERT)
 
 -- @refer insert.object
 -- @function insert_object
-crud.insert_object = insert.object
+crud.insert_object = stats.wrap(insert.object, stats.op.INSERT)
 
 -- @refer get.call
 -- @function get
-crud.get = get.call
+crud.get = stats.wrap(get.call, stats.op.GET)
 
 -- @refer replace.tuple
 -- @function replace
-crud.replace = replace.tuple
+crud.replace = stats.wrap(replace.tuple, stats.op.REPLACE)
 
 -- @refer replace.object
 -- @function replace_object
-crud.replace_object = replace.object
+crud.replace_object = stats.wrap(replace.object, stats.op.REPLACE)
 
 -- @refer update.call
 -- @function update
-crud.update = update.call
+crud.update = stats.wrap(update.call, stats.op.UPDATE)
 
 -- @refer upsert.tuple
 -- @function upsert
-crud.upsert = upsert.tuple
+crud.upsert = stats.wrap(upsert.tuple, stats.op.UPSERT)
 
 -- @refer upsert.object
 -- @function upsert
-crud.upsert_object = upsert.object
+crud.upsert_object = stats.wrap(upsert.object, stats.op.UPSERT)
 
 -- @refer delete.call
 -- @function delete
-crud.delete = delete.call
+crud.delete = stats.wrap(delete.call, stats.op.DELETE)
 
 -- @refer select.call
 -- @function select
-crud.select = select.call
+crud.select = stats.wrap(select.call, stats.op.SELECT)
 
 -- @refer select.pairs
 -- @function pairs
-crud.pairs = select.pairs
+crud.pairs = stats.wrap(select.pairs, stats.op.SELECT, { pairs = true })
 
 -- @refer utils.unflatten_rows
 -- @function unflatten_rows
 crud.unflatten_rows = utils.unflatten_rows
 
 -- @refer truncate.call
 -- @function truncate
-crud.truncate = truncate.call
+crud.truncate = stats.wrap(truncate.call, stats.op.TRUNCATE)
 
 -- @refer len.call
 -- @function len
-crud.len = len.call
+crud.len = stats.wrap(len.call, stats.op.LEN)
 
 -- @refer count.call
 -- @function count
-crud.count = count.call
+crud.count = stats.wrap(count.call, stats.op.COUNT)
 
 -- @refer borders.min
 -- @function min
-crud.min = borders.min
+crud.min = stats.wrap(borders.min, stats.op.BORDERS)
 
 -- @refer borders.max
 -- @function max
-crud.max = borders.max
+crud.max = stats.wrap(borders.max, stats.op.BORDERS)
 
 -- @refer utils.cut_rows
 -- @function cut_rows
@@ -97,6 +99,18 @@ crud.cut_rows = utils.cut_rows
 -- @function cut_objects
 crud.cut_objects = utils.cut_objects
 
+-- @refer cfg.cfg
+-- @function cfg
+crud.cfg = cfg.cfg
+
+-- @refer stats.get
+-- @function stats
+crud.stats = stats.get
+
+-- @refer stats.reset
+-- @function reset_stats
+crud.reset_stats = stats.reset
+
 --- Initializes crud on node
 --
 -- Exports all functions that are used for calls

crud/cfg.lua

@@ -0,0 +1,70 @@
+---- Module for CRUD configuration.
+-- @module crud.cfg
+--
+
+local checks = require('checks')
+local errors = require('errors')
+
+local stash = require('crud.common.stash')
+local stats = require('crud.stats')
+
+local CfgError = errors.new_class('CfgError', {capture_stack = false})
+
+local cfg_module = {}
+
+local function set_defaults_if_empty(cfg)
+    if cfg.stats == nil then
+        cfg.stats = false
+    end
+
+    return cfg
+end
+
+local cfg = set_defaults_if_empty(stash.get(stash.name.cfg))
+
+--- Configure CRUD module.
+--
+-- @function __call
+--
+-- @tab self
+--
+-- @tab[opt] opts
+--
+-- @bool[opt] opts.stats
+--  Enable or disable statistics collect.
+--  Statistics are observed only on router instances.
+--
+-- @return Configuration table.
+--
+local function __call(self, opts)
+    checks('table', { stats = '?boolean' })
+
+    opts = opts or {}
+
+    if opts.stats ~= nil then
+        if opts.stats == true then
+            stats.enable()
+        else
+            stats.disable()
+        end
+
+        rawset(cfg, 'stats', opts.stats)
+    end
+
+    return self
+end
+
+local function __newindex()
+    CfgError:assert(false, 'Use crud.cfg{} instead')
+end
+
+-- Iterating through `crud.cfg` with pairs is not supported
+-- yet, refer to tarantool/crud#265.
+cfg_module.cfg = setmetatable({}, {
+    __index = cfg,
+    __newindex = __newindex,
+    __call = __call,
+    __serialize = function() return cfg end
+})
+
+return cfg_module

crud/common/stash.lua

@@ -0,0 +1,63 @@
+---- Module for preserving data between reloads.
+-- @module crud.common.stash
+--
+local dev_checks = require('crud.common.dev_checks')
+
+local stash = {}
+
+--- Available stashes list.
+--
+-- @tfield string cfg
+--  Stash for CRUD module configuration.
+--
+-- @tfield string stats_internal
+--  Stash for main stats module.
+--
+-- @tfield string stats_local_registry
+--  Stash for local metrics registry.
+--
+stash.name = {
+    cfg = '__crud_cfg',
+    stats_internal = '__crud_stats_internal',
+    stats_local_registry = '__crud_stats_local_registry'
+}
+
+--- Setup Tarantool Cartridge reload.
+--
+--  Call on Tarantool Cartridge roles that are expected
+--  to use stashes.
+--
+-- @function setup_cartridge_reload
+--
+-- @return Returns
+--
+function stash.setup_cartridge_reload()
+    local hotreload = require('cartridge.hotreload')
+    for _, name in pairs(stash.name) do
+        hotreload.whitelist_globals({ name })
+    end
+end
+
+--- Get a stash instance, initialize if needed.
+--
+--  Stashes are persistent to package reload.
+--  To use them with Cartridge roles reload,
+--  call `stash.setup_cartridge_reload` in role.
+--
+-- @function get
+--
+-- @string name
+--  Stash identifier. Use one from `stash.name` table.
+--
+-- @treturn table A stash instance.
+--
+function stash.get(name)
+    dev_checks('string')
+
+    local instance = rawget(_G, name) or {}
+    rawset(_G, name, instance)
+
+    return instance
+end
+
+return stash