erlwitness

Copyright (c) 2015 Guilherme Andrade

Version: 1.0.0

Authors: Guilherme Andrade (erlwitness(at)gandrade(dot)net).

erlwitness: Semantic process groups watchtower.

---

What does it do?

erlwitness allows one to funnel both gen_server events (init, calls, casts, infos, state changes) and lager calls specific to identified entities into arbitrary watcher processes.

Why?

For example: remote activity tracers for sizeable software development teams (instead of hey, may you please check the log?)

Main features

• Multiple watchers can spy on the same entity;
• A single watcher can spy on multiple entities;
• An entity can consist of zero or more processes that come and go;
• Watching works transparently on both local and remote nodes;
• Watching can be initiated both before and after entity processes have been spawned.
• Lager calls are (optionally) watchable through a parse_transform.

How do I use it?

There are two main parts to an ordinary setup:

1. Adapt your existing entities' gen_server:start / gen_server:start_link / init calls to make use of the relevant registration flow:

% ....
-behaviour(gen_server).
% ...
%% Uncomment to allow for lager tracing
%-compile([{parse_transform, erlwitness_transform}]).

start_link(Person, Files, LuckyNumber) ->
    {WrappedArgs, StartOptions} = erlwitness:get_start_extras({person, Person},
                                                              person_file_serv,
                                                              [Files, LuckyNumber]),
    gen_server:start_link(?MODULE, WrappedArgs, StartOptions).

init(WrappedArgs) ->
    [Files, LuckyNumber] = erlwitness:unwrap_init_args(WrappedArgs),
    InitResult = {ok, #state{files = Files,
                             lucky_number = LuckyNumber}},
    erlwitness:finalize_init(WrappedArgs, InitResult).
% ....

2. Code your own watcher process which will implement both gen_server and erlwitness_watcher behaviours:

% ...
-behaviour(gen_server).
-behaviour(erlwitness_watcher).
% ...

start_link(Person) ->
    erlwitness_watcher:start_link([{person, Person}], ?MODULE, []).

% ...

handle_gencall_event(Timestamp, {person, Person}, PersonPid, PersonProcType,
                     PersonProcName, Call, From, State) ->
    io:format("Here's a call: ~p~n", [{Timestamp, Person, PersonPid,
                                       PersonProcType, Call, From}]),
    {noreply, State}.

handle_gencast_event(Timestamp, {person, Person}, PersonPid, PersonProcType,
                     PersonProcName, Cast, State) ->
    io:format("Here's a cast: ~p~n", [{Timestamp, Person, PersonPid,
                                       PersonProcType, Cast}]),
    {noreply, State}.

handle_geninfo_event(Timestamp, {person, Person}, PersonPid, PersonProcType,
                     PersonProcName, Info, State) ->
    io:format("Here's an info: ~p~n", [{Timestamp, Person, PersonPid,
                                       PersonProcType, Info}]),
    {noreply, State}.

handle_newstate_event(Timestamp, {person, Person}, PersonPid, PersonProcType,
                      PersonProcName, PersonState, State) ->
    io:format("Here's a new state: ~p~n", [{Timestamp, Person, PersonPid,
                                            PersonProcType, PersonState}]),
    {noreply, State}.

handle_lager_event(Timestamp, {person, Person}, PersonPid, PersonProcType,
                   PersonProcName, LagerMFA, LagerDebugInfo, State) ->
    {_LagerModule, LagerLevel, LagerArgs} = LagerMFA,
    {CodeModule, CodeFunction, CodeLine} = LagerDebugInfo,
    io:format("Here's a lager message from ~p: ~p @ ~p~n",
             [{Timestamp, Person, PersonPid, PersonProcType, PersonState},
              {LagerLevel, LagerArgs}, {CodeModule, CodeFunction, CodeLine}]),
    {noreply, State}.

% ..

.. and optionally, 3) Reuse your existing entity registration code by implementing the erlwitness_lookup behaviour and adjusting erlwitness app.config acordingly.

% ...
-behaviour(erlwitness_lookup).
% ...

lookup_global_entity({person, Person}) ->
   PeopleInfo = mnesia:dirty_read(person_info, Person),
   [{person_file_serv, PersonInfo#person_info.file_serv_pid} || PersonInfo <- Pids].
% ...

Full implementations under example/.

How do I configure it?

app.config:

[{erlwitness, [
            % Optional; module implementing 'erlwitness_lookup' behaviour
            %{entity_lookup_module, erlwitness_index_serv},
            %
            % Optional; defaults to 10 * NumberOfSchedulers
            %{erlwitness_index_serv_count, N :: pos_integer()}
            ]}
    ].

How do I globally enable the lager events parse_transform?

Compiler flags:

{parse_transform, erlwitness_transform}

What about the overhead?

This software is built on the premise that watchers are few, rare, and mostly limited to development environments; therefore watching is heavy and privacy is light.

• Unwatched processes: basic ETS lookup just before the process starts + a single process dictionary check for every lager call;
• Watched processes: one OTP debug fun per watcher (see sys(3)) + lager calls redirection.

Why are there no termination events?

This software is a building block; tracking and monitoring entity processes on a watcher is left as an exercise for the reader.

What's the deal with entity registration?

erlwitness_index_serv is bundled as an out-of-the-box solution; it might not, however, suit your needs. Please mind that it will, for each spawned entity:

• 1. Write a new entry to an ETS table (on erlwitness:finalize_init/2);
• 2. Create a monitor from a specific worker on the indexing pool;
• 3. Trigger the monitor on termination in order to unregister the entity.

In order to lax this potential bottleneck, the indexing pool will spawn (10 x NumberOfSchedulers) monitoring processes by default, with one separate ETS table per indexer, and NumberOfSchedulers generally (but not always) corresponding to the number of CPU thread queues (usually NumOfCPUs x NumOfCoresPerCPU x (1 + HyperthreadPerCore)).

However, if you already have your own global process directory in place, it's recommended that you use it instead.

Future plans

• Clean up the current lager events mess;
• Offline watcher scheduling with later history retrieving.

Modules

erlwitness

erlwitness_app

erlwitness_conf

erlwitness_entity

erlwitness_index_serv

erlwitness_lobby

erlwitness_lookup

erlwitness_sup

erlwitness_transform

erlwitness_watcher