Implement LuaJIT memory profiler #5442

Buristan · 2020-10-26T06:17:22Z

As a result of the research regarding uJIT pros and cons, LuaJIT memory profiler turned out to be a highly requested feature, so we are going to extend our LuaJIT fork API with the following interfaces:

Lua module (similar to ujit.memprof) having methods to start and stop memory profiler.
C API method to start and stop memory profiler able to use custom writer function.
C and Lua API method to traverse and dump information about all objects from Lua universe.

Moreover utility to parse dumped information (similar to ujit-parse-memprof) should be added (#5490).

Also need integrate profiler with Tarantool with a custom writer taking into account possible high-load of discs.

Feedback and features enrich for memory profiler #5657 Furthermore, in scope of this ticket, a RFC for the feature should be sent to our ML, so we can enrich the memory profiler API considering users demand.

The text was updated successfully, but these errors were encountered:

That patch allows including headers by absolute path to them from source directory like: "utils/uleb128.h". Part of tarantool/tarantool#5442

This patch introduces module for reading and writing leb128 compression. It will be used for streaming profiling events writing, that will be added at the next patches. Part of tarantool/tarantool#5442

This patch introduces module for writing profile data. Its usage will be added at the next patches. Part of tarantool/tarantool#5442

This patch adds profile writer that writes all necessary Lua functions prototypes info like GCproto address, name of the chunk this function was defined in and number of the first line of it. Usage of this module will be added at the next patches. Part of tarantool/tarantool#5442

This patch slivers LJ_VMST_LFUNC and LJ_VMST_FFUNC from LJ_VMST_INERP to allow to determine context of vm execution. Also LJ_VMST_C is renamed to LJ_VMST_CFUNC for naming consistence with newer vmstates. Part of tarantool/tarantool#5442

This patch adds new field to global_State structure named mem_L. It will be used in the next patches by memory profiler to determine which coroutine triggers memory allocation. Part of tarantool/tarantool#5442

This patch renames debug_frameline to lj_debug_frameline and moves it to public API. It will be used for memory profiler in the next patches. Part of tarantool/tarantool#5442

This patch adds memory profile module. Lua and C API for it will be added at the next patches. When VM executes a trace, vmstate is set to the trace number. Also this patch defines special macro LJ_VMST_TRACE equaled to LJ_VMST__MAX to encapsulate all different traces into one vmstate when profiled. Part of tarantool/tarantool#5442

This patch introduces both C and Lua API for LuaJIT memory profiler implemented in scope of the previous patch. FIXME: build for amalg and so on Part of tarantool/tarantool#5442

Part of #5442

This patch introduces module for writing profile data. Its usage will be added at the next patches. Part of tarantool/tarantool#5442

This patch adds profile writer that writes all necessary Lua functions prototypes info like GCproto address, name of the chunk this function was defined in and number of the first line of it. See <ljp_symtab.h> for details. Usage of this module will be added at the next patches. Part of tarantool/tarantool#5442

This patch slivers LJ_VMST_LFUNC and LJ_VMST_FFUNC from LJ_VMST_INERP to allow to determine the context of vm execution for x86/x64 arches. Also, LJ_VMST_C is renamed to LJ_VMST_CFUNC for naming consistence with newer vmstates. Also, this patch adjusts stack layout for x86/x64 arches to save vmstate to avoid inconsistent state while stack unwinding when an error is raised. Part of tarantool/tarantool#5442

This patch adds new field to global_State structure named mem_L. It will be used in the next patches by memory profiler to determine which coroutine triggers memory allocation. Part of tarantool/tarantool#5442

This patch renames debug_frameline to lj_debug_frameline and moves it to public <lj_debug.h> module API (does not provide it with LUA_API). It will be used for memory profiler in the next patches. Part of tarantool/tarantool#5442

This patch adds memory profile module. Lua and C API for it will be added at the next patches. When VM executes a trace, vmstate is set to the trace number. Also this patch defines special macro LJ_VMST_TRACE equaled to LJ_VMST__MAX to encapsulate all different traces into one vmstate when profiled. See <ljp_memprof.h> for details. Part of tarantool/tarantool#5442

This patch introduces both C and Lua API for LuaJIT memory profiler implemented in the scope of the previous patch. Profiler returns `true` if start successfully, returns nil on failure (plus an error message as a second result and a system-dependent error code as a third result) and some true value on success. <lj_errmsg.h> have adjusted with two new errors PROF_ISRUNNING/PROF_NOTRUNNING returned in case when profiler has started/stopped already correspondingly. Part of tarantool/tarantool#5442

This patch adds parser for profiler dumped binary data. It provides a script that parses given binary file. It parses symtab using ffi first and after map memory events with this symtab. Finally, it renders the data in human-readable format. Part of tarantool/tarantool#5442 Part of tarantool/tarantool#5490

This patch adds new directory contained all toolling for LuaJIT. Part of tarantool/tarantool#5442 Part of tarantool/tarantool#5490

This patch adds parser for profiler dumped binary data. It provides a script that parses given binary file. It parses symtab using ffi first and after map memory events with this symtab. Finally, it renders the data in human-readable format. Part of tarantool/tarantool#5442 Part of tarantool/tarantool#5490

Most of the numeric data written by the memory profiler is encoded via LEB128 compression. This patch introduces the module for encoding and decoding 64bit number to LEB128 form. Part of tarantool/tarantool#5442 Reviewed-by: Sergey Ostanevich <sergos@tarantool.org> Reviewed-by: Igor Munkin <imun@tarantool.org> Signed-off-by: Igor Munkin <imun@tarantool.org>

This patch introduces the standalone module for writing data to the file, socket or memory (and so on) via the special buffer. The module provides the API for buffer initial setup and its convenient usage. Part of tarantool/tarantool#5442 Reviewed-by: Sergey Ostanevich <sergos@tarantool.org> Reviewed-by: Igor Munkin <imun@tarantool.org> Signed-off-by: Igor Munkin <imun@tarantool.org>

This patch introduces LJ_VMST_LFUNC and LJ_VMST_FFUNC VM states separated from LJ_VMST_INERP. New VM states allow to determine the context of Lua VM execution for x86 and x64 arches. Also, LJ_VMST_C is renamed to LJ_VMST_CFUNC for naming consistence with new VM states. Also, this patch adjusts stack layout for x86 and x64 arches to save VM state for its consistency while stack unwinding when error is raised. To group all traces into the one vmstate, a special macro LJ_VMST_TRACE equal to LJ_VMST__MAX is introduced. Part of tarantool/tarantool#5442 Reviewed-by: Sergey Ostanevich <sergos@tarantool.org> Reviewed-by: Igor Munkin <imun@tarantool.org> Signed-off-by: Igor Munkin <imun@tarantool.org>

This patch introduces memory profiler for Lua machine. To determine currently allocating coroutine (that may not be equal to currently executed one) a new field mem_L is added to the global_State structure. This field is set on each allocation event and stores the coroutine address that is used for allocation. First of all profiler dumps the definitions of all loaded Lua functions (symtab) via the write buffer introduced in one of the previous patches. Profiler replaces the old allocation function with the instrumented one after symtab is dumped. This new function reports all allocations, reallocations or deallocations events via the write buffer during profiling. Subsequent content depends on the function's type (LFUNC, FFUNC or CFUNC). When profiling is over, a special epilogue event header is written and the old allocation function is restored back. This change also makes debug_frameline function LuaJIT-wide visible to be used in the memory profiler. For more information, see <lj_memprof.h>. Part of tarantool/tarantool#5442 Reviewed-by: Sergey Ostanevich <sergos@tarantool.org> Reviewed-by: Igor Munkin <imun@tarantool.org> Signed-off-by: Igor Munkin <imun@tarantool.org>

This patch introduces Lua API for LuaJIT memory profiler implemented in the scope of the previous patch. Profiler returns true value if started/stopped successfully, returns nil on failure (plus an error message as a second result and a system-dependent error code as a third result). If LuaJIT is build without memory profiler both return false. <lj_errmsg.h> has adjusted with three new errors PROF_MISUSE/PROF_ISRUNNING/PROF_NOTRUNNING returned in case when profiler has used incorrectly/started/stopped already correspondingly. Part of tarantool/tarantool#5442 Reviewed-by: Sergey Ostanevich <sergos@tarantool.org> Reviewed-by: Igor Munkin <imun@tarantool.org> Signed-off-by: Igor Munkin <imun@tarantool.org>

This patch adds a parser for binary data dumped via the memory profiler. It is a set of the following Lua modules: * utils/bufread.lua: read binary data from a file. * utils/symtab.lua: symbol table decode functions * memprof/parse.lua: decode the memory profiler event stream * memprof/humanize.lua: display decoded data in human readable format * memprof.lua: Lua script to display data There is also a stand-alone bash script <luajit-parse-memprof> that displays human readable parsed data to a stdout. It calls <memprof.lua> with a corresponding LUA_PATH. Part of tarantool/tarantool#5442 Part of tarantool/tarantool#5490 Reviewed-by: Sergey Ostanevich <sergos@tarantool.org> Reviewed-by: Igor Munkin <imun@tarantool.org> Signed-off-by: Igor Munkin <imun@tarantool.org>

This patch fixes the regression introduced in scope of b4e6bf0 ('misc: add Lua API for memory profiler'). Build fails with disabled memory profiler because related error messages are not defined. This patch fixes build by making LJ_ERR_PROF_MISUSE visible and avoiding usage of other profiler-related errors when the profiler is disabled. Follows up tarantool/tarantool#5442 Reviewed-by: Igor Munkin <imun@tarantool.org> Reviewed-by: Sergey Ostanevich <sergos@tarantool.org> Signed-off-by: Igor Munkin <imun@tarantool.org>

When memory profiler fails to start with PROFILE_ERRRUN status both stream and ctx are not released. At the same time when memory profiler fails to start with the PROFILE_ERRIO status both stream and ctx are released twice. Both cases occur due to invalid return status checking. To fix the leakage on_stop callback is called manually inside the profiler when error on start occurs. Checks in misc.memprof.start() are omitted. Follows up tarantool/tarantool#5442 Reviewed-by: Igor Munkin <imun@tarantool.org> Reviewed-by: Sergey Ostanevich <sergos@tarantool.org> Signed-off-by: Igor Munkin <imun@tarantool.org>

There are cases when the memory profiler attempts to attribute allocations triggered by JIT engine recording phase with a Lua function to be recorded. At this case lj_debug_frameline() may return BC_NOPOS (i.e. a negative value) so the assertion in the Lua writer memprof_write_lfunc() is violated. This patch removes this assertion. For negative returned line value profiler is reported zero frameline. Follows up tarantool/tarantool#5442 Reviewed-by: Igor Munkin <imun@tarantool.org> Reviewed-by: Sergey Ostanevich <sergos@tarantool.org> Signed-off-by: Igor Munkin <imun@tarantool.org>

Resizing of the Lua stack is not reported as internal allocation as it should. As a result, it may lead to crash inside Lua or FF frames. When the memory profiler runs, reallocation occurs first, and after profiler reports the corresponding event. When the stack is resized for local function arguments, the link to the previous frame is invalid in the case of reallocation. Therefore, the assertion in `debug_framepc()` fails. Resolves tarantool/tarantool#5842 Follows up tarantool/tarantool#5442 Reviewed-by: Igor Munkin <imun@tarantool.org> Reviewed-by: Sergey Ostanevich <sergos@tarantool.org> Signed-off-by: Igor Munkin <imun@tarantool.org>