Skip to content

Commit

Permalink
add heaptrack support
Browse files Browse the repository at this point in the history 
This commit is extracted from the next commits, with the purpose to
update the 'heaptrack_support' branch:

cd54ac3
577ad68
  • Loading branch information
@Ivansete-status @gabrielmer
Ivansete-status authored and gabrielmer committed May 16, 2024
1 parent 19fdbfc commit 03fb74d
Show file tree
Hide file tree
Showing 2 changed files with 56 additions and 216 deletions.
14 changes: 14 additions & 0 deletions lib/system/alloc.nim
View file
Original file line number Diff line number Diff line change
Expand Up @@ -691,6 +691,15 @@ else:
template trackSize(x) = discard
template untrackSize(x) = discard

when defined(heaptracker):
const heaptrackLib =
when defined(heaptracker_inject):
"libheaptrack_inject.so"
else:
"libheaptrack_preload.so"
proc heaptrack_malloc(a: pointer, size: int) {.cdecl, importc, dynlib:heaptrackLib.}
proc heaptrack_free(a: pointer) {.cdecl, importc, dynlib:heaptrackLib.}

proc rawAlloc(a: var MemRegion, requestedSize: int): pointer =
when defined(nimTypeNames):
inc(a.allocCounter)
Expand Down Expand Up @@ -765,13 +774,18 @@ proc rawAlloc(a: var MemRegion, requestedSize: int): pointer =
sysAssert(allocInv(a), "rawAlloc: end")
when logAlloc: cprintf("var pointer_%p = alloc(%ld)\n", result, requestedSize)

when defined(heaptracker):
heaptrack_malloc(result, requestedSize)

proc rawAlloc0(a: var MemRegion, requestedSize: int): pointer =
result = rawAlloc(a, requestedSize)
zeroMem(result, requestedSize)

proc rawDealloc(a: var MemRegion, p: pointer) =
when defined(nimTypeNames):
inc(a.deallocCounter)
when defined(heaptracker):
heaptrack_free(p)
#sysAssert(isAllocatedPtr(a, p), "rawDealloc: no allocated pointer")
sysAssert(allocInv(a), "rawDealloc: begin")
var c = pageAddr(p)
Expand Down
258 changes: 42 additions & 216 deletions readme.md
View file
Original file line number Diff line number Diff line change
@@ -1,233 +1,59 @@
# <img src="https://raw.githubusercontent.com/nim-lang/assets/master/Art/logo-crown.png" height="28px"/> Nim
# Heaptrack for Nim

[![Build Status](https://dev.azure.com/nim-lang/Nim/_apis/build/status/nim-lang.Nim?branchName=devel)](https://dev.azure.com/nim-lang/Nim/_build/latest?definitionId=1&branchName=devel)
This branch allows you to use [heaptrack](https://github.com/KDE/heaptrack) on nim programs.

This repository contains the Nim compiler, Nim's stdlib, tools, and documentation.
For more information about Nim, including downloads and documentation for
the latest release, check out [Nim's website][nim-site] or [bleeding edge docs](https://nim-lang.github.io/Nim/).
## Heaptrack
Heaptrack is a neat little tool used for RAM usage debugging, allowing to find leaks, allocation hotspots, etc.

## Community
It's usable in two ways:
- Preload: ex, `heaptrack ls`, will preload `libheaptrack_preload.so` into ls, and record every allocation during the entire program
- Inject: ex, `ls &; heaptrack -p $(pidof ls)`, will attach to the running program using `gdb`, inject `libheaptrack_inject.so`, and record every allocation from this point until the program finishes or `heaptrack` is killed.

[![Join the IRC chat][badge-nim-irc]][nim-irc]
[![Join the Discord server][badge-nim-discord]][nim-discord]
[![Join the Gitter chat][badge-nim-gitter]][nim-gitter]
[![Get help][badge-nim-forum-gethelp]][nim-forum]
[![View Nim posts on Stack Overflow][badge-nim-stackoverflow]][nim-stackoverflow-newest]
[![Follow @nim_lang on Twitter][badge-nim-twitter]][nim-twitter]
Preload is more useful for short-running application, and allow to detect memory leaks as well.

* The [forum][nim-forum] - the best place to ask questions and to discuss Nim.
* [#nim IRC Channel (Libera Chat)][nim-irc] - a place to discuss Nim in real-time.
Also where most development decisions get made.
* [Discord][nim-discord] - an additional place to discuss Nim in real-time. Most
channels there are bridged to IRC.
* [Gitter][nim-gitter] - an additional place to discuss Nim in real-time. There
is a bridge between Gitter and the IRC channel.
* [Telegram][nim-telegram] - an additional place to discuss Nim in real-time. There
is the official Telegram channel. Not bridged to IRC.
* [Stack Overflow][nim-stackoverflow] - a popular Q/A site for programming related
topics that includes posts about Nim.
* [Github Wiki][nim-wiki] - Misc user-contributed content.
Inject is useful to monitor a precise moment of a long running application, and can't detected memory leaks (since the program could `free` the allocated memory after we've detached, or have `alloc`ated before we attached)

## Compiling

The compiler currently officially supports the following platform and
architecture combinations:

* Windows (Windows XP or greater) - x86 and x86_64
* Linux (most, if not all, distributions) - x86, x86_64, ppc64 and armv6l
* Mac OS X (10.04 or greater) - x86, x86_64, ppc64 and Apple Silicon (based on the ARM64 architecture)

More platforms are supported, however, they are not tested regularly and they
may not be as stable as the above-listed platforms.

Compiling the Nim compiler is quite straightforward if you follow these steps:

First, the C source of an older version of the Nim compiler is needed to
bootstrap the latest version because the Nim compiler itself is written in the
Nim programming language. Those C sources are available within the
[``nim-lang/csources_v1``][csources-v1-repo] repository.

Next, to build from source you will need:

* A C compiler such as ``gcc`` 3.x/later or an alternative such as ``clang``,
``Visual C++`` or ``Intel C++``. It is recommended to use ``gcc`` 3.x or
later.
* Either ``git`` or ``wget`` to download the needed source repositories.
* The ``build-essential`` package when using ``gcc`` on Ubuntu (and likely
other distros as well).
* On Windows MinGW 4.3.0 (GCC 8.10) is the minimum recommended compiler.
* Nim hosts a known working MinGW distribution:
* [MinGW32.7z](https://nim-lang.org/download/mingw32.7z)
* [MinGW64.7z](https://nim-lang.org/download/mingw64.7z)

**Windows Note: Cygwin and similar POSIX runtime environments are not supported.**

Then, if you are on a \*nix system or Windows, the following steps should compile
Nim from source using ``gcc``, ``git``, and the ``koch`` build tool.

**Note: The following commands are for the development version of the compiler.**
For most users, installing the latest stable version is enough. Check out
the installation instructions on the website to do so: https://nim-lang.org/install.html.

For package maintainers: see [packaging guidelines](https://nim-lang.github.io/Nim/packaging.html).

First, get Nim from github:
Internally, `heaptrack` works by writing into a file each allocation and dellocation done by the program, along with the stacktrace for each allocation, eg:

```
git clone https://github.com/nim-lang/Nim.git
cd Nim
+ 55c7e2ff17ea 15 #Allocated 21 bytes into address "55c7e2ff17ea"
- 55c7e2ff17ea #Deallocated pointer "55c7e2ff17ea"
```
(note, this is a simplification of the actual format, which is optimized to be machine-readable but not human readable)

Next, run the appropriate build shell script for your platform:

* `build_all.sh` (Linux, Mac)
* `build_all.bat` (Windows)

Finally, once you have finished the build steps (on Windows, Mac, or Linux) you
should add the ``bin`` directory to your PATH.

See also [rebuilding the compiler](doc/intern.rst#rebuilding-the-compiler).

See also [reproducible builds](doc/intern.rst#reproducible-builds).

## Koch

``koch`` is the build tool used to build various parts of Nim and to generate
documentation and the website, among other things. The ``koch`` tool can also
be used to run the Nim test suite.

Assuming that you added Nim's ``bin`` directory to your PATH, you may execute
the tests using ``./koch tests``. The tests take a while to run, but you
can run a subset of tests by specifying a category (for example
``./koch tests cat async``).

For more information on the ``koch`` build tool please see the documentation
within the [doc/koch.rst](doc/koch.rst) file.

## Nimble

``nimble`` is Nim's package manager. To learn more about it, see the
[``nim-lang/nimble``][nimble-repo] repository.
For most of programs, it just wraps `malloc` and `free` into custom function to create this file, but obviously, nim doesn't use `malloc` and `free`, so some trickery is required.

## Contributors

This project exists thanks to all the people who contribute.
<a href="https://github.com/nim-lang/Nim/graphs/contributors"><img src="https://opencollective.com/Nim/contributors.svg?width=890" /></a>

## Contributing

[![Backers on Open Collective](https://opencollective.com/nim/backers/badge.svg)](#backers) [![Sponsors on Open Collective](https://opencollective.com/nim/sponsors/badge.svg)](#sponsors)
[![Setup a bounty via Bountysource][badge-nim-bountysource]][nim-bountysource]
[![Donate Bitcoins][badge-nim-bitcoin]][nim-bitcoin]
[![Open Source Helpers](https://www.codetriage.com/nim-lang/nim/badges/users.svg)](https://www.codetriage.com/nim-lang/nim)

See [detailed contributing guidelines](https://nim-lang.github.io/Nim/contributing.html).
We welcome all contributions to Nim regardless of how small or large
they are. Everything from spelling fixes to new modules to be included in the
standard library are welcomed and appreciated. Before you start contributing,
you should familiarize yourself with the following repository structure:

* ``bin/``, ``build/`` - these directories are empty, but are used when Nim is built.
* ``compiler/`` - the compiler source code. Also includes nimfix, and plugins within
``compiler/nimfix`` and ``compiler/plugins`` respectively.
* ``nimsuggest`` - the nimsuggest tool that previously lived in the [``nim-lang/nimsuggest``][nimsuggest-repo] repository.
* ``config/`` - the configuration for the compiler and documentation generator.
* ``doc/`` - the documentation files in reStructuredText format.
* ``lib/`` - the standard library, including:
* ``pure/`` - modules in the standard library written in pure Nim.
* ``impure/`` - modules in the standard library written in pure Nim with
dependencies written in other languages.
* ``wrappers/`` - modules that wrap dependencies written in other languages.
* ``tests/`` - contains categorized tests for the compiler and standard library.
* ``tools/`` - the tools including ``niminst`` and ``nimweb`` (mostly invoked via
``koch``).
* ``koch.nim`` - the tool used to bootstrap Nim, generate C sources, build the website,
and generate the documentation.

If you are not familiar with making a pull request using GitHub and/or git, please
read [this guide][pull-request-instructions].

Ideally, you should make sure that all tests pass before submitting a pull request.
However, if you are short on time, you can just run the tests specific to your
changes by only running the corresponding categories of tests. Travis CI verifies
that all tests pass before allowing the pull request to be accepted, so only
running specific tests should be harmless.
Integration tests should go in ``tests/untestable``.

If you're looking for ways to contribute, please look at our [issue tracker][nim-issues].
There are always plenty of issues labeled [``Easy``][nim-issues-easy]; these should
be a good starting point for an initial contribution to Nim.

You can also help with the development of Nim by making donations. Donations can be
made using:

* [Open Collective](https://opencollective.com/nim)
* [Bountysource][nim-bountysource]
* [Bitcoin][nim-bitcoin]

If you have any questions feel free to submit a question on the
[Nim forum][nim-forum], or via IRC on [the \#nim channel][nim-irc].


## Backers

Thank you to all our backers! [[Become a backer](https://opencollective.com/Nim#backer)]

<a href="https://opencollective.com/Nim#backers" target="_blank"><img src="https://opencollective.com/Nim/backers.svg?width=890"></a>


## Sponsors

Support this project by becoming a sponsor. Your logo will show up here with a link to your website. [[Become a sponsor](https://opencollective.com/Nim#sponsor)]
## Heaptrack in Nim
This branch simply plugs the `libheaptrack`'s procs into the nim gc, allowing to use heaptrack for nim programs.
```bash
nim c -d:heaptracker test.nim
heaptrack ./test #works as any program
```

<a href="https://opencollective.com/Nim/sponsor/0/website" target="_blank"><img src="https://opencollective.com/Nim/sponsor/0/avatar.svg"></a>
<a href="https://opencollective.com/Nim/sponsor/1/website" target="_blank"><img src="https://opencollective.com/Nim/sponsor/1/avatar.svg"></a>
<a href="https://opencollective.com/Nim/sponsor/2/website" target="_blank"><img src="https://opencollective.com/Nim/sponsor/2/avatar.svg"></a>
<a href="https://opencollective.com/Nim/sponsor/3/website" target="_blank"><img src="https://opencollective.com/Nim/sponsor/3/avatar.svg"></a>
<a href="https://opencollective.com/Nim/sponsor/4/website" target="_blank"><img src="https://opencollective.com/Nim/sponsor/4/avatar.svg"></a>
<a href="https://opencollective.com/Nim/sponsor/5/website" target="_blank"><img src="https://opencollective.com/Nim/sponsor/5/avatar.svg"></a>
<a href="https://opencollective.com/Nim/sponsor/6/website" target="_blank"><img src="https://opencollective.com/Nim/sponsor/6/avatar.svg"></a>
<a href="https://opencollective.com/Nim/sponsor/7/website" target="_blank"><img src="https://opencollective.com/Nim/sponsor/7/avatar.svg"></a>
<a href="https://opencollective.com/Nim/sponsor/8/website" target="_blank"><img src="https://opencollective.com/Nim/sponsor/8/avatar.svg"></a>
<a href="https://opencollective.com/Nim/sponsor/9/website" target="_blank"><img src="https://opencollective.com/Nim/sponsor/9/avatar.svg"></a>
Be careful, to use the `inject` mode, the `libheaptrack_inject.so` must be used instead of `libheaptrack_preload.so` during compilation:
```bash
nim c -d:heaptracker -d:heaptracker_inject test.nim
LD_LIBRARY_PATH=/usr/local/lib/heaptrack ./test &
heaptrack -p $(pidof test)
```
The binary must use the same `.so` as `heaptrack`, so an `inject` binary won't work with `prelude`, and a `prelude` binary won't work with `inject`

You can also see a list of all our sponsors/backers from various payment services on the [sponsors page](https://nim-lang.org/sponsors.html) of our website.
Be also careful, once a binary has been compiled with `-d:heaptracker`, it won't run on a device where `heaptrack` is not installed.
For `inject`, you also need to specify the `LD_LIBRARY_PATH` to locate the `so` files.

## License
The compiler and the standard library are licensed under the MIT license, except
for some modules which explicitly state otherwise. As a result, you may use any
compatible license (essentially any license) for your own programs developed with
Nim. You are explicitly permitted to develop commercial applications using Nim.
## Heaptrack for Nimbus
To build nimbus with heaptrack enabled:
Add to the Makefile the required flags, as [shown here](https://github.com/status-im/nimbus-eth2/commit/90eca6d54309db669f03866fb8d432b739687f70)

Please read the [copying.txt](copying.txt) file for more details.
Change to build Dockerfile to add `apt install heaptrack` and set the `LD_LIBRARY_PATH` properly

Copyright © 2006-2023 Andreas Rumpf, all rights reserved.
Add `gdb` capabilities to the docker-compose.yml
```yaml
cap_add:
- 'SYS_PTRACE'
security_opt:
- 'seccomp:unconfined'
```
[nim-site]: https://nim-lang.org
[nim-forum]: https://forum.nim-lang.org
[nim-issues]: https://github.com/nim-lang/Nim/issues
[nim-issues-easy]: https://github.com/nim-lang/Nim/labels/Easy
[nim-irc]: https://web.libera.chat/#nim
[nim-twitter]: https://twitter.com/nim_lang
[nim-stackoverflow]: https://stackoverflow.com/questions/tagged/nim-lang
[nim-stackoverflow-newest]: https://stackoverflow.com/questions/tagged/nim-lang?sort=newest&pageSize=15
[nim-discord]: https://discord.gg/nim
[nim-gitter]: https://gitter.im/nim-lang/Nim
[nim-telegram]: https://t.me/nim_lang
[nim-bountysource]: https://www.bountysource.com/teams/nim
[nim-bitcoin]: https://blockchain.info/address/1BXfuKM2uvoD6mbx4g5xM3eQhLzkCK77tJ
[nimble-repo]: https://github.com/nim-lang/nimble
[nimsuggest-repo]: https://github.com/nim-lang/nimsuggest
[csources-repo-deprecated]: https://github.com/nim-lang/csources
[csources-v1-repo]: https://github.com/nim-lang/csources_v1
[badge-nim-travisci]: https://img.shields.io/travis/nim-lang/Nim/devel.svg?style=flat-square
[badge-nim-irc]: https://img.shields.io/badge/chat-on_irc-blue.svg?style=flat-square
[badge-nim-discord]: https://img.shields.io/discord/371759389889003530?color=blue&label=discord&logo=discord&logoColor=gold&style=flat-square
[badge-nim-gitter]: https://img.shields.io/badge/chat-on_gitter-blue.svg?style=flat-square
[badge-nim-forum-gethelp]: https://img.shields.io/badge/Forum-get%20help-4eb899.svg?style=flat-square
[badge-nim-twitter]: https://img.shields.io/twitter/follow/nim_lang.svg?style=social
[badge-nim-stackoverflow]: https://img.shields.io/badge/stackoverflow-nim_tag-yellow.svg?style=flat-square
[badge-nim-bountysource]: https://img.shields.io/bountysource/team/nim/activity.svg?style=flat-square
[badge-nim-bitcoin]: https://img.shields.io/badge/bitcoin-1BXfuKM2uvoD6mbx4g5xM3eQhLzkCK77tJ-D69134.svg?style=flat-square
[pull-request-instructions]: https://help.github.com/articles/using-pull-requests/
[nim-wiki]: https://github.com/nim-lang/Nim/wiki
Once everything is setup, you can go into the container, install gdb, and run `heaptrack -p 1` as usual.
When you're happy with your capture, stop heaptrack, retrieve the file on your computer for analysis, and voilà!

0 comments on commit 03fb74d

Please sign in to comment.