commit-graph: avoid leaking topo_levels slab in write_commit_graph()

[ahunt]

write_commit_graph initialises topo_levels using init_topo_level_slab(),
next it calls compute_topological_levels() which can cause the slab to
grow, we therefore need to clear the slab again using
clear_topo_level_slab() when we're done.

First introduced in 72a2bfc - which
is currently only in master and not on maint.

LeakSanitizer output:

==1026==ERROR: LeakSanitizer: detected memory leaks

Direct leak of 8 byte(s) in 1 object(s) allocated from:
#0 0x498ae9 in realloc /src/llvm-project/compiler-rt/lib/asan/asan_malloc_linux.cpp:164:3
#1 0xafbed8 in xrealloc /src/git/wrapper.c:126:8
#2 0x7966d1 in topo_level_slab_at_peek /src/git/commit-graph.c:71:1
#3 0x7965e0 in topo_level_slab_at /src/git/commit-graph.c:71:1
#4 0x78fbf5 in compute_topological_levels /src/git/commit-graph.c:1472:12
#5 0x78c5c3 in write_commit_graph /src/git/commit-graph.c:2456:2
#6 0x535c5f in graph_write /src/git/builtin/commit-graph.c:299:6
#7 0x5350ca in cmd_commit_graph /src/git/builtin/commit-graph.c:337:11
#8 0x4cddb1 in run_builtin /src/git/git.c:453:11
#9 0x4cabe2 in handle_builtin /src/git/git.c:704:3
#10 0x4cd084 in run_argv /src/git/git.c:771:4
#11 0x4ca424 in cmd_main /src/git/git.c:902:19
#12 0x707fb6 in main /src/git/common-main.c:52:11
#13 0x7fee4249383f in __libc_start_main (/lib/x86_64-linux-gnu/libc.so.6+0x2083f)

Indirect leak of 524256 byte(s) in 1 object(s) allocated from:
#0 0x498942 in calloc /src/llvm-project/compiler-rt/lib/asan/asan_malloc_linux.cpp:154:3
#1 0xafc088 in xcalloc /src/git/wrapper.c:140:8
#2 0x796870 in topo_level_slab_at_peek /src/git/commit-graph.c:71:1
#3 0x7965e0 in topo_level_slab_at /src/git/commit-graph.c:71:1
#4 0x78fbf5 in compute_topological_levels /src/git/commit-graph.c:1472:12
#5 0x78c5c3 in write_commit_graph /src/git/commit-graph.c:2456:2
#6 0x535c5f in graph_write /src/git/builtin/commit-graph.c:299:6
#7 0x5350ca in cmd_commit_graph /src/git/builtin/commit-graph.c:337:11
#8 0x4cddb1 in run_builtin /src/git/git.c:453:11
#9 0x4cabe2 in handle_builtin /src/git/git.c:704:3
#10 0x4cd084 in run_argv /src/git/git.c:771:4
#11 0x4ca424 in cmd_main /src/git/git.c:902:19
#12 0x707fb6 in main /src/git/common-main.c:52:11
#13 0x7fee4249383f in __libc_start_main (/lib/x86_64-linux-gnu/libc.so.6+0x2083f)

SUMMARY: AddressSanitizer: 524264 byte(s) leaked in 2 allocation(s).

Thanks for taking the time to contribute to Git! Please be advised that the
Git community does not use github.com for their contributions. Instead, we use
a mailing list (git@vger.kernel.org) for code submissions, code reviews, and
bug reports. Nevertheless, you can use GitGitGadget (https://gitgitgadget.github.io/)
to conveniently send your Pull Requests commits to our mailing list.

Please read the "guidelines for contributing" linked above!

cc: Derrick Stolee stolee@gmail.com

[gitgitgadget]

Welcome to GitGitGadget

Hi @ahunt, and welcome to GitGitGadget, the GitHub App to send patch series to the Git mailing list from GitHub Pull Requests.

Please make sure that your Pull Request has a good description, as it will be used as cover letter.

Also, it is a good idea to review the commit messages one last time, as the Git project expects them in a quite specific form:

• the lines should not exceed 76 columns,
• the first line should be like a header and typically start with a prefix like "tests:" or "commit:", and
• the commit messages' body should be describing the "why?" of the change.
• Finally, the commit messages should end in a Signed-off-by: line matching the commits' author.

It is in general a good idea to await the automated test ("Checks") in this Pull Request before contributing the patches, e.g. to avoid trivial issues such as unportable code.

Contributing the patches

Before you can contribute the patches, your GitHub username needs to be added to the list of permitted users. Any already-permitted user can do that, by adding a comment to your PR of the form /allow. A good way to find other contributors is to locate recent pull requests where someone has been /allowed:

• Search: is:pr is:open "/allow"

Both the person who commented /allow and the PR author are able to /allow you.

An alternative is the channel #git-devel on the FreeNode IRC network:

<newcontributor> I've just created my first PR, could someone please /allow me? https://github.com/gitgitgadget/git/pull/12345
<veteran> newcontributor: it is done
<newcontributor> thanks!

Once on the list of permitted usernames, you can contribute the patches to the Git mailing list by adding a PR comment /submit.

If you want to see what email(s) would be sent for a /submit request, add a PR comment /preview to have the email(s) sent to you. You must have a public GitHub email address for this.

After you submit, GitGitGadget will respond with another comment that contains the link to the cover letter mail in the Git mailing list archive. Please make sure to monitor the discussion in that thread and to address comments and suggestions (while the comments and suggestions will be mirrored into the PR by GitGitGadget, you will still want to reply via mail).

If you do not want to subscribe to the Git mailing list just to be able to respond to a mail, you can download the mbox from the Git mailing list archive (click the (raw) link), then import it into your mail program. If you use GMail, you can do this via:

curl -g --user "<EMailAddress>:<Password>" \
    --url "imaps://imap.gmail.com/INBOX" -T /path/to/raw.txt

To iterate on your change, i.e. send a revised patch or patch series, you will first want to (force-)push to the same branch. You probably also want to modify your Pull Request description (or title). It is a good idea to summarize the revision by adding something like this to the cover letter (read: by editing the first comment on the PR, i.e. the PR description):

Changes since v1:
- Fixed a typo in the commit message (found by ...)
- Added a code comment to ... as suggested by ...
...

To send a new iteration, just add another PR comment with the contents: /submit.

Need help?

New contributors who want advice are encouraged to join git-mentoring@googlegroups.com, where volunteers who regularly contribute to Git are willing to answer newbie questions, give advice, or otherwise provide mentoring to interested contributors. You must join in order to post or view messages, but anyone can join.

You may also be able to find help in real time in the developer IRC channel, #git-devel on Freenode. Remember that IRC does not support offline messaging, so if you send someone a private message and log out, they cannot respond to you. The scrollback of #git-devel is archived, though.

[gitgitgadget]

There is an issue in commit a5c1654:
Commit not signed off

[derrickstolee]

/allow

[derrickstolee]

Thanks for finding this!

[gitgitgadget]

User ahunt is now allowed to use GitGitGadget.

WARNING: ahunt has no public email address set on GitHub

[ahunt]

/preview

[gitgitgadget]

Error: Could not determine public email of ahunt

[ahunt]

/preview

[gitgitgadget]

Preview email sent as pull.881.git.1613765496607.gitgitgadget@gmail.com

[ahunt]

/submit

[gitgitgadget]

Submitted as pull.881.git.1613765590412.gitgitgadget@gmail.com

To fetch this version into FETCH_HEAD:

git fetch https://github.com/gitgitgadget/git pr-881/ahunt/commit-graph-leak-v1

To fetch this version to local tag pr-881/ahunt/commit-graph-leak-v1:

git fetch --no-tags https://github.com/gitgitgadget/git tag pr-881/ahunt/commit-graph-leak-v1

[derrickstolee]

@ahunt: I should have warned you, and maybe your /preview email would have made this clear: your PR Description becomes a "comment" block between your commit message and the patch diff. This means your submission has some unnecessary noise. Just remove it before a v2, if necessary.

[gitgitgadget]

On the Git mailing list, Junio C Hamano wrote (reply to this):

"Andrzej Hunt via GitGitGadget" <gitgitgadget@gmail.com> writes:

> From: Andrzej Hunt <ajrhunt@google.com>
>
> write_commit_graph initialises topo_levels using init_topo_level_slab(),
> next it calls compute_topological_levels() which can cause the slab to
> grow, we therefore need to clear the slab again using
> clear_topo_level_slab() when we're done.
>
> First introduced in 72a2bfcaf01860ce8dd6921490d903dc0ad59c89 - which
> is currently only in master and not on maint.

Thanks.  

Forwarding to those who were involved in the said commit for
insights.



> LeakSanitizer output:
>
> ==1026==ERROR: LeakSanitizer: detected memory leaks
>
> Direct leak of 8 byte(s) in 1 object(s) allocated from:
>     #0 0x498ae9 in realloc /src/llvm-project/compiler-rt/lib/asan/asan_malloc_linux.cpp:164:3
>     #1 0xafbed8 in xrealloc /src/git/wrapper.c:126:8
>     #2 0x7966d1 in topo_level_slab_at_peek /src/git/commit-graph.c:71:1
>     #3 0x7965e0 in topo_level_slab_at /src/git/commit-graph.c:71:1
>     #4 0x78fbf5 in compute_topological_levels /src/git/commit-graph.c:1472:12
>     #5 0x78c5c3 in write_commit_graph /src/git/commit-graph.c:2456:2
>     #6 0x535c5f in graph_write /src/git/builtin/commit-graph.c:299:6
>     #7 0x5350ca in cmd_commit_graph /src/git/builtin/commit-graph.c:337:11
>     #8 0x4cddb1 in run_builtin /src/git/git.c:453:11
>     #9 0x4cabe2 in handle_builtin /src/git/git.c:704:3
>     #10 0x4cd084 in run_argv /src/git/git.c:771:4
>     #11 0x4ca424 in cmd_main /src/git/git.c:902:19
>     #12 0x707fb6 in main /src/git/common-main.c:52:11
>     #13 0x7fee4249383f in __libc_start_main (/lib/x86_64-linux-gnu/libc.so.6+0x2083f)
>
> Indirect leak of 524256 byte(s) in 1 object(s) allocated from:
>     #0 0x498942 in calloc /src/llvm-project/compiler-rt/lib/asan/asan_malloc_linux.cpp:154:3
>     #1 0xafc088 in xcalloc /src/git/wrapper.c:140:8
>     #2 0x796870 in topo_level_slab_at_peek /src/git/commit-graph.c:71:1
>     #3 0x7965e0 in topo_level_slab_at /src/git/commit-graph.c:71:1
>     #4 0x78fbf5 in compute_topological_levels /src/git/commit-graph.c:1472:12
>     #5 0x78c5c3 in write_commit_graph /src/git/commit-graph.c:2456:2
>     #6 0x535c5f in graph_write /src/git/builtin/commit-graph.c:299:6
>     #7 0x5350ca in cmd_commit_graph /src/git/builtin/commit-graph.c:337:11
>     #8 0x4cddb1 in run_builtin /src/git/git.c:453:11
>     #9 0x4cabe2 in handle_builtin /src/git/git.c:704:3
>     #10 0x4cd084 in run_argv /src/git/git.c:771:4
>     #11 0x4ca424 in cmd_main /src/git/git.c:902:19
>     #12 0x707fb6 in main /src/git/common-main.c:52:11
>     #13 0x7fee4249383f in __libc_start_main (/lib/x86_64-linux-gnu/libc.so.6+0x2083f)
>
> SUMMARY: AddressSanitizer: 524264 byte(s) leaked in 2 allocation(s).
>
> Signed-off-by: Andrzej Hunt <ajrhunt@google.com>
> ---
>     commit-graph: avoid leaking topo_levels slab in write_commit_graph()
>     
>     write_commit_graph initialises topo_levels using init_topo_level_slab(),
>     next it calls compute_topological_levels() which can cause the slab to
>     grow, we therefore need to clear the slab again using
>     clear_topo_level_slab() when we're done.
>     
>     First introduced in 72a2bfcaf01860ce8dd6921490d903dc0ad59c89 - which is
>     currently only in master and not on maint.
>     
>     LeakSanitizer output:
>     
>     ==1026==ERROR: LeakSanitizer: detected memory leaks
>     
>     Direct leak of 8 byte(s) in 1 object(s) allocated from: #0 0x498ae9 in
>     realloc
>     /src/llvm-project/compiler-rt/lib/asan/asan_malloc_linux.cpp:164:3 #1
>     0xafbed8 in xrealloc /src/git/wrapper.c:126:8 #2 0x7966d1 in
>     topo_level_slab_at_peek /src/git/commit-graph.c:71:1 #3 0x7965e0 in
>     topo_level_slab_at /src/git/commit-graph.c:71:1 #4 0x78fbf5 in
>     compute_topological_levels /src/git/commit-graph.c:1472:12 #5 0x78c5c3
>     in write_commit_graph /src/git/commit-graph.c:2456:2 #6 0x535c5f in
>     graph_write /src/git/builtin/commit-graph.c:299:6 #7 0x5350ca in
>     cmd_commit_graph /src/git/builtin/commit-graph.c:337:11 #8 0x4cddb1 in
>     run_builtin /src/git/git.c:453:11 #9 0x4cabe2 in handle_builtin
>     /src/git/git.c:704:3 #10 0x4cd084 in run_argv /src/git/git.c:771:4 #11
>     0x4ca424 in cmd_main /src/git/git.c:902:19 #12 0x707fb6 in main
>     /src/git/common-main.c:52:11 #13 0x7fee4249383f in __libc_start_main
>     (/lib/x86_64-linux-gnu/libc.so.6+0x2083f)
>     
>     Indirect leak of 524256 byte(s) in 1 object(s) allocated from: #0
>     0x498942 in calloc
>     /src/llvm-project/compiler-rt/lib/asan/asan_malloc_linux.cpp:154:3 #1
>     0xafc088 in xcalloc /src/git/wrapper.c:140:8 #2 0x796870 in
>     topo_level_slab_at_peek /src/git/commit-graph.c:71:1 #3 0x7965e0 in
>     topo_level_slab_at /src/git/commit-graph.c:71:1 #4 0x78fbf5 in
>     compute_topological_levels /src/git/commit-graph.c:1472:12 #5 0x78c5c3
>     in write_commit_graph /src/git/commit-graph.c:2456:2 #6 0x535c5f in
>     graph_write /src/git/builtin/commit-graph.c:299:6 #7 0x5350ca in
>     cmd_commit_graph /src/git/builtin/commit-graph.c:337:11 #8 0x4cddb1 in
>     run_builtin /src/git/git.c:453:11 #9 0x4cabe2 in handle_builtin
>     /src/git/git.c:704:3 #10 0x4cd084 in run_argv /src/git/git.c:771:4 #11
>     0x4ca424 in cmd_main /src/git/git.c:902:19 #12 0x707fb6 in main
>     /src/git/common-main.c:52:11 #13 0x7fee4249383f in __libc_start_main
>     (/lib/x86_64-linux-gnu/libc.so.6+0x2083f)
>     
>     SUMMARY: AddressSanitizer: 524264 byte(s) leaked in 2 allocation(s).
>
> Published-As: https://github.com/gitgitgadget/git/releases/tag/pr-881%2Fahunt%2Fcommit-graph-leak-v1
> Fetch-It-Via: git fetch https://github.com/gitgitgadget/git pr-881/ahunt/commit-graph-leak-v1
> Pull-Request: https://github.com/gitgitgadget/git/pull/881
>
>  commit-graph.c | 1 +
>  1 file changed, 1 insertion(+)
>
> diff --git a/commit-graph.c b/commit-graph.c
> index ed31843fa522..9529ec552139 100644
> --- a/commit-graph.c
> +++ b/commit-graph.c
> @@ -2471,6 +2471,7 @@ int write_commit_graph(struct object_directory *odb,
>  	free(ctx->graph_name);
>  	free(ctx->commits.list);
>  	oid_array_clear(&ctx->oids);
> +	clear_topo_level_slab(&topo_levels);
>  
>  	if (ctx->commit_graph_filenames_after) {
>  		for (i = 0; i < ctx->num_commit_graphs_after; i++) {
>
> base-commit: 2283e0e9af55689215afa39c03beb2315ce18e83

[ahunt]

your PR Description becomes a "comment" block between your commit message and the patch diff.

Ughh, sorry - and thanks for the hint! IIRC GitGitGadget did actually explain this, and I even saw it in the preview - but my brain didn't put 2 and 2 together until now.

[gitgitgadget]

On the Git mailing list, Derrick Stolee wrote (reply to this):

On 2/19/2021 10:36 PM, Junio C Hamano wrote:
> "Andrzej Hunt via GitGitGadget" <gitgitgadget@gmail.com> writes:
> 
>> From: Andrzej Hunt <ajrhunt@google.com>
>>
>> write_commit_graph initialises topo_levels using init_topo_level_slab(),
>> next it calls compute_topological_levels() which can cause the slab to
>> grow, we therefore need to clear the slab again using
>> clear_topo_level_slab() when we're done.
>>
>> First introduced in 72a2bfcaf01860ce8dd6921490d903dc0ad59c89 - which
>> is currently only in master and not on maint.
> 
> Thanks.  
> 
> Forwarding to those who were involved in the said commit for
> insights.

>> index ed31843fa522..9529ec552139 100644
>> --- a/commit-graph.c
>> +++ b/commit-graph.c
>> @@ -2471,6 +2471,7 @@ int write_commit_graph(struct object_directory *odb,
>>  	free(ctx->graph_name);
>>  	free(ctx->commits.list);
>>  	oid_array_clear(&ctx->oids);
>> +	clear_topo_level_slab(&topo_levels);

This change looks like a sane change to me. It definitely fixes a leak.
The leak "wasn't hurting anybody" because write_commit_graph() is only
called at most once per process, and the process closes itself out
shortly after. Still, it's good to have good memory hygiene here.

Thanks,
-Stolee

[gitgitgadget]

User Derrick Stolee <stolee@gmail.com> has been added to the cc: list.

[gitgitgadget]

On the Git mailing list, Andrzej Hunt wrote (reply to this):

On 22/02/2021 15:15, Derrick Stolee wrote:
> This change looks like a sane change to me. It definitely fixes a leak.
> The leak "wasn't hurting anybody" because write_commit_graph() is only
> called at most once per process, and the process closes itself out
> shortly after. Still, it's good to have good memory hygiene here.

Good to know - thank you! As I become more familiar with git, I'm 
beginning to realise that most leaks are unlikely to be much importance 
(even though I personally err on the side of fixing any and all issues).


One thing I forgot to mention: in this specific case the leak was 
causing a build failure when trying to build git's fuzzers within 
oss-fuzz locally*. Specifically the following command would fail (see 
also fuzz failure reproduction instructions which describe the setup [1]).

  $ python infra/helper.py build_fuzzers --sanitizer address git

As far as I can tell the issue is that: a copy of git built with ASAN is 
used to produce the fuzzing corpus as part of the git-specific build 
script [2] - the leak warning causes the script to fail. (It's possible 
to argue that the build script should disable ASAN's leak checking when 
running git, via detect_leaks=0 to reduce the risk of such breakage - I 
may try to suggest such a change to oss-fuzz.)

ATB,
   Andrzej


* Given that oss-fuzz is building via docker, I would intuitively 
suspect that the same issue occurs in automation - I'm not sure how to 
verify this myself.

[1] 
https://google.github.io/oss-fuzz/advanced-topics/reproducing/#building-using-docker
[2] 
https://github.com/google/oss-fuzz/blob/1b0115eefd70491376cf3cb6f88e49632c78ee18/projects/git/build.sh#L37

[gitgitgadget]

This branch is now known as ah/commit-graph-leakplug.

[gitgitgadget]

This patch series was integrated into seen via git@ac4b453.

[gitgitgadget]

This patch series was integrated into next via git@1330b45.

[gitgitgadget]

This patch series was integrated into seen via git@94a07a2.

[gitgitgadget]

This patch series was integrated into seen via git@412cc8d.

[gitgitgadget]

This patch series was integrated into seen via git@3c8e6dd.

[gitgitgadget]

This patch series was integrated into next via git@3c8e6dd.

[gitgitgadget]

This patch series was integrated into master via git@3c8e6dd.

[gitgitgadget]

Closed via 3c8e6dd.