Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

support jemalloc profiler #2737

Merged
merged 17 commits into from
Sep 23, 2024
Merged

Conversation

GreateCode
Copy link
Contributor

@GreateCode GreateCode commented Aug 15, 2024

What problem does this PR solve?

Issue Number: resolve #1962

Problem Summary: 支持jemalloc profiler

What is changed and the side effects?

Changed: 增加jemalloc mallctl控制profile。

用法:

  1. 编译jemalloc需--enable-prof
  2. 启动进程:MALLOC_CONF="prof:true" LD_PRELOAD=/xxx/lib/libjemalloc.so ./bin/test_server
  3. 可使用FLAGS_je_prof_active在进程running期间开启/关闭采样;
  4. 改变FLAGS_je_prof_dump会dump heap文件;
  5. FLAGS_je_prof_reset可清理已采样数据和重置prof选项,并且动态设置采样率,默认2^19B;
  6. 若要做memory leak,需MALLOC_CONF="prof:true,prof_leak:true,prof_final:true",进程退出时生成heap文件

注:每次dump的都是从采样至今的所有数据,除非reset,接来下dump的是从reset至今的所有数据,方便做diff。

样例:
image
image
image

Side effects:

  • Performance effects(性能影响):

  • Breaking backward compatibility(向后兼容性):


Check List:

  • Please make sure your changes are compilable(请确保你的更改可以通过编译).
  • When providing us with a new feature, it is best to add related tests(如果你向我们增加一个新的功能, 请添加相关测试).
  • Please follow Contributor Covenant Code of Conduct.(请遵循贡献者准则).

@wwbmmm
Copy link
Contributor

wwbmmm commented Aug 19, 2024

感谢贡献!
能否顺便更新一下使用文档?

@GreateCode
Copy link
Contributor Author

好的,我近期更新下

@GreateCode
Copy link
Contributor Author

感谢贡献! 能否顺便更新一下使用文档?

done

@yanglimingcn
Copy link
Contributor

bazel编译,也是动态链接libjemalloc.so库吗

@GreateCode
Copy link
Contributor Author

bazel编译,也是动态链接libjemalloc.so库吗

不强制,和编译方式无关,只需profiler时能找到mallctl即可。
所以静态链接jemalloc时,不用LD_PRELOAD=/xxx/lib/libjemalloc.so,直接MALLOC_CONF="prof:true" bin/xxx。

@yanglimingcn
Copy link
Contributor

LGTM

@GreateCode
Copy link
Contributor Author

GreateCode commented Aug 22, 2024

补充下,用静态方式链接jemalloc,进程加载其他so库比如dlopen("/xxx/libjvm.so", xx),so里函数申请的内存,可能不会被profiler采样到。

@wwbmmm
Copy link
Contributor

wwbmmm commented Aug 26, 2024

LGTM

Copy link
Contributor

@chenBright chenBright left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

可以集成到内置服务heap页面吗?

src/brpc/details/jemalloc_profiler.cpp Outdated Show resolved Hide resolved
@GreateCode
Copy link
Contributor Author

可以集成到内置服务heap页面吗?

尝试了,放弃了。。
你帮忙改下?

@GreateCode GreateCode requested a review from chenBright August 27, 2024 08:09
Copy link
Contributor

@chenBright chenBright left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

可以集成到内置服务heap页面吗?

尝试了,放弃了。。 你帮忙改下?

那后面再看看怎么支持吧

@GreateCode
Copy link
Contributor Author

@wwbmmm @yanglimingcn @chenBright 接口新增了jemalloc stats信息,帮助查看memory细节如 各arena,metadata等,麻烦看下还有问题吗?谢谢~
image

@chenBright
Copy link
Contributor

MemoryService也支持一下jemalloc stats?

@GreateCode
Copy link
Contributor Author

MemoryService也支持一下jemalloc stats?

done

@yanglimingcn
Copy link
Contributor

这个PR还有别的问题吗 @chenBright

Copy link
Contributor

@chenBright chenBright left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@yanglimingcn yanglimingcn merged commit 36353ef into apache:master Sep 23, 2024
20 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

brpc support memory profiling using jemalloc
4 participants