Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Increase the speed of incremental compilation #34616

Merged

Conversation

zhwesky2010
Copy link
Contributor

@zhwesky2010 zhwesky2010 commented Aug 4, 2021

PR types

Performance optimization

PR changes

Others

Describe

提速目前框架的增量编译,原因如下:

框架中有不少 自定义command 的写法,这种写法会导致增量编译时,自定义command 默认会re-run,降低了二次增量编译速度。

infoflow 2021-08-04 18-24-23

从实验中可以看到,当0改动 二次增量编译时,触发了 290个重新re-run(占总1200个target的约1/4)。

因此将这些 自定义command 写为OUTPUT的文件形式,在二次增量编译时,则会判定这些OUTPUT的文件已经存在,不会 re-run 自定义command。可以参考 https://lowlevelbits.org/bottom-up-cmake-introduction/ 的分析。

infoflow 2021-08-06 18-30-56

该PR只修复了部分增量编译问题,剩余未修复部分需要分析其他原因并优化。

@zhwesky2010 zhwesky2010 force-pushed the improve_incremental_compile branch 2 times, most recently from 3ca8f88 to 563d6c3 Compare August 5, 2021 02:52
@zhwesky2010 zhwesky2010 force-pushed the improve_incremental_compile branch 2 times, most recently from aaa7611 to bfb2ace Compare August 5, 2021 06:38
@zhwesky2010 zhwesky2010 closed this Aug 5, 2021
@zhwesky2010 zhwesky2010 reopened this Aug 5, 2021
Copy link
Contributor

@wanghuancoder wanghuancoder left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@zhwesky2010 zhwesky2010 merged commit aab4d6e into PaddlePaddle:develop Aug 9, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants