Reduce long duration for the exit -6 re-run process.
#9400
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
Thanks for your contribution! |
waliwali777
force-pushed
the
reduce_ci_exit_250
branch
from
86de410 to
69609bc
Compare
Codecov ReportAll modified and coverable lines are covered by tests ✅
Additional details and impacted files@@ Coverage Diff @@
## develop #9400 +/- ##
===========================================
- Coverage 53.01% 52.97% -0.04%
===========================================
Files 678 676 -2
Lines 108787 107838 -949
===========================================
- Hits 57668 57124 -544
+ Misses 51119 50714 -405 ☔ View full report in Codecov by Sentry. 🚨 Try these New Features:
|
liym27
reviewed
Nov 11, 2024
| @@ -117,6 +117,8 @@ function llm_qwen_case_list_auto() { | |||
|
|
|||
| function llama_dygraph_auto_bs8_fp32_DP2() { | |||
| echo "=========== $FUNCNAME run begin ===========" | |||
| export_env | |||
| cd ${llama_case_path} | |||
There was a problem hiding this comment.
为什么每个函数里要再调用一次 export_env 和 cd ${llama_case_path}，建议在run_ci.sh 中执行一次hook即可
| llama_align_dy2st_fthenb_and_vpp_auto_bs2_fp32_DP1-MP1-PP4 | ||
| llama_align_dygraph_dy2st_pir_auto_pp_bs2_bf16_DP1-MP1-PP4 | ||
| ) | ||
| restore_func $fun_list |
There was a problem hiding this comment.
建议将 restore_llama_case_list_auto_func 与 llama_case_list_auto 合并为一个，仅维护一个列表。
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
PR types
Others
PR changes
Others
Description
线上自动并行
PaddleNLP-CI-gpt-3CI 会在测试执行结束后出现随机挂exit -6的现象。原本的处理方法是结束CI，并提示用户通过re-run的方式来解决。但随机挂概率较高，导致用户负担较重。因此，本PR主要是通过自动re-run的方式解决该问题：test_case需要通过子进程方式将test函数集合写入function.txt中，父进程读取function.txt中的函数名称，再通过子进程的方式进行调用执行。因此，父进程可以捕获每个测试的ExitCodeExitCode = 250则代表出现随机挂exit -6，该测试会自动re-run一次，如果问题复现，则日志中只提示用户该测试多次出现随机挂问题，不再中断CI，继续执行剩余测试test_case执行完后，统计当前case的总测试数目、成功测试数目、运行失败测试数目、精度校验失败测试数目、随机挂测试数目