[Feature][Executor] Add resume to batch engine

[Jasmin3q]

Description

This pull request adds the resume function for batch run in batch engine.

The most important changes are:

1. Addition of new feature:

   • src/promptflow/promptflow/batch/_batch_engine.py: Added logic to handle resuming from a previous run. In resume run, the previous completed and failed lines are dropped and the other lines are processed. Added _copy_previous_run_result to load previous flow run output from output.jsonl, copy image files to output_dir, and extract aggregation inputs for aggregation node.

2. Addition of new utility functions:

   • src/promptflow/promptflow/_utils/utils.py: Added load_list_from_jsonl function to load a list from a jsonl file and copy_file_except function to copy all files from one directory to another excluding a specific file. [1] [2]

3. Rearrange functions:

   • src/promptflow/promptflow/executor/flow_executor.py: Moved _extract_aggregation_inputs and _extract_aggregation_input functions to utils to be used in more general cases.

4. Changes to test files:

   • src/promptflow/tests/executor/e2etests/test_batch_engine.py: Added e2etest to resume batch run and resume batch run with aggregation.

All Promptflow Contribution checklist:

• The pull request does not introduce [breaking changes].
• CHANGELOG is updated for new features, bug fixes or other significant changes.
• I have read the contribution guidelines.
• Create an issue and link to the pull request to get dedicated review from promptflow team. Learn more: suggested workflow.

General Guidelines and Best Practices

• Title of the pull request is clear and informative.
• There are a small number of commits, each of which have an informative message. This means that previously merged commits do not appear in the history of the PR. For more information on cleaning up the commits in your PR, see this page.

Testing Guidelines

• Pull request includes test coverage for the included changes.

[github-actions]

SDK CLI Global Config Test Result shimin/resume

2 tests   2 ✅  50s ⏱️
1 suites  0 💤
1 files    0 ❌

Results for commit 4f81337.

♻️ This comment has been updated with latest results.

[github-actions]

promptflow SDK CLI Azure E2E Test Result shimin/resume

  4 files    4 suites   3m 20s ⏱️
149 tests 128 ✅ 21 💤 0 ❌
596 runs  512 ✅ 84 💤 0 ❌

Results for commit 4f81337.

♻️ This comment has been updated with latest results.

[github-actions]

SDK CLI Test Result shimin/resume

   12 files     12 suites   41m 5s ⏱️
  424 tests   408 ✅ 16 💤 0 ❌
1 696 runs  1 632 ✅ 64 💤 0 ❌

Results for commit 4f81337.

♻️ This comment has been updated with latest results.

[github-actions]

Executor Unit Test Result shimin/resume

735 tests   735 ✅  45s ⏱️
  1 suites    0 💤
  1 files      0 ❌

Results for commit 4f81337.

♻️ This comment has been updated with latest results.

[github-actions]

Executor E2E Test Result shimin/resume

198 tests   196 ✅  4m 12s ⏱️
  1 suites    2 💤
  1 files      0 ❌

Results for commit 4f81337.

♻️ This comment has been updated with latest results.