[WIP] Work stealing in Nemo (alternative implementation)

[hwarim-hyun]

JIRA: NEMO-466 Dynamic Work Stealing on Nemo for handling skews

Major changes:

• created WorkStealingStatePass and WorkStealingSubSplitPass to enable annotations related to work stealing.
  • WorkStealingStatePass annotates three phases of work stealing process: DEFAULT, SPLIT, and MERGE. These are tagged to the vertex based on the name assigned to beam transform by the user.
  • Sub-tasks are generated in SPLIT phase and merged in MERGE phase.
  • WorkStealingSubSplitPass annotates the number to split the tasks in SPLIT phase.
• Added WorkStealingPolicy, which executes WorkStealingPass and WorkStealingSubSplitPass additionally to DefaultCompositePass.
• Changed notation of the task id and block id to differentiate between work stealing tasks and default tasks.
  • task id
    • previous: [stage id] - [task index] - [task attempt]
    • current: [stage id] - [task index] - [* or sub index for work stealing tasks] - [task attempt]
  • block id
    • previous: [runtime edge id] - [producer task index] - [producer task attempt]
    • current: [runtime edge id] - [producer task index] - [producer task sub index] - [producer task attempt] for blocks generated by work stealing tasks.

Minor changes to note:

• Changed PipelineTranslator to read the user tags given to the transform.
• Extended PlanStateManager and BlockInputReader to manage work stealing tasks and non-split tasks together.

Tests for the changes:

• Added e2e tests to test WorkStealingPolicy. Note that in order to execute WorkStealing policy, the application pipeline should provide appropriate splitting and merging strategy.
• Added unit tests to test WorkStealingCompositePass which encapsulates WorkStealingStatePass and WorkStealingSubSplitPass
• Modified ParentTaskDataFetcherTest and TaskExecutorTest, reflecting the changes in BlockInputReader

Other comments:

• This is the PR for the GSoC2021 project.

Closes #321

[sonarcloud]

Kudos, SonarCloud Quality Gate passed!   

0 Bugs
0 Vulnerabilities
0 Security Hotspots
0 Code Smells

No Coverage information
No Duplication information