[Offline Nodes] Adds new library for offline tasks

[linuxpi]

Description

• Adds a new library offline-tasks hosting all the base classes and interfaces to be used for Background Tasks

Related Issues

Resolves #13575

Check List

• New functionality includes testing.
  • All tests pass
• New functionality has been documented.
  • New functionality has javadoc added
• Failing checks are inspected and point to the corresponding known issue(s) (See: Troubleshooting Failing Builds)
• Commits are signed per the DCO using --signoff
• Commit changes are listed out in CHANGELOG.md file (See: Changelog)
• Public documentation issue/PR created

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

[github-actions]

❌ Gradle check result for a444dd6: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

[github-actions]

❌ Gradle check result for bf65b92: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

[github-actions]

❌ Gradle check result for 68ae28f: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

[github-actions]

✅ Gradle check result for 6470881: SUCCESS

[github-actions]

✅ Gradle check result for 3a84e95: SUCCESS

[github-actions]

❌ Gradle check result for 0ed8334: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

[github-actions]

❕ Gradle check result for a1e3bc3: UNSTABLE

Please review all flaky tests that succeeded after retry and create an issue if one does not already exist to track the flaky failure.

[dblock]

Naively, this feels like a subset of capabilities of job-scheduler (https://github.com/opensearch-project/job-scheduler). Are we duplicating the functionality?

[github-actions]

❌ Gradle check result for 21abf2a: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

[linuxpi]

Naively, this feels like a subset of capabilities of job-scheduler (https://github.com/opensearch-project/job-scheduler). Are we duplicating the functionality?

Thanks for the feedback @dblock . The idea behind this PR is to add generic interfaces for Task Manager and a Task Definition structure which can be leveraged to execute any background Task. The concrete implementation(TaskManager) of the interfaces will come in subsequent PRs.

job-scheduler today helps with scheduling periodic jobs and defines a job definition in JobDetail.java which is tied to job-scheduler and the way it operates and assumes index as a store for the jobs while for background tasks we might not want to take a dependency on the indexes on data nodes.

When concrete implementation for the interfaces comes up, job-scheduler can/will be one of the options which can be considered to support the Task Manager implementation.

[github-actions]

✅ Gradle check result for 21abf2a: SUCCESS

[dblock]

When concrete implementation for the interfaces comes up, job-scheduler can/will be one of the options which can be considered to support the Task Manager implementation.

Makes sense. I'm just asking to think about this a little more. From my birds eye view I think this feature/interface should be behind a feature flag until we come up with concrete implementations, and resolve any differences with job scheduler by either evolving the interface, or changing the existing job scheduler. Logically I'd want to be able to queue and schedule offline tasks of all kinds, some backed by an index, others backed by a different mechanism. This is because we've discussed moving job scheduler to core as "the" way of doing offline/remote/delayed tasks, and the proposal here seems to define "the way".

The other option is that all this code ships as a separate plugin, then it "competes" somewhat with job scheduler.

[github-actions]

❌ Gradle check result for cf26f01: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

[linuxpi]

From my birds eye view I think this feature/interface should be behind a feature flag until we come up with concrete implementations, and resolve any differences with job scheduler by either evolving the interface, or changing the existing job scheduler.

Yes, makes sense, although the interfaces are experimental, I have added a feature flag to guard the loading of the implementations in Node.java. Any future implementation can also use the same flag.

[github-actions]

✅ Gradle check result for 9a87245: SUCCESS