Fallback to internal scheduler when index creation failed

[noCharger]

Description

Failback to internal scheduler when index creation failed. This could happen during the feature deployment where the backend role is not updated for the correspond data source

Check List

• Implemented unit tests
• New added source code should include a copyright header
• Commits are signed per the DCO using --signoff

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

[dai-chen]

np: add some comments to explain why this check here
also just to confirm if the scheduler index exists, that means sufficient permission?

[noCharger]

also just to confirm if the scheduler index exists, that means sufficient permission?

• If the check exist throws an exception, it indicates insufficient permissions.
• If the index doesn't exist, it's considered accessible (returns true).

[dai-chen]

Please check CI failure.

[noCharger]

One issue on IT, addressed 5085615

24/10/31 22:30:57 ERROR MicroBatchExecution: Query flint_spark_catalog_default_mv_test_metrics [id = 3dab7d47-2d38-4951-bda0-f03e39e64dc7, runId = 7152280d-9848-41cf-b7e1-14c1f9d3ae1a] terminated with error
java.io.IOException: Cannot run program "chmod": error=24, Too many open files

[opensearch-trigger-bot]

The backport to 0.5 failed:

The process '/usr/bin/git' failed with exit code 128

To backport manually, run these commands in your terminal:

# Navigate to the root of your repository
cd $(git rev-parse --show-toplevel)
# Fetch latest updates from GitHub
git fetch
# Create a new working tree
git worktree add ../.worktrees/opensearch-spark/backport-0.5 0.5
# Navigate to the new working tree
pushd ../.worktrees/opensearch-spark/backport-0.5
# Create a new branch
git switch --create backport/backport-850-to-0.5
# Cherry-pick the merged commit of this pull request and resolve the conflicts
git cherry-pick -x --mainline 1 faae818ae028ac43eb79ff291d8744e6c590eb82
# Push it to GitHub
git push --set-upstream origin backport/backport-850-to-0.5
# Go back to the original working tree
popd
# Delete the working tree
git worktree remove ../.worktrees/opensearch-spark/backport-0.5

Then, create a pull request where the base branch is 0.5 and the compare/head branch is backport/backport-850-to-0.5.

[opensearch-trigger-bot]

The backport to 0.5-nexus failed:

The process '/usr/bin/git' failed with exit code 1

To backport manually, run these commands in your terminal:

# Navigate to the root of your repository
cd $(git rev-parse --show-toplevel)
# Fetch latest updates from GitHub
git fetch
# Create a new working tree
git worktree add ../.worktrees/opensearch-spark/backport-0.5-nexus 0.5-nexus
# Navigate to the new working tree
pushd ../.worktrees/opensearch-spark/backport-0.5-nexus
# Create a new branch
git switch --create backport/backport-850-to-0.5-nexus
# Cherry-pick the merged commit of this pull request and resolve the conflicts
git cherry-pick -x --mainline 1 faae818ae028ac43eb79ff291d8744e6c590eb82
# Push it to GitHub
git push --set-upstream origin backport/backport-850-to-0.5-nexus
# Go back to the original working tree
popd
# Delete the working tree
git worktree remove ../.worktrees/opensearch-spark/backport-0.5-nexus

Then, create a pull request where the base branch is 0.5-nexus and the compare/head branch is backport/backport-850-to-0.5-nexus.