Kubernetes orchestrator caching efficiency + reliability

[schustmi]

Describe changes

• Environment variable that allows configuring how many workers the DAG runner uses
• Fetch hydrated cache candidate to avoid extra request
• Pass step runs cache to avoid unnecessary requests
• If step pod failed during startup, publish a failed step run

Pre-requisites

Please ensure you have done the following:

• I have read the CONTRIBUTING.md document.
• I have added tests to cover my changes.
• I have based my new branch on develop and the open PR is targeting develop. If your branch wasn't based on develop read Contribution guide on rebasing branch to develop.
• IMPORTANT: I made sure that my changes are reflected properly in the following resources:
  • ZenML Docs
  • Dashboard: Needs to be communicated to the frontend team.
  • Templates: Might need adjustments (that are not reflected in the template tests) in case of non-breaking changes and deprecations.
  • Projects: Depending on the version dependencies, different projects might get affected.

Types of changes

• Bug fix (non-breaking change which fixes an issue)
• New feature (non-breaking change which adds functionality)
• Breaking change (fix or feature that would cause existing functionality to change)
• Other (add details above)

[github-actions]

⚠️ This PR has been inactive for 2 weeks and has been marked as stale.
Timeline:

• Week 2 (now): First reminder - PR marked as stale
• Week 4: PR will be automatically closed if no activity
  Please update this PR or leave a comment to keep it active. Any activity will reset the timer and remove the stale label.

[github-actions]

LLM Finetuning template updates in examples/llm_finetuning have been pushed.

[github-actions]

Classification template updates in examples/mlops_starter have been pushed.

[github-actions]

E2E template updates in examples/e2e have been pushed.