Skip to content

Implement retry policy and enhance errored state handling #326

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
NiveditJain merged 21 commits into from
Aug 31, 2025
Merged

Implement retry policy and enhance errored state handling #326

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
21 commits
Select commit Hold shift + click to select a range
8f7bcc1
Implement retry policy and enhance errored state handling
NiveditJain Aug 31, 2025
a00b97c
Merge remote-tracking branch 'origin/main' into retries
NiveditJain Aug 31, 2025
f6a3bb2
Refactor retry policy and errored state handling
NiveditJain Aug 31, 2025
66c794a
Add retry policy documentation and integrate into graph configuration
NiveditJain Aug 31, 2025
728b225
Enhance retry policy error handling and validation
NiveditJain Aug 31, 2025
ba13785
Update retry policy documentation and examples
NiveditJain Aug 31, 2025
6e02d50
Enhance retry policy implementation and documentation
NiveditJain Aug 31, 2025
64592d8
Enhance errored state handling with retry state management
NiveditJain Aug 31, 2025
33e75d3
Update state-manager/app/models/db/state.py
NiveditJain Aug 31, 2025
0c16bd7
Update docs/docs/exosphere/retry-policy.md
NiveditJain Aug 31, 2025
b0cabb0
Update docs/docs/exosphere/retry-policy.md
NiveditJain Aug 31, 2025
74da1b8
Update state-manager/app/controller/errored_state.py
NiveditJain Aug 31, 2025
06ee0a3
Update state-manager/app/controller/errored_state.py
NiveditJain Aug 31, 2025
fded75b
Update state-manager/app/models/retry_policy_model.py
NiveditJain Aug 31, 2025
e56d5f5
Update max_delay description in RetryPolicyModel to clarify behavior …
NiveditJain Aug 31, 2025
42efd68
Refine documentation for retry policy and errored state handling
NiveditJain Aug 31, 2025
4ec30ab
Enhance tests for errored state and upsert graph template
NiveditJain Aug 31, 2025
b1b85d1
Refactor test for RetryPolicyModel by removing unnecessary import
NiveditJain Aug 31, 2025
3806d26
Add comprehensive tests for errored state handling in graph templates
NiveditJain Aug 31, 2025
4048228
Refactor assertions in errored state tests for clarity
NiveditJain Aug 31, 2025
462cfde
Remove Kubernetes deployment steps from the publish workflow
NiveditJain Aug 31, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
27 changes: 0 additions & 27 deletions .github/workflows/publish-state-mangaer.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -111,30 +111,3 @@ jobs:
push: true
tags: ${{ steps.meta.outputs.tags }}
labels: ${{ steps.meta.outputs.labels }}

deploy-to-k8s:
needs: publish-image
runs-on: ubuntu-latest

steps:
- name: Deploy to K8s
run: |
echo "${{ secrets.KUBE_CONFIG }}" | base64 -d > kubeconfig.yaml
export KUBECONFIG=$PWD/kubeconfig.yaml
kubectl get nodes

echo "selected image: ${{ fromJson(needs.publish-image.outputs.json).tags[1] }}"

kubectl set image deployment/exosphere-state-manager exosphere-state-manager=${{fromJson(needs.publish-image.outputs.json).tags[1]}}

kubectl rollout status deployment/exosphere-state-manager

status=$(kubectl rollout status deployment/exosphere-state-manager)

echo "$status"

if [[ "$status" != *"successfully rolled out"* ]]; then
kubectl rollout undo deployment/exosphere-state-manager
echo "❌ Deployment failed. Rolled back." >&2
exit 1
fi
42 changes: 39 additions & 3 deletions docs/docs/exosphere/create-graph.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -51,7 +51,13 @@ One can define a graph on Exosphere through a simple json config, which specifie
},
"next_nodes": []
}
]
],
"retry_policy": {
"max_retries": 3,
"strategy": "EXPONENTIAL",
"backoff_factor": 2000,
"exponent": 2
}
}
```

Expand Down Expand Up @@ -126,6 +132,24 @@ Use the `${{ ... }}` syntax to map outputs from previous nodes:
- **`${{ node_identifier.outputs.field_name }}`**: Maps output from a specific node
- **`initial`**: Static value provided when the graph is triggered
- **Direct values**: String values. In v1, numbers/booleans must be string-encoded (e.g., "42", "true").

### Retry Policy

Graphs can include a retry policy to handle transient failures automatically. The retry policy is configured at the graph level and applies to all nodes within the graph.

```json
{
"retry_policy": {
"max_retries": 3,
"strategy": "EXPONENTIAL",
"backoff_factor": 2000, // milliseconds
"exponent": 2
}
}
```

For detailed information about retry policies, including all available strategies and configuration options, see the [Retry Policy](retry-policy.md) documentation.

## Creating Graph Templates

The recommended way to create graph templates is using the Exosphere Python SDK, which provides a clean interface to the State Manager API.
Expand Down Expand Up @@ -156,7 +180,13 @@ async def create_graph_template():
result = await state_manager.upsert_graph(
graph_name="my-workflow",
graph_nodes=graph_nodes,
secrets=secrets
secrets=secrets,
retry_policy={
"max_retries": 3,
"strategy": "EXPONENTIAL",
"backoff_factor": 2000,
"exponent": 2
}
)
print("Graph template created successfully!")
print(f"Validation status: {result['validation_status']}")
Expand Down Expand Up @@ -268,7 +298,13 @@ The state manager validates your graph template:
result = await state_manager.upsert_graph(
graph_name="my-workflow",
graph_nodes=updated_nodes,
secrets=updated_secrets
secrets=updated_secrets,
retry_policy={
"max_retries": 3,
"strategy": "EXPONENTIAL",
"backoff_factor": 2000,
"exponent": 2
}
)
print("Graph template updated successfully!")
print(f"Validation status: {result['validation_status']}")
Expand Down
Loading