diff --git a/README.md b/README.md index 2674371f..ee013ec1 100644 --- a/README.md +++ b/README.md @@ -71,9 +71,11 @@ This allows developers to deploy production agents that can scale beautifully to ).start() ``` -- ### Define your first graph +- ### Define your first graph (Beta) - Graphs are then described connecting nodes with relationships in json objects. Exosphere runs graph as per defined trigger conditions. See [Graph definitions](https://docs.exosphere.host/exosphere/create-graph/) to see more examples. + Graphs can be defined using JSON objects or with the new model-based Python SDK (beta) for better type safety and validation. See [Graph definitions](https://docs.exosphere.host/exosphere/create-graph/) for more examples. + + **JSON Definition (Traditional):** ```json { "secrets": {}, @@ -83,10 +85,7 @@ This allows developers to deploy production agents that can scale beautifully to "namespace": "hello-world", "identifier": "describe_city", "inputs": { - "bucket_name": "initial", - "prefix": "initial", - "files_only": "true", - "recursive": "false" + "city": "initial" }, "next_nodes": [] } @@ -94,6 +93,39 @@ This allows developers to deploy production agents that can scale beautifully to } ``` + **Model-Based Definition (Beta):** + ```python + from exospherehost import StateManager, GraphNodeModel, RetryPolicyModel, RetryStrategyEnum + + async def create_graph(): + state_manager = StateManager(namespace="hello-world") + + graph_nodes = [ + GraphNodeModel( + node_name="MyFirstNode", + namespace="hello-world", + identifier="describe_city", + inputs={"city": "initial"}, + next_nodes=[] + ) + ] + + # Optional: Define retry policy (beta) + retry_policy = RetryPolicyModel( + max_retries=3, + strategy=RetryStrategyEnum.EXPONENTIAL, + backoff_factor=2000 + ) + + # Create graph with model-based approach (beta) + result = await state_manager.upsert_graph( + graph_name="my-first-graph", + graph_nodes=graph_nodes, + secrets={}, + retry_policy=retry_policy # beta + ) + ``` + ## Quick Start with Docker Compose Get Exosphere running locally in under 2 minutes: diff --git a/docs/docs/exosphere/api-changes.md b/docs/docs/exosphere/api-changes.md new file mode 100644 index 00000000..9b0900e4 --- /dev/null +++ b/docs/docs/exosphere/api-changes.md @@ -0,0 +1,186 @@ +# API Changes (Beta) + +This document outlines the latest beta API changes and enhancements in ExosphereHost. + +## StateManager.upsert_graph() - Model-Based Parameters (Beta) + +The `upsert_graph` method now supports model-based parameters for improved type safety, validation, and developer experience. + +### New Signature + +```python +async def upsert_graph( + self, + graph_name: str, + graph_nodes: list[GraphNodeModel], + secrets: dict[str, str], + retry_policy: RetryPolicyModel | None = None, + store_config: StoreConfigModel | None = None, + validation_timeout: int = 60, + polling_interval: int = 1 +): +``` + +### Key Changes + +1. **Model-Based Nodes**: `graph_nodes` parameter now expects a list of `GraphNodeModel` objects instead of raw dictionaries +2. **Retry Policy Model**: Optional `retry_policy` parameter using `RetryPolicyModel` with enum-based strategy selection +3. **Store Configuration**: Optional `store_config` parameter using `StoreConfigModel` for graph-level key-value store +4. **Validation Control**: New `validation_timeout` and `polling_interval` parameters for better control over graph validation + +### Migration Guide + +#### Before (Traditional) +```python +# Old dictionary-based approach +graph_nodes = [ + { + "node_name": "DataProcessor", + "namespace": "MyProject", + "identifier": "processor", + "inputs": {"data": "initial"}, + "next_nodes": [] + } +] + +retry_policy = { + "max_retries": 3, + "strategy": "EXPONENTIAL", + "backoff_factor": 2000 +} +``` + +#### After (Beta Model-Based) +```python +from exospherehost import GraphNodeModel, RetryPolicyModel, RetryStrategyEnum + +# New model-based approach +graph_nodes = [ + GraphNodeModel( + node_name="DataProcessor", + namespace="MyProject", + identifier="processor", + inputs={"data": "initial"}, + next_nodes=[] + ) +] + +retry_policy = RetryPolicyModel( + max_retries=3, + strategy=RetryStrategyEnum.EXPONENTIAL, # Use enum instead of string + backoff_factor=2000 +) +``` + +### Available Models + +#### GraphNodeModel +- **node_name** (str): Class name of the node +- **namespace** (str): Namespace where node is registered +- **identifier** (str): Unique identifier in the graph +- **inputs** (dict[str, Any]): Input values for the node +- **next_nodes** (Optional[List[str]]): List of next node identifiers +- **unites** (Optional[UnitesModel]): Unite configuration for parallel execution + +#### RetryPolicyModel (Beta) +- **max_retries** (int): Maximum number of retry attempts (default: 3) +- **strategy** (RetryStrategyEnum): Retry strategy using enum values (default: EXPONENTIAL) +- **backoff_factor** (int): Base delay in milliseconds (default: 2000) +- **exponent** (int): Exponential multiplier (default: 2) +- **max_delay** (int | None): Maximum delay cap in milliseconds (optional) + +#### StoreConfigModel (Beta) +- **required_keys** (list[str]): Keys that must be present in the store +- **default_values** (dict[str, str]): Default values for store keys + +### Retry Strategy Enums + +- `RetryStrategyEnum.EXPONENTIAL`: Pure exponential backoff +- `RetryStrategyEnum.EXPONENTIAL_FULL_JITTER`: Exponential with full randomization +- `RetryStrategyEnum.EXPONENTIAL_EQUAL_JITTER`: Exponential with 50% randomization + +- `RetryStrategyEnum.LINEAR`: Linear backoff +- `RetryStrategyEnum.LINEAR_FULL_JITTER`: Linear with full randomization +- `RetryStrategyEnum.LINEAR_EQUAL_JITTER`: Linear with 50% randomization + +- `RetryStrategyEnum.FIXED`: Fixed delay +- `RetryStrategyEnum.FIXED_FULL_JITTER`: Fixed with full randomization +- `RetryStrategyEnum.FIXED_EQUAL_JITTER`: Fixed with 50% randomization + +### Complete Example + +```python +from exospherehost import ( + StateManager, + GraphNodeModel, + RetryPolicyModel, + StoreConfigModel, + RetryStrategyEnum +) + +async def create_advanced_graph(): + state_manager = StateManager(namespace="MyProject") + + # Define nodes using models + graph_nodes = [ + GraphNodeModel( + node_name="DataLoader", + namespace="MyProject", + identifier="loader", + inputs={"source": "initial"}, + next_nodes=["processor"] + ), + GraphNodeModel( + node_name="DataProcessor", + namespace="MyProject", + identifier="processor", + inputs={"data": "${{ loader.outputs.data }}"}, + next_nodes=[] + ) + ] + + # Define retry policy with enum + retry_policy = RetryPolicyModel( + max_retries=5, + strategy=RetryStrategyEnum.EXPONENTIAL_FULL_JITTER, + backoff_factor=1000, + exponent=2, + max_delay=30000 + ) + + # Define store configuration + store_config = StoreConfigModel( + required_keys=["cursor", "batch_id"], + default_values={ + "cursor": "0", + "batch_size": "100" + } + ) + + # Create graph with all beta features + result = await state_manager.upsert_graph( + graph_name="advanced-workflow", + graph_nodes=graph_nodes, + secrets={"api_key": "your-key"}, + retry_policy=retry_policy, # beta + store_config=store_config, # beta + validation_timeout=120, + polling_interval=2 + ) + + return result +``` + +### Benefits + +1. **Type Safety**: Pydantic models catch configuration errors at definition time +2. **IDE Support**: Better autocomplete, error detection, and documentation +3. **Validation**: Automatic validation of parameters and relationships +4. **Consistency**: Standardized parameter names and types across the SDK +5. **Extensibility**: Easy to add new fields and maintain backward compatibility + +### Beta Status + +These features are currently in beta and the API may change based on user feedback. The traditional dictionary-based approach will continue to work alongside the new model-based approach. + +For questions or feedback about these beta features, please reach out through our [Discord community](https://discord.com/invite/zT92CAgvkj). \ No newline at end of file diff --git a/docs/docs/exosphere/create-graph.md b/docs/docs/exosphere/create-graph.md index 502c4db7..79ed0634 100644 --- a/docs/docs/exosphere/create-graph.md +++ b/docs/docs/exosphere/create-graph.md @@ -9,6 +9,8 @@ A graph template consists of: - **Nodes**: The processing units in your workflow with their inputs and next nodes - **Secrets**: Configuration data shared across nodes - **Input Mapping**: How data flows between nodes using `${{ ... }}` syntax +- **Retry Policy**: Optional failure handling configuration (beta) +- **Store Configuration**: Optional graph-level key-value store (beta) ## Basic Graph Example @@ -150,12 +152,12 @@ Graphs can include a retry policy to handle transient failures automatically. Th For detailed information about retry policies, including all available strategies and configuration options, see the [Retry Policy](retry-policy.md) documentation. -## Creating Graph Templates +## Creating Graph Templates (Beta) -The recommended way to create graph templates is using the Exosphere Python SDK, which provides a clean interface to the State Manager API. +The recommended way to create graph templates is using the Exosphere Python SDK with model-based parameters, which provides a clean interface to the State Manager API and includes beta features for enhanced workflow management. -```python hl_lines="5-9 23-27" -from exospherehost import StateManager +```python hl_lines="1-3 8-12 15-35 38-44 47-53 56-70" +from exospherehost import StateManager, GraphNodeModel, RetryPolicyModel, StoreConfigModel, RetryStrategyEnum async def create_graph_template(): # Initialize the State Manager @@ -165,31 +167,74 @@ async def create_graph_template(): key=EXOSPHERE_API_KEY ) - # Define the graph nodes + # Define graph nodes using models (beta) graph_nodes = [ - ... #nodes from the namespace MyProject + GraphNodeModel( + node_name="DataLoaderNode", + namespace="MyProject", + identifier="data_loader", + inputs={ + "source": "initial", + "format": "json" + }, + next_nodes=["data_processor"] + ), + GraphNodeModel( + node_name="DataProcessorNode", + namespace="MyProject", + identifier="data_processor", + inputs={ + "raw_data": "${{ data_loader.outputs.processed_data }}", + "config": "initial" + }, + next_nodes=["data_validator"] + ), + GraphNodeModel( + node_name="DataValidatorNode", + namespace="MyProject", + identifier="data_validator", + inputs={ + "data": "${{ data_processor.outputs.processed_data }}", + "validation_rules": "initial" + }, + next_nodes=[] + ) ] # Define secrets secrets = { - ... # Store real values in a secret manager or environment variables, not in code. + "openai_api_key": "your-openai-key", + "database_url": "your-database-url" + # Store real values in a secret manager or environment variables, not in code. } + # Define retry policy using model (beta) + retry_policy = RetryPolicyModel( + max_retries=3, + strategy=RetryStrategyEnum.EXPONENTIAL, + backoff_factor=2000, + exponent=2 + ) + + # Define store configuration (beta) + store_config = StoreConfigModel( + required_keys=["cursor", "batch_id"], + default_values={ + "cursor": "0", + "batch_size": "100" + } + ) + try: - # Create or update the graph template (with optional store, beta) + # Create or update the graph template (beta) result = await state_manager.upsert_graph( graph_name="my-workflow", graph_nodes=graph_nodes, secrets=secrets, - retry_policy={ - "max_retries": 3, - "strategy": "EXPONENTIAL", - "backoff_factor": 2000, - "exponent": 2 - }, - store_config={ # beta - "ttl": 7200 # seconds to keep key/values - } + retry_policy=retry_policy, # beta + store_config=store_config, # beta + validation_timeout=60, + polling_interval=1 ) print("Graph template created successfully!") print(f"Validation status: {result['validation_status']}") @@ -202,6 +247,67 @@ async def create_graph_template(): import asyncio asyncio.run(create_graph_template()) ``` + +### Model-Based Parameters (Beta) + +The new `upsert_graph` method uses Pydantic models for better type safety and validation: + +#### GraphNodeModel + +```python +from exospherehost import GraphNodeModel + +node = GraphNodeModel( + node_name="MyNode", # Class name of the node + namespace="MyProject", # Namespace where node is registered + identifier="unique_id", # Unique identifier in this graph + inputs={ # Input values for the node + "field1": "value1", + "field2": "${{ other_node.outputs.field }}" + }, + next_nodes=["next_node_id"] # List of next node identifiers +) +``` + +#### RetryPolicyModel (Beta) + +```python +from exospherehost import RetryPolicyModel, RetryStrategyEnum + +retry_policy = RetryPolicyModel( + max_retries=3, # Maximum number of retry attempts + strategy=RetryStrategyEnum.EXPONENTIAL, # Retry strategy (use enum) + backoff_factor=2000, # Base delay in milliseconds + exponent=2, # Exponential multiplier + max_delay=30000 # Maximum delay cap in milliseconds +) +``` + +**Available Retry Strategies:** +- `RetryStrategyEnum.EXPONENTIAL` +- `RetryStrategyEnum.EXPONENTIAL_FULL_JITTER` +- `RetryStrategyEnum.EXPONENTIAL_EQUAL_JITTER` +- `RetryStrategyEnum.LINEAR` +- `RetryStrategyEnum.LINEAR_FULL_JITTER` +- `RetryStrategyEnum.LINEAR_EQUAL_JITTER` +- `RetryStrategyEnum.FIXED` +- `RetryStrategyEnum.FIXED_FULL_JITTER` +- `RetryStrategyEnum.FIXED_EQUAL_JITTER` + +#### StoreConfigModel (Beta) + +```python +from exospherehost import StoreConfigModel + +store_config = StoreConfigModel( + required_keys=["cursor", "batch_id"], # Keys that must be present + default_values={ # Default values for keys + "cursor": "0", + "batch_size": "100" + } +) +``` + ## Input Mapping Patterns === "Field Mapping" @@ -277,7 +383,9 @@ The state manager validates your graph template: === "Update Graph Template" - ```python hl_lines="17 21-25" + ```python hl_lines="1-3 8-12 15-35 38-44 47-53 56-70" + from exospherehost import StateManager, GraphNodeModel, RetryPolicyModel, StoreConfigModel, RetryStrategyEnum + async def update_graph_template(): state_manager = StateManager( namespace="MyProject", @@ -285,9 +393,50 @@ The state manager validates your graph template: key=EXOSPHERE_API_KEY ) - # Updated graph nodes + # Updated graph nodes using models (beta) updated_nodes = [ - ... + GraphNodeModel( + node_name="DataLoaderNode", + namespace="MyProject", + identifier="data_loader", + inputs={ + "source": "initial", + "format": "json", + "batch_size": "200" # Updated parameter + }, + next_nodes=["data_processor"] + ), + GraphNodeModel( + node_name="DataProcessorNode", + namespace="MyProject", + identifier="data_processor", + inputs={ + "raw_data": "${{ data_loader.outputs.processed_data }}", + "config": "initial", + "optimization": "enabled" # New parameter + }, + next_nodes=["data_validator", "data_logger"] # Added new next node + ), + GraphNodeModel( + node_name="DataValidatorNode", + namespace="MyProject", + identifier="data_validator", + inputs={ + "data": "${{ data_processor.outputs.processed_data }}", + "validation_rules": "initial" + }, + next_nodes=[] + ), + GraphNodeModel( + node_name="DataLoggerNode", # New node + namespace="MyProject", + identifier="data_logger", + inputs={ + "log_data": "${{ data_processor.outputs.processed_data }}", + "log_level": "info" + }, + next_nodes=[] + ) ] # Updated secrets @@ -297,17 +446,34 @@ The state manager validates your graph template: "logging_endpoint": "your-logging-endpoint" # Added new secret } + # Updated retry policy (beta) + retry_policy = RetryPolicyModel( + max_retries=5, # Increased retries + strategy=RetryStrategyEnum.EXPONENTIAL_FULL_JITTER, + backoff_factor=1500, # Reduced base delay + exponent=2, + max_delay=60000 # Increased max delay + ) + + # Updated store configuration (beta) + store_config = StoreConfigModel( + required_keys=["cursor", "batch_id", "session_id"], # Added session_id + default_values={ + "cursor": "0", + "batch_size": "150", # Updated default + "session_id": "default" # New default + } + ) + try: result = await state_manager.upsert_graph( graph_name="my-workflow", graph_nodes=updated_nodes, secrets=updated_secrets, - retry_policy={ - "max_retries": 3, - "strategy": "EXPONENTIAL", - "backoff_factor": 2000, - "exponent": 2 - } + retry_policy=retry_policy, # beta + store_config=store_config, # beta + validation_timeout=120, # Increased timeout + polling_interval=2 # Increased polling interval ) print("Graph template updated successfully!") print(f"Validation status: {result['validation_status']}") diff --git a/docs/docs/exosphere/retry-policy.md b/docs/docs/exosphere/retry-policy.md index 639306dc..05507d8a 100644 --- a/docs/docs/exosphere/retry-policy.md +++ b/docs/docs/exosphere/retry-policy.md @@ -388,6 +388,94 @@ If a retry policy configuration is invalid: - An error will be returned during graph creation - The graph will not be saved until the configuration is corrected +## Model-Based Configuration (Beta) + +With the new Exosphere Python SDK, you can define retry policies using Pydantic models for better type safety and validation: + +```python +from exospherehost import StateManager, GraphNodeModel, RetryPolicyModel, RetryStrategyEnum + +# Define retry policy using model (beta) +retry_policy = RetryPolicyModel( + max_retries=5, + strategy=RetryStrategyEnum.EXPONENTIAL_FULL_JITTER, + backoff_factor=1000, + exponent=2, + max_delay=30000 +) + +async def create_graph_with_retry_policy(): + state_manager = StateManager(namespace="MyProject") + + graph_nodes = [ + GraphNodeModel( + node_name="ResilientNode", + namespace="MyProject", + identifier="resilient_node", + inputs={"data": "initial"}, + next_nodes=[] + ) + ] + + # Apply retry policy to the entire graph (beta) + result = await state_manager.upsert_graph( + graph_name="resilient-workflow", + graph_nodes=graph_nodes, + secrets={"api_key": "your-key"}, + retry_policy=retry_policy # beta + ) +``` + +**Benefits of Model-Based Approach:** + +- **Type Safety**: Pydantic validation catches configuration errors early +- **IDE Support**: Better autocomplete and error detection +- **Documentation**: Built-in field descriptions and validation rules +- **Consistency**: Standardized parameter names and types + +**Available Retry Strategies:** + +- `RetryStrategyEnum.EXPONENTIAL`: Pure exponential backoff +- `RetryStrategyEnum.EXPONENTIAL_FULL_JITTER`: Exponential with full randomization +- `RetryStrategyEnum.EXPONENTIAL_EQUAL_JITTER`: Exponential with 50% randomization + +- `RetryStrategyEnum.LINEAR`: Linear backoff +- `RetryStrategyEnum.LINEAR_FULL_JITTER`: Linear with full randomization +- `RetryStrategyEnum.LINEAR_EQUAL_JITTER`: Linear with 50% randomization + +- `RetryStrategyEnum.FIXED`: Fixed delay +- `RetryStrategyEnum.FIXED_FULL_JITTER`: Fixed with full randomization +- `RetryStrategyEnum.FIXED_EQUAL_JITTER`: Fixed with 50% randomization + +**Example Configurations:** + +```python +# High-concurrency scenario (recommended) +retry_policy = RetryPolicyModel( + max_retries=3, + strategy=RetryStrategyEnum.EXPONENTIAL_FULL_JITTER, + backoff_factor=1000, + exponent=2, + max_delay=30000 +) + +# Predictable timing requirements +retry_policy = RetryPolicyModel( + max_retries=5, + strategy=RetryStrategyEnum.LINEAR, + backoff_factor=2000, + exponent=1 # Not used for LINEAR +) + +# Rate limiting scenarios +retry_policy = RetryPolicyModel( + max_retries=10, + strategy=RetryStrategyEnum.FIXED, + backoff_factor=5000, # 5 second fixed delay + max_delay=5000 +) +``` + ## Integration with Signals Retry policies work alongside Exosphere's signaling system: diff --git a/python-sdk/README.md b/python-sdk/README.md index 40b6e23b..fcfbb8f6 100644 --- a/python-sdk/README.md +++ b/python-sdk/README.md @@ -202,7 +202,7 @@ The SDK provides a `StateManager` class for programmatically triggering graph ex ### StateManager Class -The `StateManager` class allows you to trigger graph executions with custom trigger states. It handles authentication and communication with the ExosphereHost state manager service. +The `StateManager` class allows you to trigger graph executions with custom trigger states and create/update graph definitions using model-based parameters. It handles authentication and communication with the ExosphereHost state manager service. #### Initialization @@ -227,6 +227,91 @@ state_manager = StateManager(namespace="MyProject") - `key` (str, optional): Your API key. If not provided, reads from `EXOSPHERE_API_KEY` environment variable - `state_manager_version` (str): The API version to use (default: "v0") +#### Creating/Updating Graph Definitions (Beta) + +```python +from exospherehost import StateManager, GraphNodeModel, RetryPolicyModel, StoreConfigModel, RetryStrategyEnum + +async def create_graph(): + state_manager = StateManager(namespace="MyProject") + + # Define graph nodes using models + graph_nodes = [ + GraphNodeModel( + node_name="DataProcessorNode", + namespace="MyProject", + identifier="data_processor", + inputs={ + "source": "initial", + "format": "json" + }, + next_nodes=["data_validator"] + ), + GraphNodeModel( + node_name="DataValidatorNode", + namespace="MyProject", + identifier="data_validator", + inputs={ + "data": "${{ data_processor.outputs.processed_data }}", + "validation_rules": "initial" + }, + next_nodes=[] + ) + ] + + # Define retry policy using model (beta) + retry_policy = RetryPolicyModel( + max_retries=3, + strategy=RetryStrategyEnum.EXPONENTIAL, + backoff_factor=2000, + exponent=2 + ) + + # Define store configuration (beta) + store_config = StoreConfigModel( + required_keys=["cursor", "batch_id"], + default_values={ + "cursor": "0", + "batch_size": "100" + } + ) + + # Create or update the graph (beta) + result = await state_manager.upsert_graph( + graph_name="my-workflow", + graph_nodes=graph_nodes, + secrets={ + "api_key": "your-api-key", + "database_url": "your-database-url" + }, + retry_policy=retry_policy, # beta + store_config=store_config, # beta + validation_timeout=60, + polling_interval=1 + ) + + print(f"Graph created/updated: {result['validation_status']}") + return result +``` + +**Parameters:** + +- `graph_name` (str): Name of the graph to create/update +- `graph_nodes` (list[GraphNodeModel]): List of graph node models defining the workflow (beta) +- `secrets` (dict[str, str]): Key/value secrets available to all nodes +- `retry_policy` (RetryPolicyModel | None): Optional retry policy configuration (beta) +- `store_config` (StoreConfigModel | None): Graph-level store configuration (beta) +- `validation_timeout` (int): Seconds to wait for validation (default: 60) +- `polling_interval` (int): Polling interval in seconds (default: 1) + +**Returns:** + +- `dict`: Validated graph object returned by the API + +**Raises:** + +- `Exception`: If validation fails or times out + #### Triggering Graph Execution ```python diff --git a/python-sdk/exospherehost/_version.py b/python-sdk/exospherehost/_version.py index 8c58f5cc..ca533538 100644 --- a/python-sdk/exospherehost/_version.py +++ b/python-sdk/exospherehost/_version.py @@ -1 +1 @@ -version = "0.0.2b5" +version = "0.0.2b6" diff --git a/python-sdk/exospherehost/models.py b/python-sdk/exospherehost/models.py index 6ab72e4c..467da4b8 100644 --- a/python-sdk/exospherehost/models.py +++ b/python-sdk/exospherehost/models.py @@ -17,9 +17,9 @@ class GraphNodeModel(BaseModel): node_name: str = Field(..., description="Name of the node") namespace: str = Field(..., description="Namespace of the node") identifier: str = Field(..., description="Identifier of the node") - inputs: dict[str, Any] = Field(..., description="Inputs of the node") - next_nodes: Optional[List[str]] = Field(None, description="Next nodes to execute") - unites: Optional[UnitesModel] = Field(None, description="Unites of the node") + inputs: dict[str, Any] = Field(default_factory=dict, description="Inputs of the node") + next_nodes: Optional[List[str]] = Field(default=None, description="Next nodes to execute") + unites: Optional[UnitesModel] = Field(default=None, description="Unites of the node") @field_validator('node_name') @classmethod