step-functions

Step Functions configuration for Metaflow

This module sets up the infrastructure to use AWS Step Functions with Metaflow.

This builds on top of the functionality provided by the computation module, which allows to execute Metaflow step code on AWS Batch. If you use computation module alone, the orchestration is done by the Metaflow task scheduler that itself needs to runs somewhere (often, your laptop, or a dedicated server). Step Functions support in Metaflow allows you to replace that scheduler by compiling your Flows to a AWS Step Functions State Machine, and deploying it to AWS.

To read more, see the Metaflow docs

Inputs

Name	Description	Type	Default	Required
active	When true step function infrastructure is provisioned.	bool	false	no
batch_job_queue_arn	Batch job queue arn	string	n/a	yes
iam_partition	IAM Partition (Select aws-us-gov for AWS GovCloud, otherwise leave as is)	string	"aws"	no
resource_prefix	Prefix given to all AWS resources to differentiate between applications	string	n/a	yes
resource_suffix	Suffix given to all AWS resources to differentiate between environment and workspace	string	n/a	yes
s3_bucket_arn	arn of the metaflow datastore s3 bucket	string	n/a	yes
s3_bucket_kms_arn	arn of the metaflow datastore s3 bucket's kms key	string	n/a	yes
standard_tags	The standard tags to apply to every AWS resource.	map(string)	n/a	yes

Outputs

Name	Description
metaflow_eventbridge_role_arn	IAM role for Amazon EventBridge to access AWS Step Functions.
metaflow_step_functions_dynamodb_policy	Policy json allowing access to the step functions dynamodb table.
metaflow_step_functions_dynamodb_table_arn	AWS DynamoDB table arn for tracking AWS Step Functions execution metadata.
metaflow_step_functions_dynamodb_table_name	AWS DynamoDB table name for tracking AWS Step Functions execution metadata.
metaflow_step_functions_role_arn	IAM role for AWS Step Functions to access AWS resources (AWS Batch, AWS DynamoDB).