Add support for Amazon States Language "ResultSelector" in Task, Map … #102

yoodan93 · 2020-10-29T21:51:57Z

Add ResultSelector support for Task, Map and Parallel states.

Issue #, if available:
#95

Description of changes

Adding new Amazon States Language payload template "ResultSelector" https://states-language.net/#payload-template
This change allows user to define a result_selector which is used to define the effective result of a state.

Testing

Added unit test for changes
Manual test:

# Create a simple workflow containing a task state with `result_selector`
from stepfunctions.inputs import StepResult

step_result = StepResult()
step_result_placeholder = {
    'Output': step_result['output']
}

state_with_result_selector = Task(
    state_id='StartState',
    parameters={'Key': 'Value'},
    result_selector=step_result_placeholder,
    resource=<resource_arn>
)

workflow = Workflow(
    name="ResultSelector_Example",
    definition=state_with_result_selector,
    role=<workflow_execution_role>
)

# Execute the workflow and confirm it has completed successfully
workflow.execute()

Workflow definition:

{
    "StartAt": "StartState",
    "States": {
        "StartState": {
            "Parameters": {
                "Key": "Value"
            },
            "ResultSelector": {
                "Output.$": "$['output']"
            },
            "Resource": <resource_arn>,
            "Type": "Task",
            "End": true
        }
    }
}

By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.

StepFunctions-Bot · 2020-10-29T21:58:33Z

AWS CodeBuild CI Report

Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

StepFunctions-Bot · 2020-10-29T23:39:41Z

AWS CodeBuild CI Report

CodeBuild project: StepFunctionsPythonSDK-integtests
Commit ID: 41f2db2
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

tests/unit/test_placeholders.py

src/stepfunctions/inputs/placeholders.py

src/stepfunctions/steps/states.py

tests/unit/test_placeholders_with_steps.py

…and Parallel States.

…ctor'

src/stepfunctions/steps/service.py

tests/unit/test_placeholders_with_steps.py

milesgranger · 2021-08-31T12:00:12Z

Any update with this? Would be fantastic to have! 👍

shivlaks

nice work @yoodan93 !!

overall approach looks good. had some minor/naive questions.
I do think we can raise the bar on testing a bit more though

doc/placeholders.rst

src/stepfunctions/inputs/placeholders.py

src/stepfunctions/steps/compute.py

tests/unit/test_placeholders.py

ca-nguyen · 2021-09-08T17:49:36Z

Taking over to help merge @yoodan93 's PR
Thank you for your review - will be addressing comments shortly

… error message

shivlaks

can you also add some details around testing - i.e.

have we tried generating a workflow and executing the state machine successfully?

at some point we should probably add some basic integ tests that exercise state machines and their support for ASL features. Presumably, we'll be adding more capabilities in the future. Probably not in scope for this PR, but worth thinking about sooner than later.

ca-nguyen · 2021-09-09T19:03:35Z

can you also add some details around testing

Added manual test details

at some point we should probably add some basic integ tests that exercise state machines and their support for ASL features. Presumably, we'll be adding more capabilities in the future. Probably not in scope for this PR, but worth thinking about sooner than later.

Agreed! let's do add integ tests in another PR

Requested changes have been addressed in the last commits

wong-a · 2021-10-23T00:09:22Z

Manual test: Generated a workflow with a Task with StepResult placeholder and executed successfully

This is very unspecific. Can you share the SDK code you used to generate that?

doc/placeholders.rst

wong-a · 2021-10-23T00:14:28Z

src/stepfunctions/steps/compute.py

@@ -98,6 +99,7 @@ def __init__(self, state_id, wait_for_completion=True, **kwargs):
            comment (str, optional): Human-readable comment or description. (default: None)
            input_path (str, optional): Path applied to the state’s raw input to select some or all of it; that selection is used by the state. (default: '$')
            parameters (dict, optional): The value of this field becomes the effective input for the state.
+            result_selector (dict, optional): The value of this field becomes the effective result of the state.


Maintaining these docstrings for every inheriting class is pretty annoying. I wonder if there anything we can do to make that easier.

tests/unit/test_placeholders_with_steps.py

src/stepfunctions/steps/states.py

doc/placeholders.rst

wong-a · 2021-10-23T00:28:03Z

doc/placeholders.rst

@@ -82,11 +105,23 @@ that returns the placeholder output for that step.
  definition = Chain([lambda_state_first, lambda_state_second])


+Step Result
+-----------
+The third mechanism is a placeholder for a step's result. The result of a step can be modified


Still wondering if it's really necessary to introduce StepResult when StepInput could also be used to achieve the same thing. I left that comment earlier but it didn't get an answer. Literally the only difference is the class name. Thoughts @shivlaks?

We can always add classes later, but removing it afterwards breaks back-compat.

lambda_result = StepInput( schema={ "Id": str, } ) lambda_state_first = LambdaStep( state_id="MyFirstLambdaStep", result_selector={ "Output": lambda_result["Id"], "Status": "Success" } )

Good question - I missed that comment earlier

While it does make sense to use the same object, I think the name StepInput could bring confusion since, in this case, we are not using the step's input, but the step's result (or output)

To be backwards compatible, we can't rename StepInput to a more general term. Alternatively, we could make StepResult inherit from StepInput instead of Placeholder and avoid code duplication.

ca-nguyen · 2021-10-23T00:46:26Z

This is very unspecific. Can you share the SDK code you used to generate that?

Agreed - this is not sufficient information to reproduce the test. I provided more details in the Testing section of the PR description

StepFunctions-Bot · 2021-10-25T17:48:15Z

AWS CodeBuild CI Report

CodeBuild project: AutoBuildProject6AEA49D1-sEHrOdk7acJc
Commit ID: 6594b1c
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

alesageProovStation · 2023-06-19T13:49:33Z

Hi! Any plans on merging this PR soon? I'm writing a stepfunction state machine which requires this... Thanks.

Base automatically changed from master to main February 25, 2021 21:10

wong-a requested review from wong-a and shivlaks April 22, 2021 19:41

wong-a linked an issue Apr 30, 2021 that may be closed by this pull request

Feature Request: Add support for ResultSelector #95

Open

wong-a previously requested changes Apr 30, 2021

View reviewed changes

Add support for Amazon States Language "ResultSelector" in Task, Map …

c05c90f

…and Parallel States.

yoodan93 force-pushed the master branch from 733cd3f to 53d0937 Compare July 29, 2021 20:43

Update Placeholder doc and add more tests for step field 'result_sele…

8667c37

…ctor'

yoodan93 force-pushed the master branch from 53d0937 to 8667c37 Compare July 29, 2021 20:46

ca-nguyen reviewed Aug 3, 2021

View reviewed changes

src/stepfunctions/steps/service.py Show resolved Hide resolved

ca-nguyen reviewed Aug 3, 2021

View reviewed changes

tests/unit/test_placeholders_with_steps.py Outdated Show resolved Hide resolved

ca-nguyen reviewed Aug 3, 2021

View reviewed changes

tests/unit/test_placeholders_with_steps.py Outdated Show resolved Hide resolved

ca-nguyen reviewed Aug 3, 2021

View reviewed changes

tests/unit/test_placeholders_with_steps.py Outdated Show resolved Hide resolved

Daniel Yoo and others added 3 commits August 11, 2021 13:38

Add result selector support for EventBridgePutEventsStep

851fc0e

Merge branch 'main' into master

3430024

Add result selector support for GlueDataBrewStartJobRunStep

e67804b

ca-nguyen previously approved these changes Aug 11, 2021

View reviewed changes

shivlaks previously requested changes Sep 2, 2021

View reviewed changes

ca-nguyen mentioned this pull request Sep 8, 2021

feat: Support placeholders for input_path and output_path for all States (except Fail) and items_path for MapState #158

Open

6 tasks

Merge branch 'main' into master

2758879

Added tests and updated placeholders documentation

9df0b8d

ca-nguyen dismissed their stale review via 9df0b8d September 8, 2021 22:46

ca-nguyen requested review from wong-a and shivlaks September 8, 2021 22:48

ca-nguyen added 2 commits September 8, 2021 16:21

Removed double negation from function description and added schema in…

ca45e9e

… error message

Reformat documentation

a7016da

shivlaks reviewed Sep 9, 2021

View reviewed changes

ca-nguyen requested a review from shivlaks September 9, 2021 19:04

Merge branch 'main' into master

eb0a82c

evscott and others added 3 commits October 4, 2021 09:54

Merge branch 'main' into master

5f5c4ca

Merge branch 'main' into master

6871e03

Merge branch 'main' into master

21595dc

Merge branch 'main' into master

f4cb764

wong-a reviewed Oct 23, 2021

View reviewed changes

Removed unused import, reformatted doc and modified tests

6594b1c

ca-nguyen requested a review from wong-a October 23, 2021 02:40

wong-a mentioned this pull request Feb 8, 2022

[aws-stepfunctions] Support for ResultSelector #185

Closed

yoodan93 closed this Mar 6, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add support for Amazon States Language "ResultSelector" in Task, Map … #102

Add support for Amazon States Language "ResultSelector" in Task, Map … #102

yoodan93 commented Oct 29, 2020 •

edited by ca-nguyen

Loading

StepFunctions-Bot commented Oct 29, 2020

StepFunctions-Bot commented Oct 29, 2020

milesgranger commented Aug 31, 2021

shivlaks left a comment

ca-nguyen commented Sep 8, 2021

shivlaks left a comment

ca-nguyen commented Sep 9, 2021

wong-a commented Oct 23, 2021

wong-a Oct 23, 2021

wong-a Oct 23, 2021

ca-nguyen Oct 23, 2021

ca-nguyen commented Oct 23, 2021

StepFunctions-Bot commented Oct 25, 2021

alesageProovStation commented Jun 19, 2023

Add support for Amazon States Language "ResultSelector" in Task, Map … #102

Add support for Amazon States Language "ResultSelector" in Task, Map … #102

Conversation

yoodan93 commented Oct 29, 2020 • edited by ca-nguyen Loading

Description of changes

Testing

StepFunctions-Bot commented Oct 29, 2020

AWS CodeBuild CI Report

StepFunctions-Bot commented Oct 29, 2020

AWS CodeBuild CI Report

milesgranger commented Aug 31, 2021

shivlaks left a comment

Choose a reason for hiding this comment

ca-nguyen commented Sep 8, 2021

shivlaks left a comment

Choose a reason for hiding this comment

ca-nguyen commented Sep 9, 2021

wong-a commented Oct 23, 2021

wong-a Oct 23, 2021

Choose a reason for hiding this comment

wong-a Oct 23, 2021

Choose a reason for hiding this comment

ca-nguyen Oct 23, 2021

Choose a reason for hiding this comment

ca-nguyen commented Oct 23, 2021

StepFunctions-Bot commented Oct 25, 2021

AWS CodeBuild CI Report

alesageProovStation commented Jun 19, 2023

yoodan93 commented Oct 29, 2020 •

edited by ca-nguyen

Loading