Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

aws_cdk.aws_stepfunctions_tasks.HttpInvoke: Get API Endpoint from state input #30749

Open
bc-venkata-edara opened this issue Jul 4, 2024 · 2 comments
Labels
@aws-cdk/aws-stepfunctions-tasks bug This issue is a bug. effort/small Small work item – less than a day of effort p2

Comments

@bc-venkata-edara
Copy link

bc-venkata-edara commented Jul 4, 2024

Describe the bug

Get API Endpoint from state input is not working throwing this error.

SCHEMA_VALIDATION_FAILED: The value for the 'ApiEndpoint' field is not valid at /States/Invoke HTTP API/Parameters' (Service: AWSStepFunctions; Status Code: 400; Error Code: InvalidDefinition

Expected Behavior

It should create stepfunction shown in below image with below definition.

Screenshot 2024-07-03 at 8 11 41 PM
"marketo-leads-creation-update": {
      "Type": "Task",
      "Resource": "arn:aws:states:::http:invoke",
      "Parameters": {
        "Authentication": {
          "ConnectionArn": "arn:aws:events:<region>:<account>:connection/<connection>
        },
        "RequestBody.$": "$.detail.object",
        "Method.$": "$.detail.object_method",
        "ApiEndpoint.$": "States.Format('https://<endpoint>/rest/v1/{}.json', $.detail.object_name)"
      }

Current Behavior

When we add code for step function httpinvoke task cdk python. it's throwing below error.

UPDATE_FAILED        | AWS::StepFunctions::StateMachine | EXAMPLERESOURCESsfMarketosync1464C05F
Resource handler returned message: "Invalid State Machine Definition: 'SCHEMA_VALIDATION_FAILED: The value for the 'ApiEndpoint' field is n
ot valid at /States/Invoke HTTP API/Parameters' (Service: AWSStepFunctions; Status Code: 400; Error Code: InvalidDefinition; Request ID: 3c
714a80-85d1-4166-9c23-a61ed3749abd; Proxy: null)" (RequestToken: 186ba6c7-912e-b3a1-e6f9-408473474a6b, HandlerErrorCode: InvalidRequest)


 ❌  EIP-Example failed: Error: The stack named EIP-Example failed to deploy: UPDATE_ROLLBACK_COMPLETE: Resource handler returned message: "Invalid State Machine Definition: 'SCHEMA_VALIDATION_FAILED: The value for the 'ApiEndpoint' field is not valid at /States/Invoke HTTP API/Parameters' (Service: AWSStepFunctions; Status Code: 400; Error Code: InvalidDefinition; Request ID: 3c714a80-85d1-4166-9c23-a61ed3749abd; Proxy: null)" (RequestToken: 186ba6c7-912e-b3a1-e6f9-408473474a6b, HandlerErrorCode: InvalidRequest)
    at FullCloudFormationDeployment.monitorDeployment (/opt/homebrew/lib/node_modules/aws-cdk/lib/index.js:451:10568)
    at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
    at async Object.deployStack2 [as deployStack] (/opt/homebrew/lib/node_modules/aws-cdk/lib/index.js:454:199716)
    at async /opt/homebrew/lib/node_modules/aws-cdk/lib/index.js:454:181438

 ❌ Deployment failed: Error: The stack named EIP-Example failed to deploy: UPDATE_ROLLBACK_COMPLETE: Resource handler returned message: "Invalid State Machine Definition: 'SCHEMA_VALIDATION_FAILED: The value for the 'ApiEndpoint' field is not valid at /States/Invoke HTTP API/Parameters' (Service: AWSStepFunctions; Status Code: 400; Error Code: InvalidDefinition; Request ID: 3c714a80-85d1-4166-9c23-a61ed3749abd; Proxy: null)" (RequestToken: 186ba6c7-912e-b3a1-e6f9-408473474a6b, HandlerErrorCode: InvalidRequest)
    at FullCloudFormationDeployment.monitorDeployment (/opt/homebrew/lib/node_modules/aws-cdk/lib/index.js:451:10568)
    at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
    at async Object.deployStack2 [as deployStack] (/opt/homebrew/lib/node_modules/aws-cdk/lib/index.js:454:199716)
    at async /opt/homebrew/lib/node_modules/aws-cdk/lib/index.js:454:181438

The stack named EIP-Example failed to deploy: UPDATE_ROLLBACK_COMPLETE: Resource handler returned message: "Invalid State Machine Definition: 'SCHEMA_VALIDATION_FAILED: The value for the 'ApiEndpoint' field is not valid at /States/Invoke HTTP API/Parameters' (Service: AWSStepFunctions; Status Code: 400; Error Code: InvalidDefinition; Request ID: 3c714a80-85d1-4166-9c23-a61ed3749abd; Proxy: null)" (RequestToken: 186ba6c7-912e-b3a1-e6f9-408473474a6b, HandlerErrorCode: InvalidRequest)

Reproduction Steps

import os
import sys  # noqa: F401
from aws_cdk import (
    Stack,
    Tags,
    aws_lambda as lambda_,
    aws_stepfunctions as sfn,
    aws_stepfunctions_tasks as tasks,
    aws_events as events,
    aws_secretsmanager as sm,
    SecretValue
)
import aws_cdk as cdk
from constructs import Construct

connection = events.Connection(self, "Connection",
            authorization=events.Authorization.basic("username", SecretValue.unsafe_plain_text("password")))

        task = tasks.HttpInvoke(self, "Invoke HTTP API",
            api_root="States.Format('<endpoint>",  # Static API root, adjust as necessary
            api_endpoint=sfn.TaskInput.from_text("rest/v1/{}.json',$.detail.object_name"), # Use the constructed URL
            body=sfn.TaskInput.from_json_path_at('$.detail.object'),
            connection=connection,
            headers=sfn.TaskInput.from_object({"Content-Type": "application/json"}),
            method=sfn.TaskInput.from_json_path_at('$.detail.object_method'),
            url_encoding_format=tasks.URLEncodingFormat.BRACKETS
        )
 
        sfMarketo_sync = sfn.StateMachine(
            self,
            "sfMarketo-sync",
            state_machine_name="sfMarketo-sync-test",
            definition_body=sfn.DefinitionBody.from_chainable(task),
            timeout=cdk.Duration.minutes(5),
            state_machine_type=sfn.StateMachineType.STANDARD,
            #role=role
        )
cdk synth
cdk deploy

Possible Solution

When we run synth or deploy on backend it creates cloud formation template in cdk.out folder. The template should add ".$" suffix to "ApiEndpoint" parameter but it's not adding. For reference see the parameter "Method" which has suffix ".$". if it add that suffix it should work.

"EXAMPLERESOURCESsfMarketosync1464C05F": {
   "Type": "AWS::StepFunctions::StateMachine",
   "Properties": {
    "DefinitionString": {
     "Fn::Join": [
      "",
      [
       "{\"StartAt\":\"Invoke HTTP API\",\"States\":{\"Invoke HTTP API\":{\"End\":true,\"Type\":\"Task\",\"Resource\":\"arn:",
       {
        "Ref": "AWS::Partition"
       },
       ":states:::http:invoke\",\"Parameters\":{\"**ApiEndpoint**\":\"States.Format('<endpoint>/rest/v1/{}.json',$.detail.object_name\",\"Authentication\":{\"ConnectionArn\":\"",
       {
        "Fn::GetAtt": [
         "EXAMPLERESOURCESConnectionF6D008B6",
         "Arn"
        ]
       },
       "\"},\"**Method.$**\":\"$.detail.object_method\",\"Headers\":{\"Content-Type\":\"application/x-www-form-urlencoded\"},\"RequestBody.$\":\"$.detail.object\",\"Transform\":{\"RequestBodyEncoding\":\"URL_ENCODED\",\"RequestEncodingOptions\":{\"ArrayFormat\":\"BRACKETS\"}}}}},\"TimeoutSeconds\":300}"
      ]
     ]
    },

Additional Information/Context

No response

CDK CLI Version

2.146.0 (build b368c78)

Framework Version

No response

Node.js Version

v22.3.0

OS

Mac OS

Language

Python

Language Version

Python 3.12.3

Other information

No response

@bc-venkata-edara bc-venkata-edara added bug This issue is a bug. needs-triage This issue or PR still needs to be triaged. labels Jul 4, 2024
@ashishdhingra ashishdhingra self-assigned this Jul 5, 2024
@ashishdhingra ashishdhingra added needs-reproduction This issue needs reproduction. and removed needs-triage This issue or PR still needs to be triaged. labels Jul 5, 2024
@ashishdhingra
Copy link
Contributor

ashishdhingra commented Jul 5, 2024

Reproducible.

Below is the equivalent code in TypeScript:

import * as cdk from 'aws-cdk-lib';
import { Construct } from 'constructs';
import * as events from 'aws-cdk-lib/aws-events';
import * as tasks from 'aws-cdk-lib/aws-stepfunctions-tasks';
import * as sfn from 'aws-cdk-lib/aws-stepfunctions';
export class TypescriptStack extends cdk.Stack {
  constructor(scope: Construct, id: string, props?: cdk.StackProps) {
    super(scope, id, props);

    const connection = new events.Connection(this, 'Connection', {
      authorization: events.Authorization.basic('username', cdk.SecretValue.unsafePlainText('password'))
    });

    const task = new tasks.HttpInvoke(this, "Invoke HTTP API", {
      apiRoot:"States.Format('https://api.stripe.com')",  // Static API root, adjust as necessary,
      apiEndpoint: sfn.TaskInput.fromText("rest/v1/{}.json',$.detail.object_name"), // Use the constructed URL
      body: sfn.TaskInput.fromJsonPathAt('$.detail.object'),
      connection: connection,
      headers: sfn.TaskInput.fromObject({"Content-Type": "application/json"}),
      method: sfn.TaskInput.fromJsonPathAt('$.detail.object_method'),
      urlEncodingFormat: tasks.URLEncodingFormat.BRACKETS
    });

    const sfMarketo_sync = new sfn.StateMachine(this, 'sfMarketo-sync', {
      stateMachineName: 'sfMarketo-sync-test',
      definitionBody: sfn.DefinitionBody.fromChainable(task),
      timeout: cdk.Duration.minutes(5),
      stateMachineType: sfn.StateMachineType.STANDARD
    });
  }
}

Running cdk synth produces the following output:

Resources:
  Connection07624BCD:
    Type: AWS::Events::Connection
    Properties:
      AuthParameters:
        BasicAuthParameters:
          Password: password
          Username: username
      AuthorizationType: BASIC
    Metadata:
      aws:cdk:path: TypescriptStack/Connection/Connection
  sfMarketosyncRole95111CA2:
    Type: AWS::IAM::Role
    Properties:
      AssumeRolePolicyDocument:
        Statement:
          - Action: sts:AssumeRole
            Effect: Allow
            Principal:
              Service: states.amazonaws.com
        Version: "2012-10-17"
    Metadata:
      aws:cdk:path: TypescriptStack/sfMarketo-sync/Role/Resource
  sfMarketosyncRoleDefaultPolicy68A30164:
    Type: AWS::IAM::Policy
    Properties:
      PolicyDocument:
        Statement:
          - Action: events:RetrieveConnectionCredentials
            Effect: Allow
            Resource:
              Fn::GetAtt:
                - Connection07624BCD
                - Arn
          - Action:
              - secretsmanager:DescribeSecret
              - secretsmanager:GetSecretValue
            Effect: Allow
            Resource:
              Fn::GetAtt:
                - Connection07624BCD
                - SecretArn
          - Action: states:InvokeHTTPEndpoint
            Condition:
              StringLike:
                states:HTTPEndpoint: States.Format('https://api.stripe.com')*
            Effect: Allow
            Resource: "*"
        Version: "2012-10-17"
      PolicyName: sfMarketosyncRoleDefaultPolicy68A30164
      Roles:
        - Ref: sfMarketosyncRole95111CA2
    Metadata:
      aws:cdk:path: TypescriptStack/sfMarketo-sync/Role/DefaultPolicy/Resource
  sfMarketosync8373672D:
    Type: AWS::StepFunctions::StateMachine
    Properties:
      DefinitionString:
        Fn::Join:
          - ""
          - - '{"StartAt":"Invoke HTTP API","States":{"Invoke HTTP API":{"End":true,"Type":"Task","Resource":"arn:'
            - Ref: AWS::Partition
            - :states:::http:invoke","Parameters":{"ApiEndpoint":"States.Format('https://api.stripe.com')/rest/v1/{}.json',$.detail.object_name","Authentication":{"ConnectionArn":"
            - Fn::GetAtt:
                - Connection07624BCD
                - Arn
            - '"},"Method.$":"$.detail.object_method","Headers":{"Content-Type":"application/x-www-form-urlencoded"},"RequestBody.$":"$.detail.object","Transform":{"RequestBodyEncoding":"URL_ENCODED","RequestEncodingOptions":{"ArrayFormat":"BRACKETS"}}}}},"TimeoutSeconds":300}'
      RoleArn:
        Fn::GetAtt:
          - sfMarketosyncRole95111CA2
          - Arn
      StateMachineName: sfMarketo-sync-test
      StateMachineType: STANDARD
    DependsOn:
      - sfMarketosyncRoleDefaultPolicy68A30164
      - sfMarketosyncRole95111CA2
    UpdateReplacePolicy: Delete
    DeletionPolicy: Delete
    Metadata:
      aws:cdk:path: TypescriptStack/sfMarketo-sync/Resource
...

The "ApiEndpoint":"States.Format('https://api.stripe.com')/rest/v1/{}.json',$.detail.object_name" doesn't appear to be correct since per HTTP Task fields, ApiEndpoint using intrinsic functions should be of format "ApiEndpoint.$":"States.Format('https://api.stripe.com/v1/customers/{}', $.customer_id)".

This could be related to the other issue #29925 as well.

@bsod85
Copy link

bsod85 commented Oct 30, 2024

I was able to work around the issue until the fix is ready with the following code in Python CDK:

state_machine_cfn_construct = state_machine.node.find_child('Resource')

definition_string = state_machine_cfn_construct.definition_string

state_machine_cfn_construct.definition_string = Fn.join("\"ApiEndpoint.$\":", Fn.split("\"ApiEndpoint\":", definition_string))

I also altered the api_root and api_endpoint values to have a valid States.Format expression after the concatenation.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
@aws-cdk/aws-stepfunctions-tasks bug This issue is a bug. effort/small Small work item – less than a day of effort p2
Projects
None yet
Development

No branches or pull requests

3 participants