Bug: `makeFunctionIdempotent` wrapper requires payload property matching `dataKeywordArgument` #1574

brnkrygs · 2023-07-03T20:31:30Z

Expected Behaviour

When I use the Function Wrapper pattern with the makeFunctionIdempotent function, the dataKeywordArgument should be used to determine which of the arguments of my wrapped function to use for the data payload logged in the idempotency persistence mechanism.

With this setup, I should be able to send a payload to my function with any JSON structure and have the idempotency mechanism run successfully.

Current Behaviour

When I send a payload that does not include a property in the root of the input that matches dataKeywordArgument, I get an error:

{
    "errorType": "Error",
    "errorMessage": "Failed to save record in progress. This error was  caused by: The \"data\" argument must be of type string or an instance of Buffer, TypedArray, or DataView. Received undefined.",
    "cause": {
        "errorType": "TypeError",
        "errorMessage": "The \"data\" argument must be of type string or an instance of Buffer, TypedArray, or DataView. Received undefined",
        "code": "ERR_INVALID_ARG_TYPE",
        "stack": [
            "TypeError [ERR_INVALID_ARG_TYPE]: The \"data\" argument must be of type string or an instance of Buffer, TypedArray, or DataView. Received undefined",
            "    at __node_internal_captureLargerStackTrace (node:internal/errors:490:5)",
            "    at new NodeError (node:internal/errors:399:5)",
            "    at Hash.update (node:internal/crypto/hash:109:11)",
            "    at lC.generateHash (/deps/f68a7c30-4824-4150-bbd4-4456da07f1f4/node_modules/@aws-lambda-powertools/idempotency/src/persistence/BasePersistenceLayer.ts:204:10)",
            "    at lC.getHashedIdempotencyKey (/deps/f68a7c30-4824-4150-bbd4-4456da07f1f4/node_modules/@aws-lambda-powertools/idempotency/src/persistence/BasePersistenceLayer.ts:259:49)",
            "    at lC.saveInProgress (/deps/f68a7c30-4824-4150-bbd4-4456da07f1f4/node_modules/@aws-lambda-powertools/idempotency/src/persistence/BasePersistenceLayer.ts:132:28)",
            "    at io.processIdempotency (/deps/f68a7c30-4824-4150-bbd4-4456da07f1f4/node_modules/@aws-lambda-powertools/idempotency/src/IdempotencyHandler.ts:145:35)",
            "    at io.handle (/deps/f68a7c30-4824-4150-bbd4-4456da07f1f4/node_modules/@aws-lambda-powertools/idempotency/src/IdempotencyHandler.ts:115:27)",
            "    at u (/deps/f68a7c30-4824-4150-bbd4-4456da07f1f4/node_modules/@aws-lambda-powertools/idempotency/src/makeFunctionIdempotent.ts:81:31)",
            "    at Runtime.lAe (/private/var/folders/4x/83k8wgzx2nj37bdnmj078nccln0f93/T/tmpg2h0cneq/app.ts:45:4)",
            "    at Runtime.handleOnceNonStreaming (file:///var/runtime/index.mjs:1086:29)"
        ]
    },
    "stack": [
        "Error: Failed to save record in progress. This error was  caused by: The \"data\" argument must be of type string or an instance of Buffer, TypedArray, or DataView. Received undefined.",
        "    at io.processIdempotency (/deps/f68a7c30-4824-4150-bbd4-4456da07f1f4/node_modules/@aws-lambda-powertools/idempotency/src/IdempotencyHandler.ts:158:15)",
        "    at io.handle (/deps/f68a7c30-4824-4150-bbd4-4456da07f1f4/node_modules/@aws-lambda-powertools/idempotency/src/IdempotencyHandler.ts:115:16)"
    ]
}

Code snippet

Lambda handler function initialized by sam init using Hello World template, TypeScript

import { APIGatewayProxyEvent, APIGatewayProxyResult } from 'aws-lambda';
import { IdempotencyConfig, makeFunctionIdempotent } from '@aws-lambda-powertools/idempotency';
import { DynamoDBPersistenceLayer } from '@aws-lambda-powertools/idempotency/dynamodb';

const IDEMPOTENCY_TABLE_NAME = process.env['IDEMPOTENCY_TABLE_NAME'] as string;
const persistenceStore = new DynamoDBPersistenceLayer({
  tableName: IDEMPOTENCY_TABLE_NAME,
});

const processingFunction = async (payload: Record<string, APIGatewayProxyEvent>): Promise<APIGatewayProxyResult> => {
  // your code goes here here
  console.log('processingFunction(request).', payload);

  try {
    return {
      statusCode: 200,
      body: JSON.stringify({
        message: 'tire kick created',
      }),
    };
  } catch (err) {
    console.log(err);
    return {
      statusCode: 500,
      body: JSON.stringify({
        message: 'some error happened',
      }),
    };
  }
};

export const lambdaHandler = async (event: APIGatewayProxyEvent): Promise<APIGatewayProxyResult> => {
  console.log('have request in handler:', event);
  const result = makeFunctionIdempotent(processingFunction, {
    persistenceStore,
    dataKeywordArgument: 'payload',
  })(event);

  return result;
};

Invoke that works

Notice property that matches dataKeywordArgument in sample

sam remote invoke --event '{"payload":"abc124"}'

Invoke that fails

sam remote invoke --event '{"something_else":"thing"}'

Steps to Reproduce

In this environment: Latest build of powertools that includes the idempotency utility, Lambda runtime nodejs 18. No other dependencies. Set dataKeywordArgument to "payload". Lambda initialized with sam init using TypeScript Hello World template.
When I issue a sam remote invoke or use the Lambda test console to send a invoke payload that does not include a property named "payload", I get an Error.
When I issue a sam remote invoke or use the Lambda test console to send an invoke payload that includes a property named "payload" in the root of the JSON, it works.

Error: Failed to save record in progress. This error was caused by: The \"data\" argument must be of type string or an instance of Buffer, TypedArray, or DataView. Received undefined.

Possible Solution

No response

Powertools for AWS Lambda (TypeScript) version

latest

AWS Lambda function runtime

18.x

Packaging format used

npm

Execution logs

For this log, `dataKeywordArgument` is set to `"payload"`

# Failure


INIT_START Runtime Version: nodejs:18.v8	Runtime Version ARN: arn:aws:lambda:us-east-2::runtime:2755dc322c8dbb64760145d6403d14432af527bf4dd3cf03713aae10e0f8b552
START RequestId: e3e527fa-472a-4ee1-a709-699dda02ea77 Version: $LATEST
2023-07-03T17:43:16.560Z	e3e527fa-472a-4ee1-a709-699dda02ea77	INFO	have request in handler: {
  transaction_id: 'eee',
  resource: '/tire-kicks',
  path: '/tire-kicks',
  httpMethod: 'PUT',
  headers: {
    Accept: '*/*',
    'CloudFront-Forwarded-Proto': 'https',
    'CloudFront-Is-Desktop-Viewer': 'true',
    'CloudFront-Is-Mobile-Viewer': 'false',
    'CloudFront-Is-SmartTV-Viewer': 'false',
    'CloudFront-Is-Tablet-Viewer': 'false',
    'CloudFront-Viewer-ASN': '7922',
    'CloudFront-Viewer-Country': 'US',
    'content-type': 'application/x-www-form-urlencoded',
    Host: '98whvcov2m.execute-api.us-east-2.amazonaws.com',
    transaction_id: '123',
    'User-Agent': 'curl/7.88.1',
    Via: '2.0 e807928bf10df259d5d9aa45c69572c8.cloudfront.net (CloudFront)',
    'X-Amz-Cf-Id': 'Bly3yGg3y0hIdLl_cCWVcbZtahuyjNPZPA_zU_YliYCqw82IOhhQ_Q==',
    'X-Amzn-Trace-Id': 'Root=1-64a2ff98-210a43b91f2d5c8b0a2d3a3d',
    'X-Forwarded-For': '68.48.152.62, 15.158.47.105',
    'X-Forwarded-Port': '443',
    'X-Forwarded-Proto': 'https'
  },
  multiValueHeaders: {
    Accept: [ '*/*' ],
    'CloudFront-Forwarded-Proto': [ 'https' ],
    'CloudFront-Is-Desktop-Viewer': [ 'true' ],
    'CloudFront-Is-Mobile-Viewer': [ 'false' ],
    'CloudFront-Is-SmartTV-Viewer': [ 'false' ],
    'CloudFront-Is-Tablet-Viewer': [ 'false' ],
    'CloudFront-Viewer-ASN': [ '7922' ],
    'CloudFront-Viewer-Country': [ 'US' ],
    'content-type': [ 'application/x-www-form-urlencoded' ],
    Host: [ '98whvcov2m.execute-api.us-east-2.amazonaws.com' ],
    transaction_id: [ '123' ],
    'User-Agent': [ 'curl/7.88.1' ],
    Via: [
      '2.0 e807928bf10df259d5d9aa45c69572c8.cloudfront.net (CloudFront)'
    ],
    'X-Amz-Cf-Id': [ 'Bly3yGg3y0hIdLl_cCWVcbZtahuyjNPZPA_zU_YliYCqw82IOhhQ_Q==' ],
    'X-Amzn-Trace-Id': [ 'Root=1-64a2ff98-210a43b91f2d5c8b0a2d3a3d' ],
    'X-Forwarded-For': [ '68.48.152.62, 15.158.47.105' ],
    'X-Forwarded-Port': [ '443' ],
    'X-Forwarded-Proto': [ 'https' ]
  },
  queryStringParameters: null,
  multiValueQueryStringParameters: null,
  pathParameters: null,
  stageVariables: null,
  requestContext: {
    resourceId: 'ox2bhw',
    resourcePath: '/tire-kicks',
    httpMethod: 'PUT',
    extendedRequestId: 'Hfzf7H9ViYcF3jA=',
    requestTime: '03/Jul/2023:17:04:24 +0000',
    path: '/Prod/tire-kicks',
    accountId: '498170313562',
    protocol: 'HTTP/1.1',
    stage: 'Prod',
    domainPrefix: '98whvcov2m',
    requestTimeEpoch: 1688403864837,
    requestId: '53bd769a-b1e4-4a74-84fa-364555b34d11',
    identity: {
      cognitoIdentityPoolId: null,
      accountId: null,
      cognitoIdentityId: null,
      caller: null,
      sourceIp: '68.48.152.62',
      principalOrgId: null,
      accessKey: null,
      cognitoAuthenticationType: null,
      cognitoAuthenticationProvider: null,
      userArn: null,
      userAgent: 'curl/7.88.1',
      user: null
    },
    domainName: '98whvcov2m.execute-api.us-east-2.amazonaws.com',
    apiId: '98whvcov2m'
  },
  body: '{"transactionId":"10a"}',
  isBase64Encoded: false
}
2023-07-03T17:43:16.579Z	e3e527fa-472a-4ee1-a709-699dda02ea77	WARN	No value found for idempotency_key. jmespath: 
2023-07-03T17:43:22.159Z	e3e527fa-472a-4ee1-a709-699dda02ea77	ERROR	Invoke Error 	
{
    "errorType": "Error",
    "errorMessage": "Failed to save record in progress. This error was  caused by: The \"data\" argument must be of type string or an instance of Buffer, TypedArray, or DataView. Received undefined.",
    "cause": {
        "errorType": "TypeError",
        "errorMessage": "The \"data\" argument must be of type string or an instance of Buffer, TypedArray, or DataView. Received undefined",
        "code": "ERR_INVALID_ARG_TYPE",
        "stack": [
            "TypeError [ERR_INVALID_ARG_TYPE]: The \"data\" argument must be of type string or an instance of Buffer, TypedArray, or DataView. Received undefined",
            "    at __node_internal_captureLargerStackTrace (node:internal/errors:490:5)",
            "    at new NodeError (node:internal/errors:399:5)",
            "    at Hash.update (node:internal/crypto/hash:109:11)",
            "    at lC.generateHash (/deps/f68a7c30-4824-4150-bbd4-4456da07f1f4/node_modules/@aws-lambda-powertools/idempotency/src/persistence/BasePersistenceLayer.ts:204:10)",
            "    at lC.getHashedIdempotencyKey (/deps/f68a7c30-4824-4150-bbd4-4456da07f1f4/node_modules/@aws-lambda-powertools/idempotency/src/persistence/BasePersistenceLayer.ts:259:49)",
            "    at lC.saveInProgress (/deps/f68a7c30-4824-4150-bbd4-4456da07f1f4/node_modules/@aws-lambda-powertools/idempotency/src/persistence/BasePersistenceLayer.ts:132:28)",
            "    at io.processIdempotency (/deps/f68a7c30-4824-4150-bbd4-4456da07f1f4/node_modules/@aws-lambda-powertools/idempotency/src/IdempotencyHandler.ts:145:35)",
            "    at io.handle (/deps/f68a7c30-4824-4150-bbd4-4456da07f1f4/node_modules/@aws-lambda-powertools/idempotency/src/IdempotencyHandler.ts:115:27)",
            "    at u (/deps/f68a7c30-4824-4150-bbd4-4456da07f1f4/node_modules/@aws-lambda-powertools/idempotency/src/makeFunctionIdempotent.ts:81:31)",
            "    at Runtime.lAe (/private/var/folders/4x/83k8wgzx2nj37bdnmj078nccln0f93/T/tmpg2h0cneq/app.ts:45:4)",
            "    at Runtime.handleOnceNonStreaming (file:///var/runtime/index.mjs:1086:29)"
        ]
    },
    "stack": [
        "Error: Failed to save record in progress. This error was  caused by: The \"data\" argument must be of type string or an instance of Buffer, TypedArray, or DataView. Received undefined.",
        "    at io.processIdempotency (/deps/f68a7c30-4824-4150-bbd4-4456da07f1f4/node_modules/@aws-lambda-powertools/idempotency/src/IdempotencyHandler.ts:158:15)",
        "    at io.handle (/deps/f68a7c30-4824-4150-bbd4-4456da07f1f4/node_modules/@aws-lambda-powertools/idempotency/src/IdempotencyHandler.ts:115:16)"
    ]
}

END RequestId: e3e527fa-472a-4ee1-a709-699dda02ea77

Success

REPORT RequestId: e3e527fa-472a-4ee1-a709-699dda02ea77	Duration: 5664.42 ms	Billed Duration: 5665 ms	Memory Size: 128 MB	Max Memory Used: 128 MB	Init Duration: 395.56 ms	
XRAY TraceId: 1-64a308b4-0e12a3155fe94234496bc4df	SegmentId: 5339526d7ed02274	Sampled: true	
START RequestId: fe90db53-cee2-4d46-ae9e-2f8eaea8d690 Version: $LATEST
2023-07-03T17:43:38.767Z	fe90db53-cee2-4d46-ae9e-2f8eaea8d690	INFO	have request in handler: {
  payload: 'eee',
  resource: '/tire-kicks',
  path: '/tire-kicks',
  httpMethod: 'PUT',
  headers: {
    Accept: '*/*',
    'CloudFront-Forwarded-Proto': 'https',
    'CloudFront-Is-Desktop-Viewer': 'true',
    'CloudFront-Is-Mobile-Viewer': 'false',
    'CloudFront-Is-SmartTV-Viewer': 'false',
    'CloudFront-Is-Tablet-Viewer': 'false',
    'CloudFront-Viewer-ASN': '7922',
    'CloudFront-Viewer-Country': 'US',
    'content-type': 'application/x-www-form-urlencoded',
    Host: '98whvcov2m.execute-api.us-east-2.amazonaws.com',
    transaction_id: '123',
    'User-Agent': 'curl/7.88.1',
    Via: '2.0 e807928bf10df259d5d9aa45c69572c8.cloudfront.net (CloudFront)',
    'X-Amz-Cf-Id': 'Bly3yGg3y0hIdLl_cCWVcbZtahuyjNPZPA_zU_YliYCqw82IOhhQ_Q==',
    'X-Amzn-Trace-Id': 'Root=1-64a2ff98-210a43b91f2d5c8b0a2d3a3d',
    'X-Forwarded-For': '68.48.152.62, 15.158.47.105',
    'X-Forwarded-Port': '443',
    'X-Forwarded-Proto': 'https'
  },
  multiValueHeaders: {
    Accept: [ '*/*' ],
    'CloudFront-Forwarded-Proto': [ 'https' ],
    'CloudFront-Is-Desktop-Viewer': [ 'true' ],
    'CloudFront-Is-Mobile-Viewer': [ 'false' ],
    'CloudFront-Is-SmartTV-Viewer': [ 'false' ],
    'CloudFront-Is-Tablet-Viewer': [ 'false' ],
    'CloudFront-Viewer-ASN': [ '7922' ],
    'CloudFront-Viewer-Country': [ 'US' ],
    'content-type': [ 'application/x-www-form-urlencoded' ],
    Host: [ '98whvcov2m.execute-api.us-east-2.amazonaws.com' ],
    transaction_id: [ '123' ],
    'User-Agent': [ 'curl/7.88.1' ],
    Via: [
      '2.0 e807928bf10df259d5d9aa45c69572c8.cloudfront.net (CloudFront)'
    ],
    'X-Amz-Cf-Id': [ 'Bly3yGg3y0hIdLl_cCWVcbZtahuyjNPZPA_zU_YliYCqw82IOhhQ_Q==' ],
    'X-Amzn-Trace-Id': [ 'Root=1-64a2ff98-210a43b91f2d5c8b0a2d3a3d' ],
    'X-Forwarded-For': [ '68.48.152.62, 15.158.47.105' ],
    'X-Forwarded-Port': [ '443' ],
    'X-Forwarded-Proto': [ 'https' ]
  },
  queryStringParameters: null,
  multiValueQueryStringParameters: null,
  pathParameters: null,
  stageVariables: null,
  requestContext: {
    resourceId: 'ox2bhw',
    resourcePath: '/tire-kicks',
    httpMethod: 'PUT',
    extendedRequestId: 'Hfzf7H9ViYcF3jA=',
    requestTime: '03/Jul/2023:17:04:24 +0000',
    path: '/Prod/tire-kicks',
    accountId: '498170313562',
    protocol: 'HTTP/1.1',
    stage: 'Prod',
    domainPrefix: '98whvcov2m',
    requestTimeEpoch: 1688403864837,
    requestId: '53bd769a-b1e4-4a74-84fa-364555b34d11',
    identity: {
      cognitoIdentityPoolId: null,
      accountId: null,
      cognitoIdentityId: null,
      caller: null,
      sourceIp: '68.48.152.62',
      principalOrgId: null,
      accessKey: null,
      cognitoAuthenticationType: null,
      cognitoAuthenticationProvider: null,
      userArn: null,
      userAgent: 'curl/7.88.1',
      user: null
    },
    domainName: '98whvcov2m.execute-api.us-east-2.amazonaws.com',
    apiId: '98whvcov2m'
  },
  body: '{"transactionId":"10a"}',
  isBase64Encoded: false
}
2023-07-03T17:43:38.768Z	fe90db53-cee2-4d46-ae9e-2f8eaea8d690	WARN	Could not determine remaining time left. Did you call registerLambdaContext on IdempotencyConfig?
END RequestId: fe90db53-cee2-4d46-ae9e-2f8eaea8d690
REPORT RequestId: fe90db53-cee2-4d46-ae9e-2f8eaea8d690	Duration: 712.45 ms	Billed Duration: 713 ms	Memory Size: 128 MB	Max Memory Used: 128 MB	
XRAY TraceId: 1-64a308ca-2c283e332802034532f10fed	SegmentId: 332106327066e687	Sampled: true

The text was updated successfully, but these errors were encountered:

dreamorosi · 2023-07-04T08:33:34Z

Hi @brnkrygs thank you for reporting this, we'll be taking a closer look and get back to you here.

dreamorosi · 2023-07-05T12:12:56Z

Hi Brian, thank you again for reporting this.

As a result of this feedback, as well as the others you have shared, we have decided to change the API and remove the dataKeywordArgument entirely, in favor of a dataIndexArgument one.

The full detailed explanation of why we made this change can be found in the PR body of #1579 (section titled dataKeywordArgument vs dataIndexArgument).

At a high level, we realized that dataKeywordArgument was confusing and actually detrimental in the sense that it was duplicating some of the function that should have been covered by eventKeyJmespath.

The main function of dataKeywordArgument should have been to tell the utility which argument of a function should be used for the idempotency, as opposed to what subset/portion of the argument (something covered by eventKeyJmespath).

With this in mind, and considering that JavaScript doesn't have a notion of keyword arguments, we have decided to replace the option with a more aptly named dataIndexArgument one. In JS the only way to identify arguments is by index (read here to understand why), so in order to allow customers to tell the utility which argument to use, they now have to specify the index of that argument. If no argument is specified, we use the first one.

This can be used in conjunction with the eventKeyJmespath option, to both tell the utility which one and what part/subset of an argument should be used for idempotency. The new README for the utility (merged in the next few hours) will include examples for all those cases.

Let me know if this helps, it'd be amazing if you could try the same implementation with the new logic and let me know if it fixes the issue (I can provide a build if needed, just ping me).

brnkrygs · 2023-07-06T20:18:40Z

Thanks for your effort on this @dreamorosi, I would love to take another run at the new implementation. I have a set of tests I want to run against it 😄

Will ping you about a build.

brnkrygs · 2023-07-11T13:37:30Z

The new build is looking good! No more exceptions about the data argument.

github-actions · 2023-07-11T13:38:41Z

⚠️ COMMENT VISIBILITY WARNING ⚠️

Comments on closed issues are hard for our team to see.
If you need more assistance, please either tag a team member or open a new issue that references this one.
If you wish to keep having a conversation with other community members under this issue feel free to do so.

brnkrygs added triage This item has not been triaged by a maintainer, please wait bug Something isn't working labels Jul 3, 2023

github-project-automation bot added this to Powertools for AWS Lambda (TypeScript) Jul 3, 2023

dreamorosi self-assigned this Jul 5, 2023

dreamorosi added idempotency This item relates to the Idempotency Utility confirmed The scope is clear, ready for implementation and removed triage This item has not been triaged by a maintainer, please wait labels Jul 5, 2023

dreamorosi moved this to Working on it in Powertools for AWS Lambda (TypeScript) Jul 5, 2023

dreamorosi closed this as completed Jul 11, 2023

github-project-automation bot moved this from Working on it to Coming soon in Powertools for AWS Lambda (TypeScript) Jul 11, 2023

dreamorosi moved this from Coming soon to Shipped in Powertools for AWS Lambda (TypeScript) Jul 11, 2023

dreamorosi linked a pull request Jul 11, 2023 that will close this issue

test(idempotency): improve integration tests for utility #1591

Merged

9 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Bug: `makeFunctionIdempotent` wrapper requires payload property matching `dataKeywordArgument` #1574

Bug: `makeFunctionIdempotent` wrapper requires payload property matching `dataKeywordArgument` #1574

brnkrygs commented Jul 3, 2023

dreamorosi commented Jul 4, 2023

dreamorosi commented Jul 5, 2023

brnkrygs commented Jul 6, 2023

brnkrygs commented Jul 11, 2023

github-actions bot commented Jul 11, 2023

Bug: makeFunctionIdempotent wrapper requires payload property matching dataKeywordArgument #1574

Bug: makeFunctionIdempotent wrapper requires payload property matching dataKeywordArgument #1574

Comments

brnkrygs commented Jul 3, 2023

Expected Behaviour

Current Behaviour

Code snippet

Invoke that works

Invoke that fails

Steps to Reproduce

Possible Solution

Powertools for AWS Lambda (TypeScript) version

AWS Lambda function runtime

Packaging format used

Execution logs

Success

dreamorosi commented Jul 4, 2023

dreamorosi commented Jul 5, 2023

brnkrygs commented Jul 6, 2023

brnkrygs commented Jul 11, 2023

github-actions bot commented Jul 11, 2023

⚠️ COMMENT VISIBILITY WARNING ⚠️

Bug: `makeFunctionIdempotent` wrapper requires payload property matching `dataKeywordArgument` #1574

Bug: `makeFunctionIdempotent` wrapper requires payload property matching `dataKeywordArgument` #1574