tasks.DynamoAttributeValue().withB double encoding value?

[eikeon]

I'm passing in a binary value to a step function which ends up getting base64 encoded somewhere along the way and it ends up getting base64 encoded a second time when I'm using withB as below. It's unclear from the current documentation if the string passed to withB should be base64 encoded already but it's type and the example seem to suggest it. If it's supposed to be a string of the raw binary is there a way decode an already base64 encoded string from the TaskInput.

The code I'm using to pass this value in as binary has been used in a couple other places using a couple different languages etc on the receiving end without issue. And no where am I knowingly encoding it a second time within the CDK state machine here.

withB's documentation shows

withB(value)🔹

public withB(value: string): DynamoAttributeValue
Parameters

value string
Returns

DynamoAttributeValue
Sets an attribute of type Binary.

For example: "B": "dGhpcyB0ZXh0IGlzIGJhc2U2NC1lbmNvZGVk"

Reproduction Steps

    new tasks.DynamoUpdateItem({
        tableName: props.content.tableName,
        partitionKey: {
            name: "ContentID",
            value: new tasks.DynamoAttributeValue().withB(
                sfn.TaskInput.fromDataAt("$.ContentID").value
            ),
        },
        ...

Error Log

Environment

• CLI Version : 1.36.1 (build 4df7dac)
• Framework Version: 1.36.1 (build 4df7dac)
• OS : macOS Catalina 10.15.5 Beta (19F62f)
• Language : Typescript

Other

---

This is 🐛 Bug Report

[eikeon]

To work around this I'm taking the base64 encoded string (that is resulting from me passing in binary to the state machine without my base64 encoding it) and decoding it, and using the binary toString. For example, I run this extra task

        const contentIDB = new sfn.Task(this, "Decode ContentID", {
            task: new tasks.EvaluateExpression({
                expression:
                    "Buffer.from($.ContentID, 'base64').toString()",
            }),
            resultPath: "$.ContentIDB",
        });

and then use $.ContentIDB instead of $.ContentID

[eikeon]

They are only starting with the right first few characters after that attempted work around.

For example should be the same length as:
AQERFArszmn0KWJPnNOEEAewUpzQXvee
So it's now starting off right but is too long:
AQERFDDvv73vv70fNGXvv71V77+9ee+/vdeNBu+/ve+/vVLvv70Pbw==

[eikeon]

I need to add encoding='binary' to the above toString for this to work. But then I get

{
  "error": "ReferenceError",
  "cause": {
    "errorType": "ReferenceError",
    "errorMessage": "encoding is not defined",
    "trace": [
      "ReferenceError: encoding is not defined",
      "    at eval (eval at handler (/var/task/index.js:8:12), <anonymous>:1:76)",
      "    at Runtime.handler (/var/task/index.js:8:12)",
      "    at Runtime.handleOnce (/var/runtime/Runtime.js:66:25)"
    ]
  }
}

Which works here docker run --rm -it --entrypoint node -v "$PWD":/var/task:ro,delegated lambci/lambda:nodejs10.x -p "Buffer.from('AQERFArszmn0KWJPnNOEEAewUpzQXvee', 'base64').toString(encoding='binary')"

Guessing EvaluateExpression is trying to do something with the encoding= bit

[eikeon]

Will try to dive into CDK code to see how one might allow for passing in the already base64 encoded string to the withB

[eikeon]

Adding the encoding='binary' positionally toString('binary') in the above workaround results in:
AQERFArDrMOOacO0KWJPwpzDk8KEEAfCsFLCnMOQXsO3wp4
vs the expected
AQERFArszmn0KWJPnNOEEAewUpzQXvee

closer... another couple characters. Perhaps the wrong flavor of base64 in this work around

[eikeon]

CDK seems to be doing its part. It's generating the following bit in the state machine definition:

        "Record FetchCompleted": {
            "Next": "Publish FetchCompleted",
            "Parameters": {
                "Key": { "ContentID": { "B.$": "$.ContentID" } },
                "TableName": "content-LabelsBA41E4AA-1XBL4K17A6P60",
                "ExpressionAttributeValues": {
                    ":now": { "N.$": "$.FetchCompletedTime" }
                },
                "UpdateExpression": "SET FetchCompletedTime = :now REMOVE FetchStartedTime"
            },
            "Type": "Task",
            "Resource": "arn:aws:states:::dynamodb:updateItem",
            "ResultPath": "$.RecordFetchCompleted"
        },

Is there somewhere to file a bug against the step function integration services? Where is should be interpreting that "$.ContentID" as a base64 encoded string and converting it to a blob that's expected there

[shivlaks]

hey @eikeon I pinged @wong-a who might have better insight here on the Step Functions behaviour you're observing

[eikeon]

@shivlaks Thank you; In the meantime I'm switching to a Lambda for this bit. But can help retest or keep digging if need be.

[eikeon]

Just reporting back that using a lambda for this task instead of the DynamoUpdateItem step functions service integration worked to avoid this issue. And I can have the lambda grab the current time instead of relying on a separate EvaluateExpression task.

[wong-a]

In the AWS SDKs, you can pass a buffer (e.g. ByteBuffer in Java and Buffer in JavaScript) and convert to base64 for you. You only need to encode the data yourself if you're using the DynamoDB Low-level API.

Since Amazon States Language is JSON-based, binary data can only be encoded as a string. What's probably happening here is that Step Functions is deserializing the string which is getting encoded again by the AWS SDK. I'll bring this up with my team so we can take a deeper look at this.

The documentation is kind of misleading as it's not clear that the data is going to be encoded again. If you pass an unencoded string (e.g. Hello World) in the ASL definition, the stored value is base64 encoded (SGVsbG8gV29ybGQ=). If you pass that as encoded a base64 encoded string, the stored value is encoded again in base64 (U0dWc2JHOGdWMjl5YkdRPQ==).

@eikeon It sounds like you're expecting The binary attribute value to behave like the Low-level API with base64 strings passed directly to DynamoDB without further encoding, is that correct?

[eikeon]

@wong-a Yes, I was expecting the binary value to behave like the low-level API. And since we’re in this JSON-based context passing binary with type string — I assumed it’d need to be expecting base64 encoded string for it to work.

I’ve been passing the binary around with the various AWS SDKs as you describe. And by the time the data gets to were I was trying to use tasks.DynamoUpdateItem above within the state machine the ContentID is already base64 encoded since ASL is JSON-based. Normally if the SDKs encode something on the way in they decode it on the way out or the other way around.

[nija-at]

Since this is not a CDK issue, I'm resolving this issue. Let us know if there's anything we can do on the CDK side to make this better.

@wong-a internal tracking: t.corp/D16302653