SQS receiveMessage not handling new lines correctly

[pkmnfrk]

Describe the bug
The SQS client seems to mishandle the first(?) new line in a message body, turning it from \n into a space. This causes the MD5 hash validation to fail and the receiveMessages call to throw.

Error: Invalid MD5 checksum on messages: 12fcd5a6-338f-485a-9b01-949ad118a6f0
    at /home/mike/code/project/service/node_modules/@aws-sdk/middleware-sdk-sqs/dist/cjs/receive-message.js:21:19
    at processTicksAndRejections (internal/process/task_queues.js:93:5)
    at async /home/mike/code/project/service/node_modules/@aws-sdk/middleware-logger/dist/cjs/loggerMiddleware.js:7:22

SDK version number
@aws-sdk/client-sqs@1.0.0-rc.10
Reproduced in @aws-sdk/client-sqs@3.2.0

Is the issue in the browser/Node.js/ReactNative?
Node.js

Details of the browser/Node.js/ReactNative version
v14.8.0

To Reproduce (observed behavior)
Send a message to an SQS queue with a new line in it. Eg:

{
    "hello": "world"
}

Then, receive that message in the code:

const AWS = require("@aws-sdk/client-sqs");
const sqs = new AWS.SQS();
sqs
  .receiveMessage({
    QueueUrl: "https://sqs.us-east-1.amazonaws.com/...",
    WaitTimeSeconds: 20,
  })
  .then((data) => {
    console.log("Got", data.length, "messages");
  })
  .catch((e) => {
    console.error(e);
  });

Expected behavior
The call succeeds, and gives us the message.

[pkmnfrk]

I did a little more digging into this, and I found some surprising results.

Top is the original message, as sent to SQS. You can see that the second byte is 0x0A (aka, \n). Additionally, it contains a trailing 0x0A.

Bottom is the message as received by SQS, obtained by adding a logger to the SQS client, and having it save to a file. The second byte is 0x20 (aka, space). It is also missing the trailing 0x0A. Calculating the hash of each file yields different results, of course, and the actual API response from SQS matches the hash of the top file.

mike@Mikescomp:~/code/project/service$ md5sum testbody2.json
d0b8560f261410878a68bbe070d81853  testbody2.json
mike@Mikescomp:~/code/project/service$ md5sum testbody.json
503d800db75e57a2e670034f30621032  testbody.json

[axiac]

It trims the space characters from the values during XML parsing.

The cause is in Aws_query.ts:

      const parsedObj = xmlParse(encoded, {
        attributeNamePrefix: "",
        ignoreAttributes: false,
        parseNodeValue: false,
        tagValueProcessor: (val, tagName) => decodeEscapedXML(val),
      });

It needs the option trimValues: true. The default value of trimValues is false.

Unfortunately, it seems that adding this option breaks the subsequent code that thinks there isn't any message in the response. :-(

[mmatheson]

Is there a workaround for this? In cases where the SQS queue is populated by an SNS topic we don't have control over the message structure, and there will be newlines in the message written by SNS (example message here).

[matwerber1]

Seeing same issue. Looks like a regression?

Works fine with SDK version 2.868.0, but having same issue as above when trying with new SDK v3.

const AWS = require("aws-sdk"); // v 2.868.0
const sqs = new AWS.SQS({region: 'us-west-2'});

sqs.receiveMessage({
    QueueUrl: "https://sqs.us-west-2.amazonaws.com/.....",
    WaitTimeSeconds: 20,
}, (err,data) => {
    console.log(err);
    console.log(data);
});

Response:

{
  ResponseMetadata: { RequestId: 'bab08679-6f3a-5168-a0ab-dea8a2fcc9d4' },
  Messages: [
    {
      MessageId: 'b6015f49-05c7-47e2-bbec-27b53d9bf31d',
      ReceiptHandle: 'xxxxx',
      MD5OfBody: '31d3a8830dfb4dd8009700b6a2c379d5',
      Body: '{\n    "hello": "world"\n}'
    }
  ]
}

[aduca98]

Any updates on this issue? I have been struggling for a day in a half with this issue and stumbled upon this thread and this is the exact issue I am having as well.

[redvivi]

Same here.

[blumamir]

Us too. After upgrading sdk from v2.780.0 to v3.8.0, the log is full of "Invalid MD5 checksum on messages" records, and the dead letter queue is stuffed with unhandled sqs messages.

This bug prevents us from using v3, as it effectively means that sqs client is not reliably able to receive messages.

[trivikr]

I'm not able to reproduce this issue while explicitly sending message with \n in them.

$ node -v
v14.15.4

package.json

"dependencies": {
  "@aws-sdk/client-sqs": "3.10.0",
  "md5": "2.3.0"
}

Code

import { SQS } from "@aws-sdk/client-sqs";
import md5 from "md5";

const region = "us-west-2";
const client = new SQS({ region });

const QueueName = `test-${Date.now()}`;
const { QueueUrl } = await client.createQueue({ QueueName });

const MessageBody = `{\n  "hello": "world"\n}`;
console.log(MessageBody);
const { MD5OfMessageBody } = await client.sendMessage({
  MessageBody,
  QueueUrl,
});

console.log(
  `\nmd5 check for sent message: ${md5(MessageBody) === MD5OfMessageBody}`
);

const response = await client.receiveMessage({ QueueUrl });
console.log(
  `md5 check for received message: ${
    md5(MessageBody) === md5(response.Messages[0].Body)
  }`
);
console.log(
  `\nStrict Equality Comparison: ${MessageBody === response.Messages[0].Body}`
);

await client.deleteQueue({ QueueUrl });

Output

$ node sqs.mjs
{
  "hello": "world"
}

md5 check for sent message: true
md5 check for received message: true

Strict Equality Comparison: true

Does this minimal repro code need any change to reproduce the issue?
@pkmnfrk @axiac @mmatheson @matwerber1 @aduca98 @blumamir

[trivikr]

The issue is reproducible if message begins with \n as follows:

$ node -v
v14.15.4

package.json

"dependencies": {
  "@aws-sdk/client-sqs": "3.10.0",
  "md5": "2.3.0"
}

Code

import { SQS } from "@aws-sdk/client-sqs";
import md5 from "md5";

const region = "us-west-2";
const client = new SQS({ region });

const QueueName = `test-${Date.now()}`;
const { QueueUrl } = await client.createQueue({ QueueName });

const MessageBody = `\n{"hello": "world"}`;
console.log(MessageBody);
const { MD5OfMessageBody } = await client.sendMessage({
  MessageBody,
  QueueUrl,
});

console.log(
  `\nmd5 check for sent message: ${md5(MessageBody) === MD5OfMessageBody}`
);

const response = await client.receiveMessage({ QueueUrl });
console.log(
  `md5 check for received message: ${
    md5(MessageBody) === md5(response.Messages[0].Body)
  }`
);
console.log(
  `\nStrict Equality Comparison: ${MessageBody === response.Messages[0].Body}`
);

await client.deleteQueue({ QueueUrl });

Output

$ node sqs.mjs

{"hello": "world"}

md5 check for sent message: true
/Users/trivikr/workspace/test/node_modules/@aws-sdk/middleware-sdk-sqs/dist/cjs/receive-message.js:21
            throw new Error("Invalid MD5 checksum on messages: " + messageIds.join(", "));
                  ^

Error: Invalid MD5 checksum on messages: d1ddc9da-4602-430b-9d15-82b6bcc4a7f0
    at /Users/trivikr/workspace/test/node_modules/@aws-sdk/middleware-sdk-sqs/dist/cjs/receive-message.js:21:19
    at processTicksAndRejections (internal/process/task_queues.js:93:5)
    at async /Users/trivikr/workspace/test/node_modules/@aws-sdk/middleware-logger/dist/cjs/loggerMiddleware.js:6:22
    at async file:///Users/trivikr/workspace/test/sqs.mjs:21:18

Verified that this issue doesn't reproduce with aws-sdk@2.875.0