We just had the issue https://github.com/aws/aws-sdk-js-v3/issues/4576 and it was recently re-opened with no comment.

[JeremieDoctrine]

We just had the issue #4576 and it was recently re-opened with no comment. We had this code running in production for more than a year with no changed. The timing is a bit weird.

Any reason it was re-opened 16 hours ago and we had the issue at the moment it was re-opened?

Thanks. @RanVaknin maybe you know more about this since you re-opened it.

[RanVaknin]

Hi @JeremieDoctrine ,

Thanks for reaching out.
The reason it was re-opened was because the customer who opened the ticket said the issue was never solved and this was brought to our attention from other reports about this problem.

The gist is that the Service is sending back a response that is not JSON causing a deserialization error.
Since the SDKs are auto-generated from the API models of the various AWS services, they adhere to the wire protocols that those services specify.

For example, S3 uses XML based wire protocol. If it sends back something other than XML, the SDK will not know how to deserialize the response and will raise a similar error.
In #4576 's case, SSM was sending back something that is not JSON.

There was an internal escalation about this, but I dont have any new information. That's why it was re-opened without any additional info.

Are you saying you started seeing this error shortly after the ticket was re-opened? Is it also related to SSM? That is indeed weird. I know that the SSM service team is rolling out a fix for this, so Im not sure if this is a coincidence.

Will keep you posted.

Thanks,
Ran~

[JeremieDoctrine]

OK thanks for the crystal clear answer.

Are you saying you started seeing this error shortly after the ticket was re-opened?

Yes that's what i'm saying. The error poped a few times few hours after the ticket was opened, with no changes on our side. And stopped after half an hour.

Is it also related to SSM?

Nope, it's related to Kinesis.

That is indeed weird.

Indeed that's why I took the liberty to randomly ping you... sorry for that.

We've added some more logs on our side but since we can't reproduce, we are not 100% sure we will catch the actual issue.

[RanVaknin]

Hi @JeremieDoctrine ,

Thanks for the additional details. #4576 was an issue was specific to SSM, as different services use different wire protocols. Since you are experiencing this with Kinesis this tells me that its just a coincidence.

Im going to ask you to open a separate bug report. In order to investigate this further, we would need a minimal code snippet that can reliably reproduce this on our end. If you can also fill in the rest of the intake form that will be extremely helpful.

EDIT: re-reading your last comment made me realize that this isnt reproducible on your end. I have seen reports about this before happening inconsistently in a very very low cadence because of a networking error. The SDK received a network error instead of a properly formulated response from the service causing a similar parsing issue.
My guess this was just a red herring.

Since it's not reproducible, and the SDK team does not have any access to any data on your end, the best course of action would be to keep notes of the failing requests, mainly logging the raw response (before its being deserialized) either with the $metadata option discussed in a #4576 , or by using a middleware.
That raw response will contain the requestID, which can be used to file a ticket with the Kinesis service to investigate this on their end.

Thanks again,
Ran~

[JeremieDoctrine]

Ok thanks so much! That's what we've done now. Will keep you posted.

[akeating]

I've see this error occasionally. As #4576 points-out, it's not handled gracefully by the SDK. Here's a test to reproduce with lambda:

import { LambdaClient, InvokeCommand, InvokeCommandInput } from '@aws-sdk/client-lambda';
import { HttpResponse } from '@aws-sdk/protocol-http';
import { describe, test, expect } from '@jest/globals';

describe('Lambda service', () => {
  // The sdk can throw non ServiceException errors
  // and these require special handling
  // https://github.com/aws/aws-sdk-js-v3/issues/4576
  describe('the attempt to invoke returns a SyntaxError', () => {
    test('unexpected html received', async () => {
      const client = new LambdaClient({
        requestHandler: new (class {
          async handle() {
            return {
              response: new HttpResponse({
                statusCode: 503,
                headers: {},
                body: Buffer.from('<html>Some unexpected html</html>'),
              }),
            };
          }
        })(),
      });

      const params: InvokeCommandInput = {
        FunctionName: 'test',
        InvocationType: 'RequestResponse',
        LogType: 'None',
        Payload: Uint8Array.from(Buffer.from(JSON.stringify({ foo: 'bar' }))),
      };

      const result = await client.send(new InvokeCommand(params)).catch((_) => _);
      expect(result).toBeInstanceOf(SyntaxError);
      const syntaxError: SyntaxError = result as SyntaxError;
      expect(syntaxError.message).toEqual(
        'Unexpected token \'<\', "<html>Some"... is not valid JSON\n' +
          '  Deserialization error: to see the raw response, inspect the hidden field {error}.$response on this object.',
      );
      // The sdk adds a hidden field to the error object
      // https://github.com/aws/aws-sdk-js-v3/pull/4592/files#diff-8700e9d6ea5f0b48b76a520f0138954d8ada56d2abb3692c49b95e0df01ce3c5
      expect(syntaxError['$response']).toBeInstanceOf(HttpResponse);
      const httpResponse: HttpResponse = syntaxError['$response'] as HttpResponse;
      expect(httpResponse.statusCode).toEqual(503);
      const body = await httpResponse.body;
      expect(body).toEqual(Buffer.from('<html>Some unexpected html</html>'));
    });
  });
});

[github-actions[bot]]

Hello! Reopening this discussion to make it searchable.