Amazon.Lambda.Serialization.SystemTextJson.LambdaJsonSerializer uses different property casing than Amazon.Lambda.Serialization.Json.JsonSerializer

[bjorg]

It appears that the default casing behavior has changed between Amazon.Lambda.Serialization.Json.JsonSerializer and Amazon.Lambda.Serialization.SystemTextJson.LambdaJsonSerializer.

Here is some sample code to test the difference:

// create an instance to serialize
var record = new Record {
    Foo = "Hello world!"
};

// show serialization with original Lambda serializer based on Newtonsoft.Json
var oldSerializer = SerializeWith(record, new Amazon.Lambda.Serialization.Json.JsonSerializer());
Console.WriteLine($"Amazon.Lambda.Serialization.Json.JsonSerializer: {oldSerializer}");

// show serialization with new Lambda serializer based on System.Text.Json
var newSerializer = SerializeWith(record, new Amazon.Lambda.Serialization.SystemTextJson.LambdaJsonSerializer());
Console.WriteLine($"Amazon.Lambda.Serialization.SystemTextJson.LambdaJsonSerializer: {newSerializer}");

// show serialization with System.Json.Text
var jsonTextSerializer = System.Text.Json.JsonSerializer.Serialize<Record>(record);
Console.WriteLine($"System.Text.Json.JsonSerializer: {jsonTextSerializer}");

// local functions
string SerializeWith<T>(T value, Amazon.Lambda.Core.ILambdaSerializer serializer) {
    using var buffer = new MemoryStream();
    serializer.Serialize<T>(value, buffer);;
    return System.Text.Encoding.UTF8.GetString(buffer.ToArray());
}

The above code produces the following output:

Amazon.Lambda.Serialization.Json.JsonSerializer: {"Foo":"Hello world!"}
Amazon.Lambda.Serialization.SystemTextJson.LambdaJsonSerializer: {"foo":"Hello world!"}
System.Text.Json.JsonSerializer: {"Foo":"Hello world!"}

[normj]

Agreed I should not have switched the casing between the 2 libraries. I think we are lacking tests for custom request and responses as the tests now mostly focus on AWS events.

Now that this has shipped changing the default behavior is really unfeasible. My suggestion is to add a new constructor that takes in an enum for casing style so you can declare the casing you want to use. I could then update the templates to use the new constructor. How do you feel about that work around?

[bjorg]

I don't know what the idea here was. I understand that you don't want to break folks who may have taken a dependency. It seems the mistake was to believe that AWS has consistency on naming of JSON fields (i.e. AWSNamingPolicy). It does not. Some services use Pascal-casing, such as CloudFormation:
https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/crpg-ref-responses.html

Automagically changing casing and not respecting the default System.Text.Json behavior is a fatal flaw, IMHO. Maybe consider releasing a Amazon.Lambda.Serialization.SystemTextJson.LambdaJsonSerializerV2 and put the old one on ice.

[bjorg]

To clarify, as I'm not familiar with how this works, the assembly attribute declaration is only used for deserialization, correct?

[assembly: LambdaSerializer(typeof(Amazon.Lambda.SystemTextJson.LambdaJsonSerializer))]

However, is it even needed if I use this handler signature?

Task<Stream> FunctionHandlerAsync(Stream stream, ILambdaContext context)

What happens if there is no LambdaSerializerAttribute declaration for the assembly or entry-point method?

[3GDXC]

@normj an option may be to add attributes to explicitly name the json properties and therefore respect the naming regardless of casing

[martincostello]

I'd agree with that suggestion - it's tedious one-off work to add them, but then they're always correct.

[3GDXC]

@normj IMHO this is an implementation issue that needs/should be address longer term as a BUG as this is a BREAKING CHANGE that was is unintentional

[normj]

@3GDXC I agree it is an implementation bug. Just thinking on the solution. Currently in the package we have one serializer called LambdaJsonSerializer. What if we add PascalCaseLambdaJsonSerializer and CamelCaseLambdaJsonSerializerserializer which extend from LambdaJsonSerializer. I could change the templates to use PascalCaseLambdaJsonSerializer to keep the existing behavior. It is sort of a more explicit version of @bjorg suggestion of having a LambdaJsonSerializerV2.

[3GDXC]

@normj IMHO it would be better an avoid confusion to add the JsonPropertyName attributes to the messages so that regardless of serializer configuration/options the resulting Json adhered to the attribute naming.

I fully understand why your reluctant to introduce a further breaking change (with the correction) within a matter of days from release of the .NET 3.1 support; and if we (royal we) had a number of UP-FOR-GRABS issues no unit/regression testing the community could assist with these to make implementation evolve and test prior to release.

Happy to assist if/when needed just say the word.

great work thus far, loving seeing aws lamba and .net core support grow

[normj]

@3GDXC Keep in mind the casing only affects return objects where we have to go from .NET object to JSON string. In the few response objects we vend I do use the JsonPropertyName, here is an example: https://github.com/aws/aws-lambda-dotnet/blob/master/Libraries/src/Amazon.Lambda.APIGatewayEvents/APIGatewayProxyResponse.cs#L18

The problem is for response objects that other people create where I don't control whether they use the JsonPropertyName attribute or not.

[3GDXC]

@normj good point; may be the advisory should also include ALWAYS use JsonPropertyName attributes to enforce explicit naming of your properties and/or data contracts (best practices)

[3GDXC]

@normj an alternative may be to abstract away the JsonSerializerOptions into a LambdaSerializerOptions class and add the options as a constructor parameter in the attribute so the serializer can have custom options that developer can override on an assembly/method level

[bjorg]

What about flagging it as a regression and fixing it as a breaking change now instead of spreading more harm? I call it harm because LambdaJsonSerializer, which is based on System.Text.Json, does not respect the default behavior of how to serialize properties. Of course using [JsonPropertyName] fixes it, but requiring everyone to do something to counter an unwanted behavior seems heavy handed.

How many people will continue to encounter this as ?!?!? moment as they adopt LambdaJsonSerializer as their standard serializer?

[panodk]

Hi there I'm encountering this problem in Step Functions after switching the lambda Tasks over to .Net 3.1 + the new serializer. It's wreaking havoc because the output now is in camelcase so the next state machine shape tries to evaluate the new JSON using the Amazon States Language and throws a Step Function exception.

There is a a kludgy workaround for the moment. By setting the LAMBDA_NET_SERIALIZER_DEBUG=true in the environment variable, the _.options is never set in the serializer causing the case to be returned untouched. I'm not sure if that will result in other repercussions other than additional JSON being emitted into the Cloudwatch Logs.
https://github.com/aws/aws-lambda-dotnet/blob/master/Libraries/src/Amazon.Lambda.Serialization.SystemTextJson/LambdaJsonSerializer.cs#L69-L90

IMO, it would be a pain to decorate all of our models with the [JsonPropertyName] decoration as our models are buried in a number of nuget libraries. Ideally I'd like the default behavior to return to the original PascalCasing as before but I'm fine with using an explicit PascalCaseLambdaJsonSerializer with the lambdas when it is called in our Step Function project.

Thanks!

[bjorg]

I'm pretty sure that kludgy workaround is a bug.

Good point on data-structures defined by upstream assemblies.