Combining custom camel case serializer with LINQ

[Liversage]

The SDK supports a custom serializer by deriving from CosmosSerializer but using a serializer that converts idiomatic C# pascal case property names to JSON camel case names break LINQ queries.

Version 3.2.0-preview2 now supports CosmosPropertyNamingPolicy.CamelCase which fixes the camel case problem. Thanks!

However, it is impossible to both have a custom serializer and also a LINQ provider that uses camel case. If I specify a custom serializer then I'm no longer allowed to use CosmosPropertyNamingPolicy.CamelCase which I need to change the behavior of the LINQ provider to match my custom serializer.

Would it be possible to both configure the LINQ provider to generate camel case SQL while also using a custom serializer? That would be awesome.

Related issues: #72, #551, #570.

And just to explain why I need a custom serializer here are some of the reasons:

• I have some code generated types that my serializer is able to serialize based on an attribute so number-like types are serialized as numbers and not strings.
• I have abstract base types and concrete derived types which JSON doesn't support but my serializer is able to deserialize to the correct derived type based on additional light-weight type information in the JSON.
• I have some custom types that I use as dictionary keys which also require extra support from the serializer.

[j82w]

The only issue I see with support both is the ambiguity of when the custom serializer is used vs the settings. Would having an option to set it directly on linq operation be better? That way the local linq setting always override the client level settings.

[Liversage]

A local LINQ setting would also work. It does seem a bit asymmetrical that the custom serializer is configured when the client is created while the camel case LINQ override is provided on each query.

[Liversage]

Here are some additional thoughts on how to handle this issue which may be relevant to users of the Cosmos SDK (people like me, not the developers of the SDK).

I'm building my system in C# and I use the type system to assist me in doing that which includes using inheritance and combatting primitive obsession with dedicated value types.

Unfortunately, there is some impedance mismatch between C# types and JSON. To handle this I have had to customize the JSON serializer and this customization has only grown over time. This has resulted in a heavy dependency on Json.NET.

The most recent release of the Cosmos SDK has good support for camel case/pascal case but then I can no longer use my customized serializer. Essentially I was still stuck at

1. littering [JsonProperty] all over my domain model (no, that doesn't scale at all),
2. writing queries directly using Cosmos SQL (no, LINQ is so much better e.g. for composition), or
3. writing my own LINQ provider (yes, I did that - interesting, but not recommended at all).

The fundamental problem here is that intersection of my domain model, my desire to use LINQ, a requirement that all JSON is camel cased to avoid client side confusion, and the Cosmos SDK is empty.

Also, having a strong dependency on Json.NET has been a concern of mine. I would like to be able to switch to System.Text.Json in the future where it might end up being the "default" serializer with only "legacy" software using Json.NET.

As Cosmos SDK works quite well for "simple" POCO types (especially with the latest addition to support camel case) I realized that I could solve my problem by inserting a "DTO layer" with "JSON friendly" types between my domain model and lower layers like JSON serialization, Cosmos SDK etc. This would remove all the trouble I'm having with using a specific JSON serializer and a specific Cosmos SDK. If I want camel case I can just generate the DTOs with camel case and handle that in my own mapper.

I don't want to write a ton of DTOs by hand so instead I've created a system where "JSON friendly" DTOs are generated at run-time (using Roslyn). I already had a mapper in place to automatically map between domain types and DTOs and my DTO generator expands on specific conventions of my type system. Creating a general library to do this is a major undertaking so I have opted to just support the specific types I use to limit the scope.

To sum up writing to Cosmos involves the following steps:

• Generate the DTO that matches the domain type involved (this involves recursive generation for nested type but generated types are cached thus only generated once).
• Map the domain object to the generated DTO object.
• Hand the DTO object over to the JSON serializer/Cosmos SDK to serialize and write the data to Cosmos.

Reading is more or less the same just going the opposite way. The most tricky part is probably rewriting the "domain layer" LINQ expressions to equivalent "DTO layer" expressions which enables LINQ queries.

I admit that there were far more devils found in the details than I expected when building this but I'm quite happy with the final results.

[nhwilly]

@Liversage Really happy to see someone took the same path as me, albeit with a lot more thought.

My clients are Dart and that means there's all kinds of issues with casing.

I, too ended up with a DTO layer and doing my own mapping. In the end, it's just less voodoo and more predictable. I finally ended up writing a code generator that takes the C# DTO and turns it into a Dart-friendly DTO, which is what I dump back to the client.

I was hoping the new SDK would allow me to get rid of the endless [JsonProperty] stuff everywhere. If I'm going to typo something, it'll be there, for sure.

Thanks for taking the time to post.

[nhwilly]

What really ended up helping me was combining

[JsonObject(NamingStrategyType = typeof(CamelCaseNamingStrategy))]

at the top of my models (instead of every property). And using:

SerializerOptions = new CosmosSerializationOptions
{
    PropertyNamingPolicy = CosmosPropertyNamingPolicy.CamelCase,
},

Now things seem to get serialized properly at all levels.

I also make use of inherited classes and the JsonConverter works.

My parent class is an abstract, custom DocumentEntity. Then I have another abstract parent class that holds the common information and then I have a subclass that is the actual implementation.

DocumentEntity->TemplateBase->EmailTemplate (or PushTemplate or SmsTemplate)

LINQ based queries seem to work fine and I get back a heterogeneous list of objects (that are sub-classed from the abstract class) from a single query.

[derekgreer]

bump

[derekgreer]

When using a custom serializer with camelcase set, we found that we still had to use [JsonProperty] attributes, specifically on the property containing the partition key. When we didn't, calls to UpsertItemAsync() would fail with "Resource with specified id or name already exists" (see Azure/azure-cosmosdb-bulkexecutor-dotnet-getting-started#13). This, of course, is itself pretty wack given and "upsert" should never fail with an error message like "resource already exists" because, well, you know ...

[martijnburgers]

We really need this. It's insane that this is still open.

I tried to work around it using custom type descriptors placing the json property attribute on the properties but the type descriptors aren't used.

[Matthewsre]

I just ran into this problem where the string values would come back correctly, but not the Guid or DateTime values. The where filters on Guid values would still work even though it wouldn't return the values.

I tried adding [JsonObject(NamingStrategyType = typeof(CamelCaseNamingStrategy))] to the models as a workaround, which did not work.

I ended up adding [JsonProperty(PropertyName = "created")] to each of the properties to get it working as a work around, I was trying to avoid using the attributes on my models and ended up needing to add for each property.

[martijnburgers]

I can confirm that [JsonObject(NamingStrategyType = typeof(CamelCaseNamingStrategy))] is not working.

[j82w]

This is should be fixed in 3.17.0 which should be released soon by PR: #2220