Refactor code to prepare support for NDJSON and CSV

[juchom]

Pull Request

The main idea of this pull request is to cleanup and refactor some code in order to easily add support for CSV and NDJSON.

The only format supported is still JSON. I have renamed the methods according to the issue.

Batches methods were a bit akward and difficult to read / understand, so I removed the BatchOperationAsync method and added an EnumerableExtensions class which gives the Chunk.

Documents Add / Update unit tests are now Theory so we can easily extends them for CSV and NDJSON.

Documentation has been updated.

@curquiza let me know if you need help to review the code.

What does this PR do?

First step for #182

PR checklist

Please check if your PR fulfills the following requirements:

• Does this PR fix an existing issue?
• Have you read the contributing guidelines?
• Have you made sure that the title is accurate and descriptive of the changes?

Thank you so much for contributing to MeiliSearch!

[juchom]

I was digging a bit further to add NDJSON and CSV support and I found something that needs to be discussed a bit.

I think there is an issue with the JsonSerializerOptions here :

meilisearch-dotnet/src/Meilisearch/MeilisearchClient.cs

Lines 22 to 26 in 8d63ed4

	public static readonly JsonSerializerOptions JsonSerializerOptions = new JsonSerializerOptions
	{
	DefaultIgnoreCondition = JsonIgnoreCondition.WhenWritingNull,
	PropertyNamingPolicy = JsonNamingPolicy.CamelCase,
	};

And this code

meilisearch-dotnet/src/Meilisearch/ObjectExtensions.cs

Lines 49 to 63 in 8d63ed4

	public static object RemoveNullValues<T>(this IEnumerable<T> objectToTransform)
	{
	var result = objectToTransform.Select(item =>
	{
	dynamic expando = new ExpandoObject();
	var x = expando as IDictionary<string, object>;
	foreach (var p in item.GetType().GetProperties().Where(p => p.GetValue(item) != null))
	{
	x[p.Name.ToLower()] = p.GetValue(item, null);
	}
	return expando;
	}).ToList().AsEnumerable();
	return result;
	}

Used here

meilisearch-dotnet/src/Meilisearch/Index.cs

Line 212 in 8d63ed4

var filteredDocuments = documents.RemoveNullValues();

First issue is the custom function RemoveNullValues<T> is doing the same as the JsonSerializerOptions DefaultIgnoreCondition = JsonIgnoreCondition.WhenWritingNull, duplicate code that can be removed easy one.

Second issue which is more serious, removing null properties is a bad decision.

Here is why :

Imagine we have this object we want to add to Meilisearch

public class UserAccount
{
    public int Id { get; set; }
    public string Name { get; set; }
    public bool? HasAcceptedLicense { get; set; }
}

When we add the object the first time we have the following values

Id = 1
Name = John
HasAcceptedLicense = true

A few days later, you change the license and require the user to accept or refuse it and you want to set HasAcceptedLicense to null which means "the user has not said if he accepts or refuses the new license".

You won't be able to do that with the current JsonSerializerOptions.

Last important thing is CSV is not as flexible as JSON when serializing, there is no optional field, the schema is the same for all records.

With the following records and actual options

Id = 1
Name = John
HasAcceptedLicense = true

Id = 2
Name = Jane
HasAcceptedLicense = null

The Jane record would not send the HasAcceptedLicense to Meilisearch with JSON or NDJSON, but it CSV it would send it.

To conclude

The best option would be to remove the custom RemoveNullValues<T> and DefaultIgnoreCondition = JsonIgnoreCondition.WhenWritingNull this would avoid the issues above.

If you're ok with that, I can add a small commit to this PR with the changes.

[juchom]

Before working on this PR, I suggest that we first finish the work to support Meilisearch 0.25.

Then, I will take some changes from this big PR and create smaller ones to prepare the support for NDJSON and CSV. It will be easier to review.

[curquiza]

Before working on this PR, I suggest that we first finish the work to support Meilisearch 0.25.

Completely agreed!