[BUG]: 4.25 breaks binary compatibility with files serialized using 4.24 and De/Serialize<T>

[NeilMacMullen]

Library Version

4.25.0

OS

windows

OS Architecture

64 bit

How to reproduce?

Repro:

1. Build the console application below using Parquet.net 4.24
2. Run vcheck.exe w test_424.parquet
3. Rebuild the application using Parquet.net 4.25
4. Run vcheck.exe w test_425.parquet
5. Run vcheck.exe r test_424.parquet

Note that the following exception is thrown...
Unhandled exception. System.IO.InvalidDataException: property 'TestProperty' is declared as 'TestProperty (System.String?)' but source data has it as 'TestProperty (System.String)'

This appears to because the default for "IsNullable" has changed between versions...

It may be that the new behaviour is more "correct" but it's unfortunate that we have a whole load of files which are suddenly unreadable without dto/code changes :-( It would be desirable if there was a ParquetOptions property that could be set to revert to the old behaviour and allow 'nullable' columns to be mapped to non-nullable properties with an exception being thrown at runtime if a null were ever encountered. (Or even to allow some null-to-value function to be supplied)

Code

using Parquet;
using Parquet.Serialization;

if (args.Length < 2)
{
    Console.WriteLine("wrong number of parameters - use vcheck CMD FILE where CMD is r/w/s");
    return;
}
   
var cmd = args[0];
var file = args[1];

if (cmd.ToLowerInvariant().StartsWith("w"))
{
    Console.WriteLine($"writing {file}...");
    var dataOut = Enumerable.Range(0, 1000).Select(i => new TestClass { TestProperty = $"Test {i}" });
    await ParquetSerializer.SerializeAsync(dataOut, file);
}

if (cmd.ToLowerInvariant().StartsWith("r"))
{
    Console.WriteLine($"reading {file}");
    var dataIn = await ParquetSerializer.DeserializeAsync<TestClass>(file);
    Console.WriteLine("Done");
}

if (cmd.ToLowerInvariant().StartsWith("s"))
{
    using(ParquetReader reader = await ParquetReader.CreateAsync(file))
    {
        Console.WriteLine($"reading schema for {file}");
        var schema = reader.Schema;
        foreach (var f in schema.DataFields)
        {
            Console.WriteLine($"type:{f.ClrType} nullable:{f.IsNullable}");
        }
    }
}

public class TestClass
{
    public string TestProperty { get; set; } = string.Empty;
}

Failing test

No response

[aloneguid]

My bad, this is caused by #537 which I'm trying to revert now.

[NeilMacMullen]

No worries - really great library by the way :-)

[aloneguid]

Please try 5.0.0-pre.2

[ErikApption]

Imo there is no reason this fails with an exception, this should be a warning (you can technically put a null into a string when it it is not marked as nullable) but removing nullable support makes it very unintuitive to load external files. Parquet. Net doesn't let you map a non nullable string column otherwise.

[aloneguid]

I'd like to integrate it back from the #537 PR manually asap, but I'll have to write some test cases covering edge case scenarios first. Rest assured this is not boing to be lost.