PARQUET-355: Add Statistics Test for Parquet Columns

[sircodesalotOfTheRound]

In response to PARQUET-251 created an integration test that generates random values and compares the statistics against the values read from a parquet file.

There are two tools classes DataGenerationContext and RandomValueGenerators which are located in the same package as the unit test. I'm sure there is a better place to put these, but I leave that to your discretion.

Thanks
Reuben

[sircodesalotOfTheRound]

A brief outline of the Code:

(1) There is one test function that is generates random data using a random block / page size.

(2) The DataGenerationContext and the nested WriteContext callback classes use IOC to generate the file writer, write the data and then finally test the code.

(3) The tests each use RandomValueGenerators which generates a random list of values, these values are then written to the file.

(4) On test, each page is read from the file. We use a PagingValidator class which reads the statistics from each of the pages.

(5) The value-generator originally used to write the values is then queried for what it believes ought to the the correct statistics for the entries on this page. These values are then compared against what is actually read from the file.

If all of the statistics match the anticipated values, then the test passes.

[rdblue]

@sircodesalotOfTheRound, thanks for working on this! I like the overall approach of using randomly-generated data, writing an entire file, and validating the pages individually. I had a few comments about how you're doing those tasks, but overall it is a great start. Thanks!

[spena]

I think we should use a constant value as seed so that we can have reproducible tests with the same random values. If we use a dynamic value, then a bug might appear randomly.

[rdblue]

I agree, I just put my comments on the commit rather than the pull request by accident. More comments from me here: sircodesalotOfTheRound@e05447e

[rdblue]

I don't think the validation should call updateStats. Otherwise, you're delegating to the implementation you're trying to test. Instead, the test should be that the min value is less-than or equal-to each value in the page. Similarly, you should count nulls and validate that the number matches up after iterating through the page values. That way we are checking the meaning of the stats values independent of the implementation.

I think I mentioned this before, but it was probably overlooked in a sea of comments.

[rdblue]

This should be UnsupportedOperationException. NotImplementedException comes from sun.reflect

[sircodesalotOfTheRound]

Ah ok. C# Convention, thanks.

[rdblue]

@sircodesalotOfTheRound, can you update the title of this issue to "PARQUET-355: Add..." so that we can merge it? Thanks!

[sircodesalotOfTheRound]

Okay, updated the title. Thanks!