Add DAT tests for Rust

[wjones127]

Description

Data Acceptance tests should be done for the Rust Arrow reader.

See: https://github.com/delta-incubator/dat

Use Case

Related Issue(s)

[microcassidy]

@wjones127 is this still current? (If so) how would you like to see this completed?

[wjones127]

Yes, this is still current. We have these for the Python bindings, but not for Rust.

For reference, here are the scripts for the Python tests:

• Download script
• Test script

So we'd want something parallel to that. This could be based on the load operation, or perhaps the DataFusion TableProvider.

It might be a bit challenging to write in Rust, since the tests need to be generated based on the contents of a directory. You can see the approach delta-kernel-rs took here: https://github.com/delta-incubator/delta-kernel-rs/blob/main/tests/dat_reader.rs

[microcassidy]

@wjones127 I have been playing around with it and it looks like a way around it might be to infer the folders from the version of the input delta table. The test could be put into a macro maybe?

#[macro_export]
macro_rules! eg_dat_test {
    ($( $test_name:ident $test:literal),*) => {
        $(
            #[tokio::test]
            async fn $test_name()->TestResult{
                //some body
                let root = Path::new("someroot");
                let path = root.join(Path::new($test));
                //etc etc....
                Ok(())
            }
        )*
    }
}

eg_dat_test!(
    test_foo "bar",
    test_ram "ram"
);

[wjones127]

Yes something like that may work. Feel free to create a draft PR for discussion.

[microcassidy]

Cool. I have put in a draft. There are some parts missing but the idea seems to be working mostly