feat: Support test spark runner in `datafusion-spark` for slt tests

[comphead]

The datafusion-spark crate is evolving rapidly and its functionallity covered by integration tests, however it would be awesome to have a local Spark runner which can verify datafusion-spark functions results.

Currently the similar exists in Comet when all the spark internal tests run on the Comet codebase, but this approach barely transferrable.

Would be good to have a separate CI or tool to run local Spark and check datafusion-spark functions and Spark ones returns the same result.

Originally posted by @comphead in #17032 (review)

[alamb]

I think @shehabgamin and @linhr discussed this as part of the datafusion spark crate discussion, but we felt the challenge of doing spark integration testing was likely outside the scope of the main DataFusion repo

[comphead]

Thanks @alamb this is true, the entire datafusion-spark crate is more niche I would say. I still think it can be a subproject under datafusion, datafusion-spark or similar.

Currently to ensure that Spark builtin functions in crate corresponds to Spark it is required to run tests in local Spark to verify the correctness, WDYT if we can introduce some CI or script to run tests using Spark inside the scope of this crate?

[alamb]

Thanks @alamb this is true, the entire datafusion-spark crate is more niche I would say. I still think it can be a subproject under datafusion, datafusion-spark or similar.

Currently to ensure that Spark builtin functions in crate corresponds to Spark it is required to run tests in local Spark to verify the correctness, WDYT if we can introduce some CI or script to run tests using Spark inside the scope of this crate?

I suspect it it would be not a good idea to add an integration test here that actually runs spark, unless it can be integrated with an existing runner (similarly to the sqllogictests runner for postgres) 🤔

[comphead]

Thanks @alamb this is true, the entire datafusion-spark crate is more niche I would say. I still think it can be a subproject under datafusion, datafusion-spark or similar.
Currently to ensure that Spark builtin functions in crate corresponds to Spark it is required to run tests in local Spark to verify the correctness, WDYT if we can introduce some CI or script to run tests using Spark inside the scope of this crate?

I suspect it it would be not a good idea to add an integration test here that actually runs spark, unless it can be integrated with an existing runner (similarly to the sqllogictests runner for postgres) 🤔

Exactly, the postgres runner was an inspiration, planning to investigate if Spark runner can be done in a similar way.

Ideally to have a separate CI which triggers only if datafusion-spark crate changed. But stepping back and look at that again the submodule more reminds a separate subproject(its not a core, niche usage, separate CI, test runner). We can keep it evolving in DF repo but later it would probably make sense to move it?

[alamb]

Ideally to have a separate CI which triggers only if datafusion-spark crate changed. But stepping back and look at that again the submodule more reminds a separate subproject(its not a core, niche usage, separate CI, test runner). We can keep it evolving in DF repo but later it would probably make sense to move it?

I think in the long term it would make sense to consider moving datafusion-spark out of the main datafusion repo, but I think having it in the main repo initially will reduce the friction / get more eyes on it

[alamb]

I linked this issue to an epic where there is more discussion of this issue:

• [EPIC] Complete datafusion-spark Spark Compatible Functions #15914

[linhr]

I brought up a similar idea in #15914 (comment).

The idea was that we could use the pyspark library to generate SLT files and we only manually define the inputs. The SLT files will be validated in CI when they change to ensure that the output actually matches the Spark behavior.

In general, I agree that some CI setup (triggered only when needed) leveraging the original Spark implementation would be helpful to ensure the correctness of test suites for datafusion-spark.