Break datafusion crate into smaller crates

[jimexist]

Is your feature request related to a problem or challenge? Please describe what you are trying to do.

a new feature: to break datafusion crate into separate smaller crates.

It helps with code management and dependency reasoning

Describe the solution you'd like

• Create new datafusion_common crate #1752
  • [split/1] split datafusion-common module #1751
  • [split/2] move column and dfschema to datafusion-common module #1758
  • [split/4] move scalar value to datafusion-common #1760
• Create new datafusion_expr crate #1753
  • [split/3] split datafusion expr module and move aggregate and window function expr #1759
  • [split/5] move window frame and operator to datafusion-expr module #1761
  • [split/6] move signature, type signature, volatility to datafusion-expr #1763
  • [split/7] move built-in scalar function to datafusion-expr #1764
  • [split/8] move Accumulator and ColumnarValue to datafusion-expr #1765
  • [split/9+12] move udf, udaf, Expr to datafusion-expr module #1762
  • [split/10] split up expr for rewriting, visiting, and simplification traits #1774
  • [split/11] split expr type and null info to be expr-schemable #1784
  • [split/9+12] move udf, udaf, Expr to datafusion-expr module #1762
  • [split/13] move rest of expr to expr_fn in datafusion-expr module #1794

---

• split datafusion-physical-expr module #1844
  • [split/14] create datafusion-physical-expr module #1843
  • [split/15] move built in window expr and partition evaluator #1865
  • [split/16] move physical plan expressions folder to datafusion-physical-expr crate #1889
  • [split/17] move the rest of physical expr to datafusion-physical-expr crate #1892
• split datafusion-datasource module #1772
  • file format
  • listing and object store
• split datafusion-logical-plan sub-module #1755
  • move logical plan builder
  • move logical plan structs
  • move move optimizers
• split datafusion-physical-plan sub-module #1754
  • move physical planner
  • move execution exprs
  • move physical plan optimizer
• split datafusion-execution sub-module
  • move memory and disk manager
  • move stats collection

Describe alternatives you've considered
A clear and concise description of any alternative solutions or features you've considered.

Additional context
Add any other context or screenshots about the feature request here.

[jimexist]

cc @alamb @houqp @Dandandan what do you think?

[alamb]

I like this idea @jimexist 👍 There is some related commentary / information here: #348 as well

Some other crates that might be useful to consider;

• datafusion_core (DataFusionError, DFSchema, etc)
• datafusion_datasource (the built in parquet, avro, csv and json readers and supporting logic)

[andygrove]

This will be helpful for use cases where users are looking to use DataFusion in a similar fashion to Calcite, for query parsing and planning, but not for execution. I like this idea.

[houqp]

Thank you @jimexist for taking on this! I think this is the right path forward.

cc @jorgecarleitao and @yjshen since they have proposed the similar ideas before.

[yahoNanJing]

Hi @jimexist, is it possible to make an exclusive crate for the data source so that it will be easy for the integration of the different kinds of remote object store?

Suppose I hope to introduce the hdfs as one of the remote object store. To be less intrusive, it's better to make an independent datafusion-objectstore-hdfs crate which depends on the datasource crate of datafusion. Then it would be much easier for other crates to decide whether to depend on the datafusion-hdfs crate or not without cyclic dependencies.

[jimexist]

Hi @jimexist, is it possible to make an exclusive crate for the data source so that it will be easy for the integration of the different kinds of remote object store?

Suppose I hope to introduce the hdfs as one of the remote object store. To be less intrusive, it's better to make an independent datafusion-objectstore-hdfs crate which depends on the datasource crate of datafusion. Then it would be much easier for other crates to decide whether to depend on the datafusion-hdfs crate or not without cyclic dependencies.

yes it's a good idea and i believe it's included in the list already

[yahoNanJing]

Thanks @jimexist

[jimexist]

my current plan is to finish items for:

• datafusion-common
• datafusion-expr

before release 7:

• Release of DataFusion: 7.0.0 #1587

and finish the rest after the release, to cap the amount of changes in a release

[Igosuki]

@jimexist could this lead to a smaller ballista client crate as well ? Potentially, this could greatly speed up compilation of programs who just want to be datafusion clients and not run the entire stack.

[jimexist]

Contributor

i'm not sure as of now - but probably if we split up logical/physical planning further - but that won't happen soon

[alamb]

Related comment: #1762 (comment)

[jimexist]

i wanted to continue iterating on this split but obviously i didn't have time to keep up. a lot has happened in the last few months. i wonder if this is still relevant or it shall be closed now. @alamb and @andygrove any suggestions?

[alamb]

@jimexist welcome back!

I would say that the main "break datafusion into smaller crates" has been completed. There are still some items in the description of this ticket that might not be done - I would personally recommend filing a new issue with any items that you think would still be good to work on and then closing this one.

[alamb]

Closing this ticket and we can track further splitting in follow on issues. Again 👏

[alamb]

I am thinking of trying to push us to the next level here #4181