split datafusion-physical-plan sub-module

[jimexist]

Is your feature request related to a problem or challenge? Please describe what you are trying to do.
A clear and concise description of what the problem is. Ex. I'm always frustrated when [...]
(This section helps Arrow developers understand the context and why for this feature, in addition to the what)

Describe the solution you'd like
A clear and concise description of what you want to happen.

Describe alternatives you've considered
A clear and concise description of any alternative solutions or features you've considered.

Additional context
Add any other context or screenshots about the feature request here.

[alamb]

My eventual plan is to have a dependency hierarchy like:

core --> (physical_plan) --> (datasource) --> (execution)    

I am trying to extract execution first, then I will extract datasource, and then physical plan

[alamb]

End state:

• SessionContext and SessionState stay in datafusion-core
• RuntimeEnv and TaskContext are moved into datafusion-execution

Specific dependencies I need to do this:

• move object registries and table_factories off RuntimeEnv and into SessionState directly (this is so they can be populated from the SessionConfig constructor instead) and RuntimeEnv can move into datafusion-execution
• move TaskContext into datafusion_execution
• remove SessionState reference from file_format (replace with TaskContext)
• Move ExecutionProps on to TaskContext (needed for planning)
• remove SessionState reference from TableProvider (replace with TaskContext)
• Move datasource into its own crate
• move physical plan into its own crate

[alamb]

Update 2023-03-27:

I have moved TaskContext into datafusion-execution 🎉 and updated the list in #1754 (comment)

To move datasource into its own crate I need to do the following two non trivial API changes (added to the list above)

• remove SessionState reference from file_format (replace with TaskContext)
• remove SessionState reference from TableProvider (replace with TaskContext)

Here are the file_format uses (I don't think this will be major)

    async fn infer_stats(
        &self,
        _state: &SessionState,
        _store: &Arc<dyn ObjectStore>,
        _table_schema: SchemaRef,
        _object: &ObjectMeta,
    ) -> Result<Statistics> {
        Ok(Statistics::default())
    }

The TableProvider will be major I think:

    /// Create an ExecutionPlan that will scan the table.
    /// The table provider will be usually responsible of grouping
    /// the source data into partitions that can be efficiently
    /// parallelized or distributed.
    async fn scan(
        &self,
        state: &SessionState,
        projection: Option<&Vec<usize>>,
        filters: &[Expr],
        // limit can be used to reduce the amount scanned
        // from the datasource as a performance optimization.
        // If set, it contains the amount of rows needed by the `LogicalPlan`,
        // The datasource should return *at least* this number of rows if available.
        limit: Option<usize>,
    ) -> Result<Arc<dyn ExecutionPlan>>;

[alamb]

I am actively working on this -- after a few tries my current plan is to to end up with this dependency diagram (aka datasource and listing table need to depend on physical_plan I think)

More PR to come

(core) --> (datasource) --> (physical_plan)
    \---------(physical_optimizer) --/

[alamb]

I tried extracting physical_plan again after #6516 and here is the list of things I needed to do (which can be done as smaller PRs):

• Move physical planner out of physical_plan and into datafusion/core/src/physical_planner: Move PhysicalPlanner to physical_planer module #6570
• MoveStringifiedPlan, ToStringifiedPlan, PlanType out of logical_plan and into datafusion_common and update uses: Minor: Move PlanType, StringifiedPlan and ToStringifiedPlan datafusion_common  #6571
• Move JoinType out of logical_plan and into datafusion_common (break dependence on logical plan): Move JoinType and JoinCondition to datafusion_common #6572
• Update uses of crate::error to datafusion_common (e.g. use crate::error::{DataFusionError, Result}; to use datafusion_common::{DataFusionError, Result};) : Minor: Remove some use crate:: uses in physical_plan #6573
• Move StreamingTableExec into datasource as well (get datasource out of physical plan)
• Move from_slice out of core (or remove it entirely): Remove FromSlice in favor of From impl in upstream arrow-rs code #6587

Also I need to figure out how to remove all uses of SessionContext in tests (maybe by moving tests to sql integration), but that in theory could come as a follow on PR