[Feature] load code refactoring

[freemandealer]

Search before asking

• I had searched in the issues and found no similar issues.

Description

Motivation

Doris now supports many load methods. The implementation code of these methods is intertwined and the attributes overlap. To improve maintainability, we plan to refactor the code.

Goals

long-term: unify all data loading methods (brokerload, insert, etc) into one
short-term:

1. unify the attribute specification method and parsing method and default values of various loads;
2. reuse single code base as much as possible and delete duplicate code;

Plan Overview

We plan to use insert stmt as the common backend for all load methods. Various loads retain their original user interface, but will eventually be converted into insert stmt for planning and subsequent execution.

Task Breakdown

Phase1: make S3/Broker load work on insert stmt

1. [Feature] insert stmt maintain information for all loading properties #18477
2. [Feature] run S3 load with insert stmt as its backend #18480
3. [Feature] insert stmt support async job management #18478

Phase2: make all other load method work on insert stmt

1. run streamload with insert stmt planner
2. support mysql load
3. support routine load
   ...

Phase3: fully support all use cases in all load method

1. insert stmt support writing multi-tables in single transaction
2. insert stmt support sequence column v1
   ...

Use case

No response

Related issues

No response

Are you willing to submit PR?

• Yes I am willing to submit a PR!

Code of Conduct

• I agree to follow this project's Code of Conduct

[TangSiyang2001]

May I have a try ?