feat: Distributed copy into V1 #11840

JackTan25 · 2023-06-22T16:03:29Z

I hereby agree to the terms of the CLA available at: https://databend.rs/dev/policies/cla/

Summary

Summary about this PR
this pr is the first version for distributed copy into, this just supports 'copy into table from stage',and just support 'csv,tsv' files which are not compressed.As to other occasions will be given in the later pre.
the design details are below:

Closes #issue
#11752 #8594 #6395

vercel · 2023-06-22T16:03:33Z

The latest updates on your projects. Learn more about Vercel for Git ↗︎

1 Ignored Deployment

Name	Status	Preview	Comments	Updated (UTC)
databend	⬜️ Ignored (Inspect)	Visit Preview		Jun 28, 2023 1:39pm

mergify · 2023-06-22T16:04:08Z

This pull request's title is not fulfill the requirements. @JackTan25 please update it 🙏.

Valid format:

fix(query): fix group by string bug
  ^         ^---------------------^
  |         |
  |         +-> Summary in present tense.
  |
  +-------> Type: rfc, feat, fix, refactor, ci, docs, chore

Valid types:

rfc: this PR proposes a new RFC
feat: this PR introduces a new feature to the codebase
fix: this PR patches a bug in codebase
refactor: this PR changes the code base without new features or bugfix
ci: this PR changes build/testing/ci steps
docs: this PR changes the documents or websites
chore: this PR only has small changes that no need to record

JackTan25 · 2023-06-22T16:04:29Z

let me give more tests and test the performance in the real cluster environment.

BohuTANG · 2023-06-22T16:15:58Z

This is great.

But to split one stage file is very complex, hard for parquet file.
How about to distribute by files? The flow like:

COPY from stage has [file1, file2, file3]
Distributed file1 and file2 and file3 to nodes of the cluster

If a COPY only has a single file, don't distributed it by design.

cc @dantengsky @youngsofun

JackTan25 · 2023-06-22T16:20:30Z

@BohuTANG I think split for parquet file is not in my plan for now (but I'm plan to do research on parquet, maybe it will be ok in the future, I don't know), and I think splitting a csv or tsv file is a very good idea, it will improve performance very much, also when it has only one file, if the file is very large, but just simply use only one node to insert, this is bad (for me). By the way, for my implementation, if it's only one file and not too large, it will has only one split, so actually it's not distributed. By the way, if there are multi files like 3 parquet files, it will result 3 splits naturally, so it's distributed for multi files, don't worry.

BohuTANG · 2023-06-22T23:26:26Z

Need some logic tests here to make sure this PR is working as expected.

JackTan25 · 2023-06-23T02:55:32Z

How should I test the all copy into grammar in the real cluster nodes? Is there a config file for me to use Databend's cluster environment to get the performance figure? And there are some bug for now. I will fix it later.

BohuTANG · 2023-06-23T06:35:16Z

For reference, Distributed delete test https://github.com/datafuselabs/databend/pull/11777/files#diff-898c866ebb6ce89a237d7d8f7973779ff8c4f1c0c1d28f4d03939e14f937f479

youngsofun · 2023-06-25T01:09:17Z

distributed copy and splittable are different

CSV is not splittable; NDJson is not splittable(done); TSV has no stands, Hive Text is splittable, Mysql TSV is not(current impl need improve for this)
currently a split is a whole file or part of a file(if splittable); distributed based on splits
a pr for Distributed only need to based on current split implementation

src/query/pipeline/sources/src/input_formats/input_format_text.rs

JackTan25 · 2023-06-25T01:28:08Z

I will revert my split implementation

src/query/pipeline/sources/src/input_formats/input_format_text.rs

youngsofun · 2023-06-25T01:41:29Z

it is not high priority to split of the text file:

many requirements： no compression（ a few exceptions), text file with no enclosing and line terminator escaped
compression is common for text file to save space
big single files are more common log processing with spark/hadoop；big structured data are more commonly stored as parquet；

JackTan25 · 2023-06-25T01:48:30Z

text file with no enclosing

get it, I have revert all my split logic

tests/suites/1_stateful/00_copy/00_0011_dirtributed_copy_from_stage_for_text_file.sh

src/query/sql/src/executor/physical_plan.rs

… into distributed_copy_into

github-actions · 2023-06-27T08:18:25Z

Docker Image for PR

tag: pr-11840-f726d5f5f928afd0858d55190f5375156bbfd42f

… into distributed_copy_into

tests/suites/1_stateful/00_copy/00_0011_distributed_copy_into_table_exection_test.sh

chore: tidy up test scripts

fix: 00_0008_copy_with_projection test failure

fix: incorrect stage name use in 00_0011_distributed_copy

dantengsky

LGTM. distributed copy is disabled by default, let's merge

tests/suites/1_stateful/00_copy/00_0000_copy_from_stage2.py

remove `set enable_distributed_copy_into..` from py test script

* distributed copy into table from stage * delete useless codes * fix zero num_threads * add more files * add order * add more tests * correct test * rename to copy_into_table * reduce test data * fix test file * refactor code style, make it more clear * add explain pipeline for copy stmt * add explain test * remove eprintln * fix explain test result * fix explain test result * add purege * correct explain result * correct explain result * use box to avoid stack overflow * make sure distributed copy test * refactor code style * correct result * remove explain * drop products * modify tests * feat: new setting "enable_distributed_copy_into" * add more tests * add more * correct test * distributed copy into profile: forbidded * fix check * fix clippy * modify test * correct local id * remove error codes * add config to test * modify transform * modify test * fix test * update test * update test * tidy up test scripts * fix 00_0008_copy_with_projection test failure * fix 00_0011_distributed_copy_into_table_exection_test.sh * fix: incorrect stage name use in 00_0011_distributed_copy * remove `set enable_distributed_copy_into..` from py test script --------- Co-authored-by: dantengsky <dantengsky@gmail.com> Co-authored-by: everpcpc <everpcpc@users.noreply.github.com>

JackTan25 changed the title ~~Distributed copy into~~ feat: Distributed copy into Jun 22, 2023

mergify bot added the pr-feature this PR introduces a new feature to the codebase label Jun 22, 2023

JackTan25 changed the title ~~feat: Distributed copy into~~ feat: Distributed copy into V1 Jun 22, 2023

BohuTANG requested review from youngsofun and dantengsky June 22, 2023 23:24

JackTan25 force-pushed the distributed_copy_into branch 3 times, most recently from c1e08c3 to 0a05f2c Compare June 24, 2023 12:31

youngsofun reviewed Jun 25, 2023

View reviewed changes

src/query/pipeline/sources/src/input_formats/input_format_text.rs Outdated Show resolved Hide resolved

youngsofun reviewed Jun 25, 2023

View reviewed changes

src/query/pipeline/sources/src/input_formats/input_format_text.rs Outdated Show resolved Hide resolved

JackTan25 force-pushed the distributed_copy_into branch from 06b0efc to 727b3b3 Compare June 25, 2023 01:44

BohuTANG reviewed Jun 25, 2023

View reviewed changes

tests/suites/1_stateful/00_copy/00_0011_dirtributed_copy_from_stage_for_text_file.sh Outdated Show resolved Hide resolved

BohuTANG reviewed Jun 25, 2023

View reviewed changes

src/query/sql/src/executor/physical_plan.rs Outdated Show resolved Hide resolved

JackTan25 force-pushed the distributed_copy_into branch 2 times, most recently from de15bf7 to b85bd5d Compare June 25, 2023 12:25

JackTan25 added 3 commits June 25, 2023 23:07

distributed copy into table from stage

770514c

delete useless codes

7576941

fix zero num_threads

d2b785f

JackTan25 and others added 4 commits June 27, 2023 12:51

add config to test

d826dc6

modify transform

e25fd98

modify test

5087813

Merge branch 'main' into distributed_copy_into

5eeed88

everpcpc added the ci-cloud Build docker image for cloud test label Jun 27, 2023

JackTan25 added 2 commits June 27, 2023 15:42

fix test

b49b16f

Merge branch 'distributed_copy_into' of github.com:JackTan25/databend…

7fd42e6

… into distributed_copy_into

vercel bot deployed to Preview June 27, 2023 07:43 View deployment

JackTan25 and others added 2 commits June 27, 2023 15:53

update test

b829a59

Merge branch 'main' into distributed_copy_into

0a2ec1e

everpcpc added ci-cloud Build docker image for cloud test and removed ci-cloud Build docker image for cloud test labels Jun 27, 2023

JackTan25 added 2 commits June 27, 2023 16:46

update test

3a2d497

Merge branch 'distributed_copy_into' of github.com:JackTan25/databend…

743f673

… into distributed_copy_into

youngsofun reviewed Jun 28, 2023

View reviewed changes

tests/suites/1_stateful/00_copy/00_0011_distributed_copy_into_table_exection_test.sh Outdated Show resolved Hide resolved

youngsofun approved these changes Jun 28, 2023

View reviewed changes

dantengsky and others added 7 commits June 28, 2023 15:34

tidy up test scripts

b4171a2

Merge pull request #6 from dantengsky/dist-copy

c5a8ccb

chore: tidy up test scripts

fix 00_0008_copy_with_projection test failure

56f72c6

fix 00_0011_distributed_copy_into_table_exection_test.sh

a765029

Merge pull request #7 from dantengsky/dist-copy

34ef4e5

fix: 00_0008_copy_with_projection test failure

fix: incorrect stage name use in 00_0011_distributed_copy

8dd565c

Merge pull request #8 from dantengsky/dist-copy

ca94fe4

fix: incorrect stage name use in 00_0011_distributed_copy

dantengsky approved these changes Jun 28, 2023

View reviewed changes

dantengsky reviewed Jun 28, 2023

View reviewed changes

tests/suites/1_stateful/00_copy/00_0000_copy_from_stage2.py Outdated Show resolved Hide resolved

dantengsky and others added 2 commits June 28, 2023 21:36

remove set enable_distributed_copy_into.. from py test script

f518941

Merge pull request #9 from dantengsky/dist-copy

aaecb54

remove `set enable_distributed_copy_into..` from py test script

dantengsky merged commit 02adab2 into databendlabs:main Jun 28, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat: Distributed copy into V1 #11840

feat: Distributed copy into V1 #11840

JackTan25 commented Jun 22, 2023

vercel bot commented Jun 22, 2023 •

edited

Loading

mergify bot commented Jun 22, 2023

JackTan25 commented Jun 22, 2023

BohuTANG commented Jun 22, 2023

JackTan25 commented Jun 22, 2023 •

edited

Loading

BohuTANG commented Jun 22, 2023

JackTan25 commented Jun 23, 2023

BohuTANG commented Jun 23, 2023

youngsofun commented Jun 25, 2023 •

edited

Loading

JackTan25 commented Jun 25, 2023

youngsofun commented Jun 25, 2023

JackTan25 commented Jun 25, 2023

github-actions bot commented Jun 27, 2023

dantengsky left a comment

feat: Distributed copy into V1 #11840

feat: Distributed copy into V1 #11840

Conversation

JackTan25 commented Jun 22, 2023

Summary

vercel bot commented Jun 22, 2023 • edited Loading

mergify bot commented Jun 22, 2023

JackTan25 commented Jun 22, 2023

BohuTANG commented Jun 22, 2023

JackTan25 commented Jun 22, 2023 • edited Loading

BohuTANG commented Jun 22, 2023

JackTan25 commented Jun 23, 2023

BohuTANG commented Jun 23, 2023

youngsofun commented Jun 25, 2023 • edited Loading

JackTan25 commented Jun 25, 2023

youngsofun commented Jun 25, 2023

JackTan25 commented Jun 25, 2023

github-actions bot commented Jun 27, 2023

Docker Image for PR

dantengsky left a comment

Choose a reason for hiding this comment

vercel bot commented Jun 22, 2023 •

edited

Loading

JackTan25 commented Jun 22, 2023 •

edited

Loading

youngsofun commented Jun 25, 2023 •

edited

Loading