-
Notifications
You must be signed in to change notification settings - Fork 762
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
External source with new processor #4277
External source with new processor #4277
Conversation
This pull request is being automatically deployed with Vercel (learn more). 🔍 Inspect: https://vercel.com/databend/databend/BEq46q8qenDYxCcNczTJxqztVq6G [Deployment for afbe5ac canceled] |
Thanks for the contribution! Please review the labels and make any necessary changes. |
7c2289a
to
5d9ffee
Compare
933cc6e
to
f2ef6ac
Compare
TableSource(TableInfo), | ||
|
||
// S3 external source, 's3://'. | ||
S3ExternalSource(S3ExternalTableInfo), |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Why not make s3
as a special table?
Such as a table function that holds the temporary table.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Make s3 as table engine, we can do the unloading as copy from table into s3://
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
It seems not extensible enough, considering we will support azblob
, gcs
, and other locations.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
But table function can also support insert.
copy from table into s3://
select * from s3://
as_table
to generate a temporary table.- Select or insert works with the table.
I do think it's better to introduce external table function
, like:
select * from external(stage_name, ...)
This may works for s3
, azblob
, ... anything stage supports.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I have add a desc for why make it as a table engine.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
BTW, for select * from @stage ...
, we will convert the @stage
alias to the external table info which get from the metasrv(after we support stage) in the sql parser phase, it will be same as select * from s3://
did.
@zhang2014 @sundy-li, PTAL |
I hereby agree to the terms of the CLA available at: https://databend.rs/dev/policies/cla/
Summary
Summary about this PR:
COPY INTO t1 FROM 's3://'
, useread2()
apiCOPY INTO s3:// FROM t1
, useappend_data()
apitruncate()
apiChangelog
Related Issues
Part of #3586
Test Plan
Unit Tests
Stateless Tests