Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

feat: impl create dynamic table sql parser and planner #39

Closed
wants to merge 755 commits into from
Closed

feat: impl create dynamic table sql parser and planner #39

wants to merge 755 commits into from

Conversation

zhyass
Copy link

@zhyass zhyass commented Apr 17, 2024

I hereby agree to the terms of the CLA available at: https://docs.databend.com/dev/policies/cla/

Summary

Briefly describe what this PR aims to solve. Include background context that will help reviewers understand the purpose of the PR.

Fixes #[Link the issue here]

Tests

  • Unit Test
  • Logic Test
  • Benchmark Test
  • No Test - Explain why

Type of change

  • Bug Fix (non-breaking change which fixes an issue)
  • New Feature (non-breaking change which adds functionality)
  • Breaking Change (fix or feature that could cause existing functionality not to work as expected)
  • Documentation Update
  • Refactoring
  • Performance Improvement
  • Other (please describe):

drmingdrmer and others added 30 commits March 7, 2024 22:10
@drmingdrmer
refactor: enable openraft feature flag generic-snapshot-data (databen…
965d085 
…dlabs#14881)
@drmingdrmer
refactor: use SettingIdent to build string key to access settings (da…
38e2091 
…tabendlabs#14883)
@zhang2014
feat(query): support queries queue (databendlabs#14878)
0e3b8a6 
* feat(query): support queries queue

* feat(query): support queries queue

* feat(query): support queries queue

* feat(query): support queries queue
@andylokandy
fix: allow insert in multiline statment (databendlabs#14882)
6ed9a60 
* fix: allow insert in multiline statment

* fix

* fix

* fix
@andylokandy
refactor(ast): migrate to derive-visitor (databendlabs#14829)
de30d07 
* refactor(ast): migrate to derive-visitor

* add more

* add more

* add more

* fix

* fix

* fix

* fix

* fix

* fix

* fix

* fix

* fix

* fix

* fix
@PsiACE
fix: rewrite ifnull and nvl as coalesce (databendlabs#14877)
eeaf97c 
* fix: try to make nvl great again

Signed-off-by: Chojan Shang <psiace@apache.org>

* fix: try to make ifnull great again

Signed-off-by: Chojan Shang <psiace@apache.org>

* fix: make sqllogictest happy

Signed-off-by: Chojan Shang <psiace@apache.org>

* fix: workaround for fewer than expected number of null

Signed-off-by: Chojan Shang <psiace@apache.org>

* fix: make sqllogictest happy

Signed-off-by: Chojan Shang <psiace@apache.org>

* fix: make sqllogictest happy

Signed-off-by: Chojan Shang <psiace@apache.org>

---------

Signed-off-by: Chojan Shang <psiace@apache.org>
@b41sh
feat(query): create drop inverted index (databendlabs#14859)
a9d1fac 
* feat(query): create drop inverted index

* fix typos

* fix tests

* fix tests

* rewrite as table index

* index use column ids

* check column id exist
@TCeason
feat(query): show grants support mget table/db name by table/db id. (d…
e54cd2f 
…atabendlabs#14862)

* show grants support mget table/db name by table/db id.

* fix test

* refactor function name: list replace to mget

* fix error

* rename new api

* modify func name && modify err msg

* use slice

* fmt code
@ariesdevil
fix: fix incorrect schema in window expr in subquery with lazy_materi…
4cc7064 
…alization (databendlabs#14895)

fix incorrect schema in window expr in subquery with lazy_materialization
@andylokandy
refactor(ast): remove databend-common-functions dependency (databendl…
a95ac62 
…abs#14893)
@TCeason
fix(ci): fix internal benchmark (databendlabs#14897)
8e4b242
@dantengsky
fix: Display of merge into .. insert * (databendlabs#14892)
1732d8d 
* ut case of merget into ... insert *

* fix: displays "*" if necessary for insert clause of merge-into
@drmingdrmer
fix: check_and_upgrade_to_pb() should return MetaError, add quota (d…
6f7e7a1 
…atabendlabs#14899)

When writing back upgraded json-to-pb data to meta-service,
it should return `MetaError`, instead of business error such as
`ErrorCode::UnknownRole`.

Add quota limit to `check_and_upgrade_to_pb`: for each API calls, it
upgrades upto 100 records. So that an API call won't be delay a lot.

After upgrading, it should returns the `seq` number after upgrading.
@zhang2014
feat(query): support settings admin api for global level settings (da…
1ee4707 
…tabendlabs#14901)
@zhyass
chore(storage): split mutation_log file (databendlabs#14903)
48030ad 
* fix

* chore: split mutation log file
@chienguo @dantengsky
feat: add a new ratio config instead of table_data_deserialized_data_…
038d522 
…bytes (databendlabs#14896)

* add a new ratio config for data cache

* Update src/query/config/src/config.rs

Co-authored-by: dantengsky <dantengsky@gmail.com>

* Update src/query/config/src/inner.rs

Co-authored-by: dantengsky <dantengsky@gmail.com>

* add a new ratio config for data cache

---------

Co-authored-by: dantengsky <dantengsky@gmail.com>
@andylokandy
feat(ast): add parser for script (databendlabs#14891)
34c44b5 
* feat(ast): add parser for script

* add resultset

* fix
@drmingdrmer
refactor: upgrade to openraft 0.9.0 (databendlabs#14907)
991f2ca 
Openraft 0.9.0 release:
https://github.com/datafuselabs/openraft/releases/tag/v0.9.0

Implement chunk-based snapshot transmission outside openraft core.
This makes it possible to switch to another snapshot transmission
protocol in a compatible manner.
@xudong963
feat: support cross join spilling (databendlabs#14887)
0538cfa 
* feat: support cross join spilling

* trigger probe spill

* fix probe input with 0 col but non-zero row
@SkyFan2002
fix: Display of compact AST (databendlabs#14914)
cda2985 
* fix: Display of compact AST

* modify ut
@TCeason
chore(query): create user/role name can not contain single/double quo…
551c726 
…te (databendlabs#14902)
@TCeason
chore(ci): benchmark local need set meta client timeout (databendlabs…
683a3a2 
…#14910)
@drmingdrmer
refactor: use kvapi::Key to define meta-service key for QuotaMgr (dat…
f248f21 
…abendlabs#14915)

* refactor: use kvapi::Key to define meta-service key for QuotaMgr

* chore: fix time tolerance in kv_meta test suites
@xudong963
refactor: reduce hash join build spilling files (databendlabs#14919)
7003b1f
@drmingdrmer
refactor: define kvapi::Key for PasswordPolicyMgr (databendlabs#14920)
e7b1ae0 
`PasswordPolicyMgr::new()` does not have to return error because the
input tenant is never empty.

Use `KVPbApi` to access protobuf encoded values in meta-service.
Get rid of manually encoding/decoding.
@sundy-li
feat: add more function alias names (databendlabs#14924)
a5d8a6a 
* chore(query): add allow_streaming_insert_source in ast parser

* chore(query): add allow_streaming_insert_source in ast parser

* chore(query): add allow_streaming_insert_source in ast parser

* chore(query): add allow_streaming_insert_source in ast parser

* chore(query): add allow_streaming_insert_source in ast parser

* update

* update

* update

* update

* add more function alias
@Freejww @zhang2014
feat(query): support spill for new agg hashtable (databendlabs#14905)
31e9ad0 
* test

* test

* finish singleton spill

* cluster spill

* Update src/query/service/src/pipelines/processors/transforms/aggregator/aggregate_meta.rs

Co-authored-by: Winter Zhang <coswde@gmail.com>

* add Result<> for func return type

* fix bug

* fix decimal bug

* remove output columns(AggregatePartial) to pass ci

* revert set

---------

Co-authored-by: jw <freejw@gmail.com>
Co-authored-by: Winter Zhang <coswde@gmail.com>
@drmingdrmer
chore: extend time tolerance for meta-service test (databendlabs#14927)
df50add 
fixup: extend time tolerance for meta-service test
@TCeason
refactor: reduce meta call (databendlabs#14917)
aac5a23 
* refactor(privilege access): reduce meta call

* modify err msg display
@xudong963
chore: add step logs for hash join build side (databendlabs#14930)
1a36d87
TCeason and others added 26 commits April 9, 2024 22:01
@TCeason
fix(query): set setting case insensitive (databendlabs#15199)
c2e3ee6 
E.g.

```sql
SET GLOBAL data_retention_time_in_days=0; -- Need work
```
@TCeason
fix(query): show grants panic when role/user has immute db/table priv…
614c0bc 
…ilege (databendlabs#15188)

* fix(query): show grants panic when role/user has immute db/table

* Unify function parameter styles
@zhang2014
chore(executor): add end_of_stream flag for executor stream (databend…
5d6069a 
…labs#15202)
@Dousir9
chore(planner): refine explain read size (databendlabs#15204)
02e930d 
* chore: refine explain read size

* chore: handle read size 0

* chore: remove eb test

* chore: update ee test

* chore: update cluster test

* chore: update test

* chore: update test

* chore: update test

* chore: fix broken test

* chore: fix merge into test

* chore: fix test

* chore: fix cluster test

* chore: update cluster test
@kkk25641463
feat(query): implement ST_MAKEPOLYGON (databendlabs#15194)
24c14ff 
implement ST_MAKEPOLYGON

Signed-off-by: Fan Yang <yangfanlinux@gmail.com>
@drmingdrmer
refactor: simplify DatabaseFactory, just static dispatch (databendl…
515f99e 
…abs#15203)

* refactor: simplify `DatabaseFactory`, just static dispatch

* chore: try reduce usage of Tenant::new_nonempty()

This methods is a transitional method, to build a Tenant from a
NonEmptyString. This method will be removed in future because a Tenant
must contain a in-meta-service per-tenant config to work correctly,
e.g., determine the version of path to store a value: with tenant prefix
or without.
@youngsofun
feat: distinguish early close from kill/cancel with new LogType::Clos…
a883bc0 
…ed. (databendlabs#15206)

* feat: distinguish early close from kill/cancel with new LogType::Closed.

* extract Executor::get_response_state

* feat: http handler allow redundant final/kill.

* revert databendlabs#15022

databendlabs#15094 (comment)

* fix test.

* ignore flaky test.
@drmingdrmer
fix: geos-sys compile error databendlabs#15200 (databendlabs#15209)
14ff386 
- Fix: databendlabs#15200
@zhang2014
refactor(query): move api and metrics to servers mod (databendlabs#15210
0a00792 
)

* refactor(query): move api to servers

* refactor(query): move metrics to servers

* refactor(query): move metrics to servers
@BohuTANG
chore: change parquet_fast_read_bytes setting from 0 to 16MB (databen…
4c27f29 
…dlabs#15212)
@Dousir9
chore(planner): fix decorrelate exists subquery (databendlabs#15208)
80778f0 
* chore: fix decorrelate exists subquery

* chore: fix subquery explain sqllogictest
@TCeason
refactor: add view system tables (databendlabs#15184)
e557a7d 
* refactor: add view system tables

add system tables: system.view and system.view_with_history

The view engine no need to display stat info.

And fuse engine does not have view_query.

* fix test err
@TCeason
fix(query): alter modify col comment need modify when type not modify (…
a2bb0b5 
…databendlabs#15217)

* fix(query): alter modify col comment need modify when type not modify

* fix err
@TracyZYJ
chore: old op LogicalGet modify to new op name Scan (databendlabs#15222)
bb4464b
@andylokandy
feat: add EXECUTE IMMEDIATE statement (databendlabs#15144)
41ba9d1 
* feat: add EXECUTE IMMEDIATE statement

* fix

* fix

* fix

* fix

* fix

* fix

* fix

* fix

* fix

* fix

* fix

* fix

* fix

* add test
@sundy-li
chore(query): parser error need to log error (databendlabs#15219)
270b2f9
@drmingdrmer
refactor: introduce UninitTenant and TIdentRaw (databendlabs#15207)
a4de86c 
* refactor: introduce `UninitTenant` and `TIdentRaw`

`Tenant` is a meta-service key component representing a tenant,
with a embedded per-tenant config.

`UninitTenant` is a shadow of `TIdent` but without config thus it can
not be used as a key.
`UninitTenant` can be initialized to build a `Tenant` for use as a
meta-service key.

Correspondingly, this commit introduce `TIdentRaw` as an uninitialized
version of `TIdent`.
`TIdentRaw` does not cantain per-tenant config and is only used inside
meta-service values(not keys), which could be `serde`.

Other changes:

Populate usage of `Tennat` to `Catalog` implementation.

* chore: fix wrong url in share

* chore: fix rebase
@zhang2014
fix(query): fix negative session_running_acquired_queries (databendla…
d8e8220 
…bs#15223)
@BohuTANG
chore: vacuum drop table for external location table (databendlabs#15220
9925143 
)

* chore: vacuum drop table for external location table

* fix test_fuse_do_vacuum_drop_table_external_storage
@sundy-li
chore(query): fix nullable array domain (databendlabs#15224)
9b1887a
@flaneur2020 @BohuTANG
feat(http): get client ip from headers (databendlabs#15205)
2a4febd 
* use get-connection-ip

* fix get_client_ip

* refactor the code

* fix

* fix test

* middleware.rs: &req -> req

---------

Co-authored-by: Bohu <overred.shuttler@gmail.com>
@zhang2014
chore(query): add query queued duration for query log (databendlabs#1…
2071b02 
…5225)

* chore(query): add query queued duration for query log

* chore(query): add query queued duration for query log

* chore(query): add query queued duration for query log

* chore(query): add query queued duration for query log
@TCeason
feat(query): support alter table comment (databendlabs#15227)
6453c48
@ariesdevil
fix: fix map get return double nested nullable (databendlabs#15230)
13afda7 
fix map get return type
@drmingdrmer
refactor: define DatabaseNameIdent with TIdent (databendlabs#15235)
e8f88e7
@zhyass
create dynamic table
3ed2bf0
@zhyass zhyass requested a review from drmingdrmer as a code owner April 17, 2024 04:30
@zhyass zhyass closed this Apr 17, 2024
@zhyass
Copy link
Author

zhyass commented Apr 17, 2024

Sorry, pulling the wrong branch.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@drmingdrmer drmingdrmer Awaiting requested review from drmingdrmer drmingdrmer is a code owner

Assignees
No one assigned
Labels
pr-feature
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.