[Docs][Connector-V2][Doris] Reconstruct the Doris connector document #4903
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -2,11 +2,24 @@ | |
|
|
||
| > Doris sink connector | ||
|
|
||
| ## Support Those Engines | ||
|
|
||
| > Spark<br/> | ||
| > Flink<br/> | ||
| > SeaTunnel Zeta<br/> | ||
|
|
||
| ## Key Features | ||
|
|
||
| - [x] [exactly-once](../../concept/connector-v2-features.md) | ||
| - [x] [cdc](../../concept/connector-v2-features.md) | ||
|
|
||
| ## Description | ||
|
|
||
| Used to send data to Doris. Both support streaming and batch mode. | ||
| The internal implementation of Doris sink connector is cached and imported by stream load in batches. | ||
|
|
||
| ## Supported DataSource Info | ||
|
|
||
| :::tip | ||
|
|
||
| Version Supported | ||
|
|
@@ -17,67 +30,186 @@ Version Supported | |
|
|
||
| ::: | ||
|
|
||
| ## Sink Options | ||
|
|
||
| | Name | Type | Required | Default | Description | | ||
| |---------------------|--------|----------|------------|----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| | ||
| | fenodes | String | Yes | - | `Doris` cluster fenodes address, the format is `"fe_ip:fe_http_port, ..."` | | ||
| | username | String | Yes | - | `Doris` user username | | ||
| | password | String | Yes | - | `Doris` user password | | ||
| | table.identifier | String | Yes | - | The name of `Doris` table | | ||
| | sink.label-prefix | String | Yes | - | The label prefix used by stream load imports. In the 2pc scenario, global uniqueness is required to ensure the EOS semantics of SeaTunnel. | | ||
| | sink.enable-2pc | bool | No | - | Whether to enable two-phase commit (2pc), the default is true, to ensure Exactly-Once semantics. For two-phase commit, please refer to [here](https://doris.apache.org/docs/dev/sql-manual/sql-reference/Data-Manipulation-Statements/Load/STREAM-LOAD). | | ||
| | sink.enable-delete | bool | No | - | Whether to enable deletion. This option requires Doris table to enable batch delete function (0.15+ version is enabled by default), and only supports Unique model. you can get more detail at this [link](https://doris.apache.org/docs/dev/data-operate/update-delete/batch-delete-manual) | | ||
| | sink.check-interval | int | No | 10000 | check exception with the interval while loading | | ||
| | sink.max-retries | int | No | 3 | the max retry times if writing records to database failed | | ||
| | sink.buffer-size | int | No | 256 * 1024 | the buffer size to cache data for stream load. | | ||
| | sink.buffer-count | int | No | 3 | the buffer count to cache data for stream load. | | ||
| | doris.config | map | yes | - | This option is used to support operations such as `insert`, `delete`, and `update` when automatically generate sql,and supported formats. | | ||
|
|
||
| ## Data Type Mapping | ||
|
|
||
| | Doris Data type | SeaTunnel Data type | | ||
| |-----------------|-----------------------------------------| | ||
| | BOOLEAN | BOOLEAN | | ||
| | TINYINT | TINYINT | | ||
| | SMALLINT | SMALLINT<br/>TINYINT | | ||
| | INT | INT<br/>SMALLINT<br/>TINYINT | | ||
| | BIGINT | BIGINT<br/>INT<br/>SMALLINT<br/>TINYINT | | ||
| | LARGEINT | BIGINT<br/>INT<br/>SMALLINT<br/>TINYINT | | ||
| | FLOAT | FLOAT | | ||
| | DOUBLE | DOUBLE<br/>FLOAT | | ||
| | DECIMAL | DECIMAL<br/>DOUBLE<br/>FLOAT | | ||
| | DATE | DATE | | ||
| | DATETIME | TIMESTAMP | | ||
| | CHAR | STRING | | ||
| | VARCHAR | STRING | | ||
| | STRING | STRING | | ||
| | ARRAY | ARRAY | | ||
| | MAP | MAP | | ||
| | JSON | STRING | | ||
| | HLL | Not supported yet | | ||
| | BITMAP | Not supported yet | | ||
| | QUANTILE_STATE | Not supported yet | | ||
| | STRUCT | Not supported yet | | ||
|
|
||
| #### Supported import data formats | ||
|
|
||
| The supported formats include CSV and JSON | ||
|
|
||
| ## Task Example | ||
|
|
||
| ### Simple: | ||
|
|
||
| > The following example describes writing multiple data types to Doris, and users need to create corresponding tables downstream | ||
|
|
||
| ```hocon | ||
| env { | ||
| parallelism = 1 | ||
| job.mode = "BATCH" | ||
| checkpoint.interval = 10000 | ||
| } | ||
|
|
||
| source { | ||
| FakeSource { | ||
| row.num = 10 | ||
| map.size = 10 | ||
| array.size = 10 | ||
| bytes.length = 10 | ||
| string.length = 10 | ||
| schema = { | ||
| fields { | ||
| c_map = "map<string, array<int>>" | ||
| c_array = "array<int>" | ||
| c_string = string | ||
| c_boolean = boolean | ||
| c_tinyint = tinyint | ||
| c_smallint = smallint | ||
| c_int = int | ||
| c_bigint = bigint | ||
| c_float = float | ||
| c_double = double | ||
| c_decimal = "decimal(16, 1)" | ||
| c_null = "null" | ||
| c_bytes = bytes | ||
| c_date = date | ||
| c_timestamp = timestamp | ||
| } | ||
| } | ||
| } | ||
| } | ||
|
|
||
| sink { | ||
| Doris { | ||
| fenodes = "doris_cdc_e2e:8030" | ||
| username = root | ||
| password = "" | ||
| table.identifier = "test.e2e_table_sink" | ||
| sink.label-prefix = "test-cdc" | ||
| sink.enable-2pc = "true" | ||
| sink.enable-delete = "true" | ||
| doris.config { | ||
| format = "json" | ||
| read_json_by_line = "true" | ||
| } | ||
| } | ||
| } | ||
| ``` | ||
|
|
||
| ### CDC(Change Data Capture) Event: | ||
|
|
||
| > This example defines a SeaTunnel synchronization task that automatically generates data through FakeSource and sends it to Doris Sink,FakeSource simulates CDC data with schema, score (int type),Doris needs to create a table sink named test.e2e_table_sink and a corresponding table for it. | ||
|
|
||
| ```hocon | ||
| env { | ||
| parallelism = 1 | ||
| job.mode = "BATCH" | ||
| checkpoint.interval = 10000 | ||
| } | ||
|
|
||
| source { | ||
|
There was a problem hiding this comment. This data source type is more comprehensive There was a problem hiding this comment. Not every version of doris supports complex types There was a problem hiding this comment. PTAL: @TaoZex @EricJoy2048 |
||
| FakeSource { | ||
| schema = { | ||
| fields { | ||
| pk_id = bigint | ||
|
There was a problem hiding this comment. Add all the data types, it's more friendly There was a problem hiding this comment. I have added as many types as possible that are supported by all versions |
||
| name = string | ||
| score = int | ||
| sex = boolean | ||
| number = tinyint | ||
| height = float | ||
| sight = double | ||
| create_time = date | ||
| update_time = timestamp | ||
| } | ||
| } | ||
| rows = [ | ||
| { | ||
| kind = INSERT | ||
| fields = [1, "A", 100, true, 1, 170.0, 4.3, "2020-02-02", "2020-02-02T02:02:02"] | ||
| }, | ||
| { | ||
| kind = INSERT | ||
| fields = [2, "B", 100, true, 1, 170.0, 4.3, "2020-02-02", "2020-02-02T02:02:02"] | ||
| }, | ||
| { | ||
| kind = INSERT | ||
| fields = [3, "C", 100, true, 1, 170.0, 4.3, "2020-02-02", "2020-02-02T02:02:02"] | ||
| }, | ||
| { | ||
| kind = UPDATE_BEFORE | ||
| fields = [1, "A", 100, true, 1, 170.0, 4.3, "2020-02-02", "2020-02-02T02:02:02"] | ||
| }, | ||
| { | ||
| kind = UPDATE_AFTER | ||
| fields = [1, "A_1", 100, true, 1, 170.0, 4.3, "2020-02-02", "2020-02-02T02:02:02"] | ||
| }, | ||
| { | ||
| kind = DELETE | ||
| fields = [2, "B", 100, true, 1, 170.0, 4.3, "2020-02-02", "2020-02-02T02:02:02"] | ||
| } | ||
| ] | ||
| } | ||
| } | ||
|
|
||
| sink { | ||
| Doris { | ||
| fenodes = "doris_cdc_e2e:8030" | ||
| username = root | ||
| password = "" | ||
| table.identifier = "test.e2e_table_sink" | ||
| sink.label-prefix = "test-cdc" | ||
| sink.enable-2pc = "true" | ||
| sink.enable-delete = "true" | ||
| doris.config { | ||
| format = "json" | ||
| read_json_by_line = "true" | ||
| } | ||
| } | ||
| } | ||
|
|
||
| ``` | ||
|
|
||
| ### Use JSON format to import data | ||
|
There was a problem hiding this comment. "CSV and JSON, respectively. |
||
|
|
||
| ``` | ||
| sink { | ||
|
|
@@ -97,7 +229,7 @@ sink { | |
|
|
||
| ``` | ||
|
|
||
| ### Use CSV format to import data | ||
|
|
||
| ``` | ||
| sink { | ||
|
|
||
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Supported Import Data Formats?There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This is the original category and can be deleted if not needed