[Doris On ES] Split /_cluster/state to [indexName/_mappings, indexName/_search_shards] #3454

stalary · 2020-05-02T16:15:41Z

Split /_cluster/state into /_mapping and /_search_shards requests to reduce permissions and make the logic clearer
Rename part es related objects to make their representation more accurate
Simply support docValue and Fields in alias mode, and take the first one by default

#3311

…_shards]

imay · 2020-05-05T07:05:35Z

@wuyunfeng
Could you please review this PR?

@stalary
You'd better add more information about this change list in commit message

wuyunfeng · 2020-05-05T13:44:46Z

OK

docs/en/sql-reference/sql-statements/Data Types/VARCHAR.md

fe/src/main/java/org/apache/doris/external/EsIndexState.java

fe/src/main/java/org/apache/doris/external/EsRestClient.java

fe/src/main/java/org/apache/doris/external/EsStateStore.java

…to stalary_enhancement

wuyunfeng

Thanks @stalary , I left some comments.

fe/src/main/java/org/apache/doris/external/EsRestClient.java

fe/src/main/java/org/apache/doris/external/EsStateStore.java

…to stalary_enhancement

fe/src/main/java/org/apache/doris/external/EsRestClient.java

fe/src/main/java/org/apache/doris/external/EsStateStore.java

# Conflicts: # fe/src/main/java/org/apache/doris/external/elasticsearch/EsRestClient.java # fe/src/main/java/org/apache/doris/external/elasticsearch/EsStateStore.java # fe/src/test/java/org/apache/doris/es/EsStateStoreTest.java

…to stalary_enhancement # Conflicts: # fe/src/main/java/org/apache/doris/external/elasticsearch/EsRestClient.java # fe/src/main/java/org/apache/doris/external/elasticsearch/EsStateStore.java # fe/src/test/java/org/apache/doris/es/EsStateStoreTest.java

fe/src/main/java/org/apache/doris/external/elasticsearch/EsFieldInfo.java

fe/src/main/java/org/apache/doris/external/elasticsearch/EsRestClient.java

fe/src/main/java/org/apache/doris/external/elasticsearch/EsFieldInfo.java

fe/src/main/java/org/apache/doris/external/elasticsearch/EsRestClient.java

fe/src/main/java/org/apache/doris/external/elasticsearch/EsIndexState.java

BabySid

加下必要的注释提高review的效率和质量

fe/src/main/java/org/apache/doris/external/elasticsearch/EsRestClient.java

fe/src/main/java/org/apache/doris/external/elasticsearch/EsShardPartitions.java

fe/src/main/java/org/apache/doris/external/elasticsearch/EsStateStore.java

fe/src/main/java/org/apache/doris/external/elasticsearch/EsRepository.java

morningman

LGTM

fe/src/main/java/org/apache/doris/catalog/EsTable.java

morningman

LGTM

…e/_search_shards] (apache#3454) 1. Split /_cluster/state into /_mapping and /_search_shards requests to reduce permissions and make the logic clearer 2. Rename part es related objects to make their representation more accurate 3. Simply support docValue and Fields in alias mode, and take the first one by default apache#3311

…ade FE (#3961) ISSUE:#3960 PR #3454 introduce the caching for EsClient, but the initialization of the client was only during editlog replay, all this work should done also during image replay. This happens when restart or upgrade FE BTW: modify a UT failure for metric

…ade FE (apache#3961) ISSUE:apache#3960 PR apache#3454 introduce the caching for EsClient, but the initialization of the client was only during editlog replay, all this work should done also during image replay. This happens when restart or upgrade FE BTW: modify a UT failure for metric

…4012) After PR #3454 was merged, we should refactor and reorganize some logic for long-term sustainable iteration for Doris On ES. To facilitate code review，I would divided into this work to multiple PRs (some other WIP work I also need to think carefully) This PR include: 1. introduce SearchContext for all state we needed 2. divide meta-sync logic into three phase 3. modify some logic processing 4. introduce version detect logic for future using

…pache#4012) After PR apache#3454 was merged, we should refactor and reorganize some logic for long-term sustainable iteration for Doris On ES. To facilitate code review，I would divided into this work to multiple PRs (some other WIP work I also need to think carefully) This PR include: 1. introduce SearchContext for all state we needed 2. divide meta-sync logic into three phase 3. modify some logic processing 4. introduce version detect logic for future using

stalary added 3 commits May 2, 2020 11:31

MOD: fix varchar max error

e2d09f5

MOD: Split /_cluster/state to [indexName/_mappings, indexName/_search…

b6ff6b2

…_shards]

MOD: Mod some details

d858a74

imay self-assigned this May 5, 2020

BabySid reviewed May 6, 2020

View reviewed changes

docs/en/sql-reference/sql-statements/Data Types/VARCHAR.md Outdated Show resolved Hide resolved

BabySid reviewed May 6, 2020

View reviewed changes

fe/src/main/java/org/apache/doris/external/EsIndexState.java Show resolved Hide resolved

fe/src/main/java/org/apache/doris/external/EsIndexState.java Outdated Show resolved Hide resolved

MOD: Cancel the folding line，Remove the changes to varchar.md

a3d2928

wuyunfeng requested changes May 7, 2020

View reviewed changes

stalary added 2 commits May 18, 2020 22:28

Merge branch 'master' of https://github.com/apache/incubator-doris in…

558b8ee

…to stalary_enhancement

MOD: Make some changes based on comments

d25b21f

wuyunfeng requested changes May 19, 2020

View reviewed changes

fe/src/main/java/org/apache/doris/external/EsRestClient.java Outdated Show resolved Hide resolved

fe/src/main/java/org/apache/doris/external/EsStateStore.java Outdated Show resolved Hide resolved

fe/src/main/java/org/apache/doris/external/EsStateStore.java Outdated Show resolved Hide resolved

stalary added 6 commits May 24, 2020 12:38

MOD: make some changes based on the comments

407e913

FIX: fix dovValueField bug

f06f846

MOD: remove useless import

a381b68

MOD

8400246

Merge branch 'master' of https://github.com/apache/incubator-doris in…

7b79445

…to stalary_enhancement

MOD: Supplement Doris on es documentation

451b9a6

wuyunfeng reviewed May 26, 2020

View reviewed changes

fe/src/main/java/org/apache/doris/external/EsRestClient.java Outdated Show resolved Hide resolved

wuyunfeng requested changes May 26, 2020

View reviewed changes

stalary added 5 commits June 10, 2020 17:58

MOD: refactor code

3e768c8

Merge branch 'master' into stalary_enhancement

2ed77b6

# Conflicts: # fe/src/main/java/org/apache/doris/external/elasticsearch/EsRestClient.java # fe/src/main/java/org/apache/doris/external/elasticsearch/EsStateStore.java # fe/src/test/java/org/apache/doris/es/EsStateStoreTest.java

MOD: Adjust import order, resolve conflicts

b975c7a

MOD: adjust import order

3782f4b

wuyunfeng requested changes Jun 12, 2020

View reviewed changes

MOD: Abstract code

97b2338

wuyunfeng requested changes Jun 16, 2020

View reviewed changes

BabySid reviewed Jun 19, 2020

View reviewed changes

wuyunfeng requested changes Jun 19, 2020

View reviewed changes

MOD: Refactor some places

95b5117

wuyunfeng requested changes Jun 22, 2020

View reviewed changes

fe/src/main/java/org/apache/doris/external/elasticsearch/EsRepository.java Outdated Show resolved Hide resolved

fe/src/main/java/org/apache/doris/external/elasticsearch/EsRepository.java Outdated Show resolved Hide resolved

MOD: EsRepository logic moves to EsTable

5e42b5b

wuyunfeng requested changes Jun 23, 2020

View reviewed changes

fe/src/main/java/org/apache/doris/external/elasticsearch/EsRepository.java Outdated Show resolved Hide resolved

fe/src/main/java/org/apache/doris/external/elasticsearch/EsRepository.java Outdated Show resolved Hide resolved

stalary added 4 commits June 23, 2020 13:58

Merge branch 'master' into stalary_enhancement

e3748ec

MOD: Abstract code, change the function name

a7ba9d1

MOD: format

303d0a2

MOD: mod error info

15ba2d3

morningman previously approved these changes Jun 23, 2020

View reviewed changes

morningman added approved Indicates a PR has been approved by one committer. area/doris-on-es Issues or PRs related to Doris on ElasticSearch kind/improvement labels Jun 23, 2020

wuyunfeng previously approved these changes Jun 23, 2020

View reviewed changes

wutiangan reviewed Jun 24, 2020

View reviewed changes

fe/src/main/java/org/apache/doris/catalog/EsTable.java Outdated Show resolved Hide resolved

MOD: format code

5fbae71

stalary dismissed stale reviews from wuyunfeng and morningman via 5fbae71 June 24, 2020 08:57

morningman approved these changes Jun 26, 2020

View reviewed changes

morningman merged commit a894b1e into apache:master Jun 26, 2020

This was referenced Jun 28, 2020

[Doris On ES][Bug-fix] Sync ES metadata failure after restart or upgrade FE #3961

Merged

[Doris On ES] Support ES alias and wildcard index #3967

Closed

wuyunfeng mentioned this pull request Jul 3, 2020

[Doris On ES][Refactor] refactor and enchanment ES sync meta logic #4012

Merged

stalary deleted the stalary_enhancement branch July 19, 2020 08:05

[Doris On ES] Split /_cluster/state to [indexName/_mappings, indexName/_search_shards] #3454

[Doris On ES] Split /_cluster/state to [indexName/_mappings, indexName/_search_shards] #3454

Uh oh!

Conversation

stalary commented May 2, 2020 • edited by morningman Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

imay commented May 5, 2020

Uh oh!

wuyunfeng commented May 5, 2020

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

wuyunfeng left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

BabySid left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

morningman left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

morningman left a comment

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

stalary commented May 2, 2020 •

edited by morningman

Loading