Interdependent Message #553

wycccccc · 2022-08-06T13:10:32Z

#541 實作

目前挑選target partition的邏輯爲選擇log size最小的Broker，再選中該Broker中log size最小的partition。
target partition在選擇後不會改變直到關閉Interdependent function。再此開啓會重新選擇target partition。不開啓則爲正常所選擇Dispatcher功能。

在Interface中新增了jmxAddress()以在Interface中獲取jmx address。
修改測試中的RequireBrokerCluster讓其能重新設置NUMBER_OF_BROKERS。原因在於叢集中的三個節點都會共用一臺機器，導致獲取到的Broker metrics無法鑑別是在哪一個節點上。考慮到正常情況應該是一個節點一臺機器，故爲了保證測試正常進行，讓其能夠修改NUMBER_OF_BROKERS。

chia7712

@wycccccc 有一個跟設計有關的建議，請看一下

另外可以把一些跟PR無關的變更拿掉，這樣我們可以集中討論

app/src/main/java/org/astraea/app/producer/Builder.java

app/src/main/java/org/astraea/app/partitioner/Dispatcher.java

wycccccc · 2022-08-09T11:06:56Z

另外可以把一些跟PR無關的變更拿掉，這樣我們可以集中討論

好啊可是這些改動大多都和測試相關。少了這些測試會動不了。我先把相關改動切成小pr？

chia7712 · 2022-08-09T12:09:59Z

好啊可是這些改動大多都和測試相關。少了這些測試會動不了。我先把相關改動切成小pr？

這樣很好

wycccccc · 2022-08-12T11:14:43Z

衝突已解決，再麻煩review

chia7712

@wycccccc 不好意思，在設計上我有幾個較大的想法，請看一下

app/src/main/java/org/astraea/app/partitioner/Dispatcher.java

wycccccc

我將其按照上述討論修改簡化，額外的議題我會再開issue討論。還有一個小問題想要確認一下。

app/src/main/java/org/astraea/app/producer/Builder.java

chia7712

@wycccccc Thanks for this patch. there are some minor comments. PTAL

app/src/main/java/org/astraea/app/common/Utils.java

app/src/main/java/org/astraea/app/partitioner/Dispatcher.java

wycccccc · 2022-08-16T08:38:18Z

上述conversation已跟進，後續議題開設在 #591，再麻煩學長review。

app/src/main/java/org/astraea/app/partitioner/Dispatcher.java

wycccccc · 2022-08-25T07:48:15Z

我好好的修改了一下，確保了所有Dispatcher都能正確使用Interdependent功能，再麻煩review。

chia7712

@wycccccc 一樣有設計上的問題，請再多思考一下

app/src/main/java/org/astraea/app/partitioner/Dispatcher.java

wycccccc · 2022-08-29T08:27:40Z

已更新，現在一個producer的每一個thread都會有其自己的Interdependent。

chia7712 · 2022-09-07T10:47:19Z

@wycccccc 不好意思，麻煩修正一下衝突

wycccccc · 2022-09-11T11:45:28Z

衝突已解決

chia7712

@wycccccc 一樣針對設計的部分還有多執行緒的用法有一些疑問，請看一下

common/src/main/java/org/astraea/common/partitioner/Dispatcher.java

…rpendentMessage

chia7712

@wycccccc 程式碼風格還有一些建議，請看一下

common/src/main/java/org/astraea/common/partitioner/Dispatcher.java

chia7712

@wycccccc 感謝更新，應該是最後幾個建議了，感謝耐心

chia7712 · 2022-09-14T11:58:53Z

common/src/main/java/org/astraea/common/partitioner/Dispatcher.java

+            valueBytes == null ? new byte[0] : valueBytes,
+            CLUSTER_CACHE.computeIfAbsent(cluster, ignored -> ClusterInfo.of(cluster)));
+    interdependent.targetPartitions = target;
+    return interdependent.isInterdependent ? interdependent.targetPartitions : target;


這邊的判斷是多餘的，同一個執行緒下 interdependent.targetPartitions 和 target 的值會是相同的

chia7712 · 2022-09-14T12:00:01Z

common/src/main/java/org/astraea/common/partitioner/Dispatcher.java

+   * @param producer Kafka producer
+   */
+  static void endInterdependent(Producer<Key, Value> producer) {
+    THREAD_LOCAL.get().targetPartitions = -1;


用remove吧，這樣之後如果加入更多狀態也不需要新增額外的初始化邏輯

chia7712 · 2022-09-14T12:00:18Z

common/src/test/java/org/astraea/common/partitioner/DispatcherTest.java

+
+  @Test
+  void multipleThreadTest() {
+    var admin = Admin.of(bootstrapServers());


try-release

chia7712 · 2022-09-14T12:01:24Z

common/src/main/java/org/astraea/common/Utils.java

+   * @param attribute attribute name
+   * @return attribute
+   */
+  public static Object reflectionAttribute(Object object, String attribute) {


麻煩移除這個方法並改用Utils.member

wycccccc · 2022-09-15T09:49:05Z

已经修正完毕，感谢学长耐心的review

chia7712 · 2022-09-15T11:50:42Z

@chinghongfang 你方便幫忙測試下這隻的效能嗎？

chinghongfang · 2022-09-15T12:16:07Z

你方便幫忙測試下這隻的效能嗎？

沒問題，結果我再紀錄在 doc/dispatcher/experiments 裡面。

Interdependent Message

425fa91

wycccccc requested review from chia7712 and chinghongfang August 6, 2022 13:10

wycccccc added 2 commits August 6, 2022 21:17

spotless

4f5b2bc

remove some unnecessary changes

ac0010b

wycccccc mentioned this pull request Aug 6, 2022

Order records #223

Closed

chia7712 reviewed Aug 7, 2022

View reviewed changes

app/src/main/java/org/astraea/app/producer/Builder.java Outdated Show resolved Hide resolved

app/src/main/java/org/astraea/app/partitioner/Dispatcher.java Outdated Show resolved Hide resolved

wycccccc mentioned this pull request Aug 10, 2022

Log size partition cost #569

Merged

wycccccc added 5 commits August 10, 2022 16:31

replace single broker cluster

259ff16

private

1c1bf17

Use interface method to replace

9a2f35c

resolve conflicting

0d3ccae

simplify test

aa3810f

chia7712 reviewed Aug 12, 2022

View reviewed changes

app/src/main/java/org/astraea/app/partitioner/Dispatcher.java Outdated Show resolved Hide resolved

app/src/main/java/org/astraea/app/partitioner/Dispatcher.java Outdated Show resolved Hide resolved

app/src/main/java/org/astraea/app/partitioner/Dispatcher.java Outdated Show resolved Hide resolved

wycccccc added 2 commits August 13, 2022 16:13

simplify dispatcher

728b4e9

spotless

4f05f56

wycccccc commented Aug 13, 2022

View reviewed changes

app/src/main/java/org/astraea/app/producer/Builder.java Outdated Show resolved Hide resolved

chia7712 reviewed Aug 13, 2022

View reviewed changes

wycccccc added 4 commits August 15, 2022 22:50

multipleProducers

a3a527f

fixBug

c72612a

fixBug

8942573

remove excess synchronized

2cdfe03

chia7712 reviewed Aug 17, 2022

View reviewed changes

app/src/main/java/org/astraea/app/partitioner/Dispatcher.java Outdated Show resolved Hide resolved

atomicBoolean

cf51e0e

chia7712 reviewed Aug 24, 2022

View reviewed changes

app/src/main/java/org/astraea/app/partitioner/Dispatcher.java Outdated Show resolved Hide resolved

app/src/main/java/org/astraea/app/partitioner/Dispatcher.java Outdated Show resolved Hide resolved

INTERDEPENDENT

6400a17

alter (!(

5936907

swrr

1f1ffd2

chia7712 reviewed Aug 26, 2022

View reviewed changes

app/src/main/java/org/astraea/app/partitioner/Dispatcher.java Outdated Show resolved Hide resolved

app/src/main/java/org/astraea/app/partitioner/Dispatcher.java Outdated Show resolved Hide resolved

wycccccc added 4 commits August 28, 2022 16:43

compareAnd

6423c42

spotless

99ff3a2

each thread has a correspongding status

6003040

resolve conflict

3cd4387

resolve conflict

f790fcb

chia7712 reviewed Sep 12, 2022

View reviewed changes

chia7712 mentioned this pull request Sep 12, 2022

Fix raw type of Dispatcher #699

Merged

wycccccc added 3 commits September 13, 2022 15:08

Merge branch 'main' of https://github.com/skiptests/astraea into Inte…

4e30de0

…rpendentMessage

One producer, one thread

5fba898

add todo

90baa3a

chia7712 reviewed Sep 13, 2022

View reviewed changes

simplify

e0b5647

chia7712 reviewed Sep 14, 2022

View reviewed changes

InterpendentMessage

730b2ef

chia7712 merged commit ec68577 into main Sep 15, 2022

chinghongfang mentioned this pull request Sep 16, 2022

Interdependent message 使用問題 #719

Open

3 tasks

chinghongfang deleted the InterpendentMessage branch September 19, 2022 03:09

chinghongfang mentioned this pull request Sep 19, 2022

Performance with interdependent #732

Merged

chinghongfang mentioned this pull request Oct 3, 2022

Experiments on StrictCostDispatcher #816

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Interdependent Message #553

Interdependent Message #553

wycccccc commented Aug 6, 2022

chia7712 left a comment

wycccccc commented Aug 9, 2022 •

edited

Loading

chia7712 commented Aug 9, 2022

wycccccc commented Aug 12, 2022

chia7712 left a comment

wycccccc left a comment

chia7712 left a comment •

edited

Loading

wycccccc commented Aug 16, 2022

wycccccc commented Aug 25, 2022

chia7712 left a comment

wycccccc commented Aug 29, 2022

chia7712 commented Sep 7, 2022

wycccccc commented Sep 11, 2022

chia7712 left a comment

chia7712 left a comment

chia7712 left a comment

chia7712 Sep 14, 2022

chia7712 Sep 14, 2022

chia7712 Sep 14, 2022

chia7712 Sep 14, 2022

wycccccc commented Sep 15, 2022

chia7712 commented Sep 15, 2022

chinghongfang commented Sep 15, 2022

Interdependent Message #553

Interdependent Message #553

Conversation

wycccccc commented Aug 6, 2022

chia7712 left a comment

Choose a reason for hiding this comment

wycccccc commented Aug 9, 2022 • edited Loading

chia7712 commented Aug 9, 2022

wycccccc commented Aug 12, 2022

chia7712 left a comment

Choose a reason for hiding this comment

wycccccc left a comment

Choose a reason for hiding this comment

chia7712 left a comment • edited Loading

Choose a reason for hiding this comment

wycccccc commented Aug 16, 2022

wycccccc commented Aug 25, 2022

chia7712 left a comment

Choose a reason for hiding this comment

wycccccc commented Aug 29, 2022

chia7712 commented Sep 7, 2022

wycccccc commented Sep 11, 2022

chia7712 left a comment

Choose a reason for hiding this comment

chia7712 left a comment

Choose a reason for hiding this comment

chia7712 left a comment

Choose a reason for hiding this comment

chia7712 Sep 14, 2022

Choose a reason for hiding this comment

chia7712 Sep 14, 2022

Choose a reason for hiding this comment

chia7712 Sep 14, 2022

Choose a reason for hiding this comment

chia7712 Sep 14, 2022

Choose a reason for hiding this comment

wycccccc commented Sep 15, 2022

chia7712 commented Sep 15, 2022

chinghongfang commented Sep 15, 2022

wycccccc commented Aug 9, 2022 •

edited

Loading

chia7712 left a comment •

edited

Loading