Random record and producing duration setting #106

chinghongfang · 2021-11-12T05:48:02Z

Implement functions in #26.
In this pull request, changes are

Add record producing duration.
Random record size.
Random record content.
Change default producing time and records. By default, it runs for about 1 hour. And produce 1,000,000,000 many records.
Add unit test of random payload generator.
Change consumer completed rate to ratio of number of consumed and number of produced

Add record producing duration. Random record size. Random record content. Change default producing time and records. By default, it runs for about 68 years (Integer.MAX_VLUE seconds).

chia7712 · 2021-11-12T07:58:50Z

app/src/main/java/org/astraea/performance/Performance.java

        long start = System.currentTimeMillis();
        producer
            .sender()
            .topic(param.topic)
-            .value(payload)
+            .value(randomPayload.payload())


這邊之後要考慮加上key and headers，這樣比較算是完整的record

喔我的意思是之後要發PR去新增key and headers，如果只是填寫null or empty的話那不如先不要

好的，我下一隻PR把key, header放進去。

chia7712

@chinghongfang 感謝更新，幾個小問題請看一下

另外這隻PR除了random data以外還有新增duration，可否更新一下標題包含duration?

app/src/main/java/org/astraea/performance/Performance.java

By default, produce 10^9 records or produce for 1 hour. Depend on the first completed condition. Explicitly set the headers and key to "empty" and "null"

chia7712 · 2021-11-13T17:21:35Z

@chinghongfang 那個百分比是不是要修一下？如果是依照時間來停止的話

chinghongfang · 2021-11-14T00:34:27Z

@chia7712 好的，現在的做法是雙限制，任一條件到達，就會停止。
可改成輸出百分比較高的(時間%, 數量%)，但目前想到可能會發生 consumer完成度 > producer完成度。
我再思考如何解決。

chinghongfang · 2021-11-16T06:49:56Z

但目前想到可能會發生 consumer完成度 > producer完成度。

想過以後發現，並不會出現 "consumer完成度 > producer完成度" 的情況。

但是有一個問題，依時間來顯示百分比，會不會讓人以為 consumer"完全跟上" producer 的速度？

chia7712 · 2021-11-16T07:13:24Z

依時間來顯示百分比，會不會讓人以為 consumer"完全跟上" producer 的速度？

或許兩者（producer and consumer）應該要用不同邏輯，對producer而言，應該要傳到時間結束為止。而對consumer而言，則應該收到所有這次producers送出去的資料為止

* Change the meaning of Consumer completion rate. * Producers produce data until duration end or num of records reach * Consumers consume data until all data producers produced is consumed.

chinghongfang

修改輸出內容。

chinghongfang · 2021-11-18T15:34:38Z

app/src/main/java/org/astraea/performance/Tracker.java

-    System.out.printf(
-        "consumer完成度: %.2f%%%n", ((double) result.completedRecords * 100.0 / (double) records));
+    // Print out percentage of (consumed records) and (produced records)
+    var percentage = result.completedRecords * 100D / dataManager.produced();


consumer 完成度改成 "完成和已produce的比例"。

chia7712

@chinghongfang 這個版本看起來不錯，幾個小問題麻煩看一下

app/src/main/java/org/astraea/performance/DataManager.java

app/src/main/java/org/astraea/performance/Tracker.java

app/src/main/java/org/astraea/performance/Performance.java

chinghongfang · 2021-11-19T03:55:29Z

關於 "何時結束" producing/consuming 應該要統一地方做判斷，不把判斷分散各地，可以使程式更清楚明瞭。
目前打算把判斷放在DataManager內。

chia7712

@chinghongfang 感謝更新，一些設計建議請看一下，謝謝

app/src/main/java/org/astraea/performance/Manager.java

app/src/main/java/org/astraea/performance/Performance.java

app/src/main/java/org/astraea/performance/Manager.java

Performance benchmarking tool runs for "number of records" or "a duration".

chinghongfang · 2021-11-28T01:36:08Z

這次主要更新

記錄producer是否完成
更改consumer thread停止條件
更改API，改成指定"records數量"或"持續時間"
避免重複的運算（metrics, manager重複紀錄record數量）
Metrics 實作Biconsumer<>

chia7712

@chinghongfang 幾個小建議請看一下

app/src/main/java/org/astraea/performance/Manager.java

app/src/main/java/org/astraea/performance/Performance.java

chia7712

@chinghongfang 感謝更新，還有幾個意見麻煩看一下

app/src/main/java/org/astraea/performance/Performance.java

app/src/main/java/org/astraea/performance/Manager.java

app/src/main/java/org/astraea/performance/Performance.java

chia7712

@chinghongfang 我用這隻PR測試了一下100000records，然後用我們的offset工具來看，資料筆數會不一樣，可否麻煩你檢查一下

chia7712 · 2021-11-30T07:38:06Z

app/src/main/java/org/astraea/performance/Tracker.java

-    System.out.printf(
-        "consumer完成度: %.2f%%%n", ((double) result.completedRecords * 100.0 / (double) records));
+    // Print out percentage of (consumed records) and (produced records)
+    var percentage = result.completedRecords * 100D / manager.producedRecords();


這個算法是不是怪怪的？producedRecords是會隨時間增長的值，拿它來算比例怪怪的

因為無法確切的知道總共會有幾筆資料要被產出(當我們指定執行時間時)，所以這裡改成了和目前生產量的比例。

還是要改回之前的比例；而當我們指定執行時間時，就不輸出完成度？

有點忘了之前的討論結果，在時間模式下已經有時間的輸出了？如果是的話，或許這邊就不用再輸出比例了，不然很容易誤會

時間模式的話，會持續consume 直到所有的資料都收到。

會持續consume 直到所有的資料都收到。

這個是指持續到producers都完成，且所有送出去的資料都收到吧？

這個是指持續到producers都完成，且所有送出去的資料都收到吧？

是的。

Fix `CountDownLatch` misuse.

chia7712 · 2021-11-30T16:16:15Z

@chinghongfang 感謝這次的改良！

chinghongfang added 2 commits November 11, 2021 23:33

Add producing duration.

c5b34f6

Randomly create record

7bcc864

Add record producing duration. Random record size. Random record content. Change default producing time and records. By default, it runs for about 68 years (Integer.MAX_VLUE seconds).

chinghongfang requested a review from chia7712 November 12, 2021 05:48

chia7712 reviewed Nov 12, 2021

View reviewed changes

app/src/main/java/org/astraea/performance/Performance.java Outdated Show resolved Hide resolved

chinghongfang changed the title ~~Random record~~ Random record and producing duration setting Nov 12, 2021

Shorten default records and duration

bc39b46

By default, produce 10^9 records or produce for 1 hour. Depend on the first completed condition. Explicitly set the headers and key to "empty" and "null"

Merge remote-tracking branch 'upstream/main' into randomRecord

26a17d2

chinghongfang mentioned this pull request Nov 17, 2021

效能測試程式改良 #26

Closed

7 tasks

chinghongfang added 4 commits November 17, 2021 20:08

DataManager handles data producing.

0d80d13

* Change the meaning of Consumer completion rate. * Producers produce data until duration end or num of records reach * Consumers consume data until all data producers produced is consumed.

Resolve conflict

ac7a32f

Merge remote-tracking branch 'upstream/main' into randomRecord

43f43c8

Fix unnecessary change

a4ca8cf

chinghongfang commented Nov 18, 2021

View reviewed changes

chia7712 reviewed Nov 18, 2021

View reviewed changes

chinghongfang added 2 commits November 21, 2021 20:37

Manager controls producing/consuming time

20c4345

Merge remote-tracking branch 'upstream/main' into randomRecord

ddb6337

chia7712 reviewed Nov 21, 2021

View reviewed changes

app/src/main/java/org/astraea/performance/Manager.java Outdated Show resolved Hide resolved

chinghongfang added 4 commits November 27, 2021 22:31

Change performance api

f9ed2d5

Performance benchmarking tool runs for "number of records" or "a duration".

Merge remote-tracking branch 'upstream/main' into randomRecord

5d447aa

Resolve conflict

17650ac

Modify README file

4b87a00

chia7712 reviewed Nov 28, 2021

View reviewed changes

chinghongfang added 2 commits November 29, 2021 15:01

Change api format

3d14ea7

Merge remote-tracking branch 'upstream/main' into randomRecord

0d7d217

chia7712 reviewed Nov 29, 2021

View reviewed changes

Reduce code

6f33dba

chia7712 reviewed Nov 30, 2021

View reviewed changes

Produce exact records

683cd64

Fix `CountDownLatch` misuse.

chia7712 merged commit a38072b into opensource4you:main Nov 30, 2021

chinghongfang deleted the randomRecord branch March 2, 2022 09:01

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Random record and producing duration setting #106

Random record and producing duration setting #106

chinghongfang commented Nov 12, 2021 •

edited

Loading

chia7712 Nov 12, 2021

chia7712 Nov 13, 2021

chinghongfang Nov 14, 2021 •

edited

Loading

chia7712 left a comment

chia7712 commented Nov 13, 2021

chinghongfang commented Nov 14, 2021

chinghongfang commented Nov 16, 2021

chia7712 commented Nov 16, 2021

chinghongfang left a comment

chinghongfang Nov 18, 2021

chia7712 left a comment

chinghongfang commented Nov 19, 2021

chia7712 left a comment

chinghongfang commented Nov 28, 2021

chia7712 left a comment

chia7712 left a comment

chia7712 left a comment

chia7712 Nov 30, 2021

chinghongfang Nov 30, 2021

chia7712 Nov 30, 2021

chinghongfang Nov 30, 2021

chia7712 Nov 30, 2021

chinghongfang Nov 30, 2021

chia7712 commented Nov 30, 2021

Random record and producing duration setting #106

Random record and producing duration setting #106

Conversation

chinghongfang commented Nov 12, 2021 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

chinghongfang Nov 14, 2021 • edited Loading

Choose a reason for hiding this comment

chia7712 left a comment

Choose a reason for hiding this comment

chia7712 commented Nov 13, 2021

chinghongfang commented Nov 14, 2021

chinghongfang commented Nov 16, 2021

chia7712 commented Nov 16, 2021

chinghongfang left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

chia7712 left a comment

Choose a reason for hiding this comment

chinghongfang commented Nov 19, 2021

chia7712 left a comment

Choose a reason for hiding this comment

chinghongfang commented Nov 28, 2021

chia7712 left a comment

Choose a reason for hiding this comment

chia7712 left a comment

Choose a reason for hiding this comment

chia7712 left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

chia7712 commented Nov 30, 2021

chinghongfang commented Nov 12, 2021 •

edited

Loading

chinghongfang Nov 14, 2021 •

edited

Loading