HDDS-11120. Rich rebalancing status info #6911

juncevich · 2024-07-08T16:19:55Z

What changes were proposed in this pull request?

Problem: rebalancing status show information only about running rebalancing.

Need to add additional information in rebalancing status cli command.

Rebalancing start time
Rebalancing parameters
How many containers need to be balanced and how many containers from then still unbalanced
How many data in Gb need to transfered in general and how many untransfered yet
Which datanodes took a part in rebalancing

As idea to use flags for command to get more information about rebalancing status. For example use v or -verbose.

What is the link to the Apache JIRA

https://issues.apache.org/jira/browse/HDDS-11120

How was this patch tested?

Patch tested by unit tests.

kerneltime · 2024-07-08T17:09:23Z

@juncevich thank you for taking the time to post a PR, can you please follow the contributing process and create a Jira describing your change and then update the description of the PR to reflect the PR specific details?

juncevich · 2024-07-08T20:48:24Z

@juncevich thank you for taking the time to post a PR, can you please follow the contributing process and create a Jira describing your change and then update the description of the PR to reflect the PR specific details?

@kerneltime thank you for notice about PR. I tried to change PR describe information. But i have a question. Is it critical, that branch doesn't contain task number? Do i have reopen PR with properly named branch?

adoroszlai · 2024-07-08T20:55:58Z

Is it critical, that branch doesn't contain task number? Do i have reopen PR with properly named branch?

No need to reopen, you are free to use any branch name. Including Jira issue ID in the branch name helps folks with multiple tasks organize their own branches.

adoroszlai

Thanks @juncevich for the patch, and for iterating on it as draft.

I haven't checked the balancer-specific part, am only commenting on some general items.

hadoop-hdds/interface-admin/src/main/resources/proto.lock

hadoop-hdds/interface-admin/src/main/proto/ScmAdminProtocol.proto

hadoop-ozone/dist/src/main/smoketest/balancer/testBalancer.robot

...ds/tools/src/main/java/org/apache/hadoop/hdds/scm/cli/ContainerBalancerStatusSubcommand.java

hadoop-hdds/tools/pom.xml

juncevich · 2024-07-13T07:56:59Z

Thanks @juncevich for the patch, and for iterating on it as draft.

I haven't checked the balancer-specific part, am only commenting on some general items.

Hi, @adoroszlai! Cool!. Thank you for your review. Will fix notices.

adoroszlai

Thanks @juncevich for updating the patch.

Most of the changes in response to previous review comments are fine.

...ds/tools/src/main/java/org/apache/hadoop/hdds/scm/cli/ContainerBalancerStatusSubcommand.java

hadoop-ozone/dist/src/main/smoketest/balancer/testBalancer.robot

adoroszlai · 2024-07-19T08:34:39Z

@juncevich GitHub allows restricting re-run to failed checks, which saves time and resources compared to full run. It also avoids the situation where each new full run may have different intermittent failures. So please don't push empty commit to restart checks.

juncevich · 2024-07-19T08:56:09Z

@juncevich GitHub allows restricting re-run to failed checks, which saves time and resources compared to full run. It also avoids the situation where each new full run may have different intermittent failures. So please don't push empty commit to restart checks.

Sorry, bit i didn't found that buttons in build.

adoroszlai · 2024-07-19T09:02:39Z

Sorry, bit i didn't foind that buttons in build

It requires write access to the repository, so committers can re-run failed tests for you.

juncevich · 2024-07-19T09:14:06Z

Sorry, bit i didn't foind that buttons in build

It requires write access to the repository, so committers can re-run failed tests for you.

Have I ask any comitter to rerun task ?

adoroszlai · 2024-07-19T09:19:41Z

Have I ask any comitter to rerun task ?

In this case it's not needed, because the new commit requires a new full run anyway, which I have already triggered.

ivandika3

@juncevich Thanks for the improvement. Overall LGTM. Only a few nitpicks.

...erver-scm/src/main/java/org/apache/hadoop/hdds/scm/container/balancer/ContainerBalancer.java

hadoop-hdds/interface-admin/src/main/proto/ScmAdminProtocol.proto

ivandika3

Thanks for the update, LGTM +1.

ivandika3 · 2024-07-22T10:47:48Z

Thanks @juncevich for the patch, @adoroszlai for the review.

…-delete * HDDS-10239-container-reconciliation: (184 commits) HDDS-10373. Implement framework for capturing Merkle Tree Metrics. (apache#6864) HDDS-11188. Initial setup for new UI layout and enable users to switch to new UI (apache#6953) HDDS-11120. Rich rebalancing status info (apache#6911) HDDS-11187. Fix Event Handling in Recon OMDBUpdatesHandler to Prevent ClassCastException. (apache#6950) HDDS-11213. Bump commons-daemon to 1.4.0 (apache#6971) HDDS-11212. Bump commons-net to 3.11.1 (apache#6973) HDDS-11211. Bump assertj-core to 3.26.3 (apache#6972) HDDS-11210. Bump log4j2 to 2.23.1 (apache#6970) HDDS-11150. Recon Overview page crashes due to failed API Calls (apache#6944) HDDS-11183. Keys from DeletedTable and DeletedDirTable of AOS should be deleted on batch operation while creating a snapshot (apache#6946) HDDS-11198. Fix Typescript configs for Recon (apache#6961) HDDS-11180. Simplify HttpServer2#inferMimeType return statement (apache#6963) HDDS-11194. OM missing audit log for upgrade (apache#6958) HDDS-10389. Implement a search feature for users to locate open keys within the Open Keys Insights section. (apache#6231) HDDS-10561. Dashboard for delete key metrics (apache#6948) HDDS-11192. Increase SPNEGO URL test coverage (apache#6956) HDDS-11179. DBConfigFromFile#readFromFile result of toIOException not thrown (apache#6957) HDDS-11186. First container log missing from bundle (apache#6952) HDDS-10844. Clarify snapshot create error message. (apache#6955) HDDS-11166. Switch to Rocky Linux-based ozone-runner (apache#6942) ...

siddhantsangwan · 2024-08-29T11:53:29Z

@juncevich thanks for contributing this, it's a very useful feature. I opened https://issues.apache.org/jira/browse/HDDS-11385 to fix bugs and make improvements in this area. Please see if you're interested in working on any of the open tasks!

juncevich · 2024-08-29T11:58:51Z

@siddhantsangwan, thanks. I can take this task. Can you describe problems in Jira task?

siddhantsangwan · 2024-08-29T13:36:52Z

@siddhantsangwan, thanks. I can take this task. Can you describe problems in Jira task?

I've already described problems in sub-tasks. Let me know if you have any questions. Also opened a PR - #7134.

juncevich · 2024-08-29T13:43:28Z

https://issues.apache.org/jira/browse/HDDS-11385

Thanks. Didn't see subtask list.

(cherry picked from commit a5e420c)

Init commit

5bf295e

juncevich marked this pull request as draft July 8, 2024 16:20

juncevich changed the title ~~Rich balancing status info~~ HDDS-11120. Rich rebalancing status info Jul 8, 2024

adoroszlai requested a review from siddhantsangwan July 8, 2024 20:56

juncevich added 16 commits July 9, 2024 09:56

HDDS-11120. Fix compilation

9dafda8

HDDS-11120. Fix checkstyle errors

8d78b36

HDDS-11120. Fix findbugs errors

04fb871

HDDS-11120. Fix findbugs errors

b25c6c0

HDDS-11120. Fix rat check

648dfde

HDDS-11120. Try to fix robot balancer test

7b3eb0c

HDDS-11120. Try to fix robot balancer test

a0e92f1

HDDS-11120. Try to fix robot balancer test

2f0b05e

HDDS-11120. Try to fix robot balancer test

a9b9864

HDDS-11120. Try to fix robot balancer test

1dda8b3

HDDS-11120. Fix checkstyle

0bbe031

HDDS-11120. Try to fix robot balancer test

20cf496

HDDS-11120. Try to fix robot balancer test

947ae3b

HDDS-11120. TFix test + refactoring + modify balancer robot test

e7cd45b

HDDS-11120. Fix formatting.

7c035be

HDDS-11120. Fix balancing test.

b6ea9a5

juncevich marked this pull request as ready for review July 12, 2024 10:00

adoroszlai reviewed Jul 13, 2024

View reviewed changes

juncevich added 4 commits July 14, 2024 11:52

HDDS-11120. Fix review notices.

68b12b1

HDDS-11120. Fix review notices.

1538ee9

HDDS-11120. Fix testBalancer.

04bb57e

HDDS-11120. Change -h command description.

4a39571

juncevich requested a review from adoroszlai July 15, 2024 06:26

adoroszlai reviewed Jul 15, 2024

View reviewed changes

juncevich added 2 commits July 17, 2024 10:32

HDDS-11120. Fix review notices.

2207557

HDDS-11120. Change test name.

daa7da4

juncevich requested a review from adoroszlai July 17, 2024 09:55

juncevich added 3 commits July 17, 2024 18:58

HDDS-11120. Remove testBalancerStatus test.

9b51a85

HDDS-11120. Fix testBalancer test.

ffafee0

Empty commit message for restart pr checker

fe663c1

adoroszlai requested review from sumitagrawl and ivandika3 July 19, 2024 13:50

ivandika3 reviewed Jul 20, 2024

View reviewed changes

...erver-scm/src/main/java/org/apache/hadoop/hdds/scm/container/balancer/ContainerBalancer.java Outdated Show resolved Hide resolved

...erver-scm/src/main/java/org/apache/hadoop/hdds/scm/container/balancer/ContainerBalancer.java Outdated Show resolved Hide resolved

ivandika3 reviewed Jul 20, 2024

View reviewed changes

hadoop-hdds/interface-admin/src/main/proto/ScmAdminProtocol.proto Outdated Show resolved Hide resolved

HDDS-11120. Fix review notices.

b59431b

ivandika3 approved these changes Jul 22, 2024

View reviewed changes

ivandika3 merged commit a5e420c into apache:master Jul 22, 2024
39 checks passed

juncevich deleted the rich-status-info branch July 22, 2024 13:48

xichen01 pushed a commit to xichen01/ozone that referenced this pull request Sep 16, 2024

HDDS-11120. Rich rebalancing status info (apache#6911)

4945688

(cherry picked from commit a5e420c)

xichen01 pushed a commit to xichen01/ozone that referenced this pull request Sep 18, 2024

HDDS-11120. Rich rebalancing status info (apache#6911)

360bb28

(cherry picked from commit a5e420c)

xichen01 mentioned this pull request Sep 19, 2024

[DO NOT MERGE] Backport some fixes and compatibility commits from master to ozone-1.4 #7218

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

HDDS-11120. Rich rebalancing status info #6911

HDDS-11120. Rich rebalancing status info #6911

juncevich commented Jul 8, 2024 •

edited

Loading

kerneltime commented Jul 8, 2024

juncevich commented Jul 8, 2024

adoroszlai commented Jul 8, 2024

adoroszlai left a comment

juncevich commented Jul 13, 2024

adoroszlai left a comment

adoroszlai commented Jul 19, 2024

juncevich commented Jul 19, 2024 •

edited

Loading

adoroszlai commented Jul 19, 2024

juncevich commented Jul 19, 2024

adoroszlai commented Jul 19, 2024 •

edited

Loading

ivandika3 left a comment

ivandika3 left a comment

ivandika3 commented Jul 22, 2024

siddhantsangwan commented Aug 29, 2024

juncevich commented Aug 29, 2024

siddhantsangwan commented Aug 29, 2024

juncevich commented Aug 29, 2024

HDDS-11120. Rich rebalancing status info #6911

HDDS-11120. Rich rebalancing status info #6911

Conversation

juncevich commented Jul 8, 2024 • edited Loading

What changes were proposed in this pull request?

What is the link to the Apache JIRA

How was this patch tested?

kerneltime commented Jul 8, 2024

juncevich commented Jul 8, 2024

adoroszlai commented Jul 8, 2024

adoroszlai left a comment

Choose a reason for hiding this comment

juncevich commented Jul 13, 2024

adoroszlai left a comment

Choose a reason for hiding this comment

adoroszlai commented Jul 19, 2024

juncevich commented Jul 19, 2024 • edited Loading

adoroszlai commented Jul 19, 2024

juncevich commented Jul 19, 2024

adoroszlai commented Jul 19, 2024 • edited Loading

ivandika3 left a comment

Choose a reason for hiding this comment

ivandika3 left a comment

Choose a reason for hiding this comment

ivandika3 commented Jul 22, 2024

siddhantsangwan commented Aug 29, 2024

juncevich commented Aug 29, 2024

siddhantsangwan commented Aug 29, 2024

juncevich commented Aug 29, 2024

juncevich commented Jul 8, 2024 •

edited

Loading

juncevich commented Jul 19, 2024 •

edited

Loading

adoroszlai commented Jul 19, 2024 •

edited

Loading