DAOS-16477 mgmt: return suspect engines for pool healthy query #15458

wangshilong · 2024-11-06T06:38:46Z

After significant failures, the system may leave behind some suspect engines that were marked as DEAD by the SWIM protocol, but were not excluded from the system to prevent data loss. An administrator can bring these ranks back online by restarting them.

This PR aims to provide an administrative interface for querying suspect engines following a massive failure. These suspect engines can be retrieved using the daos/dmg --health-only command.

An example of output of dmg pool query --health-only:

Pool 6f450a68-8c7d-4da9-8900-02691650f6a2, ntarget=8, disabled=2, leader=3, version=4, state=Degraded Pool health info:

Disabled ranks: 1
Suspect ranks: 2
Rebuild busy, 0 objs, 0 recs

Features: DmgPoolQueryRanks
Required-githooks: true

Before requesting gatekeeper:

Two review approvals and any prior change requests have been resolved.
Testing is complete and all tests passed or there is a reason documented in the PR why it should be force landed and forced-landing tag is set.
Features: (or Test-tag*) commit pragma was used or there is a reason documented that there are no appropriate tags for this PR.
Commit messages follows the guidelines outlined here.
Any tests skipped by the ticket being addressed have been run and passed in the PR.

Gatekeeper:

After significant failures, the system may leave behind some suspect engines that were marked as DEAD by the SWIM protocol, but were not excluded from the system to prevent data loss. An administrator can bring these ranks back online by restarting them. This PR aims to provide an administrative interface for querying suspect engines following a massive failure. These suspect engines can be retrieved using the daos/dmg --health-only command. An example of output of dmg pool query --health-only: Pool 6f450a68-8c7d-4da9-8900-02691650f6a2, ntarget=8, disabled=2, leader=3, version=4, state=Degraded Pool health info: - Disabled ranks: 1 - Suspect ranks: 2 - Rebuild busy, 0 objs, 0 recs Features: DmgPoolQueryRanks Required-githooks: true Signed-off-by: Wang Shilong <shilong.wang@intel.com>

github-actions · 2024-11-06T06:39:30Z

Ticket title is 'Provide admin interface to query hanging engines after massive failure'
Status is 'In Review'
https://daosio.atlassian.net/browse/DAOS-16477

wangshilong · 2024-11-07T00:50:10Z

To reviewers: This PR landed before but got reverted because of conflicts with MD-on-SSD phase2 PR, i refreshed the PR with master and removed a walkaround in the pool query tests(since rebuild bug fixed)

liw

C changes look good to me.

daosbuild1 · 2024-11-07T14:47:33Z

Test stage Functional Hardware Medium completed with status FAILURE. https://build.hpdd.intel.com//job/daos-stack/job/daos/view/change-requests/job/PR-15458/1/execution/node/1478/log

phender

There is at least one fix needed. I also have a couple questions about the intent of some changes.

src/tests/ftest/control/dmg_pool_query_ranks.py

src/tests/ftest/util/server_utils_params.py

Required-githooks: true Features: DmgPoolQueryRanks Signed-off-by: Wang Shilong <shilong.wang@intel.com>

daosbuild1 · 2024-11-11T04:33:14Z

Test stage NLT on EL 8.8 completed with status UNSTABLE. https://build.hpdd.intel.com/job/daos-stack/job/daos//view/change-requests/job/PR-15458/2/testReport/

daosbuild1 · 2024-11-11T08:38:41Z

Test stage Functional on EL 8.8 completed with status FAILURE. https://build.hpdd.intel.com//job/daos-stack/job/daos/view/change-requests/job/PR-15458/2/execution/node/1196/log

wangshilong · 2024-11-12T01:08:38Z

@phender would you help fix CI env issue? thanks!

Features: DmgPoolQueryRanks Signed-off-by: Phil Henderson <phillip.henderson@intel.com>

daltonbohning · 2024-11-13T18:23:30Z

src/tests/ftest/util/server_utils_params.py

-            "CRT_EVENT_DELAY=1",
-            "COVFILE=/tmp/test.cov"],
+            EnvironmentalVariable("D_LOG_FILE_APPEND_PID", "1"),
+            EnvironmentalVariable("DAOS_POOL_RF", "4", True),


Why is this necessary? Couldn't we just let all of these be overridden and then we don't need the new class EnvironmentalVariable at all?

daosbuild1 · 2024-11-13T18:33:37Z

Test stage NLT on EL 8.8 completed with status UNSTABLE. https://build.hpdd.intel.com/job/daos-stack/job/daos//view/change-requests/job/PR-15458/3/testReport/

daosbuild1 · 2024-11-13T20:49:20Z

Test stage Test RPMs on EL 8.6 completed with status FAILURE. https://build.hpdd.intel.com//job/daos-stack/job/daos/view/change-requests/job/PR-15458/3/execution/node/1150/log

src/tests/ftest/control/dmg_pool_query_ranks.py

daltonbohning

ftest LGTM

kjacque

C and Go changes LGTM. Comment is minor.

kjacque · 2024-11-16T01:35:58Z

src/mgmt/tests/srv_drpc_tests.c

+	flags |= DEFAULT_QUERY_BITS;
+	assert_int_equal(ds_mgmt_pool_query_info_in.pi_bits, DEFAULT_QUERY_BITS | flags);


Redundant bitwise OR

Suggested change

flags |= DEFAULT_QUERY_BITS;

assert_int_equal(ds_mgmt_pool_query_info_in.pi_bits, DEFAULT_QUERY_BITS | flags);

assert_int_equal(ds_mgmt_pool_query_info_in.pi_bits, DEFAULT_QUERY_BITS | flags);

* DAOS-16477 mgmt: return suspect engines for pool healthy query After significant failures, the system may leave behind some suspect engines that were marked as DEAD by the SWIM protocol, but were not excluded from the system to prevent data loss. An administrator can bring these ranks back online by restarting them. This PR aims to provide an administrative interface for querying suspect engines following a massive failure. These suspect engines can be retrieved using the daos/dmg --health-only command. An example of output of dmg pool query --health-only: Pool 6f450a68-8c7d-4da9-8900-02691650f6a2, ntarget=8, disabled=2, leader=3, version=4, state=Degraded Pool health info: - Disabled ranks: 1 - Suspect ranks: 2 - Rebuild busy, 0 objs, 0 recs Signed-off-by: Wang Shilong <shilong.wang@intel.com> Signed-off-by: Phil Henderson <phillip.henderson@intel.com> Co-authored-by: Phil Henderson <phillip.henderson@intel.com>

* DAOS-16477 mgmt: return suspect engines for pool healthy query After significant failures, the system may leave behind some suspect engines that were marked as DEAD by the SWIM protocol, but were not excluded from the system to prevent data loss. An administrator can bring these ranks back online by restarting them. This PR aims to provide an administrative interface for querying suspect engines following a massive failure. These suspect engines can be retrieved using the daos/dmg --health-only command. An example of output of dmg pool query --health-only: Pool 6f450a68-8c7d-4da9-8900-02691650f6a2, ntarget=8, disabled=2, leader=3, version=4, state=Degraded Pool health info: - Disabled ranks: 1 - Suspect ranks: 2 - Rebuild busy, 0 objs, 0 recs Features: DmgPoolQueryRanks Required-githooks: true Signed-off-by: Wang Shilong <shilong.wang@intel.com> Signed-off-by: Phil Henderson <phillip.henderson@intel.com> Co-authored-by: Phil Henderson <phillip.henderson@intel.com>

* DAOS-16477 mgmt: return suspect engines for pool healthy query After significant failures, the system may leave behind some suspect engines that were marked as DEAD by the SWIM protocol, but were not excluded from the system to prevent data loss. An administrator can bring these ranks back online by restarting them. This PR aims to provide an administrative interface for querying suspect engines following a massive failure. These suspect engines can be retrieved using the daos/dmg --health-only command. An example of output of dmg pool query --health-only: Pool 6f450a68-8c7d-4da9-8900-02691650f6a2, ntarget=8, disabled=2, leader=3, version=4, state=Degraded Pool health info: - Disabled ranks: 1 - Suspect ranks: 2 - Rebuild busy, 0 objs, 0 recs Features: DmgPoolQueryRanks skip-nlt: true Required-githooks: true Signed-off-by: Wang Shilong <shilong.wang@intel.com> Signed-off-by: Phil Henderson <phillip.henderson@intel.com> Co-authored-by: Phil Henderson <phillip.henderson@intel.com>

* DAOS-16477 mgmt: return suspect engines for pool healthy query After significant failures, the system may leave behind some suspect engines that were marked as DEAD by the SWIM protocol, but were not excluded from the system to prevent data loss. An administrator can bring these ranks back online by restarting them. This PR aims to provide an administrative interface for querying suspect engines following a massive failure. These suspect engines can be retrieved using the daos/dmg --health-only command. An example of output of dmg pool query --health-only: Pool 6f450a68-8c7d-4da9-8900-02691650f6a2, ntarget=8, disabled=2, leader=3, version=4, state=Degraded Pool health info: - Disabled ranks: 1 - Suspect ranks: 2 - Rebuild busy, 0 objs, 0 recs Features: DmgPoolQueryRanks Required-githooks: true Signed-off-by: Wang Shilong <shilong.wang@intel.com> Signed-off-by: Phil Henderson <phillip.henderson@intel.com> Co-authored-by: Phil Henderson <phillip.henderson@intel.com>

* DAOS-16477 mgmt: return suspect engines for pool healthy query After significant failures, the system may leave behind some suspect engines that were marked as DEAD by the SWIM protocol, but were not excluded from the system to prevent data loss. An administrator can bring these ranks back online by restarting them. This PR aims to provide an administrative interface for querying suspect engines following a massive failure. These suspect engines can be retrieved using the daos/dmg --health-only command. An example of output of dmg pool query --health-only: Pool 6f450a68-8c7d-4da9-8900-02691650f6a2, ntarget=8, disabled=2, leader=3, version=4, state=Degraded Pool health info: - Disabled ranks: 1 - Suspect ranks: 2 - Rebuild busy, 0 objs, 0 recs Features: DmgPoolQueryRanks Signed-off-by: Wang Shilong <shilong.wang@intel.com> Signed-off-by: Phil Henderson <phillip.henderson@intel.com> Co-authored-by: Phil Henderson <phillip.henderson@intel.com>

* DAOS-16477 mgmt: return suspect engines for pool healthy query After significant failures, the system may leave behind some suspect engines that were marked as DEAD by the SWIM protocol, but were not excluded from the system to prevent data loss. An administrator can bring these ranks back online by restarting them. This PR aims to provide an administrative interface for querying suspect engines following a massive failure. These suspect engines can be retrieved using the daos/dmg --health-only command. An example of output of dmg pool query --health-only: Pool 6f450a68-8c7d-4da9-8900-02691650f6a2, ntarget=8, disabled=2, leader=3, version=4, state=Degraded Pool health info: - Disabled ranks: 1 - Suspect ranks: 2 - Rebuild busy, 0 objs, 0 recs Features: DmgPoolQueryRanks skip-nlt: true Required-githooks: true Signed-off-by: Wang Shilong <shilong.wang@intel.com> Signed-off-by: Phil Henderson <phillip.henderson@intel.com> Co-authored-by: Phil Henderson <phillip.henderson@intel.com>

After significant failures, the system may leave behind some suspect engines that were marked as DEAD by the SWIM protocol, but were not excluded from the system to prevent data loss. An administrator can bring these ranks back online by restarting them. This PR aims to provide an administrative interface for querying suspect engines following a massive failure. These suspect engines can be retrieved using the daos/dmg --health-only command. An example of output of dmg pool query --health-only: Pool 6f450a68-8c7d-4da9-8900-02691650f6a2, ntarget=8, disabled=2, leader=3, version=4, state=Degraded Pool health info: - Disabled ranks: 1 - Suspect ranks: 2 - Rebuild busy, 0 objs, 0 recs Required-githooks: true Signed-off-by: Wang Shilong <shilong.wang@intel.com> Signed-off-by: Phil Henderson <phillip.henderson@intel.com> Co-authored-by: Phil Henderson <phillip.henderson@intel.com>

… (#15512) After significant failures, the system may leave behind some suspect engines that were marked as DEAD by the SWIM protocol, but were not excluded from the system to prevent data loss. An administrator can bring these ranks back online by restarting them. This PR aims to provide an administrative interface for querying suspect engines following a massive failure. These suspect engines can be retrieved using the daos/dmg --health-only command. An example of output of dmg pool query --health-only: Pool 6f450a68-8c7d-4da9-8900-02691650f6a2, ntarget=8, disabled=2, leader=3, version=4, state=Degraded Pool health info: - Disabled ranks: 1 - Suspect ranks: 2 - Rebuild busy, 0 objs, 0 recs Signed-off-by: Wang Shilong <shilong.wang@intel.com> Co-authored-by: Phil Henderson <phillip.henderson@intel.com>

wangshilong marked this pull request as ready for review November 7, 2024 00:47

wangshilong requested review from a team as code owners November 7, 2024 00:47

wangshilong requested review from tanabarr, liw and kjacque November 7, 2024 00:47

liw previously approved these changes Nov 7, 2024

View reviewed changes

tanabarr previously approved these changes Nov 7, 2024

View reviewed changes

phender requested changes Nov 7, 2024

View reviewed changes

address some comments

92c6882

Required-githooks: true Features: DmgPoolQueryRanks Signed-off-by: Wang Shilong <shilong.wang@intel.com>

wangshilong dismissed stale reviews from tanabarr and liw via 92c6882 November 11, 2024 03:51

Add support for required server env vars that can be overridden.

006b523

Features: DmgPoolQueryRanks Signed-off-by: Phil Henderson <phillip.henderson@intel.com>

daltonbohning reviewed Nov 13, 2024

View reviewed changes

phender approved these changes Nov 15, 2024

View reviewed changes

phender requested review from daltonbohning, tanabarr and liw November 15, 2024 22:38

daltonbohning reviewed Nov 15, 2024

View reviewed changes

src/tests/ftest/control/dmg_pool_query_ranks.py Show resolved Hide resolved

daltonbohning reviewed Nov 15, 2024

View reviewed changes

kjacque approved these changes Nov 16, 2024

View reviewed changes

wangshilong requested a review from a team November 16, 2024 03:07

gnailzenh approved these changes Nov 16, 2024

View reviewed changes

gnailzenh merged commit f512f48 into master Nov 16, 2024
53 of 54 checks passed

gnailzenh deleted the shilongw/DAOS-16477_1 branch November 16, 2024 06:18

mjmac mentioned this pull request Dec 12, 2024

mjmac/dead ranks #15609

Merged

mjmac mentioned this pull request Jan 13, 2025

Merge release/2.6 into google/2.6 #15725

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

DAOS-16477 mgmt: return suspect engines for pool healthy query #15458

DAOS-16477 mgmt: return suspect engines for pool healthy query #15458

wangshilong commented Nov 6, 2024

github-actions bot commented Nov 6, 2024

wangshilong commented Nov 7, 2024

liw left a comment

daosbuild1 commented Nov 7, 2024

phender left a comment

daosbuild1 commented Nov 11, 2024

daosbuild1 commented Nov 11, 2024

wangshilong commented Nov 12, 2024

daltonbohning Nov 13, 2024

daosbuild1 commented Nov 13, 2024

daosbuild1 commented Nov 13, 2024

daltonbohning left a comment

kjacque left a comment

kjacque Nov 16, 2024

		flags \|= DEFAULT_QUERY_BITS;
		assert_int_equal(ds_mgmt_pool_query_info_in.pi_bits, DEFAULT_QUERY_BITS \| flags);

DAOS-16477 mgmt: return suspect engines for pool healthy query #15458

DAOS-16477 mgmt: return suspect engines for pool healthy query #15458

Conversation

wangshilong commented Nov 6, 2024

Before requesting gatekeeper:

Gatekeeper:

github-actions bot commented Nov 6, 2024

wangshilong commented Nov 7, 2024

liw left a comment

Choose a reason for hiding this comment

daosbuild1 commented Nov 7, 2024

phender left a comment

Choose a reason for hiding this comment

daosbuild1 commented Nov 11, 2024

daosbuild1 commented Nov 11, 2024

wangshilong commented Nov 12, 2024

daltonbohning Nov 13, 2024

Choose a reason for hiding this comment

daosbuild1 commented Nov 13, 2024

daosbuild1 commented Nov 13, 2024

daltonbohning left a comment

Choose a reason for hiding this comment

kjacque left a comment

Choose a reason for hiding this comment

kjacque Nov 16, 2024

Choose a reason for hiding this comment