HDDS-9534. Support namespace summaries (du, dist & counts) for LEGACY buckets with file system disabled #5517

ArafatKhan2198 · 2023-10-31T05:58:18Z

What changes were proposed in this pull request?

Recon NSSummaries currently support FSO, OBS buckets, and Legacy buckets with the FSEnabled flag set to true. When FSEnabled is false, Legacy buckets act like OBS buckets, requiring a similar approach for managing keys. We've adapted our OBS bucket handling methods for these cases, maximizing code reuse.

Here's a breakdown of the changes :-

NSSummaryTaskWithLegacy.java: This includes methods to process keys in Legacy buckets, distinguishing between filesystem and object store layouts to generate NSSummaries accurately.
BucketHandler.java: Depending on whether FileSystemPaths is enabled, we assign the appropriate handler for each bucket. For Legacy buckets with disabled FileSystemPaths, we employ OBS bucket handler, thanks to their identical key management approach.
EntityHandler.java: This class is activated by the NSSummaryEndpoint and determines which handler to assign based on the path provided by the user. For Legacy buckets with the FileSystemPaths option disabled, we directly employ the OBSHandler and its associated methods. We have introduced specific normalization methods for OBS keys. OBS keys follow the format volumeName/bucketName/keyName, where the keyName can include any characters, even multiple slashes, such as “////KeyName.” There's no need to normalize the keyName itself, but it's essential to ensure the path up to the bucket is correctly normalized. This is because volumeName and bucketName must not contain multiple slashes or spaces. Our standard normalization methods for the FSO layout normalize the entire path, including the keyName, which could alter the keyName and lead to invalid output. To address this, I've added a method in the OmUtils.java class that normalizes the path up to the bucket level.

What is the link to the Apache JIRA

https://issues.apache.org/jira/browse/HDDS-9534

How was this patch tested?

Manual Testing and Unit Testing

ArafatKhan2198 · 2023-11-02T06:38:24Z

@devmadhuu @adoroszlai @GeorgeJahad Could you please take a look !!

...p-ozone/recon/src/main/java/org/apache/hadoop/ozone/recon/tasks/NSSummaryTaskWithLegacy.java

sumitagrawl

@ArafatKhan2198 Thanks for working over this, can add testcases for OBS Legacy type

...zone/recon/src/main/java/org/apache/hadoop/ozone/recon/api/handlers/LegacyBucketHandler.java

...p-ozone/recon/src/main/java/org/apache/hadoop/ozone/recon/tasks/NSSummaryTaskWithLegacy.java

ArafatKhan2198 · 2024-01-17T09:09:10Z

Thanks for the comments @sumitagrawl as informed I will have to merge this one first #4245 and then move on to incorporate the comments posted by you on this PR.

… buckets with file system disabled.

…y buckets with OBS layout

… for Legacy buckets

sumitagrawl

@ArafatKhan2198 Plz check, given few minor comments

hadoop-ozone/common/src/main/java/org/apache/hadoop/ozone/OmUtils.java

hadoop-ozone/recon/src/main/java/org/apache/hadoop/ozone/recon/api/handlers/EntityHandler.java

sumitagrawl

LGTM+1

dombizita

Thanks for the effort on this @ArafatKhan2198, it looks good to me, I have one minor comment.

...p-ozone/recon/src/main/java/org/apache/hadoop/ozone/recon/tasks/NSSummaryTaskWithLegacy.java

devmadhuu

Thanks @ArafatKhan2198 for continue working on this patch. Just few Nits

hadoop-ozone/common/src/main/java/org/apache/hadoop/ozone/OmUtils.java

devmadhuu · 2024-03-29T10:31:41Z

hadoop-ozone/common/src/main/java/org/apache/hadoop/ozone/OmUtils.java

+    // Add remaining segments as the key
+    if (segments.length > 2) {
+      normalizedPath.append("/").append(
+          String.join("/", Arrays.copyOfRange(segments, 2, segments.length)));


Suggested change

String.join("/", Arrays.copyOfRange(segments, 2, segments.length)));

String.join(OM_KEY_PREFIX, Arrays.copyOfRange(segments, 2, segments.length)));

...p-ozone/recon/src/main/java/org/apache/hadoop/ozone/recon/tasks/NSSummaryTaskWithLegacy.java

devmadhuu · 2024-03-29T13:48:50Z

@ArafatKhan2198 , you can ignore If (LOG.isDebugEnabled()) comments, as current logger library being used is already having check of If (LOG.isDebugEnabled()) , so you can fix rest of other comments.

devmadhuu

Thanks @ArafatKhan2198 . LGTM +1

devmadhuu · 2024-03-29T16:58:30Z

Thanks @dombizita @sumitagrawl for review.

… buckets with file system disabled (apache#5517) (cherry picked from commit cb5d519)

… for LEGACY buckets with file system disabled (apache#5517) (cherry picked from commit cb5d519) Change-Id: I6c05f74e99cd1628d0f8489f71f1f092b9ca896f

… buckets with file system disabled (apache#5517) (cherry picked from commit cb5d519)

adoroszlai added the recon label Oct 31, 2023

devmadhuu reviewed Nov 3, 2023

View reviewed changes

...p-ozone/recon/src/main/java/org/apache/hadoop/ozone/recon/tasks/NSSummaryTaskWithLegacy.java Show resolved Hide resolved

sumitagrawl reviewed Jan 11, 2024

View reviewed changes

ArafatKhan2198 added 4 commits March 1, 2024 12:16

HDDS-9534. Support namespace summaries (du, dist & counts) for LEGACY…

a19f92a

… buckets with file system disabled.

Removed the extra parameter added to NSSummary

fdad043

Modified the process method of NSSummaryTaskWitLegacy to handle Legac…

0e1fe0d

…y buckets with OBS layout

Added more java docs and refactoring

c86182e

ArafatKhan2198 force-pushed the HDDS-9534 branch from 521d9ef to c86182e Compare March 1, 2024 06:47

ArafatKhan2198 added 8 commits March 5, 2024 15:38

Added java docs and Logs

beabd5a

Using OBSBucketHanlder in case fileSystemPathEnabled flag set to true…

a3db461

… for Legacy buckets

Fixes a few edge cases

f9dc5cf

Added Unit tests

f40e121

Added unit tests for NSSummaryEndpoint With OBS and Legacy buckets

e56c3bf

Fixed config flag for legacy buckets

76c0262

Added more unit tests for the key level

cb83bda

Added licence header and fixed checkstyle issues

d15231f

ArafatKhan2198 requested review from sumitagrawl and devmadhuu March 7, 2024 16:51

ArafatKhan2198 added 3 commits March 7, 2024 22:24

Renamed a variable

ab9cb00

Fixed failing UT's

f637dd6

Fixed checkstyle issues

3e5d5e6

sumitagrawl reviewed Mar 25, 2024

View reviewed changes

hadoop-ozone/common/src/main/java/org/apache/hadoop/ozone/OmUtils.java Outdated Show resolved Hide resolved

hadoop-ozone/recon/src/main/java/org/apache/hadoop/ozone/recon/api/handlers/EntityHandler.java Show resolved Hide resolved

Added a test to test out the utility method normalizePathUptilBucket

14a0382

ArafatKhan2198 requested a review from sumitagrawl March 26, 2024 07:12

Fixed checkstyle issue

0b1f99c

sumitagrawl approved these changes Mar 28, 2024

View reviewed changes

dombizita reviewed Mar 28, 2024

View reviewed changes

...p-ozone/recon/src/main/java/org/apache/hadoop/ozone/recon/tasks/NSSummaryTaskWithLegacy.java Outdated Show resolved Hide resolved

Renamed setParentDirectoryId to setKeyParentID

efe3ebf

ArafatKhan2198 requested a review from dombizita March 28, 2024 12:33

dombizita approved these changes Mar 28, 2024

View reviewed changes

dombizita changed the title ~~HDDS-9534 Support namespace summaries (du, dist & counts) for LEGACY buckets with file system disabled~~ HDDS-9534. Support namespace summaries (du, dist & counts) for LEGACY buckets with file system disabled Mar 28, 2024

devmadhuu reviewed Mar 29, 2024

View reviewed changes

Made review changes

b7df717

devmadhuu approved these changes Mar 29, 2024

View reviewed changes

devmadhuu merged commit cb5d519 into apache:master Mar 29, 2024
38 checks passed

jojochuang pushed a commit to jojochuang/ozone that referenced this pull request May 29, 2024

HDDS-9534. Support namespace summaries (du, dist & counts) for LEGACY…

6f9e416

… buckets with file system disabled (apache#5517) (cherry picked from commit cb5d519)

xichen01 pushed a commit to xichen01/ozone that referenced this pull request Jul 17, 2024

HDDS-9534. Support namespace summaries (du, dist & counts) for LEGACY…

6a4ff53

… buckets with file system disabled (apache#5517) (cherry picked from commit cb5d519)

xichen01 pushed a commit to xichen01/ozone that referenced this pull request Jul 17, 2024

HDDS-9534. Support namespace summaries (du, dist & counts) for LEGACY…

3fada4b

… buckets with file system disabled (apache#5517) (cherry picked from commit cb5d519)

xichen01 pushed a commit to xichen01/ozone that referenced this pull request Jul 17, 2024

HDDS-9534. Support namespace summaries (du, dist & counts) for LEGACY…

bd80ce9

… buckets with file system disabled (apache#5517) (cherry picked from commit cb5d519)

xichen01 pushed a commit to xichen01/ozone that referenced this pull request Jul 18, 2024

HDDS-9534. Support namespace summaries (du, dist & counts) for LEGACY…

fb259c3

… buckets with file system disabled (apache#5517) (cherry picked from commit cb5d519)

xichen01 pushed a commit to xichen01/ozone that referenced this pull request Jul 18, 2024

HDDS-9534. Support namespace summaries (du, dist & counts) for LEGACY…

6eb81f3

… buckets with file system disabled (apache#5517) (cherry picked from commit cb5d519)

xichen01 mentioned this pull request Jul 18, 2024

[DO NOT MERGE] Backport some fixes, performance optimizations from master to ozone-1.4 #6929 #6964

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

HDDS-9534. Support namespace summaries (du, dist & counts) for LEGACY buckets with file system disabled #5517

HDDS-9534. Support namespace summaries (du, dist & counts) for LEGACY buckets with file system disabled #5517

ArafatKhan2198 commented Oct 31, 2023 •

edited

Loading

ArafatKhan2198 commented Nov 2, 2023

sumitagrawl left a comment

ArafatKhan2198 commented Jan 17, 2024

sumitagrawl left a comment

sumitagrawl left a comment

dombizita left a comment

devmadhuu left a comment

devmadhuu Mar 29, 2024

devmadhuu commented Mar 29, 2024

devmadhuu left a comment

devmadhuu commented Mar 29, 2024

	String.join("/", Arrays.copyOfRange(segments, 2, segments.length)));
	String.join(OM_KEY_PREFIX, Arrays.copyOfRange(segments, 2, segments.length)));

HDDS-9534. Support namespace summaries (du, dist & counts) for LEGACY buckets with file system disabled #5517

HDDS-9534. Support namespace summaries (du, dist & counts) for LEGACY buckets with file system disabled #5517

Conversation

ArafatKhan2198 commented Oct 31, 2023 • edited Loading

What changes were proposed in this pull request?

What is the link to the Apache JIRA

How was this patch tested?

ArafatKhan2198 commented Nov 2, 2023

sumitagrawl left a comment

Choose a reason for hiding this comment

ArafatKhan2198 commented Jan 17, 2024

sumitagrawl left a comment

Choose a reason for hiding this comment

sumitagrawl left a comment

Choose a reason for hiding this comment

dombizita left a comment

Choose a reason for hiding this comment

devmadhuu left a comment

Choose a reason for hiding this comment

devmadhuu Mar 29, 2024

Choose a reason for hiding this comment

devmadhuu commented Mar 29, 2024

devmadhuu left a comment

Choose a reason for hiding this comment

devmadhuu commented Mar 29, 2024

ArafatKhan2198 commented Oct 31, 2023 •

edited

Loading