HADOOP-19474: [ABFS][FnsOverBlob] Listing Optimizations to avoid multiple iteration over list response. #7421

anujmodi2021 · 2025-02-21T05:01:03Z

Description of PR

On blob endpoint, there are a couple of handling that is needed to be done on client side.
This involves:

Parsing of xml response and converting them to VersionedFileStatus list
Removing duplicate entries for non-empty explicit directories coming due to presence of the marker files
Trigerring Rename recovery on the previously failed rename indicated by the presence of pending json file.

Currently all three are done in a separate iteration over whole list. This is to pbring all those things to a common place so that single iteration over list reposne can handle all three.

How was this patch tested?

For code changes:

...p-azure/src/main/java/org/apache/hadoop/fs/azurebfs/contracts/services/ListResponseData.java

anmolanmol1234 · 2025-03-06T06:44:48Z

...-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/services/AbfsBlobClient.java

+        throw new AbfsDriverException(e);
+      }
+    } catch (IOException e) {
+      LOG.error("Unable to deserialize list results", e);


Given we have the uri now, should we include that in the error log as well ?

Nice, taken

...-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/services/AbfsBlobClient.java

...tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/AzureBlobFileSystemStore.java

...p-azure/src/main/java/org/apache/hadoop/fs/azurebfs/contracts/services/ListResponseData.java

.../hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/services/AbfsAHCHttpOperation.java

...ols/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/services/AbfsHttpOperation.java

...s/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/services/VersionedFileStatus.java

...hadoop-azure/src/test/java/org/apache/hadoop/fs/azurebfs/ITestAzureBlobFileSystemRename.java

...-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/services/AbfsBlobClient.java

...s/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/services/VersionedFileStatus.java

hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/services/AbfsClient.java

...p-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/services/AbfsDfsClient.java

...ols/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/services/AbfsHttpOperation.java

...op-azure/src/test/java/org/apache/hadoop/fs/azurebfs/ITestAzureBlobFileSystemListStatus.java

anmolanmol1234 · 2025-03-26T11:50:44Z

...op-azure/src/test/java/org/apache/hadoop/fs/azurebfs/ITestAzureBlobFileSystemListStatus.java

+    assertDirectoryFileStatus(fileStatus, qualifiedPath);
+    DirectoryStateHelper.isImplicitDirectory(qualifiedPath, getFileSystem(),
+        getTestTracingContext(getFileSystem(), true));
+    Assertions.assertThat(fileStatus.getModificationTime()).isEqualTo(0);


Assertion statements are missing for all tests

...hadoop-azure/src/test/java/org/apache/hadoop/fs/azurebfs/ITestAzureBlobFileSystemRename.java

...ools/hadoop-azure/src/test/java/org/apache/hadoop/fs/azurebfs/ITestAbfsCustomEncryption.java

...s/hadoop-azure/src/test/java/org/apache/hadoop/fs/azurebfs/services/TestListActionTaker.java

...-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/services/AbfsBlobClient.java

hadoop-yetus · 2025-03-26T14:40:22Z

🎊 +1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 🆗	reexec	0m 53s		Docker mode activated.
			_ Prechecks _
+1 💚	dupname	0m 0s		No case conflicting files found.
+0 🆗	codespell	0m 0s		codespell was not available.
+0 🆗	detsecrets	0m 0s		detect-secrets was not available.
+0 🆗	xmllint	0m 0s		xmllint was not available.
+1 💚	@author	0m 1s		The patch does not contain any @author tags.
+1 💚	test4tests	0m 0s		The patch appears to include 11 new or modified test files.
			_ trunk Compile Tests _
+1 💚	mvninstall	39m 45s		trunk passed
+1 💚	compile	0m 41s		trunk passed with JDK Ubuntu-11.0.26+4-post-Ubuntu-1ubuntu120.04
+1 💚	compile	0m 36s		trunk passed with JDK Private Build-1.8.0_442-8u442-b06~~us1-0ubuntu1~~20.04-b06
+1 💚	checkstyle	0m 33s		trunk passed
+1 💚	mvnsite	0m 40s		trunk passed
+1 💚	javadoc	0m 41s		trunk passed with JDK Ubuntu-11.0.26+4-post-Ubuntu-1ubuntu120.04
+1 💚	javadoc	0m 34s		trunk passed with JDK Private Build-1.8.0_442-8u442-b06~~us1-0ubuntu1~~20.04-b06
+1 💚	spotbugs	1m 9s		trunk passed
+1 💚	shadedclient	38m 46s		branch has no errors when building and testing our client artifacts.
-0 ⚠️	patch	39m 8s		Used diff version of patch file. Binary files and potentially other changes not applied. Please rebase and squash commits if necessary.
			_ Patch Compile Tests _
+1 💚	mvninstall	0m 30s		the patch passed
+1 💚	compile	0m 33s		the patch passed with JDK Ubuntu-11.0.26+4-post-Ubuntu-1ubuntu120.04
+1 💚	javac	0m 33s		the patch passed
+1 💚	compile	0m 28s		the patch passed with JDK Private Build-1.8.0_442-8u442-b06~~us1-0ubuntu1~~20.04-b06
+1 💚	javac	0m 28s		the patch passed
+1 💚	blanks	0m 0s		The patch has no blanks issues.
+1 💚	checkstyle	0m 21s		the patch passed
+1 💚	mvnsite	0m 32s		the patch passed
+1 💚	javadoc	0m 29s		the patch passed with JDK Ubuntu-11.0.26+4-post-Ubuntu-1ubuntu120.04
+1 💚	javadoc	0m 25s		the patch passed with JDK Private Build-1.8.0_442-8u442-b06~~us1-0ubuntu1~~20.04-b06
+1 💚	spotbugs	1m 7s		the patch passed
+1 💚	shadedclient	38m 54s		patch has no errors when building and testing our client artifacts.
			_ Other Tests _
+1 💚	unit	3m 0s		hadoop-azure in the patch passed.
+1 💚	asflicense	0m 36s		The patch does not generate ASF License warnings.
		132m 28s

Subsystem	Report/Notes
Docker	ClientAPI=1.48 ServerAPI=1.48 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7421/12/artifact/out/Dockerfile
GITHUB PR	#7421
JIRA Issue	HADOOP-19474
Optional Tests	dupname asflicense codespell detsecrets xmllint compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle
uname	Linux f5d50c8ccc68 5.15.0-131-generic #141-Ubuntu SMP Fri Jan 10 21:18:28 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/bin/hadoop.sh
git revision	trunk / `9728899`
Default Java	Private Build-1.8.0_442-8u442-b06~~us1-0ubuntu1~~20.04-b06
Multi-JDK versions	/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.26+4-post-Ubuntu-1ubuntu120.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_442-8u442-b06~~us1-0ubuntu1~~20.04-b06
Test Results	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7421/12/testReport/
Max. process+thread count	529 (vs. ulimit of 5500)
modules	C: hadoop-tools/hadoop-azure U: hadoop-tools/hadoop-azure
Console output	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7421/12/console
versions	git=2.25.1 maven=3.6.3 spotbugs=4.2.2
Powered by	Apache Yetus 0.14.0 https://yetus.apache.org

This message was automatically generated.

anujmodi2021 · 2025-03-27T10:16:01Z

============================================================
HNS-OAuth-DFS

[WARNING] Tests run: 174, Failures: 0, Errors: 0, Skipped: 4
[WARNING] Tests run: 796, Failures: 0, Errors: 0, Skipped: 162
[WARNING] Tests run: 154, Failures: 0, Errors: 0, Skipped: 4
[WARNING] Tests run: 272, Failures: 0, Errors: 0, Skipped: 23

============================================================
HNS-SharedKey-DFS

[WARNING] Tests run: 174, Failures: 0, Errors: 0, Skipped: 5
[WARNING] Tests run: 799, Failures: 0, Errors: 0, Skipped: 131
[WARNING] Tests run: 154, Failures: 0, Errors: 0, Skipped: 4
[WARNING] Tests run: 272, Failures: 0, Errors: 0, Skipped: 10

============================================================
NonHNS-SharedKey-DFS

[WARNING] Tests run: 174, Failures: 0, Errors: 0, Skipped: 11
[WARNING] Tests run: 638, Failures: 0, Errors: 0, Skipped: 214
[WARNING] Tests run: 154, Failures: 0, Errors: 0, Skipped: 5
[WARNING] Tests run: 272, Failures: 0, Errors: 0, Skipped: 11

============================================================
AppendBlob-HNS-OAuth-DFS

[WARNING] Tests run: 174, Failures: 0, Errors: 0, Skipped: 4
[WARNING] Tests run: 796, Failures: 0, Errors: 0, Skipped: 169
[WARNING] Tests run: 131, Failures: 0, Errors: 0, Skipped: 5
[WARNING] Tests run: 272, Failures: 0, Errors: 0, Skipped: 23

============================================================
NonHNS-SharedKey-Blob

[WARNING] Tests run: 174, Failures: 0, Errors: 0, Skipped: 11
[WARNING] Tests run: 641, Failures: 0, Errors: 0, Skipped: 144
[WARNING] Tests run: 154, Failures: 0, Errors: 0, Skipped: 3
[WARNING] Tests run: 272, Failures: 0, Errors: 0, Skipped: 11

============================================================
NonHNS-OAuth-DFS

[WARNING] Tests run: 174, Failures: 0, Errors: 0, Skipped: 11
[WARNING] Tests run: 635, Failures: 0, Errors: 0, Skipped: 215
[WARNING] Tests run: 154, Failures: 0, Errors: 0, Skipped: 5
[WARNING] Tests run: 272, Failures: 0, Errors: 0, Skipped: 24

============================================================
NonHNS-OAuth-Blob

[WARNING] Tests run: 174, Failures: 0, Errors: 0, Skipped: 11
[WARNING] Tests run: 638, Failures: 0, Errors: 0, Skipped: 145
[WARNING] Tests run: 154, Failures: 0, Errors: 0, Skipped: 3
[WARNING] Tests run: 272, Failures: 0, Errors: 0, Skipped: 24

============================================================
AppendBlob-NonHNS-OAuth-Blob

[WARNING] Tests run: 174, Failures: 0, Errors: 0, Skipped: 11
[WARNING] Tests run: 636, Failures: 0, Errors: 0, Skipped: 163
[WARNING] Tests run: 131, Failures: 0, Errors: 0, Skipped: 4
[WARNING] Tests run: 272, Failures: 0, Errors: 0, Skipped: 24

============================================================
HNS-Oauth-DFS-IngressBlob

[WARNING] Tests run: 174, Failures: 0, Errors: 0, Skipped: 4
[WARNING] Tests run: 670, Failures: 0, Errors: 0, Skipped: 165
[WARNING] Tests run: 154, Failures: 0, Errors: 0, Skipped: 4
[WARNING] Tests run: 272, Failures: 0, Errors: 0, Skipped: 23

============================================================
NonHNS-OAuth-DFS-IngressBlob

[WARNING] Tests run: 174, Failures: 0, Errors: 0, Skipped: 11
[WARNING] Tests run: 635, Failures: 0, Errors: 0, Skipped: 213
[WARNING] Tests run: 154, Failures: 0, Errors: 0, Skipped: 5
[WARNING] Tests run: 272, Failures: 0, Errors: 0, Skipped: 24

Branch: HADOOP-19234_followup, Commit: 9728899

…iple iteration over list response. (apache#7421) Contributed by Anuj Modi Reviewed by: Anmol Asrani, Manish Bhatt, Manika Joshi Signed off by: Anuj Modi<anujmodi@apache.org>

…iple iteration over list response. (#7421) (#7581) Contributed by Anuj Modi Reviewed by: Anmol Asrani, Manish Bhatt, Manika Joshi Signed off by: Anuj Modi<anujmodi@apache.org>

anujmodi2021 added 5 commits February 13, 2025 23:41

Refactoring ListPath Call

0121838

Test Changes'

4eb7c8a

More refraction

5a4576e

Removing Unnecessary code changes

a12335c

Fixing HNS Test

604b900

github-actions bot added trunk TOOLS ABFS labels Feb 21, 2025

This comment was marked as outdated.

Sign in to view

anujmodi2021 added 2 commits February 26, 2025 21:34

Merge branch 'trunk' into HADOOP-19234_followup

c4db0bf

Test Fixes

f3c4fe6

This comment was marked as outdated.

Sign in to view

anujmodi2021 changed the title ~~Hadoop 19234 followup~~ HADOOP-19474: [ABFS][FnsOverBlob] Listing Optimizations to avoid multiple iteration over list response. Mar 3, 2025

anujmodi2021 added 3 commits March 2, 2025 21:58

Removing Unused Import

a0bf5f1

Test fixes

b37c3d2

Fix tests

ec1419b

This comment was marked as outdated.

Sign in to view

Yetus Checks

3edab0f

anujmodi2021 marked this pull request as ready for review March 4, 2025 08:47

This comment was marked as outdated.

Sign in to view

anmolanmol1234 reviewed Mar 6, 2025

View reviewed changes

...p-azure/src/main/java/org/apache/hadoop/fs/azurebfs/contracts/services/ListResponseData.java Outdated Show resolved Hide resolved

anmolanmol1234 reviewed Mar 6, 2025

View reviewed changes

...-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/services/AbfsBlobClient.java Show resolved Hide resolved

anujmodi2021 added 3 commits March 15, 2025 22:48

Merge branch 'trunk' into HADOOP-19234_followup

db0d589

Fixing Failing Tests

10163c8

Added Negative Scenarios Test

2b13bbd

This comment was marked as outdated.

Sign in to view

PR Checks

5d34f65

This comment was marked as outdated.

Sign in to view

Fixing Faling test

21f590a

This comment was marked as outdated.

Sign in to view

anujmodi2021 added 3 commits March 25, 2025 23:36

Improved logging

38f7269

Logging on DFS Listing

4e19d7b

Unused Import

e5a021a

This comment was marked as outdated.

Sign in to view

bhattmanish98 reviewed Mar 26, 2025

View reviewed changes

anmolanmol1234 reviewed Mar 26, 2025

View reviewed changes

...s/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/services/VersionedFileStatus.java Show resolved Hide resolved

anmolanmol1234 reviewed Mar 26, 2025

View reviewed changes

hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/services/AbfsClient.java Show resolved Hide resolved

anmolanmol1234 reviewed Mar 26, 2025

View reviewed changes

hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/services/AbfsClient.java Show resolved Hide resolved

anmolanmol1234 reviewed Mar 26, 2025

View reviewed changes

hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/services/AbfsClient.java Show resolved Hide resolved

anmolanmol1234 reviewed Mar 26, 2025

View reviewed changes

...p-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/services/AbfsDfsClient.java Outdated Show resolved Hide resolved

anmolanmol1234 reviewed Mar 26, 2025

View reviewed changes

...ols/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/services/AbfsHttpOperation.java Show resolved Hide resolved

anmolanmol1234 reviewed Mar 26, 2025

View reviewed changes

...op-azure/src/test/java/org/apache/hadoop/fs/azurebfs/ITestAzureBlobFileSystemListStatus.java Outdated Show resolved Hide resolved

anmolanmol1234 reviewed Mar 26, 2025

View reviewed changes

...hadoop-azure/src/test/java/org/apache/hadoop/fs/azurebfs/ITestAzureBlobFileSystemRename.java Outdated Show resolved Hide resolved

manika137 reviewed Mar 26, 2025

View reviewed changes

...ools/hadoop-azure/src/test/java/org/apache/hadoop/fs/azurebfs/ITestAbfsCustomEncryption.java Show resolved Hide resolved

manika137 reviewed Mar 26, 2025

View reviewed changes

...ools/hadoop-azure/src/test/java/org/apache/hadoop/fs/azurebfs/ITestAbfsCustomEncryption.java Show resolved Hide resolved

manika137 reviewed Mar 26, 2025

View reviewed changes

...s/hadoop-azure/src/test/java/org/apache/hadoop/fs/azurebfs/services/TestListActionTaker.java Show resolved Hide resolved

Addressing Comments

9728899

anmolanmol1234 reviewed Mar 26, 2025

View reviewed changes

...-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/services/AbfsBlobClient.java Show resolved Hide resolved

anmolanmol1234 approved these changes Mar 26, 2025

View reviewed changes

manika137 approved these changes Mar 27, 2025

View reviewed changes

bhattmanish98 approved these changes Mar 27, 2025

View reviewed changes

anujmodi2021 merged commit 140f047 into apache:trunk Mar 27, 2025
4 checks passed

anujmodi2021 deleted the HADOOP-19234_followup branch April 10, 2025 06:31

HADOOP-19474: [ABFS][FnsOverBlob] Listing Optimizations to avoid multiple iteration over list response. #7421

HADOOP-19474: [ABFS][FnsOverBlob] Listing Optimizations to avoid multiple iteration over list response. #7421

Uh oh!

Conversation

anujmodi2021 commented Feb 21, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description of PR

How was this patch tested?

For code changes:

Uh oh!

This comment was marked as outdated.

This comment was marked as outdated.

This comment was marked as outdated.

This comment was marked as outdated.

This comment was marked as outdated.

Uh oh!

anmolanmol1234 Mar 6, 2025

Choose a reason for hiding this comment

Uh oh!

anujmodi2021 Mar 20, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!

This comment was marked as outdated.

This comment was marked as outdated.

This comment was marked as outdated.

This comment was marked as outdated.

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

anmolanmol1234 Mar 26, 2025

Choose a reason for hiding this comment

Uh oh!

anujmodi2021 Mar 26, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

hadoop-yetus commented Mar 26, 2025

Uh oh!

anujmodi2021 commented Mar 27, 2025

============================================================ HNS-OAuth-DFS

============================================================ HNS-SharedKey-DFS

============================================================ NonHNS-SharedKey-DFS

============================================================ AppendBlob-HNS-OAuth-DFS

============================================================ NonHNS-SharedKey-Blob

============================================================ NonHNS-OAuth-DFS

============================================================ NonHNS-OAuth-Blob

============================================================ AppendBlob-NonHNS-OAuth-Blob

============================================================ HNS-Oauth-DFS-IngressBlob

============================================================ NonHNS-OAuth-DFS-IngressBlob

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

anujmodi2021 commented Feb 21, 2025 •

edited

Loading

============================================================
HNS-OAuth-DFS

============================================================
HNS-SharedKey-DFS

============================================================
NonHNS-SharedKey-DFS

============================================================
AppendBlob-HNS-OAuth-DFS

============================================================
NonHNS-SharedKey-Blob

============================================================
NonHNS-OAuth-DFS

============================================================
NonHNS-OAuth-Blob

============================================================
AppendBlob-NonHNS-OAuth-Blob

============================================================
HNS-Oauth-DFS-IngressBlob

============================================================
NonHNS-OAuth-DFS-IngressBlob