Skip to content

HADOOP-18146: ABFS: Added changes for expect hundred continue header #4039 #5516

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
steveloughran merged 2 commits into from
Mar 28, 2023
Merged

HADOOP-18146: ABFS: Added changes for expect hundred continue header #4039 #5516

steveloughran merged 2 commits into from
Mar 28, 2023

Conversation

@anmolanmol1234
Copy link
Contributor

@anmolanmol1234 anmolanmol1234 commented Mar 28, 2023 •
edited
Loading

Added Expect Hundred Continue Header with all append requests
-> Heavy load from a Hadoop cluster lead to high resource utilization at FE nodes. Investigations from the server side indicate payload buffering at Http.Sys as the cause. Payload of requests that eventually fail due to throttling limits are also getting buffered, as its triggered before FE could start request processing.

Approach: Client sends Append Http request with Expect header, but holds back on payload transmission until server replies back with HTTP 100. We add this header for all append requests so as to reduce load on the server.

@anmolanmol1234
HADOOP-18146: ABFS: Added changes for expect hundred continue header (a…
c88011c 
…pache#4039)


This change lets the client react pre-emptively to server load without getting to 503 and the exponential backoff
which follows. This stops performance suffering so much as capacity limits are approached for an account.

Contributed by Anmol Asranii
@anmolanmol1234
RestOperation fix
b40fd6a
@anmolanmol1234
Copy link
Contributor Author

anmolanmol1234 commented Mar 28, 2023

:::: AGGREGATED TEST RESULT ::::

HNS-OAuth

[INFO] Results:
[INFO]
[ERROR] Failures:
[ERROR] TestAccountConfiguration.testConfigPropNotFound:386->testMissingConfigKey:399 Expected a org.apache.hadoop.fs.azurebfs.contracts.exceptions.TokenAccessProviderException to be thrown, but got the result: : "org.apache.hadoop.fs.azurebfs.oauth2.ClientCredsTokenProvider"
[INFO]
[ERROR] Tests run: 106, Failures: 1, Errors: 0, Skipped: 4
[INFO] Results:
[INFO]
[WARNING] Tests run: 580, Failures: 0, Errors: 0, Skipped: 54
[INFO] Results:
[INFO]
[ERROR] Failures:
[ERROR] ITestAbfsFileSystemContractSeek.testSeekAndReadWithReadAhead:130->assertNoIncrementInRemoteReadOps:258 [Number of remote read ops shouldn't increase] expected:<[1]L> but was:<[2]L>
[INFO]
[ERROR] Tests run: 336, Failures: 1, Errors: 0, Skipped: 41

HNS-SharedKey

[INFO] Results:
[INFO]
[ERROR] Failures:
[ERROR] TestAccountConfiguration.testConfigPropNotFound:386->testMissingConfigKey:399 Expected a org.apache.hadoop.fs.azurebfs.contracts.exceptions.TokenAccessProviderException to be thrown, but got the result: : "org.apache.hadoop.fs.azurebfs.oauth2.ClientCredsTokenProvider"
[INFO]
[ERROR] Tests run: 106, Failures: 1, Errors: 0, Skipped: 4
[INFO] Results:
[INFO]
[WARNING] Tests run: 580, Failures: 0, Errors: 0, Skipped: 54
[INFO] Results:
[INFO]
[ERROR] Failures:
[ERROR] ITestAbfsFileSystemContractSeek.testSeekAndReadWithReadAhead:130->assertNoIncrementInRemoteReadOps:258 [Number of remote read ops shouldn't increase] expected:<[1]L> but was:<[2]L>
[INFO]
[ERROR] Tests run: 336, Failures: 1, Errors: 0, Skipped: 41

NonHNS-SharedKey

[INFO] Results:
[INFO]
[ERROR] Failures:
[ERROR] TestAccountConfiguration.testConfigPropNotFound:386->testMissingConfigKey:399 Expected a org.apache.hadoop.fs.azurebfs.contracts.exceptions.TokenAccessProviderException to be thrown, but got the result: : "org.apache.hadoop.fs.azurebfs.oauth2.ClientCredsTokenProvider"
[INFO]
[ERROR] Tests run: 106, Failures: 1, Errors: 0, Skipped: 4
[INFO] Results:
[INFO]
[WARNING] Tests run: 580, Failures: 0, Errors: 0, Skipped: 277
[INFO] Results:
[INFO]
[ERROR] Failures:
[ERROR] ITestAbfsTerasort.test_110_teragen:244->executeStage:211->Assert.assertEquals:647->Assert.failNotEquals:835->Assert.fail:89 teragen(1000, abfs://testfnsrename@anmolasranicinonhnstest.dfs.core.windows.net/ITestAbfsTerasort/sortin) failed expected:<0> but was:<1>
[ERROR] Errors:
[ERROR] ITestAbfsJobThroughManifestCommitter.test_0420_validateJob » OutputValidation ...
[ERROR] ITestAbfsManifestCommitProtocol.testCommitLifecycle » OutputValidation abfs:/... [ERROR] ITestAbfsManifestCommitProtocol.testCommitterWithDuplicatedCommit » OutputValidation [ERROR] ITestAbfsManifestCommitProtocol.testConcurrentCommitTaskWithSubDir » OutputValidation [ERROR] ITestAbfsManifestCommitProtocol.testMapFileOutputCommitter » OutputValidation ... [ERROR] ITestAbfsManifestCommitProtocol.testOutputFormatIntegration » OutputValidation [ERROR] ITestAbfsManifestCommitProtocol.testParallelJobsToAdjacentPaths » OutputValidation [ERROR] ITestAbfsManifestCommitProtocol.testTwoTaskAttemptsCommit » OutputValidation ...
[INFO]
[ERROR] Tests run: 336, Failures: 1, Errors: 8, Skipped: 46

NonHNS-OAuth

[INFO] Results:
[INFO]
[ERROR] Failures:
[ERROR] TestAccountConfiguration.testConfigPropNotFound:386->testMissingConfigKey:399 Expected a org.apache.hadoop.fs.azurebfs.contracts.exceptions.TokenAccessProviderException to be thrown, but got the result: : "org.apache.hadoop.fs.azurebfs.oauth2.ClientCredsTokenProvider"
[INFO]
[ERROR] Tests run: 106, Failures: 1, Errors: 0, Skipped: 4
[INFO] Results:
[INFO]
[WARNING] Tests run: 580, Failures: 0, Errors: 0, Skipped: 277
[INFO] Results:
[INFO]
[ERROR] Failures:
[ERROR] ITestAbfsTerasort.test_110_teragen:244->executeStage:211->Assert.assertEquals:647->Assert.failNotEquals:835->Assert.fail:89 teragen(1000, abfs://testfnsrename@anmolasranicinonhnstest.dfs.core.windows.net/ITestAbfsTerasort/sortin) failed expected:<0> but was:<1>
[ERROR] Errors:
[ERROR] ITestAbfsJobThroughManifestCommitter.test_0420_validateJob » OutputValidation ...
[ERROR] ITestAbfsManifestCommitProtocol.testCommitLifecycle » OutputValidation abfs:/... [ERROR] ITestAbfsManifestCommitProtocol.testCommitterWithDuplicatedCommit » OutputValidation [ERROR] ITestAbfsManifestCommitProtocol.testConcurrentCommitTaskWithSubDir » OutputValidation [ERROR] ITestAbfsManifestCommitProtocol.testMapFileOutputCommitter » OutputValidation ... [ERROR] ITestAbfsManifestCommitProtocol.testOutputFormatIntegration » OutputValidation [ERROR] ITestAbfsManifestCommitProtocol.testParallelJobsToAdjacentPaths » OutputValidation [ERROR] ITestAbfsManifestCommitProtocol.testTwoTaskAttemptsCommit » OutputValidation ...
[INFO]
[ERROR] Tests run: 336, Failures: 1, Errors: 8, Skipped: 46

AppendBlob-HNS-OAuth

[INFO] Results:
[INFO]
[ERROR] Failures:
[ERROR] TestAccountConfiguration.testConfigPropNotFound:386->testMissingConfigKey:399 Expected a org.apache.hadoop.fs.azurebfs.contracts.exceptions.TokenAccessProviderException to be thrown, but got the result: : "org.apache.hadoop.fs.azurebfs.oauth2.ClientCredsTokenProvider"
[INFO]
[ERROR] Tests run: 106, Failures: 1, Errors: 0, Skipped: 4
[INFO] Results:
[INFO]
[WARNING] Tests run: 580, Failures: 0, Errors: 0, Skipped: 54
[INFO] Results:
[INFO]
[ERROR] Failures:
[ERROR] ITestAbfsFileSystemContractSeek.testSeekAndReadWithReadAhead:130->assertNoIncrementInRemoteReadOps:258 [Number of remote read ops shouldn't increase] expected:<[1]L> but was:<[2]L>
[INFO]
[ERROR] Tests run: 336, Failures: 1, Errors: 0, Skipped: 41

Time taken: 106 mins 44 secs.

@hadoop-yetus
Copy link

hadoop-yetus commented Mar 28, 2023

🎊 +1 overall

Vote Subsystem Runtime Logfile Comment
+0 🆗 reexec 8m 43s Docker mode activated.
_ Prechecks _
+1 💚 dupname 0m 0s No case conflicting files found.
+0 🆗 codespell 0m 0s codespell was not available.
+0 🆗 detsecrets 0m 0s detect-secrets was not available.
+0 🆗 xmllint 0m 0s xmllint was not available.
+0 🆗 markdownlint 0m 1s markdownlint was not available.
+1 💚 @author 0m 0s The patch does not contain any @author tags.
+1 💚 test4tests 0m 0s The patch appears to include 10 new or modified test files.
_ branch-3.3 Compile Tests _
+1 💚 mvninstall 36m 39s branch-3.3 passed
+1 💚 compile 0m 37s branch-3.3 passed
+1 💚 checkstyle 0m 34s branch-3.3 passed
+1 💚 mvnsite 0m 43s branch-3.3 passed
+1 💚 javadoc 0m 39s branch-3.3 passed
+1 💚 spotbugs 1m 16s branch-3.3 passed
+1 💚 shadedclient 23m 49s branch has no errors when building and testing our client artifacts.
_ Patch Compile Tests _
+1 💚 mvninstall 0m 36s the patch passed
+1 💚 compile 0m 27s the patch passed
+1 💚 javac 0m 27s the patch passed
+1 💚 blanks 0m 0s The patch has no blanks issues.
+1 💚 checkstyle 0m 19s the patch passed
+1 💚 mvnsite 0m 32s the patch passed
+1 💚 javadoc 0m 23s the patch passed
+1 💚 spotbugs 1m 3s the patch passed
+1 💚 shadedclient 23m 38s patch has no errors when building and testing our client artifacts.
_ Other Tests _
+1 💚 unit 2m 13s hadoop-azure in the patch passed.
+1 💚 asflicense 0m 38s The patch does not generate ASF License warnings.
103m 56s
Subsystem Report/Notes
Docker ClientAPI=1.42 ServerAPI=1.42 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5516/1/artifact/out/Dockerfile
GITHUB PR #5516
Optional Tests dupname asflicense codespell detsecrets xmllint compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle markdownlint
uname Linux 110828bfc452 4.15.0-206-generic #217-Ubuntu SMP Fri Feb 3 19:10:13 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux
Build tool maven
Personality dev-support/bin/hadoop.sh
git revision branch-3.3 / b40fd6a
Default Java Private Build-1.8.0_362-8u362-ga-0ubuntu1~18.04.1-b09
Test Results https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5516/1/testReport/
Max. process+thread count 609 (vs. ulimit of 5500)
modules C: hadoop-tools/hadoop-azure U: hadoop-tools/hadoop-azure
Console output https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5516/1/console
versions git=2.17.1 maven=3.6.0 spotbugs=4.2.2
Powered by Apache Yetus 0.14.0 https://yetus.apache.org

This message was automatically generated.

@anmolanmol1234 anmolanmol1234 changed the title HADOOP-18146 Backport https://github.com/apache/hadoop/pull/4039 into branch 3.3 HADOOP-18146: ABFS: Added changes for expect hundred continue header #4039 Mar 28, 2023
@anmolanmol1234
Copy link
Contributor Author

anmolanmol1234 commented Mar 28, 2023 •
edited
Loading

@steveloughran, as discussed on #4039. I have backported the change in branch-3.3 Requesting you to kindly review it. Thanks.

steveloughran
steveloughran reviewed Mar 28, 2023
View reviewed changes
Copy link
Contributor

@steveloughran steveloughran left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM
+1

@steveloughran steveloughran merged commit 6306f5b into apache:branch-3.3 Mar 28, 2023
@anmolanmol1234 anmolanmol1234 mentioned this pull request May 17, 2023
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@steveloughran steveloughran steveloughran left review comments

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

3 participants

@anmolanmol1234 @hadoop-yetus @steveloughran