Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

GenericMultipartHelper.handleException needlessly wraps SdkClientException #4356

Closed
steveloughran opened this issue Aug 28, 2023 · 3 comments · Fixed by #4453
Closed

GenericMultipartHelper.handleException needlessly wraps SdkClientException #4356

steveloughran opened this issue Aug 28, 2023 · 3 comments · Fixed by #4453
Labels
bug This issue is a bug. p2 This is a standard priority issue

Comments

@steveloughran
Copy link

Describe the bug

GenericMultipartHelper.handleException() will throw Errors, but for any other exception, it wraps it in an SdkClientException

this is

  1. needless
  2. loses information when the cause is one of the subtypes

Expected Behavior

SdkClientException to be passed directly to completeExceptionally() rather than inside a new exception

Current Behavior

exceptions of all types are wrapped.

Reproduction Steps

throw a subclass of an SdkClientException from a custom handler within the SDK. observe that it is translated. this complicates any attempt to recognise and process the custom exception; code now needs to look at the inner cause. it also makes for longer stack traces, as shown below

[ERROR] test_030_postCreationAssertions(org.apache.hadoop.fs.s3a.scale.ITestS3AHugeFilesNoMultipart)  Time elapsed: 130.969 s  <<< ERROR!
org.apache.hadoop.fs.s3a.AWSClientIOException: copyFile(tests3ascale/disk/hugefile, tests3ascale/disk/hugefileRenamed) on tests3ascale/disk/hugefile: software.amazon.awssdk.core.exception.SdkClientException: Failed to initiate multipart upload: Failed to initiate multipart upload
        at org.apache.hadoop.fs.s3a.S3AUtils.translateException(S3AUtils.java:194)
        at org.apache.hadoop.fs.s3a.Invoker.once(Invoker.java:124)
        at org.apache.hadoop.fs.s3a.Invoker.lambda$retry$4(Invoker.java:376)
        at org.apache.hadoop.fs.s3a.Invoker.retryUntranslated(Invoker.java:468)
        at org.apache.hadoop.fs.s3a.Invoker.retry(Invoker.java:372)
        at org.apache.hadoop.fs.s3a.Invoker.retry(Invoker.java:347)
        at org.apache.hadoop.fs.s3a.S3AFileSystem.copyFile(S3AFileSystem.java:4439)
        at org.apache.hadoop.fs.s3a.S3AFileSystem.access$2300(S3AFileSystem.java:283)
        at org.apache.hadoop.fs.s3a.S3AFileSystem$OperationCallbacksImpl.copyFile(S3AFileSystem.java:2432)
        at org.apache.hadoop.fs.s3a.impl.RenameOperation.copySource(RenameOperation.java:561)
        at org.apache.hadoop.fs.s3a.impl.RenameOperation.renameFileToDest(RenameOperation.java:312)
        at org.apache.hadoop.fs.s3a.impl.RenameOperation.execute(RenameOperation.java:266)
        at org.apache.hadoop.fs.s3a.S3AFileSystem.innerRename(S3AFileSystem.java:2351)
        at org.apache.hadoop.fs.s3a.S3AFileSystem.lambda$rename$8(S3AFileSystem.java:2202)
        at org.apache.hadoop.fs.statistics.impl.IOStatisticsBinding.invokeTrackingDuration(IOStatisticsBinding.java:547)
        at org.apache.hadoop.fs.statistics.impl.IOStatisticsBinding.lambda$trackDurationOfOperation$5(IOStatisticsBinding.java:528)
        at org.apache.hadoop.fs.statistics.impl.IOStatisticsBinding.trackDuration(IOStatisticsBinding.java:449)
        at org.apache.hadoop.fs.s3a.S3AFileSystem.trackDurationAndSpan(S3AFileSystem.java:2623)
        at org.apache.hadoop.fs.s3a.S3AFileSystem.rename(S3AFileSystem.java:2200)
        at org.apache.hadoop.fs.s3a.scale.ITestS3AHugeFilesNoMultipart.lambda$test_030_postCreationAssertions$0(ITestS3AHugeFilesNoMultipart.java:108)
        at org.apache.hadoop.test.LambdaTestUtils.intercept(LambdaTestUtils.java:498)
        at org.apache.hadoop.test.LambdaTestUtils.intercept(LambdaTestUtils.java:384)
        at org.apache.hadoop.fs.s3a.scale.ITestS3AHugeFilesNoMultipart.test_030_postCreationAssertions(ITestS3AHugeFilesNoMultipart.java:107)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:59)
        at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
        at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:56)
        at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
        at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
        at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
        at org.junit.rules.TestWatcher$1.evaluate(TestWatcher.java:61)
        at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:299)
        at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:293)
        at java.util.concurrent.FutureTask.run(FutureTask.java:266)
        at java.lang.Thread.run(Thread.java:750)
Caused by: software.amazon.awssdk.core.exception.SdkClientException: Failed to initiate multipart upload
        at software.amazon.awssdk.core.exception.SdkClientException$BuilderImpl.build(SdkClientException.java:111)
        at software.amazon.awssdk.core.exception.SdkClientException.create(SdkClientException.java:47)
        at software.amazon.awssdk.services.s3.internal.multipart.GenericMultipartHelper.handleException(GenericMultipartHelper.java:65)
        at software.amazon.awssdk.services.s3.internal.multipart.CopyObjectHelper.lambda$copyInParts$6(CopyObjectHelper.java:115)
        at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:774)
        at java.util.concurrent.CompletableFuture.uniWhenCompleteStage(CompletableFuture.java:792)
        at java.util.concurrent.CompletableFuture.whenComplete(CompletableFuture.java:2153)
        at software.amazon.awssdk.services.s3.internal.multipart.CopyObjectHelper.copyInParts(CopyObjectHelper.java:113)
        at software.amazon.awssdk.services.s3.internal.multipart.CopyObjectHelper.doCopyObject(CopyObjectHelper.java:98)
        at software.amazon.awssdk.services.s3.internal.multipart.CopyObjectHelper.lambda$copyObject$1(CopyObjectHelper.java:79)
        at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:774)
        at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:750)
        at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:488)
        at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1975)
        at software.amazon.awssdk.core.internal.http.pipeline.stages.AsyncApiCallMetricCollectionStage.lambda$execute$0(AsyncApiCallMetricCollectionStage.java:58)
        at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:774)
        at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:750)
        at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:488)
        at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1975)
        at software.amazon.awssdk.core.internal.http.pipeline.stages.AsyncApiCallTimeoutTrackingStage.lambda$execute$2(AsyncApiCallTimeoutTrackingStage.java:69)
        at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:774)
        at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:750)
        at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:488)
        at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1975)
        at software.amazon.awssdk.core.internal.http.pipeline.stages.AsyncRetryableStage$RetryingExecutor.lambda$attemptExecute$1(AsyncRetryableStage.java:177)
        at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:774)
        at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:750)
        at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:488)
        at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1975)
        at software.amazon.awssdk.core.internal.http.pipeline.stages.MakeAsyncHttpRequestStage.lambda$null$0(MakeAsyncHttpRequestStage.java:105)
        at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:774)
        at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:750)
        at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:488)
        at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1975)
        at software.amazon.awssdk.core.internal.http.pipeline.stages.MakeAsyncHttpRequestStage.completeResponseFuture(MakeAsyncHttpRequestStage.java:238)
        at software.amazon.awssdk.core.internal.http.pipeline.stages.MakeAsyncHttpRequestStage.lambda$executeHttpRequest$3(MakeAsyncHttpRequestStage.java:163)
        at java.util.concurrent.CompletableFuture.uniHandle(CompletableFuture.java:836)
        at java.util.concurrent.CompletableFuture$UniHandle.tryFire(CompletableFuture.java:811)
        at java.util.concurrent.CompletableFuture$Completion.run(CompletableFuture.java:456)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        ... 1 more
Caused by: org.apache.hadoop.fs.s3a.audit.AuditOperationRejectedException: Multipart IO request CreateMultipartUploadRequest(Bucket=stevel-london, ContentType=application/octet-stream, Key=tests3ascale/disk/hugefileRenamed, Metadata={}, ServerSideEncryption=aws:kms, SSEKMSKeyId=*** Sensitive Data Redacted ***) rejected https://audit.example.org/hadoop/1/op_rename/1a74e618-2e61-4fd6-ab65-9a86d4fc96d1-00000017/?op=op_rename&p1=s3a://stevel-london/tests3ascale/disk/hugefile&pr=stevel&p2=s3a://stevel-london/tests3ascale/disk/hugefileRenamed&ps=ad53d2f7-214b-4521-a535-be03c7876d22&id=1a74e618-2e61-4fd6-ab65-9a86d4fc96d1-00000017&t0=61&fs=1a74e618-2e61-4fd6-ab65-9a86d4fc96d1&t1=105&ts=1693234405164
        at org.apache.hadoop.fs.s3a.audit.impl.LoggingAuditor$LoggingAuditSpan.modifyHttpRequest(LoggingAuditor.java:405)
        at org.apache.hadoop.fs.s3a.audit.impl.ActiveAuditManagerS3A$WrappingAuditSpan.modifyHttpRequest(ActiveAuditManagerS3A.java:871)
        at org.apache.hadoop.fs.s3a.audit.impl.ActiveAuditManagerS3A.modifyHttpRequest(ActiveAuditManagerS3A.java:612)
        at software.amazon.awssdk.core.interceptor.ExecutionInterceptorChain.modifyHttpRequestAndHttpContent(ExecutionInterceptorChain.java:90)
        at software.amazon.awssdk.core.internal.handler.BaseClientHandler.runModifyHttpRequestAndHttpContentInterceptors(BaseClientHandler.java:157)
        at software.amazon.awssdk.core.internal.handler.BaseClientHandler.finalizeSdkHttpFullRequest(BaseClientHandler.java:83)
        at software.amazon.awssdk.core.internal.handler.BaseAsyncClientHandler.doExecute(BaseAsyncClientHandler.java:201)
        at software.amazon.awssdk.core.internal.handler.BaseAsyncClientHandler.lambda$execute$1(BaseAsyncClientHandler.java:82)
        at software.amazon.awssdk.core.internal.handler.BaseAsyncClientHandler.measureApiCallSuccess(BaseAsyncClientHandler.java:291)
        at software.amazon.awssdk.core.internal.handler.BaseAsyncClientHandler.execute(BaseAsyncClientHandler.java:75)
        at software.amazon.awssdk.awscore.client.handler.AwsAsyncClientHandler.execute(AwsAsyncClientHandler.java:52)
        at software.amazon.awssdk.services.s3.DefaultS3AsyncClient.createMultipartUpload(DefaultS3AsyncClient.java:1550)
        at software.amazon.awssdk.services.s3.DelegatingS3AsyncClient.lambda$createMultipartUpload$4(DelegatingS3AsyncClient.java:1273)
        at software.amazon.awssdk.services.s3.internal.multipart.MultipartS3AsyncClient$1.invokeOperation(MultipartS3AsyncClient.java:91)
        at software.amazon.awssdk.services.s3.DelegatingS3AsyncClient.createMultipartUpload(DelegatingS3AsyncClient.java:1273)
        at software.amazon.awssdk.services.s3.internal.multipart.CopyObjectHelper.copyInParts(CopyObjectHelper.java:108)
        ... 34 more

Possible Solution

don't wrap SdkClientException

Additional Information/Context

No response

AWS Java SDK version used

2.20.128

JDK version used

openjdk version "1.8.0_362" OpenJDK Runtime Environment (Zulu 8.68.0.21-CA-macos-aarch64) (build 1.8.0_362-b09) OpenJDK 64-Bit Server VM (Zulu 8.68.0.21-CA-macos-aarch64) (build 25.362-b09, mixed mode

Operating System and version

macos
@steveloughran steveloughran added bug This issue is a bug. needs-triage This issue or PR still needs to be triaged. labels Aug 28, 2023
@debora-ito debora-ito added needs-review This issue or PR needs review from the team. and removed needs-triage This issue or PR still needs to be triaged. labels Aug 30, 2023
@debora-ito
Copy link
Member

Acknowledged.

@debora-ito debora-ito added p2 This is a standard priority issue and removed needs-review This issue or PR needs review from the team. labels Sep 6, 2023
@steveloughran
Copy link
Author

steveloughran commented Sep 7, 2023 •
edited
Loading

you should be aware that this did break downstream test code and so is a regression which surfaces when moving from the v1 to v2 sdk.

HADOOP-18863. AWS SDK V2 - AuditFailureExceptions aren't being translated properly

@zoewangg zoewangg mentioned this issue Sep 19, 2023
Merged
12 tasks
@github-actions
Copy link

⚠️COMMENT VISIBILITY WARNING⚠️

Comments on closed issues are hard for our team to see.
If you need more assistance, please open a new issue that references this one.
If you wish to keep having a conversation with other community members under this issue feel free to do so.

@zoewangg zoewangg mentioned this issue Oct 23, 2023
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
bug This issue is a bug. p2 This is a standard priority issue
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Fixed the issue where SdkException is unnecessarily re-wrapped with S… aws/aws-sdk-java-v2
2 participants
@steveloughran @debora-ito