HADOOP-19242. Aliyun oss add a redirection switch #6967
+283
−0
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
🎊 +1 overall
This message was automatically generated. |
zeekling
approved these changes
Aug 6, 2024
masteryhx
reviewed
Aug 29, 2024
There was a problem hiding this comment.
Thanks for the PR.
PTAL my comments.
@steveloughran Kindly ping. Could you also help to take a look ? Thanks.
| } | ||
| } | ||
|
|
||
| private void constructOssClient(Configuration conf) throws IOException { |
There was a problem hiding this comment.
Could we avoid repeating these codes?
There was a problem hiding this comment.
such as is done in AliyunOSSTestUtils
| @@ -147,6 +147,13 @@ public void initialize(URI uri, Configuration conf, String user, | |||
| throw new IllegalArgumentException(msg); | |||
| } | |||
|
|
|||
| boolean redirectEnable = conf.getBoolean(REDIRECT_ENABLE_KEY, | |||
| REDIRECT_ENABLE_DEFAULT); | |||
| if (!redirectEnable) { | |||
There was a problem hiding this comment.
Could we just use clientConf.setRedirectEnable(redirectEnable); ?
Current implementation always assumes true is default for clientConf, right ?
There was a problem hiding this comment.
1 I will simplify the code to clientConf.setRedirectEnable(redirectEnable)
2 yes,the OSS SDK defaults the redirect setting to true
| REDIRECT_ENABLE_DEFAULT); | ||
| if (!redirectEnable) { | ||
| clientConf.setRedirectEnable(false); | ||
| LOG.info("oss redirectEnable closed"); |
There was a problem hiding this comment.
I will change the log level to debug in the next commit
| @@ -211,4 +211,8 @@ private Constants() { | |||
| public static final String LIST_VERSION = "fs.oss.list.version"; | |||
|
|
|||
| public static final int DEFAULT_LIST_VERSION = 2; | |||
|
|
|||
| // oss redirect | |||
| public static final String REDIRECT_ENABLE_KEY = "fs.oss.redirect.enable"; | |||
There was a problem hiding this comment.
add javadocs with {@value} to show the value...really helps in IDE use
There was a problem hiding this comment.
add javadocs with
{@value}to show the value...really helps in IDE use
OK。I will add javadocs in the next commit。
|
|
||
| package org.apache.hadoop.fs.aliyun.oss; | ||
|
|
||
| import org.apache.commons.lang3.StringUtils; |
There was a problem hiding this comment.
nit: import ordering, especially keeping apache stuff on their own
There was a problem hiding this comment.
nit: import ordering, especially keeping apache stuff on their own
OK. I will optimize the import ordering
| } | ||
| } | ||
|
|
||
| private void constructOssClient(Configuration conf) throws IOException { |
There was a problem hiding this comment.
such as is done in AliyunOSSTestUtils
| * Tests basic functionality for AliyunOSSInputStream, including seeking and | ||
| * reading files. | ||
| */ | ||
| public class TestAliyunOSSRedirect { |
There was a problem hiding this comment.
Is this really trying to talk to the store?
There was a problem hiding this comment.
Is this really trying to talk to the store?
This case will access the OSS storage layer for real（This test checks the redirection behavior of Aliyun OSS. When the redirection feature is available, it will access the redirected target file.）. but, it does not call seeking. The comment is inaccurate; I will rephrase it.
Currently, none of the OSS test cases have the ITest prefix. I'm not sure about the historical reasons, and I may need to discuss this with other colleagues on the Aliyun OSS team. Can this PR be considered separately from the changes to add the ITest prefix? |
|
💔 -1 overall
This message was automatically generated. |
|
💔 -1 overall
This message was automatically generated. |
|
🎊 +1 overall
This message was automatically generated. |
|
@steveloughran @masteryhx @zeekling hi，I have made the suggested changes to the code. Please take a look again. Thanks. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Description of PR
How was this patch tested?
New unit tests; cloud store tests
For code changes:
LICENSE,LICENSE-binary,NOTICE-binaryfiles?