S3 storage classes config support #24934
Merged
+50
−2
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
Thank you for your pull request and welcome to our community. We could not parse the GitHub identity of the following contributors: EdenKik.
|
EdenKik
force-pushed
the
s3-storage-classes-config-support
branch
from
8da6edb to
e541e2c
Compare
anusudarsan
reviewed
Feb 7, 2025
lib/trino-filesystem-s3/src/main/java/io/trino/filesystem/s3/S3FileSystemConfig.java
Outdated
Show resolved
Hide resolved
EdenKik
force-pushed
the
s3-storage-classes-config-support
branch
2 times, most recently
from
39c8191 to
5d02c5f
Compare
This was referenced Feb 10, 2025
anusudarsan
approved these changes
Feb 11, 2025
EdenKik
force-pushed
the
s3-storage-classes-config-support
branch
from
5d02c5f to
218a639
Compare
wendigo
reviewed
Feb 14, 2025
| STANDARD_IA, | ||
| INTELLIGENT_TIERING; | ||
|
|
||
| public static StorageClass getStorageClass(S3FileSystemConfig.StorageClassType storageClass) |
There was a problem hiding this comment.
toStorageClass()
import StorageClassType
wendigo
reviewed
Feb 14, 2025
| } | ||
|
|
||
| @Config("s3.storage-class") | ||
| @ConfigDescription("The S3 storage class to use when writing the data") |
There was a problem hiding this comment.
S3 storage class used while writing data
wendigo
reviewed
Feb 14, 2025
| @@ -49,6 +50,7 @@ public void testDefaults() | |||
| .setExternalId(null) | |||
| .setStsEndpoint(null) | |||
| .setStsRegion(null) | |||
| .setStorageClass(StorageClassType.STANDARD) | |||
There was a problem hiding this comment.
we cannot use it in this case because both RetryMode and StorageClassType have a STANDARD constant. This causes ambiguity in the code, as the compiler cannot distinguish between the two without explicitly mentioning their parent classes.
wendigo
reviewed
Feb 14, 2025
| @@ -124,6 +127,7 @@ public void testExplicitPropertyMappings() | |||
| .setExternalId("myid") | |||
| .setStsEndpoint("sts.example.com") | |||
| .setStsRegion("us-west-2") | |||
| .setStorageClass(StorageClassType.STANDARD_IA) | |||
wendigo
approved these changes
Feb 14, 2025
electrum
approved these changes
Feb 14, 2025
| @@ -30,6 +30,8 @@ support: | |||
| - Required region name for S3. | |||
| * - `s3.path-style-access` | |||
| - Use path-style access for all requests to S3 | |||
| * - `s3.storage-class` | |||
| - Set the S3 storage class to use when writing the data, defaults to STANDARD | |||
There was a problem hiding this comment.
S3 storage class to use while writing data. Defaults to
STANDARD.
Add s3.storage-class property integration in S3 operations Modify configuration tests to use storage class property Update Trino documentation to describe s3.storage-class configuration Implement whitelist for supported storage classes and map to SDK options
EdenKik
force-pushed
the
s3-storage-classes-config-support
branch
from
218a639 to
93e8ec6
Compare
wendigo
approved these changes
Feb 15, 2025
|
Thanks @EdenKik |
|
Added to release notes for all four connectors. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Description
This PR enhances Trino's compatibility with S3 storage classes and adds support for configuring the storage class of objects written to S3. By introducing the s3.storage-class property, users can now specify how objects should be stored, enabling better control over storage costs and retrieval performance.
Closes issue #24698
Release notes