feat: project setup and upload file #1712

0marperez · 2025-10-17T13:55:48Z

Issue #

N/A

Description of changes

Project setup
Publishing config
Transfer manager client
Business metric
Transfer interceptors
Upload file operation

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.

github-actions · 2025-10-17T14:02:55Z

A new generated diff is ready to view.

No codegen difference in the AWS SDK

github-actions · 2025-10-17T14:14:53Z

A new generated diff is ready to view.

No codegen difference in the AWS SDK

github-actions · 2025-10-17T14:42:33Z

A new generated diff is ready to view.

No codegen difference in the AWS SDK

github-actions · 2025-10-17T14:53:02Z

A new generated diff is ready to view.

No codegen difference in the AWS SDK

github-actions · 2025-10-17T15:11:43Z

A new generated diff is ready to view.

No codegen difference in the AWS SDK

lauzadis

Nice start

lauzadis · 2025-10-17T20:21:12Z

...mmon/src/aws/sdk/kotlin/runtime/http/interceptors/businessmetrics/AwsBusinessMetricsUtils.kt

 public enum class AwsBusinessMetric(public override val identifier: String) : BusinessMetric {
    S3_EXPRESS_BUCKET("J"),
    DDB_MAPPER("d"),
+    S3_TRANSFER("G"),


There are two new features S3_TRANSFER_UPLOAD_DIRECTORY and S3_TRANSFER_DOWNLOAD_DIRECTORY we should add here

lauzadis · 2025-10-17T20:22:24Z

hll/build.gradle.kts

 subprojects {
    group = "aws.sdk.kotlin"
-    version = hllPreviewVersion
+    version = if (name == "s3-transfer-manager") sdkVersion else hllPreviewVersion


You can override the version in the subproject rather than hacking it in here

Or better, override / set hllPreviewVersion in the submodules which need it, and use sdkVersion as the default

lauzadis · 2025-10-17T20:22:48Z

hll/s3-transfer-manager/build.gradle.kts

+ */
+
+description = "S3 Transfer Manager for the AWS SDK for Kotlin"
+extra["displayName"] = "AWS :: SDK :: Kotlin :: HLL :: S3TransferManager"


You can use spaces in the display name, S3 Transfer Manager

lauzadis · 2025-10-17T20:23:47Z

hll/s3-transfer-manager/build.gradle.kts

+        commonMain {
+            dependencies {
+                implementation(project(":aws-runtime:aws-http"))
+                implementation(libs.s3)


Instead of depending on an already-published version of S3, we should require S3 to be bootstrapped locally for S3TM to build. Like how we did it for DDB Mapper

This was an intentional choice to help prevent changes to S3 from breaking the S3 TM. We don't need to have the latest APIs from S3.

I don't think it would happen often, but I think either approach is fine. The current one is just slightly safer.

Doesn't that mean that S3TM will always depend on the n-1 S3 client? So S3TM-1.2.3 uses S3-1.2.2? That doesn't seem right.

We want to catch changes to S3 which break S3TM and fix them as soon as possible. All of our modules are a comprehensive platform and should use the same, unified set of dependency versions.

lauzadis · 2025-10-17T20:24:23Z

hll/s3-transfer-manager/jvm/test/aws/sdk/kotlin/hll/s3transfermanager/UploadFileTest.kt

+            S3TransferManager.Companion {
+                client = s3Client
+            }.uploadFile {
+                bucket = "aoperez"


These tests should either be made generic or not committed yet

Oh I see they have an @Ignore annotation, that seems fine too

lauzadis · 2025-10-17T20:41:52Z

hll/s3-transfer-manager/common/src/aws/sdk/kotlin/hll/s3transfermanager/utils/UploadFile.kt

+ * Builds a low-level S3 upload part request from a high-level upload file request
+ * and data from the S3 Transfer Manager.
+ */
+internal fun buildUploadPartRequest(


suggestion: This and the function below might be better as extension functions

lauzadis · 2025-10-17T20:44:33Z

...ransfer-manager/common/src/aws/sdk/kotlin/hll/s3transfermanager/BusinessMetricInterceptor.kt

+/**
+ * An interceptor that emits the S3 Transfer Manager business metric
+ */
+internal object BusinessMetricInterceptor : HttpInterceptor {


naming: add S3 / S3 Transfer Manager somewhere in the name for better IDE search discovery

lauzadis · 2025-10-17T20:47:14Z

hll/s3-transfer-manager/common/src/aws/sdk/kotlin/hll/s3transfermanager/S3TransferManager.kt

+
+        internal fun build(): S3TransferManager =
+            S3TransferManager(
+                client = client?.withConfig { interceptors += BusinessMetricInterceptor } ?: error("client must be set"),


correctness: the underlying client will emit a business metric for every request, whether the S3TM was used or not. Is that what we want?

withConfig makes a copy of the client with some overrides as far as I understand

lauzadis · 2025-10-17T20:49:01Z

hll/s3-transfer-manager/common/src/aws/sdk/kotlin/hll/s3transfermanager/S3TransferManager.kt

+    public suspend fun uploadFile(uploadFileRequest: UploadFileRequest): Deferred<UploadFileResponse> = coroutineScope {
+        val multiPartUpload = uploadFileRequest.contentLength >= multipartUploadThreshold
+        val uploadedParts = mutableListOf<CompletedPart>()
+        var mpuUploadId = "null"


this feels better as a lateinit var

lauzadis · 2025-10-17T20:49:48Z

hll/s3-transfer-manager/common/src/aws/sdk/kotlin/hll/s3transfermanager/S3TransferManager.kt

+            operationHook(TransferInitiated) {
+                if (multiPartUpload) {
+                    context.response = client.createMultipartUpload(context.request as CreateMultipartUploadRequest)
+                    mpuUploadId = (context.response as CreateMultipartUploadResponse).uploadId ?: throw Exception("Missing upload id in create multipart upload response")


Let's throw a better Exception here and elsewhere

This isn't a scenario the user can cause themselves, is it? This could only happen because of a bug in our S3TM implementation it seems. If so, I'm generally fine with non-specific errors since we cannot expect users to catch them and act on them.

ianbotsf · 2025-10-20T20:54:29Z

...-transfer-manager/common/src/aws/sdk/kotlin/hll/s3transfermanager/model/UploadFileRequest.kt

+    public val checksumCrc32: String?,
+    public val checksumCrc32C: String?,
+    public val checksumCrc64Nvme: String?,
+    public val checksumSha1: String?,


Correctness: The builder shouldn't be passing all of its values to the class's constructor. The constructor should accept a builder object and set its own fields that way. This results in less boilerplate:

public class UploadFileRequest(builder: UpdateFileRequest.Builder) { public val acl = builder.acl public val body = builder.body public val bucket = builder.bucket // ...etc... public companion object { public operator fun invoke(block: Builder.() -> Unit): UploadFileRequest = Builder().apply(block).build() } public class Builder { public var acl: ObjectCannedAcl? = null public var body: ByteStream? = null public var bucket: String? = null // ...etc... internal fun build(): UploadFileRequest = UploadFileRequest(this) } }

We follow this pattern in our codegenned shapes and many of our handwritten ones.

Comment applies to other types in this PR.

ianbotsf · 2025-10-20T20:57:49Z

...-transfer-manager/common/src/aws/sdk/kotlin/hll/s3transfermanager/model/UploadFileRequest.kt

+        public var tagging: String? = null
+        public var websiteRedirectLocation: String? = null
+
+        public fun build(): UploadFileRequest =


Nit: Generally our build methods are internal because we don't want users to think they have to invoke it manually inside of a DSL block:

val req = UploadFileRequest { bucket = "foo" key = "bar" build() // <-- this is unnecessary }

ianbotsf · 2025-10-20T21:01:22Z

...-transfer-manager/common/src/aws/sdk/kotlin/hll/s3transfermanager/model/UploadFileRequest.kt

+import aws.smithy.kotlin.runtime.content.ByteStream
+import aws.smithy.kotlin.runtime.time.Instant
+
+public class UploadFileRequest private constructor(


Question: These classes, their builders, their converters, etc. all seem like a lot of repeated boilerplate. Why aren't we code-generating these based on the list of fields from the spec?

ianbotsf · 2025-10-20T21:04:33Z

hll/s3-transfer-manager/common/src/aws/sdk/kotlin/hll/s3transfermanager/utils/UploadFile.kt

+/**
+ * Determines the actual part size to use for a multipart S3 upload.
+ *
+ * This function calculates the part size based on the total size
+ * of the file and the requested part size. If the requested part size is
+ * too small to allow the upload to fit within S3's 10,000-part limit, the
+ * part size will be automatically increased so that exactly 10,000 parts
+ * are uploaded.
+ */
+internal fun resolvePartSize(uploadFileRequest: UploadFileRequest, tm: S3TransferManager, logger: Logger): Long {


Nit: This function doesn't feel like it belongs in this file. Nothing else in UploadFile.kt uses it, only S3TransferManager.kt.

ianbotsf · 2025-10-20T21:05:33Z

hll/s3-transfer-manager/common/src/aws/sdk/kotlin/hll/s3transfermanager/utils/UploadFile.kt

+    val targetNumberOfParts = uploadFileRequest.contentLength / tm.targePartSize
+    return if (targetNumberOfParts > MAX_NUMBER_PARTS) {
+        ceilDiv(uploadFileRequest.contentLength, MAX_NUMBER_PARTS).also {
+            logger.debug { "Target part size is too small to meet the 10,000 S3 part limit. Increasing part size to $it" }


Nit: 10000 is already a constant in this file. Reuse that constant in this string rather than duplicating it. This'll reduce the likelihood that the values get out of sync.

ianbotsf · 2025-10-20T21:49:26Z

hll/s3-transfer-manager/common/src/aws/sdk/kotlin/hll/s3transfermanager/S3TransferManager.kt

+/**
+ * High level utility for managing transfers to Amazon S3.
+ */
+public class S3TransferManager private constructor(


Style: This class has too much logic inside of it and that won't be sustainable once we add more operations and variants. We should refactor this, possibly so that individual operations are located in a single file and maybe even represented by a class.

ianbotsf · 2025-10-20T21:50:01Z

hll/s3-transfer-manager/common/src/aws/sdk/kotlin/hll/s3transfermanager/S3TransferManager.kt

+     * all parts to fit within S3's limit of 10,000 parts, the part size will be
+     * automatically increased so that exactly 10,000 parts are uploaded.
+     */
+    public suspend fun uploadFile(uploadFileRequest: UploadFileRequest): Deferred<UploadFileResponse> = coroutineScope {


Style: Method is too long and should be refactored.

ianbotsf · 2025-10-20T21:54:24Z

hll/s3-transfer-manager/common/src/aws/sdk/kotlin/hll/s3transfermanager/S3TransferManager.kt

+     * all parts to fit within S3's limit of 10,000 parts, the part size will be
+     * automatically increased so that exactly 10,000 parts are uploaded.
+     */
+    public suspend fun uploadFile(uploadFileRequest: UploadFileRequest): Deferred<UploadFileResponse> = coroutineScope {


Correctness: This method shouldn't return Deferred<UploadFileResponse>, it should return UploadFileResponse just like our low-level APIs do. If users want this to happen asynchronously, they can wrap it in async { } themselves.

Note that coroutineScope does not return until all the work inside is completed (including child coroutines) so the Deferred value would already be filled anyway.

ianbotsf · 2025-10-20T21:57:39Z

hll/s3-transfer-manager/common/src/aws/sdk/kotlin/hll/s3transfermanager/TransferInterceptor.kt

+public interface TransferInterceptor {
+    // Transfer initialization hooks
+    public fun readBeforeTransferInitiated(context: TransferContext) {}
+    public fun modifyBeforeTransferInitiated(context: TransferContext): TransferContext = context
+    public fun readAfterTransferInitiated(context: TransferContext) {}
+    public fun modifyAfterTransferInitiated(context: TransferContext): TransferContext = context
+
+    // Byte transferring hooks
+    public fun readBeforeBytesTransferred(context: TransferContext) {}
+    public fun modifyBeforeBytesTransferred(context: TransferContext): TransferContext = context
+    public fun readAfterBytesTransferred(context: TransferContext) {}
+    public fun modifyAfterBytesTransferred(context: TransferContext): TransferContext = context
+
+    // File transfer hooks
+    public fun readBeforeFileTransferred(context: TransferContext) {}
+    public fun modifyBeforeFileTransferred(context: TransferContext): TransferContext = context
+    public fun readAfterFileTransferred(context: TransferContext) {}
+    public fun modifyAfterFileTransferred(context: TransferContext): TransferContext = context
+
+    // Transfer completion hooks
+    public fun readBeforeTransferCompleted(context: TransferContext) {}
+    public fun modifyBeforeTransferCompleted(context: TransferContext): TransferContext = context
+    public fun readAfterTransferCompleted(context: TransferContext) {}
+    public fun modifyAfterTransferCompleted(context: TransferContext): TransferContext = context
+}


Question: Why do all of these hooks have the same TransferContext parameter? Our other interceptors have unique context parameters per hook because we gradually accumulate more context over the lifetime of the request. For instance, readBeforeFileTransferred can never have an upload ID but readAfterFileTransferred can.

ianbotsf · 2025-10-20T21:59:09Z

hll/s3-transfer-manager/build.gradle.kts

+        jvmTest {
+            dependencies {
+                implementation(libs.smithy.kotlin.test.jvm)
+                implementation(libs.smithy.kotlin.testing.jvm)
+                implementation(libs.smithy.kotlin.aws.signing.common)
+            }
+        }


Correctness: We should not have JVM-only tests in most cases. S3TM must work across all targets and our test dependencies should be available on all targets.

github-actions · 2025-10-21T14:07:26Z

A new generated diff is ready to view.

No codegen difference in the AWS SDK

0marperez added 6 commits October 6, 2025 17:39

feat: s3 transfer manager client

bc2fa0a

saving work - upload file

6895fc3

self review

c9486f1

pull from main

22243e7

misc: merge from main

dbff057

finish upload file

835866a

provide static dummy credentials to tests

8edf604

0marperez added the no-changelog Indicates that a changelog entry isn't required for a pull request. Use sparingly. label Oct 17, 2025

add s3 as a test dependency?

4137873

refactor tests to be JVM only

d84516d

Depend on missing 'DefaultAwsSigner'

9d5ec07

0marperez marked this pull request as ready for review October 17, 2025 15:14

0marperez requested a review from a team as a code owner October 17, 2025 15:14

lauzadis requested changes Oct 17, 2025

View reviewed changes

Merge branch 'main' of github.com:aws/aws-sdk-kotlin into s3-tm-dev

71ea5b7

ianbotsf requested changes Oct 20, 2025

View reviewed changes

feedback

fb2417f

feat: project setup and upload file #1712

Are you sure you want to change the base?

feat: project setup and upload file #1712

Uh oh!

Conversation

0marperez commented Oct 17, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Issue #

Description of changes

Uh oh!

github-actions bot commented Oct 17, 2025

Uh oh!

github-actions bot commented Oct 17, 2025

Uh oh!

github-actions bot commented Oct 17, 2025

Uh oh!

github-actions bot commented Oct 17, 2025

Uh oh!

github-actions bot commented Oct 17, 2025

Uh oh!

lauzadis left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

0marperez Oct 20, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

0marperez commented Oct 17, 2025 •

edited

Loading

0marperez Oct 20, 2025 •

edited

Loading