Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

API presign multipart upload experimenal support for S3 #7246

Merged
merged 11 commits into from
Jan 8, 2024
Merged
Show file tree
Hide file tree
Changes from 7 commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
192 changes: 192 additions & 0 deletions api/swagger.yml
Original file line number Diff line number Diff line change
Expand Up @@ -1115,6 +1115,9 @@ components:
type: boolean
import_validity_regex:
type: string
pre_sign_multipart_upload:
type: boolean

Config:
type: object
properties:
Expand Down Expand Up @@ -1504,6 +1507,61 @@ components:
items:
$ref: "#/components/schemas/StatsEvent"

PresignMultipartUpload:
type: object
properties:
upload_id:
type: string
physical_address:
type: string
presigned_urls:
type: array
items:
type: string
required:
- upload_id
- physical_address

UploadPart:
type: object
properties:
part_number:
type: integer
etag:
type: string
required:
- part_number
- etag

CompletePresignMultipartUpload:
type: object
properties:
physical_address:
type: string
parts:
type: array
description: "List of uploaded parts, should be ordered by ascending part number"
items:
$ref: "#/components/schemas/UploadPart"
user_metadata:
type: object
additionalProperties:
type: string
content_type:
type: string
description: Object media type
required:
- physical_address
- parts

AbortPresignMultipartUpload:
type: object
properties:
physical_address:
type: string
required:
- physical_address

paths:
/setup_comm_prefs:
post:
Expand Down Expand Up @@ -3785,6 +3843,140 @@ paths:
default:
description: internal server error

/repositories/{repository}/branches/{branch}/staging/pmpu:
parameters:
- in: path
name: repository
required: true
schema:
type: string
- in: path
name: branch
required: true
schema:
type: string
- in: query
name: path
description: relative to the branch
required: true
schema:
type: string
- in: query
name: parts
description: number of presigned URL parts required to upload
schema:
type: integer
post:
tags:
- experimental
operationId: createPresignMultipartUpload
summary: Initiate a multipart upload
description: |
Initiates a multipart upload and returns an upload ID with presigned URLs for each part (optional).
Part numbers starts with 1. Each part except the last one has minimum size depends on the underlying blockstore implementation.
For example working with S3 blockstore, minimum size is 5MB (excluding the last part).
responses:
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Need 400 too. For example, 'parts' is 0

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Added the missing 400 status code for all routes.
The SDK will fail the request in case of bad request.
The number of parts for create is optional, in order to support additional API to provide presigned parts URL request for streaming.

201:
description: Presign multipart upload initiated
content:
application/json:
schema:
$ref: "#/components/schemas/PresignMultipartUpload"
400:
$ref: "#/components/responses/BadRequest"
401:
$ref: "#/components/responses/Unauthorized"
404:
$ref: "#/components/responses/NotFound"
420:
description: too many requests
default:
$ref: "#/components/responses/ServerError"

/repositories/{repository}/branches/{branch}/staging/pmpu/{uploadId}:
parameters:
- in: path
name: repository
required: true
schema:
type: string
- in: path
name: branch
required: true
schema:
type: string
- in: path
name: uploadId
required: true
schema:
type: string
- in: query
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why is the path needed when uploadID is given?
Can 2 mpus be available for the same part?
isn't uploadID unique between different paths?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We require path like S3 protocol requires key. Same for upload ID, even though it seems unique, s3 protocol requires bucket and key on each api call.

When we actually work with physical address (which we also require) we use the path for permission check on create and complete.
For each part the request is signed with all the information which include the upload id and the path.

There can be two mpu to the same logical address which means that there can be two parts upload in parallel. In our case because we provide different physical addresses these parts will have different addresses.
If you will try to provide such path to different upload request the complete will not find this part. Hope I understood the concern.

name: path
description: relative to the branch
required: true
schema:
type: string
put:
tags:
- experimental
operationId: completePresignMultipartUpload
summary: Complete a presign multipart upload request
description: Completes a presign multipart upload by assembling the uploaded parts.
requestBody:
content:
application/json:
schema:
$ref: "#/components/schemas/CompletePresignMultipartUpload"
responses:
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

400 is needed. for example. wrong parts order

200:
description: Presign multipart upload completed
content:
application/json:
schema:
$ref: "#/components/schemas/ObjectStats"
400:
$ref: "#/components/responses/BadRequest"
401:
$ref: "#/components/responses/Unauthorized"
404:
$ref: "#/components/responses/NotFound"
409:
description: conflict with a commit, try here
content:
application/json:
schema:
$ref: "#/components/schemas/StagingLocation"
420:
description: too many requests
default:
$ref: "#/components/responses/ServerError"

delete:
tags:
- experimental
operationId: abortPresignMultipartUpload
summary: Abort a presign multipart upload
description: Aborts a presign multipart upload.
requestBody:
content:
application/json:
schema:
$ref: "#/components/schemas/AbortPresignMultipartUpload"
responses:
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

400 is needed. For example, uploadID mismatch with path

204:
description: Presign multipart upload aborted
400:
$ref: "#/components/responses/BadRequest"
401:
$ref: "#/components/responses/Unauthorized"
404:
$ref: "#/components/responses/NotFound"
420:
description: too many requests
default:
$ref: "#/components/responses/ServerError"


/repositories/{repository}/branches/{branch}/staging/backing:
parameters:
- in: path
Expand Down
12 changes: 12 additions & 0 deletions clients/java-legacy/.openapi-generator/FILES

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

7 changes: 7 additions & 0 deletions clients/java-legacy/README.md

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

Loading
Loading