-
Notifications
You must be signed in to change notification settings - Fork 361
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
API presign multipart upload experimenal support for S3 #7246
Changes from 7 commits
d9ef33f
62612d7
c0a3294
fafc506
cd54fe9
bc02ce7
2bdc6fd
3a7ab8e
614c899
a6236a5
8b45119
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
Original file line number | Diff line number | Diff line change |
---|---|---|
|
@@ -1115,6 +1115,9 @@ components: | |
type: boolean | ||
import_validity_regex: | ||
type: string | ||
pre_sign_multipart_upload: | ||
type: boolean | ||
|
||
Config: | ||
type: object | ||
properties: | ||
|
@@ -1504,6 +1507,61 @@ components: | |
items: | ||
$ref: "#/components/schemas/StatsEvent" | ||
|
||
PresignMultipartUpload: | ||
type: object | ||
properties: | ||
upload_id: | ||
type: string | ||
physical_address: | ||
type: string | ||
presigned_urls: | ||
type: array | ||
items: | ||
type: string | ||
required: | ||
- upload_id | ||
- physical_address | ||
|
||
UploadPart: | ||
type: object | ||
properties: | ||
part_number: | ||
type: integer | ||
etag: | ||
type: string | ||
required: | ||
- part_number | ||
- etag | ||
|
||
CompletePresignMultipartUpload: | ||
type: object | ||
properties: | ||
physical_address: | ||
type: string | ||
parts: | ||
type: array | ||
description: "List of uploaded parts, should be ordered by ascending part number" | ||
items: | ||
$ref: "#/components/schemas/UploadPart" | ||
user_metadata: | ||
type: object | ||
additionalProperties: | ||
type: string | ||
content_type: | ||
type: string | ||
description: Object media type | ||
required: | ||
- physical_address | ||
- parts | ||
|
||
AbortPresignMultipartUpload: | ||
type: object | ||
properties: | ||
physical_address: | ||
type: string | ||
required: | ||
- physical_address | ||
|
||
paths: | ||
/setup_comm_prefs: | ||
post: | ||
|
@@ -3785,6 +3843,140 @@ paths: | |
default: | ||
description: internal server error | ||
|
||
/repositories/{repository}/branches/{branch}/staging/pmpu: | ||
parameters: | ||
- in: path | ||
name: repository | ||
required: true | ||
schema: | ||
type: string | ||
- in: path | ||
name: branch | ||
required: true | ||
schema: | ||
type: string | ||
- in: query | ||
name: path | ||
description: relative to the branch | ||
required: true | ||
schema: | ||
type: string | ||
- in: query | ||
name: parts | ||
description: number of presigned URL parts required to upload | ||
schema: | ||
type: integer | ||
post: | ||
tags: | ||
- experimental | ||
operationId: createPresignMultipartUpload | ||
summary: Initiate a multipart upload | ||
description: | | ||
Initiates a multipart upload and returns an upload ID with presigned URLs for each part (optional). | ||
Part numbers starts with 1. Each part except the last one has minimum size depends on the underlying blockstore implementation. | ||
For example working with S3 blockstore, minimum size is 5MB (excluding the last part). | ||
responses: | ||
201: | ||
description: Presign multipart upload initiated | ||
content: | ||
application/json: | ||
schema: | ||
$ref: "#/components/schemas/PresignMultipartUpload" | ||
400: | ||
$ref: "#/components/responses/BadRequest" | ||
401: | ||
$ref: "#/components/responses/Unauthorized" | ||
404: | ||
$ref: "#/components/responses/NotFound" | ||
420: | ||
description: too many requests | ||
default: | ||
$ref: "#/components/responses/ServerError" | ||
|
||
/repositories/{repository}/branches/{branch}/staging/pmpu/{uploadId}: | ||
parameters: | ||
- in: path | ||
name: repository | ||
required: true | ||
schema: | ||
type: string | ||
- in: path | ||
name: branch | ||
required: true | ||
schema: | ||
type: string | ||
- in: path | ||
name: uploadId | ||
required: true | ||
schema: | ||
type: string | ||
- in: query | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Why is the path needed when uploadID is given? There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. We require path like S3 protocol requires key. Same for upload ID, even though it seems unique, s3 protocol requires bucket and key on each api call. When we actually work with physical address (which we also require) we use the path for permission check on create and complete. There can be two mpu to the same logical address which means that there can be two parts upload in parallel. In our case because we provide different physical addresses these parts will have different addresses. |
||
name: path | ||
description: relative to the branch | ||
required: true | ||
schema: | ||
type: string | ||
put: | ||
tags: | ||
- experimental | ||
operationId: completePresignMultipartUpload | ||
summary: Complete a presign multipart upload request | ||
description: Completes a presign multipart upload by assembling the uploaded parts. | ||
requestBody: | ||
content: | ||
application/json: | ||
schema: | ||
$ref: "#/components/schemas/CompletePresignMultipartUpload" | ||
responses: | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. 400 is needed. for example. wrong parts order |
||
200: | ||
description: Presign multipart upload completed | ||
content: | ||
application/json: | ||
schema: | ||
$ref: "#/components/schemas/ObjectStats" | ||
400: | ||
$ref: "#/components/responses/BadRequest" | ||
401: | ||
$ref: "#/components/responses/Unauthorized" | ||
404: | ||
$ref: "#/components/responses/NotFound" | ||
409: | ||
description: conflict with a commit, try here | ||
content: | ||
application/json: | ||
schema: | ||
$ref: "#/components/schemas/StagingLocation" | ||
420: | ||
description: too many requests | ||
default: | ||
$ref: "#/components/responses/ServerError" | ||
|
||
delete: | ||
tags: | ||
- experimental | ||
operationId: abortPresignMultipartUpload | ||
summary: Abort a presign multipart upload | ||
description: Aborts a presign multipart upload. | ||
requestBody: | ||
content: | ||
application/json: | ||
schema: | ||
$ref: "#/components/schemas/AbortPresignMultipartUpload" | ||
responses: | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. 400 is needed. For example, uploadID mismatch with path |
||
204: | ||
description: Presign multipart upload aborted | ||
400: | ||
$ref: "#/components/responses/BadRequest" | ||
401: | ||
$ref: "#/components/responses/Unauthorized" | ||
404: | ||
$ref: "#/components/responses/NotFound" | ||
420: | ||
description: too many requests | ||
default: | ||
$ref: "#/components/responses/ServerError" | ||
|
||
|
||
/repositories/{repository}/branches/{branch}/staging/backing: | ||
parameters: | ||
- in: path | ||
|
Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.
Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Need 400 too. For example, 'parts' is 0
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Added the missing 400 status code for all routes.
The SDK will fail the request in case of bad request.
The number of parts for create is optional, in order to support additional API to provide presigned parts URL request for streaming.