Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat(storage): introduce preventOverwrite option to uploadData via HeadObject #13640

Merged
merged 5 commits into from
Jul 24, 2024
Merged

feat(storage): introduce preventOverwrite option to uploadData via HeadObject #13640

Show file tree
Hide file tree
Changes from 4 commits
Commits
Show all changes
5 commits
Select commit Hold shift + click to select a range
acfed52
feat(storage): introduce preventOverwrite operation to uploadData via…
eppjame Jul 23, 2024
36def96
Merge branch 'staging' into upload/prevent-overwrite
eppjame Jul 23, 2024
f2b9fcf
fix: add missing license and remove dependency on core in preventOver…
eppjame Jul 23, 2024
4f48477
chore: update storage:uploadData bundle size
eppjame Jul 24, 2024
1d96efe
feat: move existing object validation to before completeMultipartUpload
eppjame Jul 24, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion packages/aws-amplify/package.json
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -497,7 +497,7 @@
"name": "[Storage] uploadData (S3)",
"path": "./dist/esm/storage/index.mjs",
"import": "{ uploadData }",
"limit": "19.94 kB"
"limit": "20.05 kB"
}
]
}
68 changes: 68 additions & 0 deletions packages/storage/__tests__/providers/s3/apis/uploadData/multipartHandlers.test.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -861,6 +861,74 @@ describe('getMultipartUploadHandlers with path', () => {
expect(mockUploadPart).toHaveBeenCalledTimes(2);
expect(mockCompleteMultipartUpload).not.toHaveBeenCalled();
});

describe('overwrite prevention', () => {
beforeEach(() => {
mockHeadObject.mockReset();
mockUploadPart.mockReset();
});

it('should upload if target key is not found', async () => {
expect.assertions(7);
const notFoundError = new Error('mock message');
notFoundError.name = 'NotFound';
mockHeadObject.mockRejectedValueOnce(notFoundError);
mockMultipartUploadSuccess();

const { multipartUploadJob } = getMultipartUploadHandlers({
path: testPath,
data: new ArrayBuffer(8 * MB),
options: { preventOverwrite: true },
});
await multipartUploadJob();

expect(mockHeadObject).toHaveBeenCalledTimes(1);
await expect(mockHeadObject).toBeLastCalledWithConfigAndInput(
expect.objectContaining({
credentials,
region,
}),
expect.objectContaining({
Bucket: bucket,
Key: testPath,
}),
);
expect(mockCreateMultipartUpload).toHaveBeenCalledTimes(1);
expect(mockUploadPart).toHaveBeenCalledTimes(2);
expect(mockCompleteMultipartUpload).toHaveBeenCalledTimes(1);
});

it('should not upload if target key already exists', async () => {
expect.assertions(2);
mockHeadObject.mockResolvedValueOnce({
ContentLength: 0,
$metadata: {},
});
const { multipartUploadJob } = getMultipartUploadHandlers({
path: testPath,
data: new ArrayBuffer(8 * MB),
options: { preventOverwrite: true },
});
await expect(multipartUploadJob()).rejects.toThrow(
'At least one of the pre-conditions you specified did not hold',
);
expect(mockCreateMultipartUpload).not.toHaveBeenCalled();
});

it('should not upload if HeadObject fails with other error', async () => {
expect.assertions(2);
const accessDeniedError = new Error('mock error');
accessDeniedError.name = 'AccessDenied';
mockHeadObject.mockRejectedValueOnce(accessDeniedError);
const { multipartUploadJob } = getMultipartUploadHandlers({
path: testPath,
data: new ArrayBuffer(8 * MB),
options: { preventOverwrite: true },
});
await expect(multipartUploadJob()).rejects.toThrow('mock error');
expect(mockCreateMultipartUpload).not.toHaveBeenCalled();
});
});
});

describe('upload caching', () => {
Expand Down
81 changes: 80 additions & 1 deletion packages/storage/__tests__/providers/s3/apis/uploadData/putObjectJob.test.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,10 @@
import { AWSCredentials } from '@aws-amplify/core/internals/utils';
import { Amplify } from '@aws-amplify/core';

import { putObject } from '../../../../../src/providers/s3/utils/client/s3data';
import {
headObject,
putObject,
} from '../../../../../src/providers/s3/utils/client/s3data';
import { calculateContentMd5 } from '../../../../../src/providers/s3/utils';
import { putObjectJob } from '../../../../../src/providers/s3/apis/uploadData/putObjectJob';
import '../testUtils';
Expand Down Expand Up @@ -38,6 +41,7 @@ const credentials: AWSCredentials = {
const identityId = 'identityId';
const mockFetchAuthSession = jest.mocked(Amplify.Auth.fetchAuthSession);
const mockPutObject = jest.mocked(putObject);
const mockHeadObject = jest.mocked(headObject);

mockFetchAuthSession.mockResolvedValue({
credentials,
Expand Down Expand Up @@ -233,4 +237,79 @@ describe('putObjectJob with path', () => {
await job();
expect(calculateContentMd5).toHaveBeenCalledWith('data');
});

describe('overwrite prevention', () => {
beforeEach(() => {
mockHeadObject.mockClear();
});

it('should upload if target key is not found', async () => {
expect.assertions(3);
const notFoundError = new Error('mock message');
notFoundError.name = 'NotFound';
mockHeadObject.mockRejectedValueOnce(notFoundError);

const job = putObjectJob(
{
path: testPath,
data: 'data',
options: { preventOverwrite: true },
},
new AbortController().signal,
);
await job();

await expect(mockHeadObject).toBeLastCalledWithConfigAndInput(
{
credentials,
region: 'region',
},
{
Bucket: 'bucket',
Key: testPath,
},
);
expect(mockHeadObject).toHaveBeenCalledTimes(1);
expect(mockPutObject).toHaveBeenCalledTimes(1);
});

it('should not upload if target key already exists', async () => {
expect.assertions(3);
mockHeadObject.mockResolvedValueOnce({
ContentLength: 0,
$metadata: {},
});
const job = putObjectJob(
{
path: testPath,
data: 'data',
options: { preventOverwrite: true },
},
new AbortController().signal,
);
await expect(job()).rejects.toThrow(
'At least one of the pre-conditions you specified did not hold',
);
expect(mockHeadObject).toHaveBeenCalledTimes(1);
expect(mockPutObject).not.toHaveBeenCalled();
});

it('should not upload if HeadObject fails with other error', async () => {
AllanZhengYP marked this conversation as resolved.
Show resolved Hide resolved
expect.assertions(3);
const accessDeniedError = new Error('mock error');
accessDeniedError.name = 'AccessDenied';
mockHeadObject.mockRejectedValueOnce(accessDeniedError);
const job = putObjectJob(
{
path: testPath,
data: 'data',
options: { preventOverwrite: true },
},
new AbortController().signal,
);
await expect(job()).rejects.toThrow('mock error');
expect(mockHeadObject).toHaveBeenCalledTimes(1);
expect(mockPutObject).not.toHaveBeenCalled();
});
});
});
9 changes: 9 additions & 0 deletions packages/storage/src/providers/s3/apis/uploadData/multipart/uploadHandlers.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -29,6 +29,7 @@ import {
} from '../../../utils/client/s3data';
import { getStorageUserAgentValue } from '../../../utils/userAgent';
import { logger } from '../../../../../utils';
import { validateObjectNotExists } from '../validateObjectNotExists';

import { uploadPartExecutor } from './uploadPartExecutor';
import { getUploadsCacheKey, removeCachedUpload } from './uploadCache';
Expand Down Expand Up @@ -92,6 +93,7 @@ export const getMultipartUploadHandlers = (
contentEncoding,
contentType = 'application/octet-stream',
metadata,
preventOverwrite,
onProgress,
} = uploadDataOptions ?? {};

Expand All @@ -107,6 +109,13 @@ export const getMultipartUploadHandlers = (
resolvedAccessLevel = resolveAccessLevel(accessLevel);
}

if (preventOverwrite) {
Copy link
Member

@AllanZhengYP AllanZhengYP Jul 24, 2024 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Does it make more sense to be called right before calling complete MPU API?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, that has a better chance of catching recent uploads. Updated.

await validateObjectNotExists(resolvedS3Config, {
Bucket: resolvedBucket,
Key: finalKey,
});
}

if (!inProgressUpload) {
const { uploadId, cachedParts } = await loadOrCreateMultipartUpload({
s3Config: resolvedS3Config,
Expand Down
10 changes: 10 additions & 0 deletions packages/storage/src/providers/s3/apis/uploadData/putObjectJob.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -15,6 +15,8 @@ import { putObject } from '../../utils/client/s3data';
import { getStorageUserAgentValue } from '../../utils/userAgent';
import { STORAGE_INPUT_KEY } from '../../utils/constants';

import { validateObjectNotExists } from './validateObjectNotExists';

/**
* Get a function the returns a promise to call putObject API to S3.
*
Expand All @@ -41,10 +43,18 @@ export const putObjectJob =
contentDisposition,
contentEncoding,
contentType = 'application/octet-stream',
preventOverwrite,
metadata,
onProgress,
} = uploadDataOptions ?? {};

if (preventOverwrite) {
await validateObjectNotExists(s3Config, {
Bucket: bucket,
Key: finalKey,
});
}

const { ETag: eTag, VersionId: versionId } = await putObject(
{
...s3Config,
Expand Down
25 changes: 25 additions & 0 deletions packages/storage/src/providers/s3/apis/uploadData/validateObjectNotExists.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,25 @@
// Copyright Amazon.com, Inc. or its affiliates. All Rights Reserved.
// SPDX-License-Identifier: Apache-2.0

import { StorageError } from '../../../../errors/StorageError';
import { ResolvedS3Config } from '../../types/options';
import { HeadObjectInput, headObject } from '../../utils/client/s3data';

export const validateObjectNotExists = async (
s3Config: ResolvedS3Config,
input: HeadObjectInput,
): Promise<void> => {
try {
await headObject(s3Config, input);

throw new StorageError({
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

so this error gets thrown only when target key is already present? Can we be more clear on the error message? or is there more than that we check for here?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'll defer this quetion to @eppjame but one advantage of making the message generic is to prevent users to enumerate the resources and gain knowledge about the bucket.

name: 'PreconditionFailed',
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can we get this value as const value? — In case we are unit testing this specific error we would refer to the variable instead of the plain text.

message: 'At least one of the pre-conditions you specified did not hold',
});
} catch (error) {
const serviceError = error as StorageError;
if (serviceError.name !== 'NotFound') {
throw error;
}
AllanZhengYP marked this conversation as resolved.
Show resolved Hide resolved
}
};
5 changes: 5 additions & 0 deletions packages/storage/src/providers/s3/types/options.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -169,6 +169,11 @@ export type UploadDataOptions = CommonOptions &
* @see https://docs.aws.amazon.com/AmazonS3/latest/userguide/UsingMetadata.html#UserMetadata
*/
metadata?: Record<string, string>;
/**
* Enforces target key does not already exist in S3 before committing upload.
* @default false
*/
preventOverwrite?: boolean;
};

/** @deprecated Use {@link UploadDataOptionsWithPath} instead. */
Expand Down
Loading