Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

fix(node): deepCopy not working for large Buffers #909

Merged
merged 1 commit into from
Dec 29, 2020
Merged

fix(node): deepCopy not working for large Buffers #909

merged 1 commit into from
Dec 29, 2020

Conversation

aloisklink
Copy link
Contributor

deepCopy() does not work when copying an object with a large Buffer (see test case).

This causes issues when using multipartUpload() when file is a very big Buffer (~1GB) and with the checkpoint option, since in completeMultipartUpload we deepCopy all the options.

Example:

const file = Buffer.alloc(Math.pow(2, 30) - 1); // ~ 1GB
const savedCheckpoint = {file, ...} // made from multipartUpload#progress

// throws RangeError in utils/deepCopy, when copying options.checkpoint.file
multipartUpload("name", file, {checkpoint: savedCheckpoint});

Copying all Buffer in deepCopy() using Buffer.from seems to fix this issue.

@aloisklink
fix(node): deepCopy not working for large Buffers
93235cf 
deepCopy does not work when copying an object
with a large Buffer (see test case).

This causes issues when using multipartUpload with the
`checkpoint` option: e.g.

```js
const file = Buffer.alloc(Math.pow(2, 30) - 1);
// throws RangeError in utils/deepCopy
multipartUpload("name", file, {checkpoint: {file, ...}});
```
@PeterRao PeterRao merged commit f52cd2e into ali-sdk:develop Dec 29, 2020
PeterRao added a commit that referenced this pull request Jan 6, 2021
@PeterRao @beajer
@weiyie @aloisklink
chore: mege develop (#917)
d375157 
* feat: support non-english http header (#842)

* feat: support non-english http header

* test: Add test cases for non-English request header information

* feat: add headerEncoding option

* feat: add encoder types

* fix: remove object spread syntax

Co-authored-by: biejia <wb-bj559483@alibaba-inc.com>

* fix(browser): checkpoint.file (#795)

* merge branch master into develop

* feat: inventory (#860)

* feat: worm (#857)

* feat: auto set stsConfig when sts expires (#856)

* feat: auto set stsConfig when sts expires

* test: add auto set sts test case

* fix: host err which old image client request (#869)

* fix: use put insteadof putStream when calling multipartUpload to process small file in browser (#890)

* fix(browser):setTimeout  has minimum time interval when tab is inactive (#887)

* fix: replace process for browser

* fix: use browserify options to replace process.js (#886)

Co-authored-by: beajer <919060679@qq.com>

* fix: merge conflict #726 (#892)

* fix: merge conflict

* test: ResponseTimeoutError

* fix: cannot use ali-oss in webworker (#823)

* feat: retry when net err or timeout (#876)

* feat: retry when net err or timeout

* test: should not retry when params.stream is not readable

* test: retry in browser

Co-authored-by: beajer <919060679@qq.com>

* fix: getBucketVersions should use NextKeyMarker property (#896)

* fix: getBucketVersions should use NextKeyMarker property

* docs: getBucketVersions should use NextKeyMarker property

* docs: Compatible wrong property

* Fix multipart upload (#895)

* fix: multipartUpload

zancun

* fix: test case

* test: multipartUploadStreams

* test: cov

* fix: error method

* fix: test and destory(err) when cancel

Co-authored-by: weiyie <912881342@qq.com>

* feat: listObjectsV2 (#888)

* feat: listObjectsV2

* style: trailingComma none

* fix: check region and endpoint for ssrf (#874)

* fix: check region and endpoint for ssrf

* fix: retry in browser

Co-authored-by: beajer <919060679@qq.com>

* use shime/url replace node-url to fix split regex bug

* fix: browser test and listV2 docs (#901)

* chore(release): 6.12.0

* chore: build 6.12.0

* fix: the continuation-token params in listV2 (#906)

* fix: the continuation-token params in listV2

* docs: update headers.Expires

* docs: revise `start-after` in listV2 (#907)

* fix: should use encodeURIComponent insteadof encodeURI in copy api (#914)

* fix: use lowercase header when calculating signatures (#910)

* fix: use lowercase header when calculating signatures

* fix: should throw error when neither bucket nor cname;  docs;

* fix(node): deepCopy not working for large Buffers (#909)

deepCopy does not work when copying an object
with a large Buffer (see test case).

This causes issues when using multipartUpload with the
`checkpoint` option: e.g.

```js
const file = Buffer.alloc(Math.pow(2, 30) - 1);
// throws RangeError in utils/deepCopy
multipartUpload("name", file, {checkpoint: {file, ...}});
```

* feat: prevent caching for the get api (#916)

* feat: prevent caching for the get api

* fix: support responseCacheControl for browser `get`

* fix: unnecessary query parameters

* fix: test case because Image Process updated compression algorithm

* fix: test case of buffer replication in Nodejs8

* fix: remove test.only

* fix: do not copy buffer in completeMultipartUpload (#915)

* fix: do not copy buffer in completeMultipartUpload

* fix: test case because Image Process updated compression algorithm

* fix: test case of buffer replication in Nodejs8

* fix: remove test.only

* fix: codecov

* fix: merge conflict from master

Co-authored-by: beajer <919060679@qq.com>
Co-authored-by: biejia <wb-bj559483@alibaba-inc.com>
Co-authored-by: 冷若霜寒 <912881342@qq.com>
Co-authored-by: Alois Klink <alois.klink@gmail.com>
PeterRao added a commit that referenced this pull request Jan 11, 2021
@beajer @PeterRao
@weiyie @aloisklink
Fix/browser test case (#920)
d351b18 
* chore: mege develop (#917)

* feat: support non-english http header (#842)

* feat: support non-english http header

* test: Add test cases for non-English request header information

* feat: add headerEncoding option

* feat: add encoder types

* fix: remove object spread syntax

Co-authored-by: biejia <wb-bj559483@alibaba-inc.com>

* fix(browser): checkpoint.file (#795)

* merge branch master into develop

* feat: inventory (#860)

* feat: worm (#857)

* feat: auto set stsConfig when sts expires (#856)

* feat: auto set stsConfig when sts expires

* test: add auto set sts test case

* fix: host err which old image client request (#869)

* fix: use put insteadof putStream when calling multipartUpload to process small file in browser (#890)

* fix(browser):setTimeout  has minimum time interval when tab is inactive (#887)

* fix: replace process for browser

* fix: use browserify options to replace process.js (#886)

Co-authored-by: beajer <919060679@qq.com>

* fix: merge conflict #726 (#892)

* fix: merge conflict

* test: ResponseTimeoutError

* fix: cannot use ali-oss in webworker (#823)

* feat: retry when net err or timeout (#876)

* feat: retry when net err or timeout

* test: should not retry when params.stream is not readable

* test: retry in browser

Co-authored-by: beajer <919060679@qq.com>

* fix: getBucketVersions should use NextKeyMarker property (#896)

* fix: getBucketVersions should use NextKeyMarker property

* docs: getBucketVersions should use NextKeyMarker property

* docs: Compatible wrong property

* Fix multipart upload (#895)

* fix: multipartUpload

zancun

* fix: test case

* test: multipartUploadStreams

* test: cov

* fix: error method

* fix: test and destory(err) when cancel

Co-authored-by: weiyie <912881342@qq.com>

* feat: listObjectsV2 (#888)

* feat: listObjectsV2

* style: trailingComma none

* fix: check region and endpoint for ssrf (#874)

* fix: check region and endpoint for ssrf

* fix: retry in browser

Co-authored-by: beajer <919060679@qq.com>

* use shime/url replace node-url to fix split regex bug

* fix: browser test and listV2 docs (#901)

* chore(release): 6.12.0

* chore: build 6.12.0

* fix: the continuation-token params in listV2 (#906)

* fix: the continuation-token params in listV2

* docs: update headers.Expires

* docs: revise `start-after` in listV2 (#907)

* fix: should use encodeURIComponent insteadof encodeURI in copy api (#914)

* fix: use lowercase header when calculating signatures (#910)

* fix: use lowercase header when calculating signatures

* fix: should throw error when neither bucket nor cname;  docs;

* fix(node): deepCopy not working for large Buffers (#909)

deepCopy does not work when copying an object
with a large Buffer (see test case).

This causes issues when using multipartUpload with the
`checkpoint` option: e.g.

```js
const file = Buffer.alloc(Math.pow(2, 30) - 1);
// throws RangeError in utils/deepCopy
multipartUpload("name", file, {checkpoint: {file, ...}});
```

* feat: prevent caching for the get api (#916)

* feat: prevent caching for the get api

* fix: support responseCacheControl for browser `get`

* fix: unnecessary query parameters

* fix: test case because Image Process updated compression algorithm

* fix: test case of buffer replication in Nodejs8

* fix: remove test.only

* fix: do not copy buffer in completeMultipartUpload (#915)

* fix: do not copy buffer in completeMultipartUpload

* fix: test case because Image Process updated compression algorithm

* fix: test case of buffer replication in Nodejs8

* fix: remove test.only

* fix: codecov

* fix: merge conflict from master

Co-authored-by: beajer <919060679@qq.com>
Co-authored-by: biejia <wb-bj559483@alibaba-inc.com>
Co-authored-by: 冷若霜寒 <912881342@qq.com>
Co-authored-by: Alois Klink <alois.klink@gmail.com>

* chrome: build

* fix: responseCacheControl in Node.js (#919)

* fix: responseCacheControl in Node.js

* fix: code specification

* chore(release): 6.13.1

* fix: failed browser test case

Co-authored-by: 饶培泽 <peizerao@gmail.com>
Co-authored-by: biejia <wb-bj559483@alibaba-inc.com>
Co-authored-by: 冷若霜寒 <912881342@qq.com>
Co-authored-by: Alois Klink <alois.klink@gmail.com>
PeterRao added a commit that referenced this pull request Jan 21, 2021
@PeterRao @beajer
@weiyie @aloisklink
Fix/browser test case (#920) (#922)
3eb6a81 
* chore: mege develop (#917)

* feat: support non-english http header (#842)

* feat: support non-english http header

* test: Add test cases for non-English request header information

* feat: add headerEncoding option

* feat: add encoder types

* fix: remove object spread syntax

Co-authored-by: biejia <wb-bj559483@alibaba-inc.com>

* fix(browser): checkpoint.file (#795)

* merge branch master into develop

* feat: inventory (#860)

* feat: worm (#857)

* feat: auto set stsConfig when sts expires (#856)

* feat: auto set stsConfig when sts expires

* test: add auto set sts test case

* fix: host err which old image client request (#869)

* fix: use put insteadof putStream when calling multipartUpload to process small file in browser (#890)

* fix(browser):setTimeout  has minimum time interval when tab is inactive (#887)

* fix: replace process for browser

* fix: use browserify options to replace process.js (#886)

Co-authored-by: beajer <919060679@qq.com>

* fix: merge conflict #726 (#892)

* fix: merge conflict

* test: ResponseTimeoutError

* fix: cannot use ali-oss in webworker (#823)

* feat: retry when net err or timeout (#876)

* feat: retry when net err or timeout

* test: should not retry when params.stream is not readable

* test: retry in browser

Co-authored-by: beajer <919060679@qq.com>

* fix: getBucketVersions should use NextKeyMarker property (#896)

* fix: getBucketVersions should use NextKeyMarker property

* docs: getBucketVersions should use NextKeyMarker property

* docs: Compatible wrong property

* Fix multipart upload (#895)

* fix: multipartUpload

zancun

* fix: test case

* test: multipartUploadStreams

* test: cov

* fix: error method

* fix: test and destory(err) when cancel

Co-authored-by: weiyie <912881342@qq.com>

* feat: listObjectsV2 (#888)

* feat: listObjectsV2

* style: trailingComma none

* fix: check region and endpoint for ssrf (#874)

* fix: check region and endpoint for ssrf

* fix: retry in browser

Co-authored-by: beajer <919060679@qq.com>

* use shime/url replace node-url to fix split regex bug

* fix: browser test and listV2 docs (#901)

* chore(release): 6.12.0

* chore: build 6.12.0

* fix: the continuation-token params in listV2 (#906)

* fix: the continuation-token params in listV2

* docs: update headers.Expires

* docs: revise `start-after` in listV2 (#907)

* fix: should use encodeURIComponent insteadof encodeURI in copy api (#914)

* fix: use lowercase header when calculating signatures (#910)

* fix: use lowercase header when calculating signatures

* fix: should throw error when neither bucket nor cname;  docs;

* fix(node): deepCopy not working for large Buffers (#909)

deepCopy does not work when copying an object
with a large Buffer (see test case).

This causes issues when using multipartUpload with the
`checkpoint` option: e.g.

```js
const file = Buffer.alloc(Math.pow(2, 30) - 1);
// throws RangeError in utils/deepCopy
multipartUpload("name", file, {checkpoint: {file, ...}});
```

* feat: prevent caching for the get api (#916)

* feat: prevent caching for the get api

* fix: support responseCacheControl for browser `get`

* fix: unnecessary query parameters

* fix: test case because Image Process updated compression algorithm

* fix: test case of buffer replication in Nodejs8

* fix: remove test.only

* fix: do not copy buffer in completeMultipartUpload (#915)

* fix: do not copy buffer in completeMultipartUpload

* fix: test case because Image Process updated compression algorithm

* fix: test case of buffer replication in Nodejs8

* fix: remove test.only

* fix: codecov

* fix: merge conflict from master

Co-authored-by: beajer <919060679@qq.com>
Co-authored-by: biejia <wb-bj559483@alibaba-inc.com>
Co-authored-by: 冷若霜寒 <912881342@qq.com>
Co-authored-by: Alois Klink <alois.klink@gmail.com>

* chrome: build

* fix: responseCacheControl in Node.js (#919)

* fix: responseCacheControl in Node.js

* fix: code specification

* chore(release): 6.13.1

* fix: failed browser test case

Co-authored-by: 饶培泽 <peizerao@gmail.com>
Co-authored-by: biejia <wb-bj559483@alibaba-inc.com>
Co-authored-by: 冷若霜寒 <912881342@qq.com>
Co-authored-by: Alois Klink <alois.klink@gmail.com>

Co-authored-by: beajer <919060679@qq.com>
Co-authored-by: biejia <wb-bj559483@alibaba-inc.com>
Co-authored-by: 冷若霜寒 <912881342@qq.com>
Co-authored-by: Alois Klink <alois.klink@gmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
No reviews
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
2 participants
@aloisklink @PeterRao