Skip to content

For #5801 Use BufferedOutputStream to speed up checkpoint #5802

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
morningman merged 1 commit into from
May 13, 2021
Merged

For #5801 Use BufferedOutputStream to speed up checkpoint #5802

morningman merged 1 commit into from
May 13, 2021

Conversation

@liutang123
Copy link
Contributor

@liutang123 liutang123 commented May 12, 2021 •
edited
Loading

Proposed changes

Use BufferedOutputStream to speed up checkpoint

Types of changes

What types of changes does your code introduce to Doris?
Put an x in the boxes that apply

  • Bugfix (non-breaking change which fixes an issue)
  • New feature (non-breaking change which adds functionality)
  • Breaking change (fix or feature that would cause existing functionality to not work as expected)
  • Documentation Update (if none of the other choices apply)
  • Code refactor (Modify the code structure, format the code, etc...)

Checklist

Put an x in the boxes that apply. You can also fill these out after creating the PR. If you're unsure about any of them, don't hesitate to ask. We're here to help! This is simply a reminder of what we are going to look for before merging your code.

  • I have created an issue on (Fix [Optimize] Speed up checkpoint #5801) and described the feature there in detail
  • Compiling and unit tests pass locally with my changes
  • I have added tests that prove my fix is effective or that my feature works
  • If these changes need document changes, I have updated the document
  • Any dependent changes have been merged

After this PR, we found checkpoint IO has 15 times performance improvement.

morningman
morningman approved these changes May 12, 2021
View reviewed changes
Copy link
Contributor

@morningman morningman left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@morningman morningman added the approved Indicates a PR has been approved by one committer. label May 12, 2021
@morningman morningman merged commit e7a6d65 into apache:master May 13, 2021
stdpain pushed a commit to stdpain/incubator-doris that referenced this pull request Jul 8, 2021
@liutang123 @morningman-cmy
[Optimize] Use BufferedOutputStream to speed up checkpoint (apache#5802)
4615997 
Use BufferedOutputStream to speed up checkpoint
CalvinKirs added a commit to CalvinKirs/incubator-doris that referenced this pull request Nov 27, 2025
@CalvinKirs
[fix](rest-s3)Set AWS Request Checksum Calculation only if not set, a…
f3d6d95 
…void issues on non-S3 storage

AWS SDK for Java v2 uses the system property AwsSystemSetting.AWS_REQUEST_CHECKSUM_CALCULATION to control request payload checksum calculation.
↳

According to the official discussion AWS SDK v2 Discussion apache#5802
:
↳

By default, the SDK may automatically calculate checksums when required.
↳

For AWS S3, setting the default value "WHEN_REQUIRED" ensures checksums are calculated only when necessary, providing both compatibility and performance benefits.
↳

For non-S3 object storage (such as COS, OSS, OBS), using this default value may cause errors, for example:
↳

aws-chunked encoding is not supported with the specified x-amz-content-sha256 value

because these services do not support AWS-specific chunked encoding or SHA256 checksum.
CalvinKirs added a commit to CalvinKirs/incubator-doris that referenced this pull request Nov 27, 2025
@CalvinKirs
[fix](rest-s3)Set AWS Request Checksum Calculation only if not set, a…
8214b93 
…void issues on non-S3 storage

AWS SDK for Java v2 uses the system property AwsSystemSetting.AWS_REQUEST_CHECKSUM_CALCULATION to control request payload checksum calculation.
↳

According to the official discussion AWS SDK v2 Discussion apache#5802
:
↳

By default, the SDK may automatically calculate checksums when required.
↳

For AWS S3, setting the default value "WHEN_REQUIRED" ensures checksums are calculated only when necessary, providing both compatibility and performance benefits.
↳

For non-S3 object storage (such as COS, OSS, OBS), using this default value may cause errors, for example:
↳

aws-chunked encoding is not supported with the specified x-amz-content-sha256 value

because these services do not support AWS-specific chunked encoding or SHA256 checksum.
CalvinKirs added a commit to CalvinKirs/incubator-doris that referenced this pull request Nov 27, 2025
@CalvinKirs
[fix](rest-s3)Set AWS Request Checksum Calculation only if not set, a…
3653a50 
…void issues on non-S3 storage

AWS SDK for Java v2 uses the system property AwsSystemSetting.AWS_REQUEST_CHECKSUM_CALCULATION to control request payload checksum calculation.
↳

According to the official discussion AWS SDK v2 Discussion apache#5802
:
↳

By default, the SDK may automatically calculate checksums when required.
↳

For AWS S3, setting the default value "WHEN_REQUIRED" ensures checksums are calculated only when necessary, providing both compatibility and performance benefits.
↳

For non-S3 object storage (such as COS, OSS, OBS), using this default value may cause errors, for example:
↳

aws-chunked encoding is not supported with the specified x-amz-content-sha256 value

because these services do not support AWS-specific chunked encoding or SHA256 checksum.
CalvinKirs added a commit to CalvinKirs/incubator-doris that referenced this pull request Nov 27, 2025
@CalvinKirs
[fix](rest-s3)Set AWS Request Checksum Calculation only if not set, a…
f13b44f 
…void issues on non-S3 storage

AWS SDK for Java v2 uses the system property AwsSystemSetting.AWS_REQUEST_CHECKSUM_CALCULATION to control request payload checksum calculation.
↳

According to the official discussion AWS SDK v2 Discussion apache#5802
:
↳

By default, the SDK may automatically calculate checksums when required.
↳

For AWS S3, setting the default value "WHEN_REQUIRED" ensures checksums are calculated only when necessary, providing both compatibility and performance benefits.
↳

For non-S3 object storage (such as COS, OSS, OBS), using this default value may cause errors, for example:
↳

aws-chunked encoding is not supported with the specified x-amz-content-sha256 value

because these services do not support AWS-specific chunked encoding or SHA256 checksum.
CalvinKirs added a commit to CalvinKirs/incubator-doris that referenced this pull request Nov 27, 2025
@CalvinKirs
[fix](rest-s3)Set AWS Request Checksum Calculation only if not set, a…
1d8a5a1 
…void issues on non-S3 storage

AWS SDK for Java v2 uses the system property AwsSystemSetting.AWS_REQUEST_CHECKSUM_CALCULATION to control request payload checksum calculation.
↳

According to the official discussion AWS SDK v2 Discussion apache#5802
:
↳

By default, the SDK may automatically calculate checksums when required.
↳

For AWS S3, setting the default value "WHEN_REQUIRED" ensures checksums are calculated only when necessary, providing both compatibility and performance benefits.
↳

For non-S3 object storage (such as COS, OSS, OBS), using this default value may cause errors, for example:
↳

aws-chunked encoding is not supported with the specified x-amz-content-sha256 value

because these services do not support AWS-specific chunked encoding or SHA256 checksum.
@CalvinKirs CalvinKirs mentioned this pull request Nov 27, 2025
Merged
CalvinKirs added a commit to CalvinKirs/incubator-doris that referenced this pull request Nov 27, 2025
@CalvinKirs
[fix](rest-s3)Set AWS Request Checksum Calculation only if not set, a…
d5e6e1e 
…void issues on non-S3 storage

AWS SDK for Java v2 uses the system property AwsSystemSetting.AWS_REQUEST_CHECKSUM_CALCULATION to control request payload checksum calculation.
↳

According to the official discussion AWS SDK v2 Discussion apache#5802
:
↳

By default, the SDK may automatically calculate checksums when required.
↳

For AWS S3, setting the default value "WHEN_REQUIRED" ensures checksums are calculated only when necessary, providing both compatibility and performance benefits.
↳

For non-S3 object storage (such as COS, OSS, OBS), using this default value may cause errors, for example:
↳

aws-chunked encoding is not supported with the specified x-amz-content-sha256 value

because these services do not support AWS-specific chunked encoding or SHA256 checksum.
CalvinKirs added a commit to CalvinKirs/incubator-doris that referenced this pull request Nov 27, 2025
@CalvinKirs
[fix](rest-s3)Set AWS Request Checksum Calculation only if not set, a…
5251cb7 
…void issues on non-S3 storage

AWS SDK for Java v2 uses the system property AwsSystemSetting.AWS_REQUEST_CHECKSUM_CALCULATION to control request payload checksum calculation.

According to the official discussion AWS SDK v2 Discussion apache#5802
:

By default, the SDK may automatically calculate checksums when required.

For AWS S3, setting the default value "WHEN_REQUIRED" ensures checksums are calculated only when necessary, providing both compatibility and performance benefits.

aws-chunked encoding is not supported with the specified x-amz-content-sha256 value

because these services do not support AWS-specific chunked encoding or SHA256 checksum.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@morningman morningman morningman approved these changes

Assignees

No one assigned

Labels

approved Indicates a PR has been approved by one committer.

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

[Optimize] Speed up checkpoint

2 participants

@liutang123 @morningman