Skip to content

Commit

Permalink
Set CRC32 checksum algorithm for S3 multipart upload
Browse files Browse the repository at this point in the history
  • Loading branch information
majetideepak committed Aug 21, 2024
1 parent 74a3183 commit 6e4f98c
Showing 1 changed file with 4 additions and 0 deletions.
4 changes: 4 additions & 0 deletions velox/connectors/hive/storage_adapters/s3fs/S3FileSystem.cpp
Original file line number Diff line number Diff line change
Expand Up @@ -377,6 +377,10 @@ class S3WriteFile::Impl {
request.SetContentLength(part.size());
request.SetBody(
std::make_shared<StringViewStream>(part.data(), part.size()));
// The default algorithm used is MD5. However, MD5 is not supported with
// fips and can cause a SIGSEGV. Set CRC32 instead which is a standard for
// checksum computation and is not restricted by fips.
request.SetChecksumAlgorithm(Aws::S3::Model::ChecksumAlgorithm::CRC32);
auto outcome = client_->UploadPart(request);
VELOX_CHECK_AWS_OUTCOME(outcome, "Failed to upload", bucket_, key_);
// Append ETag and part number for this uploaded part.
Expand Down

0 comments on commit 6e4f98c

Please sign in to comment.