Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Fix LZO decompression #10123

Closed
wants to merge 3 commits into from
Closed

Fix LZO decompression #10123

wants to merge 3 commits into from

Conversation

majetideepak
Copy link
Collaborator

@majetideepak majetideepak commented Jun 10, 2024 •
edited
Loading

resolves #9618

The output is verified via Presto.

@facebook-github-bot facebook-github-bot added the CLA Signed This label is managed by the Facebook bot. Authors need to sign the CLA before a PR can be reviewed. label Jun 10, 2024
@netlify Netlify
Copy link

netlify bot commented Jun 10, 2024 •
edited
Loading

✅ Deploy Preview for meta-velox canceled.

Name Link
🔨 Latest commit ccc16bd
🔍 Latest deploy log https://app.netlify.com/sites/meta-velox/deploys/667042c7779b48000843b505

@majetideepak majetideepak mentioned this pull request Jun 10, 2024
Closed
@majetideepak majetideepak marked this pull request as ready for review June 11, 2024 16:31
@majetideepak
Copy link
Collaborator Author

@Yuhta Can you take a look? Thanks!

@majetideepak majetideepak requested a review from Yuhta June 11, 2024 16:52
@yingsu00 yingsu00 self-assigned this Jun 14, 2024
yingsu00
yingsu00 approved these changes Jun 14, 2024
View reviewed changes
Copy link
Collaborator

@yingsu00 yingsu00 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@majetideepak Thanks for fixing this! The bug also indicates insufficient tests to cover different compression schemes in existing E2EFilterTest. #8103. We will need to make the tests to loop over different compression schemes in future PRs.

velox/dwio/common/compression/Compression.cpp Show resolved Hide resolved
@majetideepak
Copy link
Collaborator Author

The bug also indicates insufficient tests to cover different compression schemes in existing E2EFilterTest

@yingsu00 We don't currently have support for lzo compression. We should add that support as well.

@majetideepak majetideepak marked this pull request as draft June 17, 2024 11:59
@majetideepak majetideepak marked this pull request as ready for review June 17, 2024 14:06
@majetideepak majetideepak requested a review from pedroerp June 17, 2024 14:06
@majetideepak
Copy link
Collaborator Author

@pedroerp can you please help review this? Thanks.

pedroerp
pedroerp approved these changes Jun 17, 2024
View reviewed changes
Copy link
Contributor

@pedroerp pedroerp left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you!

We should at some point have an actual fuzzer suite to cross test all sorts of file formats and compression schemes.

velox/dwio/parquet/tests/reader/ParquetReaderTest.cpp
TEST_F(ParquetReaderTest, testLzoDataPage) {
const std::string sample(getExampleFilePath("lzo.parquet"));

facebook::velox::dwio::common::ReaderOptions readerOptions{leafPool_.get()};
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: we should add tests to a "facebook::velox::dwio::test" sort of namespace, so you could just do "common::ReaderOptions" here

@facebook-github-bot
Copy link
Contributor

@pedroerp has imported this pull request. If you are a Meta employee, you can view this diff on Phabricator.

@facebook-github-bot
Copy link
Contributor

@pedroerp merged this pull request in e29f598.

@conbench-facebook conbench-facebook
Copy link

Conbench analyzed the 1 benchmark run on commit e29f5980.

There were no benchmark performance regressions. 🎉

The full Conbench report has more details.

@majetideepak majetideepak mentioned this pull request Jun 21, 2024
Open
@majetideepak majetideepak deleted the fix-lzo branch June 21, 2024 06:55
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@yingsu00 yingsu00 yingsu00 approved these changes

@nmahadevuni nmahadevuni nmahadevuni approved these changes

@pedroerp pedroerp pedroerp approved these changes

@Yuhta Yuhta Yuhta approved these changes

Assignees

@yingsu00 yingsu00

Labels
CLA Signed This label is managed by the Facebook bot. Authors need to sign the CLA before a PR can be reviewed. Merged
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

Parquet reading failed to decompress LZO files
6 participants
@majetideepak @facebook-github-bot @Yuhta @pedroerp @nmahadevuni @yingsu00