Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add Delta Data Skipping on Staging Area #239

Merged
merged 11 commits into from
Nov 30, 2023

Conversation

osopardo1
Copy link
Member

@osopardo1 osopardo1 commented Nov 28, 2023

Description

Fixes #235

Type of change

It is an enhancement, it does not affect the current writing algorithm/format.

Checklist:

Here is the list of things you should do before submitting this pull request:

  • New feature / bug fix has been committed following the Contribution guide.
  • Add comments to the code (make it easier for the community!).
  • Add tests.
  • Your branch is updated to the main-1.0.0 branch (dependent changes have been merged).

How Has This Been Tested? (Optional)

Added test to OTreeIndexTest to check if staging files are filtered properly.

@osopardo1 osopardo1 requested a review from cugni November 28, 2023 12:08
@osopardo1 osopardo1 marked this pull request as ready for review November 28, 2023 12:08
Copy link

codecov bot commented Nov 28, 2023

Codecov Report

Attention: 12 lines in your changes are missing coverage. Please review.

Comparison is base (2d39821) 90.86% compared to head (681518e) 91.04%.
Report is 30 commits behind head on main-1.0.0.

Files Patch % Lines
...core/transform/StringHistogramTransformation.scala 76.47% 8 Missing ⚠️
...cala/io/qbeast/spark/index/OTreeDataAnalyzer.scala 96.22% 2 Missing ⚠️
...e/src/main/scala/io/qbeast/core/model/CubeId.scala 94.73% 1 Missing ⚠️
...st/core/transform/StringHistogramTransformer.scala 88.88% 1 Missing ⚠️
Additional details and impacted files
@@              Coverage Diff               @@
##           main-1.0.0     #239      +/-   ##
==============================================
+ Coverage       90.86%   91.04%   +0.17%     
==============================================
  Files              93       93              
  Lines            2386     2445      +59     
  Branches          178      169       -9     
==============================================
+ Hits             2168     2226      +58     
- Misses            218      219       +1     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

@osopardo1 osopardo1 merged commit 4b32021 into Qbeast-io:main-1.0.0 Nov 30, 2023
3 checks passed
osopardo1 added a commit to osopardo1/qbeast-spark that referenced this pull request Dec 4, 2023
…ipping

Add Delta Data Skipping on Staging Area
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants