Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Ensuring that the TableChange is always broadcasting #73

Merged
merged 12 commits into from
Feb 11, 2022
Merged

Ensuring that the TableChange is always broadcasting #73

merged 12 commits into from
Feb 11, 2022

Conversation

cugni
Copy link
Member

@cugni cugni commented Feb 7, 2022 •
edited by osopardo1
Loading

To avoid issues when we write a large dataset, we must ensure that we always broadcast the TableChanges ( and we try to keep it small, but that is for another PR):

I've drafted the first change, but there are still a few points to resolve:

  • how to manage the state? It does not make sense to manage the state for each row independently, but we should merge it in the file meta.
  • Ensure everything works with the last changes.
  • Ensure the changes make sense and that the structure of the new metadata makes sense

cugni added 2 commits February 7, 2022 08:20
@cugni
wip on ensuring we always broadcast TableChanges
c6de5c4
@cugni
changing the default cubeSize to a more reasonable value
2823263 
We will have to change this value to something that takes into
consideraition also the size of a row
@osopardo1 osopardo1 marked this pull request as ready for review February 9, 2022 08:50
eavilaes
eavilaes approved these changes Feb 10, 2022
View reviewed changes
Copy link
Contributor

@eavilaes eavilaes left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Just a couple of comments, everything looks good to me!

core/src/main/scala/io/qbeast/core/model/PointWeightIndexer.scala Outdated Show resolved Hide resolved
src/test/scala/io/qbeast/spark/index/SparkPointWeightIndexerTest.scala Outdated Show resolved Hide resolved
@cugni cugni merged commit b89adac into Qbeast-io:main Feb 11, 2022
@cugni cugni deleted the broadcasting branch February 11, 2022 08:48
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@osopardo1 osopardo1 osopardo1 approved these changes

@eavilaes eavilaes eavilaes approved these changes

@Jiaweihu08 Jiaweihu08 Awaiting requested review from Jiaweihu08

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
3 participants
@cugni @osopardo1 @eavilaes