Snap 2358 Sorted Column Batches on partitioning keys#1054

Open

vibhaska wants to merge 392 commits intomasterfrom SNAP-2358

+2,755-96

This pull request is big! We're only showing the most recent 250 commits

Commits on Mar 7, 2018

Merge branch 'master' into li-perfomance-test
vivekwiz
committed
Merge branch 'li-perfomance-test' into vivek-try1
vivekwiz
committed
Reverted benchmark. Now use QueryBenchmark only for data generation purpose.
vivekwiz
committed
Updated test with new values
vivekwiz
committed
Merge branch 'li-perfomance-test' into vivek-try1
vivekwiz
committed
Updated test
vivekwiz
committed
test code refactoring
vivekwiz
committed
Merge branch 'li-perfomance-test' into vivek-try1
vivekwiz
committed
Updated test
vivekwiz
committed
Updated test
vivekwiz
committed
Merge branch 'li-perfomance-test' into vivek-try1
vivekwiz
committed
Updated test
vivekwiz
committed

Commits on Mar 8, 2018

Updated test to handle multiple inserts
vivekwiz
committed
Merge branch 'li-perfomance-test' into vivek-try1
vivekwiz
committed
Compilation issue
vivekwiz
committed
Added more duplicity in data
vivekwiz
committed
Merge branch 'li-perfomance-test' into vivek-try1
vivekwiz
committed
Updated params for test
vivekwiz
committed
Merge branch 'li-perfomance-test' into vivek-try1
vivekwiz
committed
Updated expectedcount
vivekwiz
committed
Merge branch 'li-perfomance-test' into vivek-try1
vivekwiz
committed
Updated expected count in test
vivekwiz
committed
Updated estimated count
vivekwiz
committed
Updated tests
vivekwiz
committed
[SNAP-2243][SNAP-2188] procedure for smart connector iteration and fixes
Sumedh Wale
committed

Commits on Mar 9, 2018

fixed test failures
Sumedh Wale
committed
changed Filters to Expressions in Relation APIs
Sumedh Wale
committed
Merge remote-tracking branch 'origin/master' into SNAP-2243
Sumedh Wale
committed
fixed issues in the new StartsWithForStats expression code
Sumedh Wale
committed
removed a debug assertion
Sumedh Wale
committed

Commits on Mar 10, 2018

correct one issue
Sumedh Wale
committed
update store link
Sumedh Wale
committed
added release for last batch in RemoteEntriesIterator
Sumedh Wale
committed
minor comment changes
Sumedh Wale
committed

Commits on Mar 11, 2018

minor change to retrieve ClusteredColumnIterator only once
Sumedh Wale
committed
[SNAP-2244] stats for delta column batches
Sumedh Wale
committed
fixed failures and few cleanups
Sumedh Wale
committed
added some code comments
Sumedh Wale
committed

Commits on Mar 12, 2018

fixing some failures
Sumedh Wale
committed
link store
vivekwiz
committed
fix a failure in SnappyRowStoreModeDUnit
Sumedh Wale
committed
Merge branch 'master' into li-perfomance-test
vivekwiz
committed
Sync spark
vivekwiz
committed
Merge branch 'li-perfomance-test' into vivek-try1
vivekwiz
committed
added tests for delta stats check; incorporate review comment
Sumedh Wale
committed
Changes to handle latest master merge
vivekwiz
committed
Changes to get rid of any customization to spark
vivekwiz
committed
Doing away with any customization in join attributes or forcing SMJ
vivekwiz
committed

Commits on Mar 13, 2018

Reintroducing customization to spark for testing purpose. Will be reverted back.
vivekwiz
committed
Customization to spark for testing purpose. Will be reverted back.
vivekwiz
committed
Disabling insert part of Put-Into
vivekwiz
committed
Changed put-into join condition to full outer.
vivekwiz
committed
Further done away with Insert part of PutInto in excution part.
vivekwiz
committed
Code refactoring
vivekwiz
committed
Sync spark
vivekwiz
committed
Sync spark
vivekwiz
committed

Commits on Mar 14, 2018

Switch to QueryBenchmark
vivekwiz
committed
Merge branch 'li-perfomance-test' into li-sorted-insert
vivekwiz
committed
Benchmark: For rate pass avg than best
vivekwiz
committed

Commits on Mar 15, 2018

Commits on Mar 16, 2018

Added Dunit based test and performance test
vivekwiz
committed
Merge branch 'li-perfomance-test' into li-sorted-insert
vivekwiz
committed
Minor update
vivekwiz
committed

Commits on Mar 19, 2018

Commits on Mar 20, 2018

Update multithreaded performance test
vivekwiz
committed
Changes done for Multithreaded performance test
vivekwiz
committed
Changes for latency tests
vivekwiz
committed
Updated number of threads
vivekwiz
committed
Refactored Multi threaded tests
vivekwiz
committed
Merge branch 'li-perfomance-test-SNAP-2243' into li-sorted-insert-SNAP-2243
vivekwiz
committed
Merge branch 'master' into li-perfomance-test-SNAP-2243
vivekwiz
committed
Merge branch 'li-perfomance-test-SNAP-2243' into li-sorted-insert-SNAP-2243
vivekwiz
committed
Refactoring of test code
vivekwiz
committed

Commits on Mar 21, 2018

Merge branch 'li-perfomance-test-SNAP-2243' into li-sorted-insert-SNAP-2243
vivekwiz
committed
Added test for join performance
vivekwiz
committed
Fix for build failure
vivekwiz
committed
Changes to generate correct data
Vivek Bhaskar
committed
Merge branch 'master' into li-perfomance-test-SNAP-2243
vivekwiz
committed
Merge branch 'li-perfomance-test-SNAP-2243' into li-sorted-insert-SNAP-2243
vivekwiz
committed
Fix a build issue
vivekwiz
committed

Commits on Mar 22, 2018

Refactored test so latency tests use Spark's Benchmark
vivekwiz
committed
Small change
vivekwiz
committed
Merge branch 'li-perfomance-test-SNAP-2243' into li-sorted-insert-SNAP-2243
vivekwiz
committed
Code refactoring and also marked as colocated table
vivekwiz
committed
Increased heap memory for test
vivekwiz
committed
Slight change
vivekwiz
committed
Merge branch 'li-perfomance-test-SNAP-2243' into li-sorted-insert-SNAP-2243
vivekwiz
committed
Added an issue
vivekwiz
committed
Merge branch 'li-perfomance-test-SNAP-2243' into li-sorted-insert-SNAP-2243
vivekwiz
committed
Merge branch 'master' into li-sorted
vivekwiz
committed
Merge branch 'li-master' into li-sorted
vivekwiz
committed

Commits on Mar 23, 2018

Merge branch 'master' into li-master
vivekwiz
committed
Small change
vivekwiz
committed
Merge branch 'li-master' into li-sorted
vivekwiz
committed
Code refactoring. Moved some code from li-sorted to li-master
vivekwiz
committed
Restored DUnit test
vivekwiz
committed
Restore DUnit test
vivekwiz
committed
Revert this. Only temporarily switch ON logs
vivekwiz
committed
Merge branch 'li-master' into li-sorted
vivekwiz
committed

Commits on Mar 26, 2018

Disabled insert into row-buffer.
vivekwiz
committed
Minor update
vivekwiz
committed
Force SMJ while performing join
vivekwiz
committed
Updated test
vivekwiz
committed
Code refactoring
vivekwiz
committed

Commits on Mar 28, 2018

Commits on Mar 29, 2018

Merge branch 'master' into li-master
vivekwiz
committed
Slight improvement over 'Temporary fix for getting column batches in
Vivek Bhaskar
committed

Commits on Apr 2, 2018

Updated join test
Vivek Bhaskar
committed
Merge branch 'li-master' into li-sorted
Vivek Bhaskar
committed
Updated insert performance test
Vivek Bhaskar
committed
Merge branch 'li-master' into li-sorted
Vivek Bhaskar
committed
Minor change
Vivek Bhaskar
committed

Commits on Apr 4, 2018

Merge branch 'master' into li-master
vivekwiz
committed
Merge branch 'li-master' into li-sorted
vivekwiz
committed

Commits on Apr 5, 2018

Merge branch 'master' into li-master
vivekwiz
committed
Merge branch 'li-master' into li-sorted
vivekwiz
committed

Commits on Apr 6, 2018

Removed redundant code
vivekwiz
committed
Follow up to: Temporary fix for insertingrows in sorted cahed batches which have no preceding rows in order in main table.
vivekwiz
committed
code refactoring
Vivek Bhaskar
committed

Commits on Apr 11, 2018

Even if Minimum or Maximum value has not changed, but delta would still change if its number of count increases or decreases
vivekwiz
committed
Fix for handling multiple incremental inserts
vivekwiz
committed
Added test for multiple inserts
vivekwiz
committed
Disable overflow of column batches while scanning
vivekwiz
committed

Commits on Apr 12, 2018

Added test for duplicates in incremenatl inserts
vivekwiz
committed
Corrected a test
vivekwiz
committed
Updated range query function
vivekwiz
committed
Updated range performance test
vivekwiz
committed
Merge branch 'li-master' into li-sorted
vivekwiz
committed

Commits on Apr 13, 2018

Correctly reflecting count of incremental insert in delta i.e. update count
vivekwiz
committed
Added test to do both update and incremental insert
vivekwiz
committed
Updated merge primitive to handle both update and insert
vivekwiz
committed
Updated test for update and insert
vivekwiz
committed

Commits on Apr 17, 2018

Fix issue with handling incremental insert and update on same ordinal.
vivekwiz
committed
Small code refactoring
vivekwiz
committed
Small code refactoring
vivekwiz
committed
Handling duplicate values incse of both insert and update
vivekwiz
committed
Updated test for update and insert
vivekwiz
committed
Updated test
vivekwiz
committed

Commits on Apr 18, 2018

Updated test
vivekwiz
committed
Always do update of delta even when there is no changes in count
vivekwiz
committed
Revert "Always do update of delta even when there is no changes in count"
vivekwiz
committed

Commits on Apr 24, 2018

Merge branch 'master' into li-master
vivekwiz
committed
Merge branch 'li-master' into li-sorted
vivekwiz
committed
Handling both update and insert on same dataset even with multiple buckets.
vivekwiz
committed
Updated test for insert and update
vivekwiz
committed

Commits on Apr 27, 2018

Added new tests
vivekwiz
committed
Adding rudimentary ColumnSortedInsertExec Node
vivekwiz
committed

Commits on May 2, 2018

Merge branch 'master' into li-master
vivekwiz
committed
Merge branch 'li-master' into li-sorted
vivekwiz
committed
Doing away with idea of adding an extra node over SMJ for insert for now
vivekwiz
committed

Commits on May 8, 2018

Merge branch 'master' into li-master
vivekwiz
committed
Merge branch 'li-master' into li-sorted
vivekwiz
committed

Commits on May 10, 2018

Adding basic delete functionality
vivekwiz
committed
Sync store
vivekwiz
committed
Merge branch 'master' into li-master
vivekwiz
committed
Merge branch 'li-master' into li-sorted
vivekwiz
committed

Commits on May 11, 2018

Merge branch 'master' into li-master
vivekwiz
committed
Merge branch 'li-master' into li-sorted
vivekwiz
committed

Commits on May 15, 2018

Changing column table scan so to handle deletes in both dictionary and delte buffer
vivekwiz
committed
Improving over commit 7983bb4 i.e. Adding basic delete functionality
vivekwiz
committed
Improving delete tests
vivekwiz
committed
Merge branch 'master' into li-master
vivekwiz
committed
Merge branch 'li-master' into li-sorted
vivekwiz
committed
Updated test
vivekwiz
committed

Commits on May 17, 2018

Commits on May 23, 2018

Merge branch 'master' into li-master
vivekwiz
committed
Merge branch 'li-master' into li-sorted
vivekwiz
committed

Commits on May 24, 2018

Removing redundant changes
vivekwiz
committed

Commits on May 29, 2018

Reverted all changes related to Put Into so far
vivekwiz
committed

Commits on May 30, 2018

DML changes for insert that will follow a new path now
vivekwiz
committed
Merge branch 'master' into SNAP-2358
vivekwiz
committed
Reverting some redundant changes
vivekwiz
committed

Commits on May 31, 2018

Changes from Sumedh to sort column batches on minimum value
vivekwiz
committed

Commits on Jun 4, 2018

Temporary changes to make sorting on column batches working
vivekwiz
committed
Merge branch 'master' into SNAP-2358
vivekwiz
committed

Commits on Jun 5, 2018

Removing local flags
vivekwiz
committed

Commits on Jun 6, 2018

Revert "Doing away with idea of adding an extra node over SMJ for insert for now"
vivekwiz
committed
Removed a hardcoded flag and also removing any dependency on changes in spark
vivekwiz
committed
Merge branch 'master' into SNAP-2358
vivekwiz
committed
Merge branch 'SNAP-2358' into SNAP-2358-v1
vivekwiz
committed

Commits on Jun 7, 2018

Remove a redundancy
vivekwiz
committed
Merge branch 'master' into SNAP-2358
vivekwiz
committed
Merge branch 'SNAP-2358' into SNAP-2358-v1
vivekwiz
committed
Removing hardcoded elimination of exceptions.
vivekwiz
committed
Added a TODO
vivekwiz
committed

Commits on Jun 12, 2018

Merge branch 'master' into SNAP-2358
vivekwiz
committed
Merge branch 'SNAP-2358' into SNAP-2358-v1
vivekwiz
committed

Commits on Jun 14, 2018

Incorporating required DDL changes so column batches would only be sorted when user has given option while create table.
vivekwiz
committed
Merge branch 'master' into SNAP-2358
vivekwiz
committed
Merge branch 'SNAP-2358' into SNAP-2358-v1
vivekwiz
committed
Updated DDL changes
vivekwiz
committed
Handle case where table is not sorted
vivekwiz
committed
small correction
vivekwiz
committed

Commits on Jun 16, 2018

First phase of changes to handle both first insert and delta insert with same syntax.
vivekwiz
committed

Commits on Jun 17, 2018

Second phase of changes to handle both first insert and delta insert with same syntax.
vivekwiz
committed
Code refactoring over last commit
vivekwiz
committed
Handling direct insert scenario
vivekwiz
committed
Added change for handling delta insert rows that falls in existing range
vivekwiz
committed

Commits on Jun 18, 2018

Commits on Jun 19, 2018

Added ways to avoid scans that are not needed while insert
vivekwiz
committed

Commits on Jun 20, 2018

Commits on Jun 21, 2018

Test refactoring. Commented out a failing test
vivekwiz
committed
Removed debug changes for update related to delta insert. may be needed for debugging.
vivekwiz
committed
Code refatoring to guard delta inserted change sunder a flag
vivekwiz
committed
Further code refactoring to make pass precheckin
vivekwiz
committed
Putting a small fix for eliminating extra sort node
vivekwiz
committed
Marked partitioning information in DeltaInsertExec
vivekwiz
committed
Removed extra debug flag
vivekwiz
committed
Disable dunit and performance tests by default
vivekwiz
committed
Code rectoring to bring code generated changes under a flag
vivekwiz
committed
Further refactoring for generated code and update
vivekwiz
committed
Further refactor column delta related code for taking under a flag
vivekwiz
committed

Commits on Jun 22, 2018

Commits on Jun 25, 2018

Merge branch 'master' into SNAP-2358
vivekwiz
committed
Removing an optimization that used iterator only for cases of join and insert.
vivekwiz
committed

Commits on Jun 27, 2018

Fix for an issue reklated to sorting
vivekwiz
committed

Commits on Aug 2, 2018

resolved merge conflict with master in SnappySessionState
hemanthmeka
committed

Commits on Aug 8, 2018

merging from master
hemanthmeka
committed