Support modulus op #510

gangliao · 2021-06-05T14:53:58Z

Which issue does this PR close?

apache/arrow-rs#98
#99
apache/arrow-rs#317

Rationale for this change

What changes are included in this PR?

Are there any user-facing changes?

no

gangliao · 2021-06-05T15:22:26Z

@alamb Is that possible to let datafusion use the latest code from arrow-rs?

alamb · 2021-06-05T15:43:25Z

Thank you for your contribution @gangliao !

@alamb Is that possible to let datafusion use the latest code from arrow-rs?

As of now, we are only using "released" versions of arrow.rs.

Your change in apache/arrow-rs#317 is currently slated to be released in 4.3.0 (eta ~ 10 or so days) at which time it will be available for datafusion to use without any additional action

Until 4.3.0 is released, for your branch you can point datafusion to a sha from the arrow github to make sure the code works (e.g. how it was done prior to #395)

Basically we should wait until 4.3.0 is released and then this PR can be merged into datafusion

gangliao · 2021-06-05T15:55:39Z

Got it! Thank you for the clarification.

Co-authored-by: Sathis Kumar <sathis.kumar@udemy.com>

) * refactor sort exec stream and combine batches * refactor async function

* 110 support group by positions * try resolve positions via array, not map * Add comment for i64 and simplify the pattern match * combine match and if condition, add more test cases * replace '0 as i64' with 0_i64

* Fix aggregate fn with invalid column * Fix error message * Fix error message * fix clippy * fix message and test * avoid unwrap in test_aggregation_with_bad_arguments * Update datafusion/tests/sql.rs Co-authored-by: Andrew Lamb <andrew@nerdnetworks.org> * Fix test_aggregation_with_bad_arguments Co-authored-by: Andrew Lamb <andrew@nerdnetworks.org>

* Use pytest * Formatting * Update GHA conf * Remove TODO note * Format * Test requirements file * Update workflow file * Merge requirements file * Update workflow file

…planning (apache#501) * closing up type checks * add fmt

…pache#532) * use logical planner in ballista building * simplify statement * fix unit test * fix per comment

…he#541) * ShuffleReaderExec now supports multiple locations per partition * Remove TODO * avoid clone

* refactor hash aggregates * remove stale comments

* turn on clippy rule for needless borrow * do a format round * use warn not deny

…e#544)

* Cleanup RepartitionExec code * cleanup metric handling * Add elapsed_nanos

* support table alias in join clause * Update datafusion/src/sql/planner.rs Co-authored-by: Andrew Lamb <andrew@nerdnetworks.org> Co-authored-by: Andrew Lamb <andrew@nerdnetworks.org>

alamb · 2021-06-14T15:40:02Z

Arrow 4.3.0 has been released so if you rebase this PR it will likely be ready to go

…nto mod

codecov-commenter · 2021-06-16T15:38:15Z

Codecov Report

Merging #510 (f63c8fe) into master (a9d04ca) will increase coverage by 0.03%.
The diff coverage is 72.05%.

❗ Current head f63c8fe differs from pull request most recent head 73902b6. Consider uploading reports for the commit 73902b6 to get more accurate results

@@            Coverage Diff             @@
##           master     #510      +/-   ##
==========================================
+ Coverage   76.07%   76.11%   +0.03%     
==========================================
  Files         155      156       +1     
  Lines       26544    27061     +517     
==========================================
+ Hits        20194    20597     +403     
- Misses       6350     6464     +114

Impacted Files	Coverage Δ
ballista/rust/client/src/context.rs	`0.00% <0.00%> (ø)`
...ta/rust/core/src/execution_plans/shuffle_reader.rs	`0.00% <0.00%> (ø)`
...sta/rust/core/src/serde/logical_plan/from_proto.rs	`35.04% <0.00%> (-0.88%)`	⬇️
...lista/rust/core/src/serde/logical_plan/to_proto.rs	`61.41% <0.00%> (-1.00%)`	⬇️
...ista/rust/core/src/serde/physical_plan/to_proto.rs	`49.38% <0.00%> (-0.93%)`	⬇️
ballista/rust/core/src/utils.rs	`25.53% <0.00%> (-2.06%)`	⬇️
ballista/rust/executor/src/flight_service.rs	`0.00% <ø> (ø)`
ballista/rust/scheduler/src/planner.rs	`66.91% <ø> (ø)`
ballista/rust/scheduler/src/state/mod.rs	`70.49% <0.00%> (ø)`
datafusion-cli/src/lib.rs	`0.00% <0.00%> (ø)`
... and 44 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update a9d04ca...73902b6. Read the comment docs.

alamb · 2021-06-16T16:38:52Z

I am not sure what has happened to this PR but the diff looks not right:

Where I would expect to only see the changes from 97fd9ff

gangliao · 2021-06-16T19:04:38Z

Yeah. Looks strange. I will resubmit a new PR tonight.

gangliao · 2021-06-17T03:02:32Z

#577

Support modulus op

97fd9ff

msathis and others added 11 commits June 5, 2021 23:34

Optimize cast function during planning stage (apache#513)

b84789a

Co-authored-by: Sathis Kumar <sathis.kumar@udemy.com>

Fix display of execution time (apache#514)

ee2b9ef

closing up type checks (apache#506)

767eeb0

refactor sort exec stream and combine batches (apache#515)

5773a03

closing up type checks (apache#518)

63accf8

Refactor window aggregation, simplify batch processing logic (apache#516

2f73e79

) * refactor sort exec stream and combine batches * refactor async function

110 support group by positions (apache#519)

e39f311

* 110 support group by positions * try resolve positions via array, not map * Add comment for i64 and simplify the pattern match * combine match and if condition, add more test cases * replace '0 as i64' with 0_i64

Define the unittests using pytest (apache#493)

42f908e

* Use pytest * Formatting * Update GHA conf * Remove TODO note * Format * Test requirements file * Update workflow file * Merge requirements file * Update workflow file

Add partition by constructs in window functions and modify logical …

d5bca0e

…planning (apache#501) * closing up type checks * add fmt

Return errors properly from RepartitionExec (apache#521)

8f84564

alamb added the datafusion Changes in the datafusion crate label Jun 10, 2021

jimexist and others added 12 commits June 10, 2021 10:55

add boundary check (apache#530)

77775b7

remove redundant into_iter() calls (apache#527)

5c88450

use nightly nightly-2021-05-10 (apache#536)

3ef7f34

Make BallistaContext::collect streaming (apache#535)

63e3045

reuse datafusion physical planner in ballista building from protobuf (a…

ad70a1e

…pache#532) * use logical planner in ballista building * simplify statement * fix unit test * fix per comment

ShuffleReaderExec now supports multiple locations per partition (apac…

8f4078d

…he#541) * ShuffleReaderExec now supports multiple locations per partition * Remove TODO * avoid clone

Refactor hash aggregates's planner building code (apache#539)

519698a

* refactor hash aggregates * remove stale comments

turn on clippy rule for needless borrow (apache#545)

738f13b

* turn on clippy rule for needless borrow * do a format round * use warn not deny

apache#420: Support for not_eq predicate in pruning predicates (apach…

2568323

…e#544)

Cleanup Repartition Exec code (apache#538)

d382854

* Cleanup RepartitionExec code * cleanup metric handling * Add elapsed_nanos

support table alias in join clause (apache#547)

91af820

* support table alias in join clause * Update datafusion/src/sql/planner.rs Co-authored-by: Andrew Lamb <andrew@nerdnetworks.org> Co-authored-by: Andrew Lamb <andrew@nerdnetworks.org>

reuse code for now function expr creation (apache#548)

fe810e9

Support modulus op

9c23de0

Merge branch 'mod' of https://github.com/DSLAM-UMD/arrow-datafusion i…

73902b6

…nto mod

gangliao closed this Jun 17, 2021

alamb mentioned this pull request Jun 6, 2022

Update sqlparser-rs to 0.18.0 #2705

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Support modulus op #510

Support modulus op #510

gangliao commented Jun 5, 2021 •

edited

Loading

gangliao commented Jun 5, 2021

alamb commented Jun 5, 2021

gangliao commented Jun 5, 2021

alamb commented Jun 14, 2021

codecov-commenter commented Jun 16, 2021

alamb commented Jun 16, 2021

gangliao commented Jun 16, 2021

gangliao commented Jun 17, 2021

Support modulus op #510

Support modulus op #510

Conversation

gangliao commented Jun 5, 2021 • edited Loading

Which issue does this PR close?

Rationale for this change

What changes are included in this PR?

Are there any user-facing changes?

gangliao commented Jun 5, 2021

alamb commented Jun 5, 2021

gangliao commented Jun 5, 2021

alamb commented Jun 14, 2021

codecov-commenter commented Jun 16, 2021

Codecov Report

alamb commented Jun 16, 2021

gangliao commented Jun 16, 2021

gangliao commented Jun 17, 2021

gangliao commented Jun 5, 2021 •

edited

Loading