Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Refactor Interval Arithmetic Updates #8276

Merged
merged 47 commits into from
Nov 21, 2023
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
47 commits
Select commit Hold shift + click to select a range
1cb0927
Interval lib can be accessible from logical plan
berkaysynnada Oct 17, 2023
3398701
committed to merge precision PR
berkaysynnada Oct 19, 2023
9744d44
Merge branch 'apache_main' into feature/interval-arithmetic-updates
berkaysynnada Oct 19, 2023
7dfa92f
minor fix after merge, adding ts-interval handling
berkaysynnada Oct 20, 2023
efdb6ec
Merge branch 'apache_main' into feature/interval-arithmetic-updates
berkaysynnada Oct 20, 2023
509189f
bound openness removed
berkaysynnada Oct 20, 2023
2469438
Remove all interval bound related code
berkaysynnada Nov 7, 2023
dfec420
test fix
berkaysynnada Nov 7, 2023
e58bebb
fix docstrings
berkaysynnada Nov 8, 2023
fee3a7a
Merge branch 'apache_main' into feature/interval-arithmetic-updates
berkaysynnada Nov 8, 2023
501ad5c
Fix after merge
berkaysynnada Nov 8, 2023
4a8f0a7
Minor changes
berkaysynnada Nov 8, 2023
394ea3f
Simplifications
mustafasrepo Nov 8, 2023
1d30998
Resolve linter errors
mustafasrepo Nov 8, 2023
7234812
Addressing reviews
berkaysynnada Nov 9, 2023
17b51a4
Fix win tests
berkaysynnada Nov 10, 2023
8db8068
Update interval_arithmetic.rs
berkaysynnada Nov 10, 2023
940f899
Merge branch 'apache_main' into feature/interval-arithmetic-updates
berkaysynnada Nov 10, 2023
c87f857
Code simplifications
berkaysynnada Nov 10, 2023
89bff3a
Merge remote-tracking branch 'origin/apache_main' into feature/interv…
ozankabak Nov 11, 2023
5217a29
Review Part 1
ozankabak Nov 12, 2023
6bf4472
Review Part 2
ozankabak Nov 13, 2023
7cae34c
Addressing Ozan's feedback
berkaysynnada Nov 13, 2023
9d97441
Merge branch 'apache_main' into feature/interval-arithmetic-updates
berkaysynnada Nov 13, 2023
789490e
Resolving conflicts
berkaysynnada Nov 13, 2023
423494c
Review Part 3
ozankabak Nov 14, 2023
1de1275
Better cardinality calculation
berkaysynnada Nov 14, 2023
9e5b7ae
fix clippy
berkaysynnada Nov 14, 2023
f3ce775
Merge branch 'apache_main' into feature/interval-arithmetic-updates
berkaysynnada Nov 15, 2023
1b090eb
Merge branch 'apache_main' into feature/interval-arithmetic-updates
berkaysynnada Nov 15, 2023
d7ee755
Review Part 4
ozankabak Nov 15, 2023
9ad2e37
type check, test polish, bug fix
berkaysynnada Nov 16, 2023
6fc867e
Constructs filter graph with datatypes
berkaysynnada Nov 17, 2023
272f39a
Merge branch 'apache_main' into feature/interval-arithmetic-updates
berkaysynnada Nov 17, 2023
d5fb068
Update Cargo.lock
berkaysynnada Nov 17, 2023
d6c7c18
Update Cargo.toml
berkaysynnada Nov 17, 2023
10d3f13
Review
ozankabak Nov 19, 2023
fe99558
Other expectations of AND
berkaysynnada Nov 20, 2023
f835c29
OR operator implementation
berkaysynnada Nov 20, 2023
341cff2
Certainly false asserting comparison operators
berkaysynnada Nov 20, 2023
8c45de8
Update interval_arithmetic.rs
berkaysynnada Nov 20, 2023
e0c4b9f
Merge branch 'apache_main' into feature/interval-arithmetic-updates
berkaysynnada Nov 20, 2023
3b6ab4a
Tests added, is_superset renamed
berkaysynnada Nov 20, 2023
d113224
Final review
ozankabak Nov 20, 2023
080637d
Merge branch 'apache_main' into feature/interval-arithmetic-updates
berkaysynnada Nov 20, 2023
c2143e4
Resolving conflicts
berkaysynnada Nov 20, 2023
5e5384c
Address review feedback
ozankabak Nov 21, 2023
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
55 changes: 23 additions & 32 deletions datafusion-cli/Cargo.lock

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

12 changes: 10 additions & 2 deletions datafusion/common/Cargo.toml
Original file line number Diff line number Diff line change
Expand Up @@ -38,14 +38,22 @@ backtrace = []
pyarrow = ["pyo3", "arrow/pyarrow", "parquet"]

[dependencies]
ahash = { version = "0.8", default-features = false, features = ["runtime-rng"] }
apache-avro = { version = "0.16", default-features = false, features = ["bzip", "snappy", "xz", "zstandard"], optional = true }
ahash = { version = "0.8", default-features = false, features = [
"runtime-rng",
] }
apache-avro = { version = "0.16", default-features = false, features = [
"bzip",
"snappy",
"xz",
"zstandard",
], optional = true }
arrow = { workspace = true }
arrow-array = { workspace = true }
arrow-buffer = { workspace = true }
arrow-schema = { workspace = true }
chrono = { workspace = true }
half = { version = "2.1", default-features = false }
libc = "0.2.140"
num_cpus = { workspace = true }
object_store = { workspace = true, optional = true }
parquet = { workspace = true, optional = true, default-features = true }
Expand Down
1 change: 1 addition & 0 deletions datafusion/common/src/lib.rs
Original file line number Diff line number Diff line change
Expand Up @@ -34,6 +34,7 @@ pub mod file_options;
pub mod format;
pub mod hash_utils;
pub mod parsers;
pub mod rounding;
pub mod scalar;
pub mod stats;
pub mod test_util;
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -22,8 +22,8 @@

use std::ops::{Add, BitAnd, Sub};

use datafusion_common::Result;
use datafusion_common::ScalarValue;
use crate::Result;
use crate::ScalarValue;

// Define constants for ARM
#[cfg(all(target_arch = "aarch64", not(target_os = "windows")))]
Expand Down Expand Up @@ -162,7 +162,7 @@ impl FloatBits for f64 {
/// # Examples
///
/// ```
/// use datafusion_physical_expr::intervals::rounding::next_up;
/// use datafusion_common::rounding::next_up;
///
/// let f: f32 = 1.0;
/// let next_f = next_up(f);
Expand Down Expand Up @@ -195,7 +195,7 @@ pub fn next_up<F: FloatBits + Copy>(float: F) -> F {
/// # Examples
///
/// ```
/// use datafusion_physical_expr::intervals::rounding::next_down;
/// use datafusion_common::rounding::next_down;
///
/// let f: f32 = 1.0;
/// let next_f = next_down(f);
Expand Down
42 changes: 42 additions & 0 deletions datafusion/common/src/scalar.rs
Original file line number Diff line number Diff line change
Expand Up @@ -988,6 +988,48 @@ impl ScalarValue {
Self::try_from_array(r.as_ref(), 0)
}

/// Wrapping multiplication of `ScalarValue`
///
/// NB: operating on `ScalarValue` directly is not efficient, performance sensitive code
/// should operate on Arrays directly, using vectorized array kernels.
pub fn mul<T: Borrow<ScalarValue>>(&self, other: T) -> Result<ScalarValue> {
let r = mul_wrapping(&self.to_scalar()?, &other.borrow().to_scalar()?)?;
Self::try_from_array(r.as_ref(), 0)
}

/// Checked multiplication of `ScalarValue`
///
/// NB: operating on `ScalarValue` directly is not efficient, performance sensitive code
/// should operate on Arrays directly, using vectorized array kernels.
pub fn mul_checked<T: Borrow<ScalarValue>>(&self, other: T) -> Result<ScalarValue> {
let r = mul(&self.to_scalar()?, &other.borrow().to_scalar()?)?;
Self::try_from_array(r.as_ref(), 0)
}

/// Performs `lhs / rhs`
///
/// Overflow or division by zero will result in an error, with exception to
/// floating point numbers, which instead follow the IEEE 754 rules.
///
/// NB: operating on `ScalarValue` directly is not efficient, performance sensitive code
/// should operate on Arrays directly, using vectorized array kernels.
pub fn div<T: Borrow<ScalarValue>>(&self, other: T) -> Result<ScalarValue> {
let r = div(&self.to_scalar()?, &other.borrow().to_scalar()?)?;
Self::try_from_array(r.as_ref(), 0)
}

/// Performs `lhs % rhs`
///
/// Overflow or division by zero will result in an error, with exception to
/// floating point numbers, which instead follow the IEEE 754 rules.
///
/// NB: operating on `ScalarValue` directly is not efficient, performance sensitive code
/// should operate on Arrays directly, using vectorized array kernels.
pub fn rem<T: Borrow<ScalarValue>>(&self, other: T) -> Result<ScalarValue> {
let r = rem(&self.to_scalar()?, &other.borrow().to_scalar()?)?;
Self::try_from_array(r.as_ref(), 0)
}

pub fn is_unsigned(&self) -> bool {
matches!(
self,
Expand Down
5 changes: 4 additions & 1 deletion datafusion/expr/Cargo.toml
Original file line number Diff line number Diff line change
Expand Up @@ -35,10 +35,13 @@ path = "src/lib.rs"
[features]

[dependencies]
ahash = { version = "0.8", default-features = false, features = ["runtime-rng"] }
ahash = { version = "0.8", default-features = false, features = [
"runtime-rng",
] }
arrow = { workspace = true }
arrow-array = { workspace = true }
datafusion-common = { workspace = true }
paste = "^1.0"
sqlparser = { workspace = true }
strum = { version = "0.25.0", features = ["derive"] }
strum_macros = "0.25.0"
Expand Down
Loading