feat: Introduce convert Expr to SQL string API and basic feature #9517

backkem · 2024-03-09T14:25:00Z

Relates to #9495.

This is an initial attempt at porting over the SQL Unparser as described in the tracking ticket.

Open question: Do we need this to be feature complete before landing?

devinjdangelo

Since this is entirely new functionality, I think it is a good idea to merge this as a start so we can move further development to the DataFusion repo, where it will be easiest to collaborate with more interested parties.

devinjdangelo · 2024-03-09T16:48:48Z

datafusion/sql/src/unparser/dialect.rs

@@ -0,0 +1,52 @@
+pub trait Dialect {


It is too bad we can't simply use the Dialect trait already in sqlparser... but it does appear that trait can't tell you the quote style directly.

Shall we make a change upstream to sqlparser? I would be happy to review such a PR (or maybe just make it?) We could leave a comment in the PR to remove this trait when the new sqlparser cate was released

devinjdangelo · 2024-03-09T16:49:21Z

datafusion/sql/src/unparser/expr.rs

+                let r = self.expr_to_sql(right.as_ref())?;
+                let op = self.op_to_sql(op)?;
+
+                Ok(self.binary_op_to_sql(l, r, op))


I have found that we lose order of operations information here, but that wrapping the output in SQLExpr::Nested fixes it.

E.g.

x+1 / 2

is not the same as

(x+1)/2

I can open a follow up PR with some tests to show this.

Does DF represent this nesting explicitly as well? Or does it use the Expr hierarchy to capture the order?

If it's the latter, do we want to inject brackets for every operations or have a heuristic to only add them when mathematically needed?

Based on playing around with round trips (AST->Logical Plan->AST) I believe that Logical Plans encode operation ordering in the tree itself while AST represents the literal ordering in the SQL string. The easiest way to guarantee mathematical equivalence would be to always add SQLExpr::Nested. A heuristic could work but would be quite a bit more complex I think.

DF represents the nesting via Expr children

In the SQL parser, there are precident rules to resolve how a + b + c into a tree and then DataFusion simply gets the same tree structure

So the difference between (a + b) + c ad a + (b + c) is a function of what the left and right of the BinaryExpr are

I agree with @devinjdangelo that we should file a ticket to track this issue and work on in a subsequent PR

devinjdangelo · 2024-03-09T16:52:16Z

datafusion/sql/src/unparser/expr.rs

+        }
+    }
+
+    fn scalar_to_sql(&self, v: &ScalarValue) -> Result<ast::Value> {


I believe that some DataFusion Scalars will require mapping to a SQLExpr. For example for date scalars, I think the only way is to convert to something like CAST(quoted_string_val to DATE).

This is a private function though, so does not need to be resolved prior to merging imo.

devinjdangelo · 2024-03-09T16:55:33Z

datafusion/sql/src/unparser/expr.rs

+    #[test]
+    fn expr_to_sql_ok() -> Result<()> {
+        let tests: Vec<(Expr, &str)> = vec![
+            (col("a").gt(lit(4)), r#"a > 4"#),


I think these tests look good to start. It might also be a good idea to have some tests that do a full round trip using SqltoRel, e.g. String -> AST -> LogicalPlan -> AST -> String and making sure the Strings and AST match.

I agree the round trip test would be good (and will make it quite mechanical to add coverage for the missing expressions)

Something like

round_trip("a"); // parse "a" to Expr and then turn back to string round_trip("a + b"); ...

I think we could do it as a follow on PR as well

alamb

First of all thank you @backkem and @devinjdangelo -- this looks great.

Open question: Do we need this to be feature complete before landing?

No

Since this is entirely new functionality, I think it is a good idea to merge this as a start so we can move further development to the DataFusion repo, where it will be easiest to collaborate with more interested parties.

I agree 100%

Typically what we do in DataFusion is incrementally develop features on main. The requirements are typically:

The new code has test coverage
It doesn't regress any existing functionality (aka make it impossible to do something that used to be possible)
Any unimplemented features return DataFusionError::NotYetImplemented rather than, for example, panics

alamb

Thank you @backkem and @devinjdangelo (again)!

I started the CI -- once that is clean I think this PR would be good to merge

Here are some follow on items that occur to me. How would you like to track them (perhaps I can file a ticket or we can put a list on the existing ticket)?

Upstream changes in datafusion/sql/src/unparser/dialect.rs to sqlparser-rs
Additional Expr variant support
Sorting out binary expr nesting
Add an example of how to use this API in expr_api.rs example
Round trip tests

alamb · 2024-03-10T11:42:12Z

datafusion/sql/src/unparser/dialect.rs

@@ -0,0 +1,52 @@
+pub trait Dialect {


Shall we make a change upstream to sqlparser? I would be happy to review such a PR (or maybe just make it?) We could leave a comment in the PR to remove this trait when the new sqlparser cate was released

datafusion/sql/src/unparser/expr.rs

alamb · 2024-03-10T11:47:33Z

datafusion/sql/src/unparser/expr.rs

+                let r = self.expr_to_sql(right.as_ref())?;
+                let op = self.op_to_sql(op)?;
+
+                Ok(self.binary_op_to_sql(l, r, op))


DF represents the nesting via Expr children

In the SQL parser, there are precident rules to resolve how a + b + c into a tree and then DataFusion simply gets the same tree structure

So the difference between (a + b) + c ad a + (b + c) is a function of what the left and right of the BinaryExpr are

I agree with @devinjdangelo that we should file a ticket to track this issue and work on in a subsequent PR

alamb · 2024-03-10T11:50:12Z

datafusion/sql/src/unparser/expr.rs

+    #[test]
+    fn expr_to_sql_ok() -> Result<()> {
+        let tests: Vec<(Expr, &str)> = vec![
+            (col("a").gt(lit(4)), r#"a > 4"#),


I agree the round trip test would be good (and will make it quite mechanical to add coverage for the missing expressions)

Something like

round_trip("a"); // parse "a" to Expr and then turn back to string round_trip("a + b"); ...

I think we could do it as a follow on PR as well

datafusion/sql/src/unparser/mod.rs

relates to apache/datafusion#9517

backkem · 2024-03-11T13:40:13Z

I believe I addressed the points that we didn't decide to ticket out. The CI indicates that some workflow still requires approval from a maintainer

backkem · 2024-03-11T17:20:38Z

I see the CI fails on some dead code. I'll look into it. There is another CI failure but it seems related to a network timeout.

datafusion/sql/src/unparser/dialect.rs

alamb · 2024-03-11T20:32:25Z

I added the follow on tasks to #9495. Thanks @backkem

alamb · 2024-03-11T21:03:48Z

Onwards and upwards (and thank you for your first contribution @backkem )

feat: convert Expr to SQL string

03d078f

github-actions bot added the sql SQL Planner label Mar 9, 2024

devinjdangelo approved these changes Mar 9, 2024

View reviewed changes

devinjdangelo mentioned this pull request Mar 10, 2024

Basic support for converting Expr to SQL string #9495

Closed

5 tasks

alamb reviewed Mar 10, 2024

View reviewed changes

alamb changed the title ~~feat: convert Expr to SQL string~~ feat: Introduce convert Expr to SQL string API and basic eature Mar 10, 2024

alamb changed the title ~~feat: Introduce convert Expr to SQL string API and basic eature~~ feat: Introduce convert Expr to SQL string API and basic feature Mar 10, 2024

alamb reviewed Mar 10, 2024

View reviewed changes

alamb approved these changes Mar 10, 2024

View reviewed changes

backkem added a commit to backkem/sqlparser-rs that referenced this pull request Mar 10, 2024

feat: add identifier quote style to dialect

05f003e

relates to apache/datafusion#9517

backkem mentioned this pull request Mar 10, 2024

Add identifier quote style to Dialect trait apache/datafusion-sqlparser-rs#1170

Merged

fix: add license headers

2cda8b0

fix: make Unparser and Dialect public

9ead24b

alamb reviewed Mar 11, 2024

View reviewed changes

datafusion/sql/src/unparser/dialect.rs Outdated Show resolved Hide resolved

alamb added 2 commits March 11, 2024 16:28

Update datafusion/sql/src/unparser/dialect.rs

07ff3e5

fmt

b6cff96

alamb merged commit 02f7e1f into apache:main Mar 11, 2024
22 of 23 checks passed

This was referenced Mar 12, 2024

[EPIC] Support LogicalPlan --> SQL String #8661

Closed

Initial support LogicalPlan to SQL String #9596

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat: Introduce convert Expr to SQL string API and basic feature #9517

feat: Introduce convert Expr to SQL string API and basic feature #9517

backkem commented Mar 9, 2024

devinjdangelo left a comment

devinjdangelo Mar 9, 2024

alamb Mar 10, 2024 •

edited

Loading

devinjdangelo Mar 9, 2024

backkem Mar 10, 2024

devinjdangelo Mar 10, 2024

alamb Mar 10, 2024

devinjdangelo Mar 9, 2024

devinjdangelo Mar 9, 2024

alamb Mar 10, 2024

alamb left a comment

alamb left a comment

alamb Mar 10, 2024 •

edited

Loading

alamb Mar 10, 2024

alamb Mar 10, 2024

backkem commented Mar 11, 2024

backkem commented Mar 11, 2024

alamb commented Mar 11, 2024

alamb commented Mar 11, 2024

feat: Introduce convert Expr to SQL string API and basic feature #9517

feat: Introduce convert Expr to SQL string API and basic feature #9517

Conversation

backkem commented Mar 9, 2024

devinjdangelo left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

alamb Mar 10, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

alamb left a comment

Choose a reason for hiding this comment

alamb left a comment

Choose a reason for hiding this comment

alamb Mar 10, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

backkem commented Mar 11, 2024

backkem commented Mar 11, 2024

alamb commented Mar 11, 2024

alamb commented Mar 11, 2024

alamb Mar 10, 2024 •

edited

Loading

alamb Mar 10, 2024 •

edited

Loading