Skip to content

Conversation

goldmedal
Copy link
Contributor

@goldmedal goldmedal commented Jul 29, 2025

Which issue does this PR close?

  • Closes #.

Rationale for this change

The default null ordering is much different in different database.
In Postgres, the null value is considered as the max value.

psql=# select * from (values (1, 'one'), (2, 'two'), (null, 'three')) t(c1, c2) order by c1 asc;
 c1 |  c2   
----+-------
  1 | one
  2 | two
    | three
(3 rows)

psql=# select * from (values (1, 'one'), (2, 'two'), (null, 'three')) t(c1, c2) order by c1 desc;
 c1 |  c2   
----+-------
    | three
  2 | two
  1 | one
(3 rows)

In DuckDB, the null value is always at the last

D select * from (values (1, 'one'), (2, 'two'), (null, 'three')) t(c1, c2) order by c1 asc;
┌───────┬─────────┐
│  c1   │   c2    │
│ int32 │ varchar │
├───────┼─────────┤
│     1 │ one     │
│     2 │ two     │
│  NULL │ three   │
└───────┴─────────┘
D select * from (values (1, 'one'), (2, 'two'), (null, 'three')) t(c1, c2) order by c1 desc;
┌───────┬─────────┐
│  c1   │   c2    │
│ int32 │ varchar │
├───────┼─────────┤
│     2 │ two     │
│     1 │ one     │
│  NULL │ three   │
└───────┴─────────┘

As a composable database engine, I think it make sense to provide a config to customize the default behavior of null ordering.

What changes are included in this PR?

This PR introduces a new parser option default_null_ordering:

        /// Specifies the default null ordering for query results. There are 4 options:
        /// - `nulls_max`: Nulls appear last in ascending order.
        /// - `nulls_min`: Nulls appear first in ascending order.
        /// - `nulls_first`: Nulls always be first in any order.
        /// - `nulls_last`: Nulls always be last in any order.
        ///
        /// By default, `nulls_max` is used to follow Postgres's behavior.
        /// postgres rule: <https://www.postgresql.org/docs/current/queries-order.html>
        pub default_null_ordering: String, default = "nulls_max".to_string()

There are 4 cases for the null ordering. nulls_max is used by default to follow the Postgres's behavior.

Are these changes tested?

Are there any user-facing changes?

add a new config.

goldmedal and others added 3 commits July 29, 2025 19:07
* add default_null_ordering config

* add test for different config

* Update datafusion/sql/src/planner.rs

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* update doc

* fix sqllogictest

---------

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
@github-actions github-actions bot added documentation Improvements or additions to documentation sql SQL Planner core Core DataFusion crate sqllogictest SQL Logic Tests (.slt) common Related to common crate labels Jul 29, 2025
Copy link
Contributor

@alamb alamb left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you @goldmedal -- this looks great to me. I had some small suggestions but nothing I think is required

@alamb alamb changed the title Allow to customize the default null ordering Add sql_parser.default_null_ordering config option to customize the default null ordering Jul 29, 2025
@goldmedal goldmedal merged commit d376a32 into apache:main Jul 31, 2025
28 checks passed
@goldmedal goldmedal deleted the feat/custom-null-ordering branch July 31, 2025 12:55
@goldmedal
Copy link
Contributor Author

Thanks @alamb for reviewing 👍

Standing-Man pushed a commit to Standing-Man/datafusion that referenced this pull request Aug 4, 2025
… default null ordering (apache#16963)

* Allow to set up the default null ordering (apache#3)

* add default_null_ordering config

* add test for different config

* Update datafusion/sql/src/planner.rs

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* update doc

* fix sqllogictest

---------

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* rename config and fix test

* update doc

* fix default

* fix doc

* fix sqllogictests

* address comments

---------

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
goldmedal added a commit to Canner/datafusion that referenced this pull request Aug 19, 2025
… default null ordering (apache#16963)

* Allow to set up the default null ordering (#3)

* add default_null_ordering config

* add test for different config

* Update datafusion/sql/src/planner.rs

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* update doc

* fix sqllogictest

---------

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* rename config and fix test

* update doc

* fix default

* fix doc

* fix sqllogictests

* address comments

---------

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

common Related to common crate core Core DataFusion crate documentation Improvements or additions to documentation sql SQL Planner sqllogictest SQL Logic Tests (.slt)

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants