DataFusion 52 release post #135

alamb · 2026-01-08T22:47:33Z

This is a draft of the DataFusion 52 release post

See rendered preview: https://datafusion.staged.apache.org/blog/2026/01/08/datafusion-52.0.0/

This was initially created using coded. Commands below

Details

We are going to write a blog post for the DataFusion 52.0.0 release

We need to cover the major features in this release. If you are unsure of any content, please leave a "TODO" note in the text and we can fill it in
later.

Please start with a copy of the previous post as a starting point: content/blog/2025-11-25-datafusion-51.0.0.md and update as needed.

The changelog is here: https://github.com/xudong963/arrow-datafusion/blob/update_version/dev/changelog/52.0.0.md

The list of major features can be found in apache/datafusion#18566 under the section "Features to mention in the blog
(if they make it)". Only include the ones that made it into the release, with a checkmark.

Please

write a blog post
leave a section for performance chart which we can fill in later
include a section for each major feature, summarizing what it is and why it is important, and the related PRs. Please try to include a diagram or
example where possible.

content/blog/2026-01-08-datafusion-52.0.0.md

Co-authored-by: Matt Butrovich <mbutrovich@users.noreply.github.com>

alamb · 2026-01-20T17:25:57Z

Thanks @mbutrovich -- any additional context / suggestions you have on the sort mergejoin improvement would be most appreciated

alamb · 2026-01-20T17:26:20Z

(this is on my list, but I am struggling to find time to finish it -- hopefully after CIDR / thursday)

…sion-site into site/datafusion_52

alamb · 2026-01-23T00:50:59Z

martin-g · 2026-01-23T06:38:05Z

content/blog/2026-01-08-datafusion-52.0.0.md

+---
+layout: post
+title: Apache DataFusion 52.0.0 Released
+date: 2026-01-08


According to:

https://lists.apache.org/thread/gt29yg6wxzx82s87drwq1xb06yhs16y6

https://crates.io/crates/datafusion/52.0.0

Suggested change

date: 2026-01-08

date: 2026-01-12

Thanks -- I think in the past we have dated the blog posts based on when the post was released rather than when the software was 🤔

content/blog/2026-01-08-datafusion-52.0.0.md

martin-g · 2026-01-23T06:39:02Z

content/blog/2026-01-08-datafusion-52.0.0.md

+TODO: confirm the release date for 52.0.0 and update the front matter if needed.
+
+[DataFusion 52.0.0]: https://crates.io/crates/datafusion/52.0.0
+[DataFusion 51.0.0]: https://datafusion.apache.org/blog/2025/11/25/datafusion-51.0.0/


geoffreyclaude · 2026-01-23T07:05:21Z

content/blog/2026-01-08-datafusion-52.0.0.md

+explained in the [Extending SQL in DataFusion Blog]. With this new API, you can
+customize DataFusion to support almost any SQL syntax, such as the following
+(which are not supported by default):


I feel that this is slightly misleading: it reads as if the RelationPlanner is what now allows extending expressions and types (and relations). Maybe something like:

In addition to the existing expression and types extension points, this new API now allows extending FROM clauses, leading DataFusion to support almost any SQL syntax, such as the following (which are not supported by default):

But reworded to be less of a run-on sentence...

This is a great call out. I reworded it in 615affd to this:

DataFusion now has an API for extending the SQL planner for relations, as
explained in the [Extending SQL in DataFusion Blog]. In addition to the existing
expression and types extension points, this new API now allows extending FROM
clauses. Using these APIs it is straightforward to provide SQL support for
almost any dialect, including vendor-specific syntax. Example use cases include:

pepijnve · 2026-01-23T12:09:07Z

content/blog/2026-01-08-datafusion-52.0.0.md

+[Apache Comet]: https://datafusion.apache.org/comet/
+[mbutrovich]: https://github.com/mbutrovich
+
+### Rewritten merge join


This section title looks very similar to the previous one. The start of the first sentence is also identical. Maybe a title that differentiates this section more from the previous one (e.g. "Optimised Output Handling of Merge Join") would be clearer.

Thank you -- I think this was the result of a bad merge conflict resolution (I had both the revised paragraph and the original). I removed the section in 1345bfb

content/blog/2026-01-08-datafusion-52.0.0.md

nuno-faria

I was looking at the changelog and this PR caught my attention: apache/datafusion#18644. Maybe it could be worth a mention as well.

nuno-faria · 2026-01-23T21:17:58Z

content/blog/2026-01-08-datafusion-52.0.0.md

+
+This release also includes several additional caching improvements.
+
+A new statistics cache for Parquet Metadata avoids repeatedly (re)calculating


nit: maybe "Parquet Metadata" -> "File Metadata"? Since there is also a separate cache for the Parquet metadata itself.

Good call -- Fixed in e9308d4

Co-authored-by: Martin Grigorov <martin-g@users.noreply.github.com>

…sion-site into site/datafusion_52

Co-authored-by: Adrian Garcia Badaracco <1755071+adriangb@users.noreply.github.com>

alamb · 2026-01-24T12:54:37Z

I was looking at the changelog and this PR caught my attention: apache/datafusion#18644. Maybe it could be worth a mention as well.

@nuno-faria another great call. I have added a section in 4e24b1f

Perhaps @2010YOUY01 can verify if I got the summary correct

xudong963 · 2026-01-24T14:01:09Z

content/blog/2026-01-08-datafusion-52.0.0.md

+[Variant shredding]: https://github.com/apache/datafusion/issues/16116
+[PhysicalExprAdapter]: https://docs.rs/datafusion/52.0.0/datafusion/physical_expr_adapter/trait.PhysicalExprAdapter.html
+
+### Sort Pushdown to Scans


content/blog/2026-01-08-datafusion-52.0.0.md

Co-authored-by: Yongting You <2010youy01@gmail.com>

…sion-site into site/datafusion_52

Initial draft (coded with codex)

68abcfb

alamb mentioned this pull request Jan 8, 2026

WIP: DataFusion 52 release post #134

Closed

updates

1686945

mbutrovich reviewed Jan 12, 2026

View reviewed changes

content/blog/2026-01-08-datafusion-52.0.0.md Outdated Show resolved Hide resolved

alamb added 3 commits January 18, 2026 21:54

Merge remote-tracking branch 'apache/main' into site/datafusion_52

63b8e12

updates

3c7dd6a

Update sql planning

2cc1f59

mbutrovich reviewed Jan 20, 2026

View reviewed changes

content/blog/2026-01-08-datafusion-52.0.0.md Outdated Show resolved Hide resolved

mbutrovich reviewed Jan 20, 2026

View reviewed changes

content/blog/2026-01-08-datafusion-52.0.0.md Outdated Show resolved Hide resolved

mbutrovich reviewed Jan 20, 2026

View reviewed changes

content/blog/2026-01-08-datafusion-52.0.0.md Outdated Show resolved Hide resolved

Apply suggestions from code review

ccc5d42

Co-authored-by: Matt Butrovich <mbutrovich@users.noreply.github.com>

alamb added 7 commits January 22, 2026 18:45

Updates

81954c5

acknowledgments

781cd62

update

790b658

updates

d63a4de

update

d38d99f

typos

b47c50d

refine

1f5b91e

alamb changed the title ~~WIP: DataFusion 52 release post~~ DataFusion 52 release post Jan 23, 2026

alamb added 2 commits January 22, 2026 19:42

Merge branch 'site/datafusion_52' of https://github.com/apache/datafu…

34cea38

…sion-site into site/datafusion_52

clean

2823de5

martin-g reviewed Jan 23, 2026

View reviewed changes

geoffreyclaude reviewed Jan 23, 2026

View reviewed changes

pepijnve reviewed Jan 23, 2026

View reviewed changes

adriangb reviewed Jan 23, 2026

View reviewed changes

content/blog/2026-01-08-datafusion-52.0.0.md Outdated Show resolved Hide resolved

nuno-faria approved these changes Jan 23, 2026

View reviewed changes

Update content/blog/2026-01-08-datafusion-52.0.0.md

1c9dadf

Co-authored-by: Martin Grigorov <martin-g@users.noreply.github.com>

alamb and others added 7 commits January 24, 2026 07:02

Clarify RelationPlanner syntax

615affd

Merge branch 'site/datafusion_52' of https://github.com/apache/datafu…

8329e6a

…sion-site into site/datafusion_52

remove extra section

1345bfb

Update content/blog/2026-01-08-datafusion-52.0.0.md

63b571c

Co-authored-by: Adrian Garcia Badaracco <1755071+adriangb@users.noreply.github.com>

Refine wording

7984011

reflow

66a9dcb

Metadata cache is general

e9308d4

alamb mentioned this pull request Jan 24, 2026

optimizer: Support dynamic filter in MIN/MAX aggregates apache/datafusion#18644

Merged

2 tasks

Add section on Min/max dynamic filters

4e24b1f

xudong963 reviewed Jan 24, 2026

View reviewed changes

zhuqi-lucas reviewed Jan 24, 2026

View reviewed changes

content/blog/2026-01-08-datafusion-52.0.0.md Outdated Show resolved Hide resolved

2010YOUY01 approved these changes Jan 25, 2026

View reviewed changes

content/blog/2026-01-08-datafusion-52.0.0.md Outdated Show resolved Hide resolved

alamb and others added 6 commits January 25, 2026 08:56

Update content/blog/2026-01-08-datafusion-52.0.0.md

d62dfab

Co-authored-by: Yongting You <2010youy01@gmail.com>

whitespace

1a77f71

Merge branch 'site/datafusion_52' of https://github.com/apache/datafu…

859d40d

…sion-site into site/datafusion_52

Improve sort pushdown description

37b12dd

capitalizaton

91f71f3

wordsmith

040b85c

zhuqi-lucas approved these changes Jan 25, 2026

View reviewed changes


		This release also includes several additional caching improvements.

		A new statistics cache for Parquet Metadata avoids repeatedly (re)calculating

DataFusion 52 release post #135

Are you sure you want to change the base?

DataFusion 52 release post #135

Uh oh!

Conversation

alamb commented Jan 8, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

alamb commented Jan 20, 2026

Uh oh!

alamb commented Jan 20, 2026

Uh oh!

alamb commented Jan 23, 2026

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

geoffreyclaude Jan 23, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

alamb Jan 24, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

nuno-faria left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

alamb commented Jan 24, 2026

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

10 participants

alamb commented Jan 8, 2026 •

edited

Loading

geoffreyclaude Jan 23, 2026 •

edited

Loading

alamb Jan 24, 2026 •

edited

Loading