Blog post for release 40.0.0 #6

alamb · 2024-07-09T12:23:54Z

phillipleblanc · 2024-07-09T13:36:59Z

_posts/2024-07-09-datafusion-40.0.0.md

+BTW it would be great if someone made a demo showing how to do this (see https://github.com/apache/datafusion/issues/9326 )
+
+
+## Parquet indexing / low latency queries


We used the Parquet indexing feature (ParquetAccessPlan) to add efficient support for reading from DeltaLake tables and handling deletion vectors. I think its a cool use-case for the indexing feature and perhaps worth calling out as an example of a real-world use-case.

_posts/2024-07-09-datafusion-40.0.0.md

Co-authored-by: Phillip LeBlanc <phillip@leblanc.tech>

…into alamb/40.0.0_blog

alamb · 2024-07-10T11:10:26Z

Thanks @phillipleblanc -- I added your suggestions. I hope to clean this up more over the next few days

alamb

I took a pass at the community growth section and performance and started on features / looking ahead.

I'll try and take another pass over the features section tomorrow

_posts/2024-07-09-datafusion-40.0.0.md

findepi · 2024-07-19T12:40:53Z

Is there a way to deploy a rendered preview?

_posts/2024-07-09-datafusion-40.0.0.md

findepi · 2024-07-19T12:41:37Z

_posts/2024-07-09-datafusion-40.0.0.md

+<!-- todo update this intro --> 
+[Apache Arrow DataFusion] is an extensible query engine, written in [Rust], that
+uses [Apache Arrow] as its in-memory format. DataFusion is used by developers to
+create new, fast data centric systems such as databases, dataframe libraries,
+machine learning and streaming applications. While [DataFusion’s primary design
+goal] is to accelerate creating other data centric systems, it has a
+reasonable experience directly out of the box as a [dataframe library] and
+[command line SQL tool].


reminder about the todo (this paragraph isn't to be reviewed yet, right?)

findepi · 2024-07-19T12:44:56Z

_posts/2024-07-09-datafusion-40.0.0.md

+group by columns which resulted in a 40% performance improvement for some
+benchmarks.


for some benchmarks

can this be a little more explicit what type of workfloads are expected to benefit?

_posts/2024-07-09-datafusion-40.0.0.md

findepi · 2024-07-19T12:52:46Z

_posts/2024-07-09-datafusion-40.0.0.md

+
+## SQL Unparser
+
+DataFusion now supports converting `Expr`s and `LogicalPlan`s BACK to SQL text.


a side question - is there assumption that every plan can be converted to SQL?
what happens with operations that are result of optimization of original SQL, like partial aggregations?

No -- I think as you correctly point out there are some LogicalPlan which can not be converted to SQL

PartialAggregation only shows up in the ExecutionPlans to my knowledge, but you could easly construct logical plans which can't be conveted to SQL (like doing multiple logical aggregations for example)

findepi · 2024-07-19T12:54:53Z

_posts/2024-07-09-datafusion-40.0.0.md

+of doing this efficiently is to minimize the number of requests made to an
+object store by caching metadata and skipping over parts of the file that are


Do we also join requests for adjacent or near-by pages (eg sibling columns a,b or siblings a,c where b is small)?

The way it works is that the initial pages are fetch in a single object store request with multiple RANGEs

_posts/2024-07-09-datafusion-40.0.0.md

findepi · 2024-07-19T12:55:48Z

_posts/2024-07-09-datafusion-40.0.0.md

+* Faster and easier to use [TreeNode API] for traversing and manipulating plans and expressions.
+* All functions now use the same [Scalar User Defined Function API], making it easier to customize
+  DataFusion's behavior without sacrificing performance. See [ticket] for more details.
+* DataFusion can now be compiled to [WASM]. 


_posts/2024-07-09-datafusion-40.0.0.md

alamb · 2024-07-22T17:20:04Z

Thank you everyone for your reviews and comments ❤️

I plan to take one final proofreading / wordsmithing pass tomorrow 2024-07-23 and publish on 2024-07-24 unless anyone else would like time to review

milenkovicm · 2024-07-22T17:25:00Z

_posts/2024-07-23-datafusion-40.0.0.md

+
+BTW it would be great if [someone made a demo] showing how easy this is to do 🎣. 
+
+[someone made a demo]: https://github.com/apache/datafusion/issues/9326 


Thanks @alamb !
One question, would it make sense to link example https://github.com/apache/datafusion/blob/main/datafusion-examples/examples/function_factory.rs ?

Oh, sweet, yes it would! I think I forgot about that one. Added in 79e8f16

Omega359 · 2024-07-22T19:01:00Z

_posts/2024-07-23-datafusion-40.0.0.md

+[apache arrow]: https://arrow.apache.org
+[rust]: https://www.rust-lang.org/
+
+DataFusion's core thesis is that as a community together, we can build much more


DataFusion's core thesis is that as a community, together we can build much more

in 6c0223f

Throne3d

(A couple of small typos)

It's awesome to see all this progress captured in a single place!

_posts/2024-07-23-datafusion-40.0.0.md

Throne3d · 2024-07-23T03:17:40Z

_posts/2024-07-23-datafusion-40.0.0.md

+
+[discussing what we will work on in the next six months]: https://github.com/apache/datafusion/issues/11442
+[aggregating "high cardinality"]: https://github.com/apache/arrow-datafusion/issues/7000
+[Improved statistics handling]: https://github.com/apache/arrow-datafusion/issues/8227


I don't think this is referenced

_posts/2024-07-23-datafusion-40.0.0.md

Co-authored-by: Phillip LeBlanc <phillip@leblanc.tech> Co-authored-by: Edward Jones <throne3d@gmail.com>

…into alamb/40.0.0_blog

alamb · 2024-07-23T15:50:13Z

Thanks everyone for the comments. I plan to publish this tomorrow

Is there any chance one of the committers could approve this PR so I can do so? @andygrove maybe?

mustafasrepo

Thanks for this post

lewiszlw · 2024-07-24T06:37:40Z

_posts/2024-07-24-datafusion-40.0.0.md

+1. DataFusion became a top level Apache Software Foundation project (read the
+   [press release] and [blog post]).
+2. We added several PMC members and new
+   committers: [@comphead], [@mustafasrepo], [@ozankabak] joined the PMC,


@waynexia also joined the PMC.

Thank you -- yes indeed you are correct and I double checked it was only in June https://lists.apache.org/thread/h58vgtb0nm7kf25o8bhjns47nkdt5nf0

For some reason I thought he had been part of the PMC longer``

alamb · 2024-07-24T11:06:48Z

The blog post is live: https://datafusion.apache.org/blog/2024/07/24/datafusion-40.0.0/ 🎉

I also made a few PRs with instructions on how to publish and work with this site that are looking for review (and approval from a committer)

Also there is a change here for fixing dates: #4

Blog post for release 40.0.0

8e38da3

alamb mentioned this pull request Jul 9, 2024

Blog post with DataFusion Jan - June 2024 apache/datafusion#9602

Closed

alamb added 2 commits July 9, 2024 08:38

add some more contents / links

d2b5112

more

e37f78b

phillipleblanc reviewed Jul 9, 2024

View reviewed changes

_posts/2024-07-09-datafusion-40.0.0.md Outdated Show resolved Hide resolved

phillipleblanc reviewed Jul 9, 2024

View reviewed changes

_posts/2024-07-09-datafusion-40.0.0.md Outdated Show resolved Hide resolved

alamb and others added 4 commits July 10, 2024 07:09

Update _posts/2024-07-09-datafusion-40.0.0.md

ed02820

Co-authored-by: Phillip LeBlanc <phillip@leblanc.tech>

Update _posts/2024-07-09-datafusion-40.0.0.md

114f983

Co-authored-by: Phillip LeBlanc <phillip@leblanc.tech>

add details of parquet indexing

5867e65

Merge branch 'alamb/40.0.0_blog' of github.com:alamb/datafusion-site …

9b2a29c

…into alamb/40.0.0_blog

alamb added 4 commits July 12, 2024 14:48

SQL unparser

7c4a9e5

Update community growth section

359c268

Hone

71757bd

Add planning time improvements

7feccdb

alamb commented Jul 18, 2024

View reviewed changes

alamb added 2 commits July 19, 2024 07:38

Flesh out more

22ee0e7

More

bd30952

findepi reviewed Jul 19, 2024

View reviewed changes

_posts/2024-07-09-datafusion-40.0.0.md Outdated Show resolved Hide resolved

mustafasrepo reviewed Jul 19, 2024

View reviewed changes

_posts/2024-07-09-datafusion-40.0.0.md Outdated Show resolved Hide resolved

findepi reviewed Jul 19, 2024

View reviewed changes

phillipleblanc reviewed Jul 19, 2024

View reviewed changes

_posts/2024-07-09-datafusion-40.0.0.md Outdated Show resolved Hide resolved

andygrove reviewed Jul 19, 2024

View reviewed changes

_posts/2024-07-09-datafusion-40.0.0.md Outdated Show resolved Hide resolved

andygrove reviewed Jul 19, 2024

View reviewed changes

_posts/2024-07-09-datafusion-40.0.0.md Outdated Show resolved Hide resolved

andygrove reviewed Jul 19, 2024

View reviewed changes

_posts/2024-07-09-datafusion-40.0.0.md Outdated Show resolved Hide resolved

andygrove reviewed Jul 19, 2024

View reviewed changes

_posts/2024-07-09-datafusion-40.0.0.md Outdated Show resolved Hide resolved

andygrove reviewed Jul 19, 2024

View reviewed changes

_posts/2024-07-09-datafusion-40.0.0.md Outdated Show resolved Hide resolved

alamb added 6 commits July 22, 2024 12:47

improve performance section

46ed5d7

hone

09e968c

hone

4cee7b9

complete first pass

0163c36

fix heading

fc900b0

aspirationally set publish date to07-23

6d3678a

alamb marked this pull request as ready for review July 22, 2024 17:11

add github names

9932ba8

milenkovicm reviewed Jul 22, 2024

View reviewed changes

Add functon factory example link

79e8f16

Omega359 reviewed Jul 22, 2024

View reviewed changes

Throne3d reviewed Jul 23, 2024

View reviewed changes

phillipleblanc approved these changes Jul 23, 2024

View reviewed changes

_posts/2024-07-23-datafusion-40.0.0.md Outdated Show resolved Hide resolved

phillipleblanc approved these changes Jul 23, 2024

View reviewed changes

_posts/2024-07-23-datafusion-40.0.0.md Outdated Show resolved Hide resolved

alamb and others added 6 commits July 23, 2024 11:21

fix comma

6c0223f

Apply suggestions from code review

a711708

Co-authored-by: Phillip LeBlanc <phillip@leblanc.tech> Co-authored-by: Edward Jones <throne3d@gmail.com>

Merge branch 'alamb/40.0.0_blog' of github.com:alamb/datafusion-site …

7b408c3

…into alamb/40.0.0_blog

wordsmith, fix heading sizes

cf84326

more wordsmithing

ec45edb

Update date to publish date

f3b420e

mustafasrepo approved these changes Jul 24, 2024

View reviewed changes

lewiszlw reviewed Jul 24, 2024

View reviewed changes

Include waynexia on the list of people joining the PMC

00ba35b

alamb merged commit ab1d356 into apache:main Jul 24, 2024

alamb deleted the alamb/40.0.0_blog branch July 24, 2024 10:23

This was referenced Jul 24, 2024

Publish DataFusion 40.0.0 post #10

Merged

Minor: add publishing instructions #11

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Blog post for release 40.0.0 #6

Blog post for release 40.0.0 #6

alamb commented Jul 9, 2024

phillipleblanc Jul 9, 2024

alamb commented Jul 10, 2024

alamb left a comment

findepi commented Jul 19, 2024

findepi Jul 19, 2024

findepi Jul 19, 2024

findepi Jul 19, 2024

alamb Jul 20, 2024

findepi Jul 19, 2024

alamb Jul 23, 2024

findepi Jul 19, 2024

alamb commented Jul 22, 2024

milenkovicm Jul 22, 2024

alamb Jul 22, 2024

Omega359 Jul 22, 2024

alamb Jul 23, 2024

Throne3d left a comment

Throne3d Jul 23, 2024

alamb commented Jul 23, 2024

mustafasrepo left a comment

lewiszlw Jul 24, 2024

alamb Jul 24, 2024

alamb commented Jul 24, 2024

		BTW it would be great if someone made a demo showing how to do this (see https://github.com/apache/datafusion/issues/9326 )


		## Parquet indexing / low latency queries

		group by columns which resulted in a 40% performance improvement for some
		benchmarks.


		## SQL Unparser

		DataFusion now supports converting `Expr`s and `LogicalPlan`s BACK to SQL text.

		of doing this efficiently is to minimize the number of requests made to an
		object store by caching metadata and skipping over parts of the file that are


		BTW it would be great if [someone made a demo] showing how easy this is to do 🎣.

		[someone made a demo]: https://github.com/apache/datafusion/issues/9326

Blog post for release 40.0.0 #6

Blog post for release 40.0.0 #6

Conversation

alamb commented Jul 9, 2024

Choose a reason for hiding this comment

alamb commented Jul 10, 2024

alamb left a comment

Choose a reason for hiding this comment

findepi commented Jul 19, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

alamb commented Jul 22, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Throne3d left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

alamb commented Jul 23, 2024

mustafasrepo left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

alamb commented Jul 24, 2024