Release DataFusion 43.0.0

[alamb]

Is your feature request related to a problem or challenge?

Tracking ticket for next release, also a place to track desired inclusions

Last release was https://crates.io/crates/datafusion/42.0.0 September 17th, 2024 so next major release would be around October 20, 2024

Prior release tickets:

• 42.0.0 Release DataFusion 42.0.0 #11902

Desired Items that would be good to get into this release:

• (Andrew's goal) Enable reading StringView by default from Parquet (schema_force_string_view) by default #11682
• (Andrew's goal) Enable parquet filter pushdown by default #3463

Items to fix before release

• Error with type coercion with CREATE TABLE AS SELECT ... inserting VALUES  #13124
• Option to disable Physical input schema should be the same as the one converted from logical schema error #13065

[andygrove]

I'm keen to start the 43.0.0 release process as soon as we have upgraded to arrow-rs 53.1.0 since it will unblock apache/datafusion-ray#10

[alamb]

I'm keen to start the 43.0.0 release process as soon as we have upgraded to arrow-rs 53.1.0 since it will unblock apache/datafusion-ray#10

The arrow upgrade is all ready here: #12724

Since the arrow upgrade is a minor version, you won't have to wait for the datafusion upgrade to get the latest version of arrow-rs

[andygrove]

It has been 28 days since the last release to crates.io, so we should start planning this release.

@alamb would you still like to wait for the items mentioned in the description?

[alamb]

Thanks @andygrove -- I think we should start planning. Here are some items I think we should include:

Required

• Bug fixes for metadata mismatches (@wiedld has a few more she will file over the next day or so): [EPIC] Schema metadata handling / bugs #12733

I would love to get these as well

• Implement special Groups for StringViews #12771 (should merge tomorrow)
• Performance: Add "read strings as binary" option for parquet #12788 (needs an arrow-rs upgrade which isn't scheduled until Nov: Release arrow-rs / parquet minor version 53.2.0 (~November~ October 2024) arrow-rs#6341 -- though I could make a release sooner if you are willing to hold the DF release until next week)

Other than #12788 we are on track I think to be ready to release in the next day or two

[andygrove]

Thanks @alamb. I am in no rush for the release myself.

A "nice to have" for me is #12969 if it is ready in time, but it should not block the release (and could potentially be back-ported to a 42.x.x release)

[alamb]

BTW I am going to accelerate the timeline to release arrow 53.2.0 so we can potentially include it in the next datafusion release: apache/arrow-rs#6341

I hope to make the RC today

[alamb]

This came up on the sync call today

I would like consider doing this item before we release (as this check has caused significant pain downstream in InfluxDB 3.0 as well as delta-rs (see links on ticket):

• Option to disable Physical input schema should be the same as the one converted from logical schema error #13065

If we did that I view everything else as nice to have, including

• [EPIC] Schema metadata handling / bugs #12733 (there are still some additional outstanding issues)
• Enable reading StringView by default from Parquet (schema_force_string_view) by default #11682 as much as I would like to turn this on by default, I think it would be a bad idea to turn it on by default right before a release. It would be better to merge it to main and give it some bake time

[alamb]

Here is the PR to use StringView when reading from Parquet files: #13101

[findepi]

There is a regression for CREATE TABLE #12864 (comment)
This should be resolved before release (or judged OK to break).

[alamb]

There is a regression for CREATE TABLE #12864 (comment) This should be resolved before release (or judged OK to break).

Filed I have filed #13124 to track the

[alamb]

I am getting worried about the number of changes that are accumulating unreleased.

Here is my proposal:

• Add the "turn off the schema check" PR: Option to disable Physical input schema should be the same as the one converted from logical schema error #13065 (any volunteers to help with this?)
• Backport/make another 42.2 release with just that fix
• Create the 43 RC

[alamb]

@andygrove what are your thoughts timingwise for 43? I feel like we are accumulating substantial API changes and it might be nice to release sooner rather than later

[timsaucer]

Do you need an assist with any portion of this?

[alamb]

Thank yoU @timsaucer ❤️

I am not sure -- I haven't heard from @andygrove -- perhaps we should ask him in Slack?

If he is willing, I think it would be great if you wanted to create the PR to update the versions and readme as documented in https://github.com/apache/datafusion/blob/main/dev/release/README.md#change-log

[andygrove]

I am here (now). I can start the RC process this evening if we are not waiting on anything else

[timsaucer]

I would like to get this in, but I have yet to find a reviewer: #13183

[alamb]

The release has been approved and published to crates.io

Minor follow on:

• minor: include datafusion-ffi in publishing instructions #13333

Tracking 44.0.0:

• Release DataFusion 44.0.0 #13334

[alamb]

Thanks again @andygrove