Release DataFusion 16

[andygrove]

Is your feature request related to a problem or challenge? Please describe what you are trying to do.
It has been ~4 weeks since the last release. I would like to release the next version soon (aiming for Friday, Jan 6).

Describe the solution you'd like

Release tasks

• Confirm that project is still compatible with Python bindings
• Run benchmarks and confirm no major regressions

Post-release tasks

• Blog post about datafusion 16 release #4804
• Publish user guide: DataFusion 16.0.0 documentation arrow-site#301

Describe alternatives you've considered
A clear and concise description of any alternative solutions or features you've considered.

Additional context
Add any other context or screenshots about the feature request here.

[tustvold]

Confirm that project is still compatible with Python bindings

Do let me know if I've broken anything and I'll try to help out. I think I'm mostly done with the major changes, I can probably defer async catalog support until after the release

[andygrove]

@tustvold I managed to bulldoze my way through getting the Python bindings working in apache/datafusion-python#115. Please take a look if you have the time.

I have one issue to fix around window functions, and I will look at that later this week.

[alamb]

I would like to write a blog post about DataFusion 16 release. I have started collecting notes on #4804

[alamb]

FYI we found some issues with 30.0.0: apache/arrow-rs#3455

[andygrove]

The first RC seems to have an issue, so I will plan on creating a new RC once #4869 is merged.

[liukun4515]

The first RC seems to have an issue, so I will plan on creating a new RC once #4869 is merged.

merged, Thanks @ygf11

You can create the RC for 16.0.0. cc @andygrove

[DDtKey]

I was able to find one more regression against current master: #4875
And one more: #4876

[alamb]

Checking them out

[alamb]

#4875 looks serious but I could not reproduce it (more details in the ticket)

I think #4876 is expected

[DDtKey]

JFYI: as it was discussed - #4875 isn't regression, but the bug existed before.
While #4876 is partially expected.

So probably it doesn't lock release, but it's up to the datafusion team 🙂

[alamb]

I think with apache/arrow-site#301 this issue can now be closed