Optimize enhancements

[wjones127]

Description

This is an umbrella issue for a variety of improvements we could make to optimize:

• Support out-of-core Z-order using DataFusion #1460
• Expose Z-order in Python #1442
• Expose compression options in optimize and write_deltalake #1465
• Add support for Sort-based optimize
• Use Parquet sort metadata to make sort-based optimize skip already sorted files

Use Case

Related Issue(s)

[mjducut-oe]

Hello @wjones127, just want to check if the compression option is already exposed in python write_deltalake? I'm not sure I have missed it somewhere or it's not yet ready. We're really hoping we could do a snappy on our outputs but it might not be available yet. In any case, could you recommend a workaround for this?

Thank you so much!

[wjones127]

the compression option is already exposed in python write_deltalake?

Nope, no one has made a PR for that yet. I don't think there's any workaround at the moment. Shouldn't be too hard to implement though if someone is motivated to contribute it.

[ion-elgreco]

the compression option is already exposed in python write_deltalake?

Nope, no one has made a PR for that yet. I don't think there's any workaround at the moment. Shouldn't be too hard to implement though if someone is motivated to contribute it.

I can maybe centralize this in a function after update, merge PRs. I also partially exposed the writer properties in those. Would be good to make this available across all the APIs

[mjducut-oe]

@ion-elgreco, sounds great! This will really help us especially with storage utilization on writes. Will keep watch and thank you! 🙏👍