Python write_deltalake: when will this be production ready?

[kk921dbg]

Description

Just wondering when python write_deltalake will be ready for production use as it says its still experimental

Use Case
We need this for use from an Azure App Service as it will save us having to deploy multiple reources

Related Issue(s)

[wjones127]

Our plan is to have it production ready for read and append workloads by end of this year.

two big things need to happen ( that are tracked in other issues):

1. Support writer protocol v2
2. Support objects stores such as S3 and Azure blob store (it only works on local file systems right now)

[vmesel]

Hey @wjones127, how are you? Any updates on the S3 writing functionality? Should I use this repo as a production tool or should I leave it and rollback to databricks?

Now I'm trying a pure AirFlow Silver layer approach, but I need to be able to save the delta table to S3. Already tried using s3fs and StringIO, but I get the following error:

     90 else:
     91     table = table_or_uri
---> 92     table_uri = table_uri = table._table.table_uri()
     94 # TODO: Pass through filesystem once it is complete
     95 # if filesystem is None:
     96 #    filesystem = pa_fs.PyFileSystem(DeltaStorageHandler(table_uri))
     98 if table:  # already exists

AttributeError: '_io.TextIOWrapper' object has no attribute '_table'

[wjones127]

Should I use this repo as a production tool or should I leave it and rollback to databricks?

We've implemented support for S3 and other object stores. I'm still working on full support for writer protocol v2: #834

Once that's complete we'll need some real world user testing from folks like you before I'd call it fully production ready :)

We don't support the s3fs package in the current release, but will in our next release. You won't be able to pass a StringIO object though; instead, you will wrap the s3fs filesystem you configure in a PyArrow filesystem with FSSpecHandler.

Have you tried configuring S3 through the settings show here (without s3fs)? That should work in the current release.
https://delta-io.github.io/delta-rs/python/usage.html#loading-a-delta-table

[tgilon]

Hi @wjones127 , Thanks for the amazing job ! Any news ? I'm also wondering to use this repo to save production data in delta table on S3.

[wjones127]

Any news ? I'm also wondering to use this repo to save production data in delta table on S3.

Right now I'd call it Beta-supported right now. We've implemented writer protocol v2 and our support for authentication is pretty well tested now. What we need most now is users trying it out and reporting any issues in their systems. We have plenty of tests here, but the real world is always more complex :) I will drop the experimental label in the next release or perhaps the one after.

[madsenwattiq]

That’s great. When will we see the next Python release to PyPI? That release should, given recent merges, incorporate a key bug fix (#685) for anybody doing time series data. I’m going to start beta testing immediately, as soon as I can build containers from a PyPI package. Thank you!