[Storage System] Support for Oracle Cloud (Object Storage)

[rohitagl91]

Hi,

we need support for oracle cloud storage system (Object Storage), to leverage Delta Lake capabilities on Oracle Cloud.

Thanks,
Rohit

[mukulmurthy]

Here are the requirements for the underlying storage system in order to maintain ACID guarantees (from
https://github.com/delta-io/delta#requirements-for-underlying-storage-systems)

Atomic visibility: There must be a way for a file to be visible in its entirety or not visible at all.
Mutual exclusion: Only one writer must be able to create (or rename) a file at the final destination.
Consistent listing: Once a file has been written in a directory, all future listings for that directory must return that file.

I don't know if Oracle cloud storage has those guarantees out of the box; if it doesn't, we'll need to write a LogStore implementation (see #39 and #40). We've had people ask a few times for Google Cloud support though, so I'm guessing that's the next cloud we'll work on supporting.

[ksranga]

Hi @mukulmurthy - OCI object store does not support directory semantics, unlike S3, if I am right. Is this a roadblock or can it be solved with a custom implementation of LogStore?

[mukulmurthy]

Hey @ksranga ,

Delta needs consistent listing in order to figure out the latest version of the table. S3 is an object storage system too, not a file system, and still supports abstractions to list a directory. Does Oracle support something similar?

[pranavanand]

Hi @ksranga ,

have you had a chance to look into this?

[ksranga]

Hi @pranavanand - yes I did and have a working prototype. Hope to have a PR out soon to resolve this issue.

[vibhaska]

I have raised a PR to address request to support Oracle Cloud Infrastructure Object Store as Delta Storage.

Kindly review. Please suggest if any changes or details are required.

[zsxwing]

We have added the support in #301