Adding support for Oracle Cloud Infrastructure (OCI) Object Store as Delta Storage #468

vibhaska · 2020-07-01T12:49:06Z

Adding support for Oracle Cloud Infrastructure (OCI) Object Store as Delta Storage by introducing OCILogStore.

Regarding Storage configuration page in Delta Documentation, I request following changes mentioned here.

pranavanand · 2020-07-02T20:10:54Z

Thanks for submitting this PR! We are currently coming up with a process for contributing third-party support and will let you know once that's ready.

ksranga · 2020-07-12T14:43:02Z

Can this be ported to delta 0.6.x version so its available for spark 2.x as well?

vibhaska · 2020-07-12T15:07:28Z

@ksranga Yes.

I guess only one line change is required: Remove line number 268 from LogStoreSuite.scala. If you want I can raise another PR for old branch after testing and verification.

README.md

vibhaska · 2020-11-18T06:22:34Z

Thanks for submitting this PR! We are currently coming up with a process for contributing third-party support and will let you know once that's ready.

@pranavanand can you please share any further update.

ddraj · 2020-12-22T21:34:40Z

Echoing @vibhaska's question .. Hi @pranavanand, any chance the PR can be merged? Could you please let us know if anything is pending on the PR that we should address?

tdas · 2021-04-21T23:18:56Z

Hey @ddraj

Our apologies for sitting on the PR for so long. We wanted to figure out what is the best we integrate slightly less battle-tested features into our repo, and we finally figured it out. We are currently in the process of converting our repo to a multi-module SBT project so that we can add other sub-modules (mapping to other maven release artifacts beyond delta-core). Then we can add a contribs module and then accept such new and experimental features.

Are you still interested in contributing to this feature?

ddraj · 2021-04-23T00:26:02Z

Hey @ddraj

Our apologies for sitting on the PR for so long. We wanted to figure out what is the best we integrate slightly less battle-tested features into our repo, and we finally figured it out. We are currently in the process of converting our repo to a multi-module SBT project so that we can add other sub-modules (mapping to other maven release artifacts beyond delta-core). Then we can add a contribs module and then accept such new and experimental features.

Are you still interested in contributing to this feature?

Thanks @tdas. Yes we are still interested. Please let us know if we need to do anything specific.

tdas · 2021-04-23T17:57:38Z

So the refactor to multi-module has been merged. Lets' work towards getting this in. Here are the next steps.

Please define a new module using the same pattern as the core module in the build.sbt file.
- Does the IBM support need additional maven dependencies? I am guessing not. In that case, we can make the module name generic like "delta-contribs" (that is, not specific to IBM). Other such zero-dependency contributions can go in that project as well. Otherwise, lets put it in a specifically named module like "delta-contribs-ibm"
Move the new log store code inside the <new-module-dir>/src/main/<package>/... and the test code in a separate file in <new-module-dir>/src/test/<package>/....

Let me know if you need help to get it working.

…Delta Storage by introducing OCILogStore. https://developercertificate.org/ Signed-off-by: Vivek Bhaskar <vivek.bhaskar@oracle.com>

vibhaska · 2021-04-26T17:26:45Z

@tdas I have updated the changes as suggested. Kindly check if this is what you wanted.

Couple of points where I need some guidance:

Will there be any root project, and jars from same will be published? Or will a separate jars be published from this module. I am not clear how.
I am also not clear why CI failed. Is this due to multi module nature?

tdas · 2021-05-06T02:54:53Z

These are good questions. My apologies for the delay, we have been trying to line up a few more features like a new, more stable LogStore API that LogStore builders like you folks can use. However, there are a few more pieces that we still need to get in for the public API to be easily usable. I dont want you folks to block on those. So we can continue working using the existing LogStore API, which we will continue supporting for at least the 1.x releases. Later we can rewrite the OCILogStore to use the new API, we are not going to worry about the 1.0 release we are targeting in a couple of weeks.

Now back to the question. We will be releasing a separate Maven artifact for each subproject. The refactoring you have done in the sbt looks good at a high-level. I will leave more detailed comments on them soon. I will also take a look at why the tests failed.

tdas · 2021-05-06T03:00:31Z

Somehow the sbt download failed. Could be a random error. Can you run the tests once again. An easy to rerun tests is to simply push an empty commit to this branch.

tdas · 2021-05-06T03:10:58Z

build.sbt

+    // mimaSettings,
+    unidocSettings,
+    releaseSettings,
+    libraryDependencies ++= Seq(


do we need to add these dependencies? I think this project automatically gets the dependencies from the core module

tdas · 2021-05-06T03:11:05Z

build.sbt

+    commonSettings,
+    scalaStyleSettings,
+    // mimaSettings,
+    unidocSettings,


i dont think we need unidoc. there is no public in this project yet.

tdas · 2021-05-06T22:35:32Z

The community has already created the contrib submodule! So all you have to do merge with the master and add your LogStore to the existing directories. No need to fix anything in the build.sbt!

tdas · 2021-05-07T17:25:51Z

@vibhaska @ddraj if you have time to update the PR, please do so quickly as we preparing to release 1.0 soon. Alternatively, if you are okay, I can do the slight refactoring as well (you still get commit authorship).

ddraj · 2021-05-07T17:44:23Z

@tdas coincidentally @vibhaska is on vacation this week and the next. So we are not able to update the PR :( I'll happily take your offer for the slight updates. Thanks a lot @tdas

tdas · 2021-05-07T23:03:53Z

@ddraj all right! I will do it. Do you mind if I change the name OracleCILogstore to make it more obvious which cloud it is? Similar to our naming of IBMCOSLogStore and AzureLogStore.

vibhaska · 2021-05-08T12:58:31Z

Sorry for the delayed response. I have checked-in refactoring changes. I have also updated the file name, as suggested.

@tdas Thanks for offering help with the changes. Kindly review and please suggest if any more changes are required.

@ddraj Thanks for your support.

tdas · 2021-05-10T18:35:05Z

@vibhaska thank you for the update. One minor correction I suggest is to rename OracleOCI to just OracleCI because the O in OCI is redundant with Oracle. I can do rename super easily if that's okay with you.

ddraj · 2021-05-10T21:09:23Z

@tdas any chance we can name it as OracleCloudLogStore. I think that would fully convey the purpose of this implementation.

tdas · 2021-05-10T23:10:38Z

Sure. As long as there won't be any other variation of LogStore needed on the Oracle Cloud, maybe for any other storage product in Oracle Cloud.

ddraj · 2021-05-11T02:54:08Z

Don’t think that’s in the offing anytime soon @tdas Lets rename it to OracleCloudLogStore

tdas · 2021-05-11T04:23:17Z

Alright!

vibhaska · 2021-05-11T18:15:22Z

@tdas @ddraj Updated the file name, as suggested.

vibhaska · 2021-05-12T20:24:32Z

@tdas Kindly suggest if any further changes are required.

ddraj · 2021-05-13T18:58:42Z

@tdas echoing what @vibhaska asked. Is there anything you need from us?

ddraj · 2021-05-18T03:00:21Z

Hi @tdas gentle ping :)

vibhaska mentioned this pull request Jul 1, 2020

[Storage System] Support for Oracle Cloud (Object Storage) #301

Closed

mikedillion reviewed Jul 12, 2020

View reviewed changes

README.md Outdated Show resolved Hide resolved

ranu010101 mentioned this pull request Dec 1, 2020

Added GcsLogStore implementation ranu010101/delta#1

Merged

Vivek Bhaskar added 2 commits April 26, 2021 13:10

Adding support for Oracle Cloud Infrastructure (OCI) Object Store as …

0da6497

…Delta Storage by introducing OCILogStore. https://developercertificate.org/ Signed-off-by: Vivek Bhaskar <vivek.bhaskar@oracle.com>

Adding support for Oracle Cloud Infrastructure (OCI) Object Store as …

58854a3

…Delta Storage by introducing OCILogStore. https://developercertificate.org/ Signed-off-by: Vivek Bhaskar <vivek.bhaskar@oracle.com>

vibhaska force-pushed the oci_pr_1 branch from d1d0184 to 58854a3 Compare April 26, 2021 07:41

Addressed review comments

f502b7e

tdas reviewed May 6, 2021

View reviewed changes

Vivek Bhaskar added 2 commits May 8, 2021 16:33

Merge branch 'master' into oci_pr_1

68ac0da

Merge branch 'master' into oci_pr_1

e7d78c4

Vivek Bhaskar added 2 commits May 11, 2021 21:59

Merge branch 'master' into oci_pr_1

f23966b

Renamed to OracleOCILogStore

f4c91fd

tdas closed this in cb46fd1 May 18, 2021

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Adding support for Oracle Cloud Infrastructure (OCI) Object Store as Delta Storage #468

Adding support for Oracle Cloud Infrastructure (OCI) Object Store as Delta Storage #468

vibhaska commented Jul 1, 2020

pranavanand commented Jul 2, 2020

ksranga commented Jul 12, 2020

vibhaska commented Jul 12, 2020

vibhaska commented Nov 18, 2020

ddraj commented Dec 22, 2020

tdas commented Apr 21, 2021

ddraj commented Apr 23, 2021

tdas commented Apr 23, 2021

vibhaska commented Apr 26, 2021 •

edited

Loading

tdas commented May 6, 2021 •

edited

Loading

tdas commented May 6, 2021

tdas May 6, 2021

tdas May 6, 2021

tdas commented May 6, 2021

tdas commented May 7, 2021

ddraj commented May 7, 2021

tdas commented May 7, 2021

vibhaska commented May 8, 2021

tdas commented May 10, 2021

ddraj commented May 10, 2021

tdas commented May 10, 2021

ddraj commented May 11, 2021

tdas commented May 11, 2021

vibhaska commented May 11, 2021

vibhaska commented May 12, 2021

ddraj commented May 13, 2021

ddraj commented May 18, 2021

Adding support for Oracle Cloud Infrastructure (OCI) Object Store as Delta Storage #468

Adding support for Oracle Cloud Infrastructure (OCI) Object Store as Delta Storage #468

Conversation

vibhaska commented Jul 1, 2020

pranavanand commented Jul 2, 2020

ksranga commented Jul 12, 2020

vibhaska commented Jul 12, 2020

vibhaska commented Nov 18, 2020

ddraj commented Dec 22, 2020

tdas commented Apr 21, 2021

ddraj commented Apr 23, 2021

tdas commented Apr 23, 2021

vibhaska commented Apr 26, 2021 • edited Loading

tdas commented May 6, 2021 • edited Loading

tdas commented May 6, 2021

tdas May 6, 2021

Choose a reason for hiding this comment

tdas May 6, 2021

Choose a reason for hiding this comment

tdas commented May 6, 2021

tdas commented May 7, 2021

ddraj commented May 7, 2021

tdas commented May 7, 2021

vibhaska commented May 8, 2021

tdas commented May 10, 2021

ddraj commented May 10, 2021

tdas commented May 10, 2021

ddraj commented May 11, 2021

tdas commented May 11, 2021

vibhaska commented May 11, 2021

vibhaska commented May 12, 2021

ddraj commented May 13, 2021

ddraj commented May 18, 2021

vibhaska commented Apr 26, 2021 •

edited

Loading

tdas commented May 6, 2021 •

edited

Loading