DynamoDBLogStore #339

mrk-its · 2020-03-03T13:43:42Z

This PR addresses issue #41 - Support for AWS S3 (multiple clusters/drivers/JVMs)

It implements few ideas from #41 discussion:

provides generic base class BaseExternalLogStore for storing listing of commit files
in external DB. This class may be easly extended for specific DB backend
stores contents of commit in temporary file and links to it in DB's row
to be able to finish uncompleted write operation while reading
provides concrete DynamoDBLogStore implementation extending BaseExternalLogStore
implementations for other DB backends should be simple to implement
(ZooKeeper implementation is almost ready, I can create separate PR if anyone is interested)

DynamoDBLogStore requirements:

To enable DynamoDBLogStore set following spark property:
spark.delta.logStore.class=io.delta.storage.DynamoDBLogStore

Single dynamodb table is required. Default table name is 'delta_log',
it may be changed by setting spark property.

Required key schema:

parentPath: String, HASH
filename: String, RANGE

Table may be created with following aws cli command:

aws --region us-east-1 dynamodb create-table \
    --table-name delta_log \
    --attribute-definitions AttributeName=tablePath,AttributeType=S \
                            AttributeName=fileName,AttributeType=S \
    --key-schema AttributeName=tablePath,KeyType=HASH \
                AttributeName=fileName,KeyType=RANGE \
    --provisioned-throughput ReadCapacityUnits=5,WriteCapacityUnits=5

Following spark properties are recognized:

spark.delta.DynamoDBLogStore.tableName - table name (defaults to 'delta_log')
spark.delta.DynamoDBLogStore.region - AWS region (defaults to 'us-east-1')

Testing

Python integration test is included :examples/python/dynamodb_logstore.py

This solution has been also stress-tested on Amazon's EMR cluster
(mutiple test jobs writing thousands of parallel transactions to single delta table)
and no data loss has beed observed so far

databricks-cla-assistant · 2020-03-03T13:43:46Z

All committers have signed the CLA.

mrk-its · 2020-03-04T10:06:34Z

@tdas @marmbrus could you take a look?

tdas · 2020-03-05T04:09:37Z

do you have any testing framework that can be run as unit tests in the PR?

I believe that with appropriate separation of contention-handling logic and dynamodb, each of them can be tested separately. for example, contention-handling logic can be tested with a pluggable in-memory store (that is, independent of dynamodb). Then we can ensure correctnes independent of any specific KV store.

mrk-its · 2020-03-05T10:34:57Z

Thanks for taking a look!

do you have any testing framework that can be run as unit tests in the PR?

Not yet.

I believe that with appropriate separation of contention-handling logic and dynamodb, each of them can be tested separately. for example, contention-handling logic can be tested with a pluggable in-memory store (that is, independent of dynamodb). Then we can ensure correctnes independent of any specific KV store.

You are right, I thought about that. BaseExternalLogStore logic can be tested independently of any KV store by subclassing it in tests. I'll work on that.

mmazek · 2020-04-14T08:29:35Z

Hi @tdas, we implemented the recommended changes and added tests. Can you please take a look and tag appropriate reviewers?

mmazek · 2020-05-14T19:42:53Z

Hi, can we get some reviews on this PR?

matthewpick · 2020-05-28T15:54:02Z

Is it possible to migrate an existing delta table to this format? Or does it have to be created from scratch with this implementation?

mrk-its · 2020-05-28T20:08:57Z

Is it possible to migrate an existing delta table to this format? Or does it have to be created from scratch with this implementation?

It is perfectly fine to start with existing delta table and empty DynamoDB table, you don't need to do any migration.

zsxwing · 2020-05-28T23:46:10Z

Sorry for the delay. We are working on adding a contrib directory and a new project to put all future LogStore implementations like this. We would like to keep the core project small because special LogStore implementations are not needed by everyone. Will report back when the project is ready.

mmazek · 2020-05-29T10:32:09Z

Hi @zsxwing, thanks for the reply. Can you share some rough estimates on when you'll be releasing those changes? Is it 1-2 months or rather some time in Q3 or Q4? We're currently running our prod workloads on the forked delta and I'd like to plan for the switch.

zsxwing · 2020-06-03T00:07:15Z

@mmazek I would avoid changing the build script as it might break the release script and block 0.7.0. So it will likely happen in Q3.

mixam24 · 2021-01-14T18:08:14Z

Hello, do you have any update on this?
Thanks.

zsxwing · 2021-04-12T21:47:57Z

Apologies for leaving this PR open for so long. We are currently working on refactoring the project to support merging LogStore implementations in a separate module, so that we don't need to pull lots of unnecessary dependencies to the core project. In addition, we are also working on a stable public LogStore API to avoid developers building something on top of private APIs. This should be done soon as we are marking these tasks release blockers for the next release. Will ping you when they are ready. Thanks again for your contribution.

mrk-its · 2021-04-21T11:26:49Z

@zsxwing thanks for update, I'll try to rebase our work on top of #644 and maybe will be able to provide some feedback.

emanuelh-cloud · 2021-05-27T09:11:19Z

Any news regarding this PR? Is it part of the 1.0.0?
thanks

mrk-its · 2021-05-27T12:39:52Z

@emanuelh-cloud I'm going to continue work on that soon, stay tuned!

mrk-its · 2021-06-02T14:01:10Z

@zsxwing @tdas

I've moved code to contribs - could you take a look?

Regarding tests:

DynamoDBLogStore code is split onto two parts: BaseExternalLogStore class containing db-independent part (for example support for recovery from failures is there) and DynamoDBLogStore class extending BaseExternalLogStore adding DynamoDB support. For tests (ExternalLogStoreSuite) we define MemoryLogStore extending BaseExternalLogStore - so DynamoDB part itself is not tested here. But important parts are still tested, like after-failure recovery.
python script examples/python/extra/dynamodb_logstore.py may be used to stress-test DynamoDBLogStore with real DynamoDB table - it writes number of transactions concurrently and verifies number of rows written.
We also tested it with spark stress test job, running number of parallel spark applications (on separate drivers) writing concurrent transactions.
This code is battle-tested on our production for over a year on few many-TB sized datasets with concurrent writes - no issues so far.

contribs/src/main/scala/io/delta/storage/DynamoDBLogStore.scala

contribs/src/main/scala/io/delta/storage/BaseExternalLogStore.scala

build.sbt

soumilshah1995 · 2022-11-25T14:25:43Z

Any Updates #1498

Thanks
have been having issue with this 🗡️

orionmoana · 2023-10-03T22:15:14Z

@mrk-its

ZooKeeper implementation is almost ready

Do you still have access to this Zookeeper implementation? Could be interested in progressing this, as a DynamoDB alternative would be very useful. #1441

mrk-its · 2023-10-04T06:45:50Z

@mrk-its

ZooKeeper implementation is almost ready

Do you still have access to this Zookeeper implementation? Could be interested in progressing this, as a DynamoDB alternative would be very useful. #1441

Yes, I have: https://github.com/mrk-its/delta-old/blob/98c34704ca4d9d264e5fad58f88b5ab379abd9c9/src/main/scala/org/apache/spark/sql/delta/storage/ZookeeperLogStore.scala

Of course it is very old (based on delta 0.5), but adopting to current version of delta should be possible.

## Description Taking inspiration from #339, this PR adds a Commit Owner Client which uses DynamoDB as the backend. Each Delta table managed by a DynamoDB instance will have one corresponding entry in a DynamoDB table. The table schema is as follows: * tableId: String --- The unique identifier for the entry. This is a UUID. * path: String --- The fully qualified path of the table in the file system. e.g. s3://bucket/path. * acceptingCommits: Boolean --- Whether the commit owner is accepting new commits. This will only * be set to false when the table is converted from managed commits to file system commits. * tableVersion: Number --- The version of the latest commit. * tableTimestamp: Number --- The inCommitTimestamp of the latest commit. * schemaVersion: Number --- The version of the schema used to store the data. * commits: --- The list of unbackfilled commits. - version: Number --- The version of the commit. - inCommitTimestamp: Number --- The inCommitTimestamp of the commit. - fsName: String --- The name of the unbackfilled file. - fsLength: Number --- The length of the unbackfilled file. - fsTimestamp: Number --- The modification time of the unbackfilled file. For a table to be managed by DynamoDB, `registerTable` must be called for that Delta table. This will create a new entry in the db for this Delta table. Every `commit` invocation appends the UUID delta file status to the `commits` list in the table entry. `commit` is performed through a conditional write in DynamoDB. ## How was this patch tested? Added a new suite called `DynamoDBCommitOwnerClient5BackfillSuite` which uses a mock DynamoDB client. + plus manual testing against a DynamoDB instance.

mrk-its force-pushed the dynamodb-logstore branch from db3e083 to fc4fa03 Compare April 6, 2020 09:06

batas force-pushed the dynamodb-logstore branch 3 times, most recently from db3e083 to 33c1fa0 Compare April 9, 2020 14:44

mrk-its force-pushed the dynamodb-logstore branch from 33c1fa0 to b2d4980 Compare April 9, 2020 14:55

batas force-pushed the dynamodb-logstore branch from 8a8b751 to 99b4cee Compare April 14, 2020 08:07

mmazek mentioned this pull request Apr 14, 2020

[Storage System] Support for AWS S3 (multiple clusters/drivers/JVMs) #41

Closed

9 tasks

ranu010101 mentioned this pull request Dec 1, 2020

Added GcsLogStore implementation ranu010101/delta#1

Merged

mrk-its force-pushed the dynamodb-logstore branch from 3085c67 to 9d35b1e Compare February 19, 2021 12:22

mrk-its force-pushed the dynamodb-logstore branch from 9d35b1e to 6cbb87d Compare May 31, 2021 14:28

andyndang reviewed Jun 2, 2021

View reviewed changes

contribs/src/main/scala/io/delta/storage/DynamoDBLogStore.scala Outdated Show resolved Hide resolved

mrk-its force-pushed the dynamodb-logstore branch from b9edd2b to 897fb59 Compare July 16, 2021 13:35

ioanluca reviewed Sep 9, 2021

View reviewed changes

contribs/src/main/scala/io/delta/storage/BaseExternalLogStore.scala Outdated Show resolved Hide resolved

mrk-its and others added 17 commits March 15, 2022 19:06

move example to contrib subdir

bb6b3dc

refactoring

82de287

add python integration test

778499e

add --jars option

f3126b2

change log levels

bd0e55f

change the test skipping logic and adding logging

cee6c0c

after-review changes

b99b289

store tablePath in db instead of _delta_log path

064b146

add example invocation of integration test

54ca4d0

move FailingDynamoDBLogStore to test package

2a8c4f8

refactor error injection

e7f63bc

changes after review

13e7817

store temp files inside of _delta_log directory

0a235ba

linting

804f486

fix docs

ffef1e6

remove unnecessary logic from getTablePath, fix tests

0ba166e

minor fix

2a51441

mrk-its force-pushed the dynamodb-logstore branch from 0ca39a2 to 2a51441 Compare March 15, 2022 18:29

move to storage-dynamodb

9d08d0d

scottsand-db reviewed Mar 15, 2022

View reviewed changes

build.sbt Show resolved Hide resolved

scottsand-db changed the base branch from master to dynamodb_logstore_scala_feature_branch March 15, 2022 20:45

storage-dynamodb project: add TODO, linting

6b054dd

mrk-its force-pushed the dynamodb-logstore branch from e6ee259 to 6b054dd Compare March 15, 2022 20:57

scottsand-db merged commit 3af698d into delta-io:dynamodb_logstore_scala_feature_branch Mar 15, 2022

orionmoana mentioned this pull request Oct 3, 2023

[Feature Request]Create a Logstore that utilizes S3 APIs but utilizes something cloud-agnostic containerized solution like postgres. #1441

Open

3 tasks

dhruvarya-db mentioned this pull request May 16, 2024

[Spark] Managed Commits: add a DynamoDB-based commit owner #3107

Merged

5 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

DynamoDBLogStore #339

DynamoDBLogStore #339

mrk-its commented Mar 3, 2020 •

edited

Loading

databricks-cla-assistant commented Mar 3, 2020 •

edited

Loading

mrk-its commented Mar 4, 2020

tdas commented Mar 5, 2020 •

edited

Loading

mrk-its commented Mar 5, 2020 •

edited

Loading

mmazek commented Apr 14, 2020

mmazek commented May 14, 2020

matthewpick commented May 28, 2020

mrk-its commented May 28, 2020

zsxwing commented May 28, 2020 •

edited

Loading

mmazek commented May 29, 2020

zsxwing commented Jun 3, 2020

mixam24 commented Jan 14, 2021

zsxwing commented Apr 12, 2021

mrk-its commented Apr 21, 2021

emanuelh-cloud commented May 27, 2021 •

edited

Loading

mrk-its commented May 27, 2021

mrk-its commented Jun 2, 2021

soumilshah1995 commented Nov 25, 2022

orionmoana commented Oct 3, 2023 •

edited

Loading

mrk-its commented Oct 4, 2023

DynamoDBLogStore #339

DynamoDBLogStore #339

Conversation

mrk-its commented Mar 3, 2020 • edited Loading

DynamoDBLogStore requirements:

Testing

databricks-cla-assistant commented Mar 3, 2020 • edited Loading

mrk-its commented Mar 4, 2020

tdas commented Mar 5, 2020 • edited Loading

mrk-its commented Mar 5, 2020 • edited Loading

mmazek commented Apr 14, 2020

mmazek commented May 14, 2020

matthewpick commented May 28, 2020

mrk-its commented May 28, 2020

zsxwing commented May 28, 2020 • edited Loading

mmazek commented May 29, 2020

zsxwing commented Jun 3, 2020

mixam24 commented Jan 14, 2021

zsxwing commented Apr 12, 2021

mrk-its commented Apr 21, 2021

emanuelh-cloud commented May 27, 2021 • edited Loading

mrk-its commented May 27, 2021

mrk-its commented Jun 2, 2021

soumilshah1995 commented Nov 25, 2022

orionmoana commented Oct 3, 2023 • edited Loading

mrk-its commented Oct 4, 2023

mrk-its commented Mar 3, 2020 •

edited

Loading

databricks-cla-assistant commented Mar 3, 2020 •

edited

Loading

tdas commented Mar 5, 2020 •

edited

Loading

mrk-its commented Mar 5, 2020 •

edited

Loading

zsxwing commented May 28, 2020 •

edited

Loading

emanuelh-cloud commented May 27, 2021 •

edited

Loading

orionmoana commented Oct 3, 2023 •

edited

Loading