Adjust the S3DynamoDBLogStore to be compatible with ScyllaDB's Alternator.

[rbushri]

Which Delta project/connector is this regarding?

• Spark
• Standalone
• Flink
• Kernel
• S3DynamoDBLogStore
• Other (fill in here)

Description

This PR aims to create a cloud-agnostic solution for the Delta Lake on S3 Multiple Writers issue using ScyllaDB's Alternator. It offers an open-source solution for S3 and S3-compatible storage lacking the putIfAbsent functionality.
The implementation includes the addition of an abstraction layer for DynamoDB LogStore (io.delta.storage.BaseDynamoDBLogStore) and introduces two implementations:

1. io.delta.storage.DynamoDBLogStore - for DynamoDB (no configuration changes for DynamoDB implementation).
2. io.delta.storage.S3ScyllaDBLogStore - for Scylla DB
   The configuration details for ScyllaDB are as follows:

spark.delta.logStore.s3a.impl=io.delta.storage.S3ScyllaDBLogStore
spark.io.delta.storage.S3ScyllaDBLogStore.ddb.endpoint=<ScyllaDB's Alternator cluster endpoint>
spark.io.delta.storage.S3ScyllaDBLogStore.credentials.provider=<The AWSCredentialsProvider used by the client, default DefaultAWSCredentialsProviderChain>
spark.io.delta.storage.S3ScyllaDBLogStore.ddb.tableName=<The name of the Scylla table to use, default delta_log>

Resolves #2411, #1336, #1441

How was this patch tested?

Unit test - SUCCEEDED
Manual test :

• Set up ScyllaDB's Alternator cluster on K8s.
• Write and read to a delta table on S3 storage with the specified configuration:

  spark.delta.logStore.s3a.impl=io.delta.storage.S3ScyllaDBLogStore
  spark.io.delta.storage.S3ScyllaDBLogStore.ddb.endpoint=<ScyllaDB's Alternator cluster endpoint>
  

• Verify that the delta_logs table is created in ScyllaDB, and Delta uses this table for reading and writing the logs.

Does this PR introduce any user-facing changes?

No

[ItaiYaffeAkamai]

@scottsand-db , @mrk-its - there were numerous discussions on Delta OSS Slack around adding S3 concurrent write support, that would not rely on AWS DynamoDB, but rather an open-source database (including the one I started some months ago - https://delta-users.slack.com/archives/CJ70UCSHM/p1689589392090319).
I think @rbushri created a very simple and elegant solution that could potentially help many Delta OSS users.

Who would be the right person to review and approve this PR?
Thanks in advance!

[scottsand-db]

Hi @rbushri - this looks great! Seems like we should rename S3DynamoDBLogStore to something more top-level then, eh? SinceScyllaDB doesn't seem to be a child of DynamoDB, right?

Seems like we could have some generic abstract parent class, and keep the child class name S3DynamoDBLogStore for existing clients, and create a new S3ScyllaDBLogStore? What do you think?

[rbushri]

@scottsand-db, thank you for your review. I've implemented the changes you suggested and updated the PR description. If these changes are acceptable to you, I'll proceed to update the documentation.

[rbushri]

@scottsand-db, would you kindly consider reviewing the changes I made?

[scottsand-db]

@rbushri - yes! sorry, thinks have been very busy focusing on the delta 3.1 release! will take a look

[rbushri]

Thanks @scottsand-db! I truly appreciate your review.

[rbushri]

@scottsand-db, I've addressed your comments and updated the documentation. could you review it, please?

[chris-aeviator]

@rbushri I'm impatiently awaiting this feature getting merged 🙏 . Does this include using delta with write locks via Scylla when interfacing with python (via delta-rs)?

UPDATE: I managed to patch in my custom endpoint into delta-rs and will contribute this to their repo once the code is more clean.

[rbushri]

@scottsand-db, Would you kindly review the changes I've made?

[hattajr]

its been 4 months passed, any updates on this PR?

[scottsand-db]

Hey all, thanks for your patience. Will try to find time to review this in the coming week. Thanks!

[obakamai]

@scottsand-db Any progress with this ? Would be much appreciated if you could push this forward