Support writes to Delta tables concurrently with Spark

[tvamsikalyan]

Delta lake supports multi cluster setup using Spark that supports concurrent writes to S3 from multiple clusters . Please see multi cluster setup for more information.

This ticket is to request similar feature from Trino.

cc: @dennyglee

[dennyglee]

thanks @tvamsikalyan - I'll be glad to work with folks on enabling this!

[tdas]

I am happy to help out as well. It would be great if Trino connector starts using Delta Standalone for reading the Delta Log. Then the DynamoDB multi-cluster write solution will be available for free, as it works with any connector using Delta Standalone.

[alexjo2144]

Switching the connector to use the DSR would be a pretty big overhaul. Is the protocol for how the engine should interact with DynamoDB to check/set table locks documented somewhere? If it's not too complicated, implementing a TransactionLogSynchronizer would probably be easier.

[alexjo2144]

Having lock compatibility with the open source Delta Spark implementation would be really great to have though.

[dennyglee]

Here's the design document for the S3DynamoDBLogstore and related issues 41, 1044. Hope this helps and please do not hesitate to ping me or @tdas for this, eh?!

[soumilshah1995]

Any updates ?delta-io/delta#1498