Utilize Amazon S3 condition write to support concurrent write

[Cpaulyz]

Description

Hi, I noticed that Amazon S3 supports condition write now (https://aws.amazon.com/about-aws/whats-new/2024/08/amazon-s3-conditional-writes/). Is it possible to utilize this feature to replace dynamodb-lock to support concurrent write natively?

[thomasfrederikhoeck]

I guess the following upstream needs to be closed first: apache/arrow-rs#6285

[danielgafni]

Just to confirm, this won't allow parallel writing (for example, writing 100 partitions at once), but will remove the dependency on DynamoDB as locking mechanism, right?

[Cpaulyz]

Just to confirm, this won't allow parallel writing (for example, writing 100 partitions at once), but will remove the dependency on DynamoDB as locking mechanism, right?

Yes. I think so.

[rtyler]

If you're up for doing some experimentation (on non-production workloads), I believe that our conditional put support will "just work" for S3.

[satyanmandavilli]

apache/arrow-rs#6285 is closed now. Do we have any update on this issue? Looking forward to remove DynamoDB dependency.

[gianghq]

Keen on this as well.

[ion-elgreco]

apache/arrow-rs#6285 is closed now. Do we have any update on this issue? Looking forward to remove DynamoDB dependency.

You will have to wait on the next arrow release that aligns with datafusion and object store. Then there are still bugs in datafusion that stops us from actually upgrading for some time.