Azure concurrent writes

[davidrobbo]

Description

Looking for a bit of advice on concurrent writes against a table in azure. I'm familiar with the requirement to have a Dynamo table to handle table locks in AWS, but reading the docs (as someone not overly familiar with azure), I'm unsure if concurrent writes against a delta table in blob/adls2 in azure could lead to data loss, or if by default the storage in azure provides 'put-if-absent' type guarantees that makes it protect against data loss from multi-app/cluster writes?

Use Case

I would like to utilise write_deltalake to write to a table on Azure (blob storage and/or adls2) from a multi-replica python app, and I want to guarantee no data loss while creating a transaction in the _delta_log directory (i.e. each replica of my application creating a transaction under _delta_log with the same name at exactly the same point in time leading to one replica overwriting the other replica's entry)

Related Issue(s)

None.

[roeap]

Good news - azure storage supports concurrent writes out of the box.

[inigohidalgo]

Apologies for necro.
@roeap could you give some info on how this is achieved? Is it based on conditional headers for read/write in blob?

I saw the azure implementation for the LogStore is basically the default, whereas the aws one has a specific LogStore implementation based on dynamoDB locking.

Thanks!

[inigohidalgo]

I think I have answered myself by trawling through various issues and PRs.

To answer myself: yes, the difference is that the Azure object store implements a simple copy_if_not_exists

https://github.com/apache/arrow-rs/blob/c2b05cdbcb37f46f170c7b5073ee6bc2178fdace/object_store/src/azure/mod.rs#L134-L137

Whereas the AWS implementation depends on the locking client

https://github.com/apache/arrow-rs/blob/c2b05cdbcb37f46f170c7b5073ee6bc2178fdace/object_store/src/aws/mod.rs#L292-L318