Overwrite creates new file

[0xdarkman]

version: delta v0.6.3
environment: cloud, azure

mode overwrite used.
no partition by option used

I see new file being created each time after write:
1-...
2-...
...
6-...

I would expect same file being overwritten so I would expect to see:

1-...

only.

import pyarrow as pa
from deltalake.writer import write_deltalake

storage_options = { "AZURE_STORAGE_ACCOUNT_NAME": account_name, "AZURE_STORAGE_ACCOUNT_KEY": account_key, }

table_path = "abfss://CONTAINERNAME@STORAGEACCOUNT.dfs.core.windows.net/TABLE_NAME"

dt = DeltaTable(table_path, storage_options=storage_options)

tb = pa.Table.from_pandas(df, preserve_index=False)

write_deltalake(table_or_uri=dt, data=tb, mode="overwrite")

[0xdarkman]

delta_table.vacuum(retention_hours=0, enforce_retention_duration=False, dry_run=False)

I have to perform vacuum after all in order to delete non latest version files.

[wjones127]

Hi @0xdarkman, this is intentional and part of the Delta protocol. ACID guarantees and time travel would not work without it.