vacuum is very slow on Cloudflare R2

[djouallah]

Environment

0.9

Binding:
Python

Environment:
Cloudflare R2

---

Bug

Edit : the issue is with vacuum, it is very slow for a delete operation

I am running a cloud function to vacuum and optimize a small delta table in Cloudflare R2, the table has currently 45 partition (per day( and every day, I insert 288 new small files.

the function take nearly 10 minute to finish, that's seems very slow, and I am not sure if it will scale later when the table increase in size

here is the code I use

from deltalake import DeltaTable
import os
delta_path = 's3://delta/scada'
storage_options = {
"Region": "us-east-1",   
"AWS_ACCESS_KEY_ID":     os.environ.get("aws_access_key_id_secret") ,
"AWS_SECRET_ACCESS_KEY": os.environ.get("aws_secret_access_key_secret")   ,   
"AWS_ENDPOINT_URL" :     os.environ.get("endpoint_url_secret") ,
"AWS_S3_ALLOW_UNSAFE_RENAME":"true"
}
def compaction(request):
    dt = DeltaTable(delta_path,storage_options=storage_options)
    dt.optimize()
    dt.vacuum(retention_hours=24,dry_run=False,  enforce_retention_duration=False)
    return 'done'

[Blajda]

Hi @djouallah,
Do you have any insights on which operation takes the longest?
In terms of optimize we are looking to support parallel processing which is outlined here #1171.
Since R2 is using the S3 API, vacuum should be deleting multiple files within in a single API call so I suspect it's not the limiting factor here.

[djouallah]

@Blajda turn out the issue is vacuum !!! I have remove it from the cloud function and it went from 10 minutes to 1 minute

[djouallah]

now i am testing vacuum alone and it did time out after 5 minutes, delete should not take that much ?

[Blajda]

How many files do you have under s3://delta/scada? Vacuum recursively lists all files under the delta root and reconciles them with the log. To determine if the scan is the cause of the issue you can set dryRun to True

[djouallah]

I vacuum and optimised already today, so I don't have much files left : but I produce 288 new files daily, anyway

running this with only 68 files is rather fast, got 3 second
dt.vacuum(retention_hours=1,dry_run=True, enforce_retention_duration=False)

[roeap]

do you have any insights on memory consumption? in my experience, whenever we seen such difference once we get to a certain size, it often relates to limitations in memory.

vacuum should be deleting multiple files within in a single API

Unfortunately, this is not yet implemented on the object-store side, but there is some previous work apache/arrow-rs#4060

[djouallah]

memory seems fine, fir that run, I think it did delete total of 288 x 4 days

[djouallah]

thank you, vacuum went from 5 minutes to 26 second, beautiful works.