Lance commits are progressively slower once a lot of fragments are written

[jaystarshot]

Benchmark file - https://gist.github.com/jaystarshot/9e16a3a58c25ece9b78056e4c55ff3db

[hpvd]

if it's not the code... looks like thermo throttling of the SSD :-)

[wjones127]

Is this on local filesystem? It might not be an issue with # of fragments so much as number of versions. You can differentiate this by calling cleanup and then continuing the benchmark.

This is an old draft PR for improving performance of finding the latest version of local fs: https://github.com/lancedb/lance/pull/4059/files so it it's # of versions then finishing that might help.

[Karin-Wang]

We see this both local and remote on cloud.

[wjones127]

On cloud, have you enabled V2 manifest paths? That can have a big performance impact. See: #2798

[jaystarshot]

Ahh true, i can try the repro on cloud too

[jaystarshot]

I can confirm the issue is also on OCI using the same benchmark.

At 800 we are already 2x worse at a batch level

[jaystarshot]

@wjones127 Is this on by default? I am on lance master for this test but we also see this in 0.383

[wjones127]

Is this on by default? I am on lance master for this test but we also see this in 0.383

It is not on by default. (We can't make it default without breaking older readers. Once enough time is passed we might change the default.)

[jaystarshot]

Thanks I also tried with (enable_v2_manifest_paths=True) + OCI
lance.write_dataset(table, uri, mode="append", storage_options=storage_options, enable_v2_manifest_paths=True)

seems like there is little change.

[wjones127]

Thanks I also tried with (enable_v2_manifest_paths=True) + OCI lance.write_dataset(table, uri, mode="append", storage_options=storage_options, enable_v2_manifest_paths=True)

You can't apply this to an existing dataset. You should either create a new one, or you can migrate your existing one using https://lancedb.github.io/lance-python-doc/all-modules.html#lance.dataset.LanceDataset.migrate_manifest_paths_v2

[jaystarshot]

@wjones127 I had created a new one (on create I had this too)
lance.write_dataset(table, uri, mode="create", storage_options=storage_options, enable_v2_manifest_paths=True)

[wjones127]

Got it. Then maybe it's something else.

Maybe collect traces? https://github.com/lancedb/lance/blob/main/python/DEVELOPMENT.md#tracing-a-python-script

[jaystarshot]

Not as bad the ssd one. (but could be for more fragments)

I will try to spend some time on this next week

[wjones127]

BTW it's faster if you re-use the dataset object instead of passing the URI each time. So instead of this:

https://gist.github.com/jaystarshot/9e16a3a58c25ece9b78056e4c55ff3db#file-benchmark_fragment_writes-py-L59

Just do:

dataset.insert(table)