Make enable_v2_manifest_paths default to True

[wojiaodoubao]

Manifest V2 enables O(1) time complexity for list operations. IMHO, there is no reason to use v1 for a new lance dataset. Shall we make manifest V2 the default behavior instead of requiring additional configuration, so that Lance’s out-of-the-box behavior becomes more sensible.

When users first use a Lance dataset, they almost always stick to the default out-of-the-box configuration, so they do not explicitly enable Manifest V2. At this stage, they won’t encounter any issues because their dataset is still small.
They may not notice the slow list operation problem until much later, by which time Lance will have already been deployed in a production environment. This will make the subsequent migration process quite cumbersome.

[westonpace]

I agree it seems like a good default. If a user already has a bunch of v1 datasets and we change the default then what will happen? Will it detect that the datasets are v1 and fallback to v1 behavior? In other words, does the default only change new datasets?

[westonpace]

Also, CC @wjones127

[wjones127]

Here is the original PR it was introduced in, for reference: #2798

For posterity, the reason we didn't change the default when we implemented it was that it was a hard breaking change in the format. Trying to read a dataset that uses V2 manifest paths using a library that doesn't understand them is undefined behavior.

The earliest Lance library version that understands V2 paths is v0.17.0. That was around September 2024.

I think it's time we can change the default to V2. 👍

[wjones127]

FYI @jackye1995 was considering introducing a V3 version: #4488

Though what we might do is re-use the "batch head" strategy for V2 and see if we can get most of the same benefits without changing the format.

[wojiaodoubao]

In other words, does the default only change new datasets?

@westonpace Yes, lance can auto-detect manifest version. The config enable_v2_manifest_paths only effects when creating new dataset.

[jackye1995]

For v3 scheme, based on the discussion, it's probably better to just have an optional hint in the v2 scheme as an optimization. Will had a good point that if commit frequency is high it could lead to the reader keep finding new manifest, and list provides a point in time snapshot which is valuable. I did not get time to work on the optimization yet, let me create an issue to track it.