perf: allow fragment scan for nearest query if there is a prefilter

[jiachengdb]

We want to create a query plan that first runs prefiltering on selected fragments.

Then for rows passing the prefilter, we probe the ANN index with filter_row_ids.

In this plan, the returned result rows will only come from the selected fragments, so there will be no violation to the query semantics.

[codecov-commenter]

Codecov Report

Attention: Patch coverage is 66.66667% with 2 lines in your changes missing coverage. Please review.

Project coverage is 79.15%. Comparing base (782350e) to head (44eb5e4).

Files	Patch %	Lines
rust/lance/src/dataset/scanner.rs	66.66%	0 Missing and 2 partials ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #2631      +/-   ##
==========================================
+ Coverage   79.13%   79.15%   +0.01%     
==========================================
  Files         218      218              
  Lines       63615    63617       +2     
  Branches    63615    63617       +2     
==========================================
+ Hits        50344    50354      +10     
+ Misses      10324    10322       -2     
+ Partials     2947     2941       -6     

Flag	Coverage Δ
unittests	79.15% <66.66%> (+0.01%)	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[westonpace]

This won't work today for filters on indexed columns. For example, if the filter is my_uuid == 17 or my_uuid != 17 and there is a scalar index on my_uuid. In this case the prefilter will be an allow list or block list with one row id.

However, we should be able to enhance the indexed prefilter logic to apply an allow list or block list based on excluded fragments.

[jiachengdb]

Got it.

Is there an easy way to extend the validation logic in this PR to only allow fragment scan if the leaf is a FileScan?

[jiachengdb]

@westonpace I added another check on the index scan to disallow the case you mentioned. PTAL.

[westonpace]

Can you move this to this spot (it's line 1539 on main right now I think):

            (Some(index_query), None, true) => {
                // The filter is completely satisfied by the index.  We
                // only need to search the index to determine the valid row
                // ids.
                let index_query = Arc::new(ScalarIndexExec::new(
                    self.dataset.clone(),
                    index_query.clone(),
                ));
                PreFilterSource::ScalarIndexQuery(index_query)
            }

The only node that doesn't handle a fragment scan is the ScalarIndexQuery node. So right here is a bit too restrictive. If we do that then we're good to go.

[jiachengdb]

Moved to line 1539. PTAL.

[westonpace]

Thanks for bearing with me.

[westonpace]

@jiachengdb

I'm going to merge this but if you find time can you add a PR with a python unit test covering this case?

[jiachengdb]

@jiachengdb

I'm going to merge this but if you find time can you add a PR with a python unit test covering this case?

Let me find time to add the test.

[jiachengdb]

@jiachengdb
I'm going to merge this but if you find time can you add a PR with a python unit test covering this case?

Let me find time to add the test.

@westonpace #2638