Random access with filter (in Python)

[pimdh]

Hi,
Suppose I have a dataset with an column "tag", taking values in some small set. I'd like to randomly access the subset of the dataset with the filter tag=X. I can do this manually, by fetching the entire tag column of the dataset, manually constructing a lookup index. To do random access, then I look up a row index and do dataset.take(row_idx).

Are there better ways of doing this? I can create a lance index on the tag column, but I can't find the Python API to access the index to accommodate this use-case.

Thanks!

[westonpace]

Are there better ways of doing this? I can create a lance index on the tag column, but I can't find the Python API to access the index to accommodate this use-case.

There are no better ways available in the current code. If you create a bitmap index on the tag column we are essentially creating the exact same lookup you describe (a map from every tag to a list of row ids) but there are no APIs in place today to access the index values directly.

So the best you can do, once you create the index, is to first query for all matching row ids (filter="tag=X", with_row_id=True) and then sample from the set of row ids that you get back and call _take_rows.