Expose filter APIs via Python

[eddyxu]

Problem Statement

Lance format itself does support nested columns projection pushdown, filter/predicates pushdown as well as LIMIT / OFFSET pushdown (#45). However, the Apache Arrow ScanOptions and Duckdb Pyarrow integration do not fully support these optimizations yet (#46).

We could expose these filters via our Python and C++ API first to make them usable first.

Desired Behavior

Enrich Python / C++ API support for nested column projection pushdown, filter / predicates pushdown as well as limit/offset clause pushdown.

Proposed python API change:

# lance/__init__.py
def dataset(
   uri: str,
   columns: Optional[list[str]] = None,
   filters: Optional[pyarrow.Expression] = None,
   limit: Optional[int] = None,
   offset: int = 0
) -> pyarrow.dataset.Dataset:
    ...

[eddyxu]

So Arrow can not directly support the (VERY) nested column (list<struct>) push down #60 . We can use our own API to by-pass the limitation.

[eddyxu]

Done in #61