Epic: move indexing to an application layer

[skshetry]

Description

i.e make it based on a feature schema and if possible, with udfs.

Subtasks

Preview Give feedback

1. Accept schema in DataChain.from_records() #246
   +0
   
2. Generator function to list bucket #244
   priority-p1 +1
   
3. Refactor .from_storage() to use listing generator #266
   priority-p1 +1
   
4. Make .get_storage() caching listing lazy #317
   +0
   
5. Replace deprecated listing / indexing with new DataChain.from_storage() #318
   +0
   
6. Refactor Catalog storages methods to use new listing mechanism #329
   +0
   
7. Remove storages from DatasetQuery #340
   housekeeping +1
   
8. Refactor back file system listing source and path #447
   priority-p1 +1
   

[ilongin]

We need to think how to deal with additional tables that are created during indexing, like buckets or partials. So this is not just normal UDF that has an output of some rows in a dataset table, but needs to insert into buckets and partials tables.
It's easy for us to implement this, but if we want users to implement their own indexing maybe we need to provide framework to do so implicitly (user should not care about those tables explicitly) ... WDYT?

[shcheklein]

I think we should start getting rid of partials. They are too complicated for the value they provide. Same with buckets / sources - I would reconsider also drop them.

Each path that we pass to from_storage can be creating a versioned dataset. We can decide to reuse those (as a way to cache things) with some expiration date, etc.

What are the major things we are loosing by getting rid of bucket, sources, partials?

[ilongin]

Partials are needed to be able to index part of a bucket and to avoid re-indexing subdirectories. I have a feeling though that this can all be done even without that partials table, just on the fly but this needs to be investigated.

[dmpetrov]

I think we should start getting rid of partials. T

and

that this can all be done even without that partials table, just on the fly but this needs to be investigated.

Both are good ideas! Let's try to simplify this as much as we can.

We need to think how to deal with additional tables that are created during indexing, like buckets or partials. So this is not just normal UDF that has an output of some rows in a dataset table

Right. We need to find a way to fit the buckets (as well as partials i if needed) into "just normal UDF" and normal datasets. I hope these datasets won't be visible to users (by default).

[shcheklein]

Prioritizing this. It's an epic. Need to add first steps.

[ilongin]

I can take over this one and make a plan / subtasks