Implement sys feature, and rename id/random columns

[skshetry]

Check individual commits.

[dmpetrov]

It feels like the logic should be opposite - the fields should be excluded by default everywhere and user can enable these in some specific API calls like from_storage(sys=True) or from_dataset(sys=True)

[skshetry]

We have a lot of internals that depend on it. Eg: all of the indexing queries, udf, join require id columns. We are also doing some ordering based on id in different places.

We use random column for chunking, etc.

Another thing is, we try to preserve data as much as possible:
Eg:

DatasetQuery(name="test").save()

This preserves all of the columns including id and random, we don't generate anything.

Similarly, DatasetQuery.generate() adds a column, but it preserves id and random.

[skshetry]

Ideally, we need a notion of default columns, in which select() returns all columns except id and random unless specifically asked.

DatasetQuery.select(C.id, C.random, *query.columns).results()
# contains `id` and `random` columns

DatasetQuery.results()
# excludes `id` and `random` column

But this is very hard to differentiate at the query level, as we might get columns from other queries, or tables.

[dmpetrov]

I'm talking about user facing API.

1. Internally we have to always save the columns.
2. For users - only when these sys columns were requested directly.

Practically, it means, the request should be likedc.map(res=lambda sys, file: ..., ) or dc.map(res=func(), params=["sys"])

If we decide to do that in the lower level calls (which is optional until we can make it in map()): @udf(params=("sys__id") ...).

[dmpetrov]

Looks good! thank you.

[dmpetrov]

Shouldn't it be collect_flatten()?

[skshetry]

What should we do for existing results and to_records?

[dmpetrov]

Do we need this? It looks smart but the usage is not clear. If it's needed - we should probably introduce this in other methods.

[skshetry]

It's just trying to imitate what we do in DatasetQuery.results(). It also makes it easier to implement to_records()?

[dmpetrov]

good idea