always include sys signals

[skshetry]

This PR

• renames DatasetQuery.results() and DatasetQuery.to_records() to db_results()/to_db_records().
• Always adds signals during select(...), and removes them on as_iterable.

TODO

• cleanup

[rlamy]

LGTM so far. I guess it makes sense to handle this only in DataChain but then we should try to remove all special handling for sys columns from DatasetQuery.

I don't like that _select() and select() are so similar - it would probably be better to add an undocumented flag, but feel free to clean-up however you like.

[dmpetrov]

An question is inline. Please answer.

In general, that's very interesting idea but I need time to think a bit more about this. What i'm not sure about right now:

1. id & rand are technical signals that are required for map and shuffle. I'm not sure we need to expose this to app level. it feels like it belongs to DB level.
2. this approach requires downloading ID & RAND all the time. In some cases it triples amount of data to transfer. In DB level it can be easily optimized.
3. An open question: can we make RAND optional? So, it's created only when user asks. It will require separate datasets and shuffled/randomized dataset.

[skshetry]

1. id & rand are technical signals that are required for map and shuffle. I'm not sure we need to expose this to app level. it feels like it belongs to DB level.

It might be hard to do in db level, due to subqueries. We won't always know if id and rand should be added to select() or not.

3. An open question: can we make RAND optional? So, it's created only when user asks. It will require separate datasets and shuffled/randomized dataset.

chunking for UDFs also require rand.

[dmpetrov]

We won't always know if id and rand should be added to select() or not.

We do know! It should be added in 100% of the cases 🙂

discussed privately