Support eager mode #2331
Support eager mode #2331
Conversation
jieguangzhou
force-pushed
the
feat/eager-mode
branch
3 times, most recently
from
dbb1caf to
b229454
Compare
|
TODO: Need to add the document |
jieguangzhou
force-pushed
the
feat/eager-mode
branch
4 times, most recently
from
a5ca324 to
00d77d3
Compare
| def replace_id(part): | ||
| find_params = part[1] | ||
| if not find_params: | ||
| find_params = ({}, {}) | ||
|
|
||
| assert isinstance(find_params[1], dict) | ||
| find_params = list(find_params) | ||
| find_params[1]['_id'] = 1 | ||
| return (part[0], tuple(find_params), part[2]) |
There was a problem hiding this comment.
Only return _id
superduper/backends/mongodb/query.py
Outdated
| def filter(self, filter): | ||
| """Return a query that filters the documents. | ||
|
|
||
| :param filter: The filter to apply. | ||
| """ | ||
|
|
||
| def replace_function(part): | ||
| find_params = part[1] | ||
| if not find_params: | ||
| find_params = ({},) | ||
|
|
||
| assert isinstance(find_params[0], dict) | ||
|
|
||
| find_params[0].update(filter) | ||
| return (part[0], tuple(find_params), part[2]) | ||
|
|
||
| return self._replace_part('find', replace_function) |
There was a problem hiding this comment.
Add the key of the filter to the list of returned fields.⬤
I think the Eager mode can be always enabled, so the returned data is not the raw data. This approach allows users to purposefully use the Eager mode. If we directly combine it with WDYT? |
superduper/backends/base/query.py
Outdated
|
|
||
| assert self.db is not None, 'No datalayer (db) provided' | ||
| query = self | ||
| if not len(query.parts): |
There was a problem hiding this comment.
This means query is Simply a Table, right?
| def _eager_call__(self, *args, **kwargs): | ||
| from superduper.misc.eager import SuperDuperData, SuperDuperDataType, TrackData | ||
|
|
||
| have_sdd, graph = SuperDuperData.detect_and_get_graph(*args, **kwargs) |
There was a problem hiding this comment.
already been called at line 910?
There was a problem hiding this comment.
Yes, the graph parameter is retrieved again because I don’t want the graph parameter to be passed from upstream. This function has very low overhead.
jieguangzhou
force-pushed
the
feat/eager-mode
branch
from
00d77d3 to
548814c
Compare
After discuss with @kartik4949 , I change to When |
docs/content/tutorials/eager_mode.md
Outdated
| When using `select.execute(eager_mode=True)`, all returned data will enter eager mode, which can be used for interactive model pipeline construction. | ||
|
|
||
| ```python | ||
| datas = list(db["documents"].select().execute(eager_mode=True)) |
There was a problem hiding this comment.
Interesting language thing: plural of "data" is "data", singular is "datum".
There was a problem hiding this comment.
Hmm, Got it, I changed all of them
superduper/backends/mongodb/query.py
Outdated
| project.update({k: 1 for k in filter_mapping_base.keys()}) | ||
|
|
||
| predict_ids_in_filter = [ | ||
| key.replace("_outputs__", "") for key in filter_mapping_outptus.keys() |
There was a problem hiding this comment.
Optimized the logic by using predict_id.
Additionally, since MongoMock does not support ‘.’ in the ‘as’ field for lookups, the key was converted to ‘outputs_’.
After the join, the complete outputs data can be queried as _outputs__{predict_id}._outputs.{predict_id} : result.
Explanations have already been added in the code.
…_ids during the application build_from_db.
jieguangzhou
force-pushed
the
feat/eager-mode
branch
3 times, most recently
from
1981447 to
eabcea8
Compare
- Support eager mode condition filter - Add eager mode tutorial - Use select.execute instead of select.datas to initiate eager mode.
Description
Related Issues
Checklist
make unit_testingandmake integration-testingsuccessfully?Additional Notes or Comments