Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Description
Addresses the following issues that were discovered during testing of DB mode with certain example notebooks:
None
(matching the behavior of mem mode) rather than causing an errorget_utterance_ids
was not being correctly called infilter_utterances_by
DBStorageManager
to avoid the use of the no-longer-supportedcollection.map_reduce
, replacing it with a more canonical solution for listing all IDs in the collectionCorpus.load_info
depending on the storage type. In mem mode the behavior is as before, whereas in DB mode it will instead use iterated batch insertion to avoid needing to load the entire info file (which can be highly memory intensive) and to improve execution speed.
separator inPromptTypes
metadata names with the__
separator, to account for the fact that.
has special meaning in MongoDBMotivation and Context
The above issues were preventing several example notebooks from running to completion, either due to outright crashes or high memory usage. This patch should help resolve that.
How has this been tested?
Ensured that the erroring parts of the notebooks in question now run without issue, and also monitored memory usage to ensure that there is the expected reduction