DM-47375: Run query_all_datasets in a single request for RemoteButler #1114
Commits on Nov 14, 2024
-
Use advanced query system in query_all_datasets
Switch from the query_datasets convenience method to the advanced query system in query_all_datasets. This lets us get the results one page at a time, which will be needed to prevent memory exhaustion when running these queries on the server.
-
Remove with_dimension_records from query-datasets
It turns out that the query-datasets CLI was not actually using dimension records, and it will simplify the implementation to not support this.
-
Restrict --order-by in query-datasets to single type
The backend for querying multiple dataset types will not support "order by", so restrict the CLI to match the implementation.
-
Remove order_by from query_all_datasets
The upcoming implementation of query_all_datasets will not support order_by, so remove it. This requires modifying the query-datasets CLI to use the single dataset type query_datasets when order by needs to be supported.
-
Make streaming query logic reusable
In preparation for implementing query_all_datasets on the server, make the streaming response and timeout logic from the existing query handler re-usable.
-
Move query streaming logic to its own file
After the refactor in the previous commit, this is somewhat independent of the query routes.
-
Move query streaming client code to its own file
This will be shared by the RemoteButler query_all_datasets implementation in an upcoming commit.
-
Define a dataclass for query_all_datasets args
This will be used in an upcoming commit to prevent excessive duplication of function parameters between implementations of query_all_datasets.
-
Add server-side implementation of query_all_datasets
query_all_datasets can potentially involve hundreds or thousands of separate dataset queries. We don't want clients slamming the server with that many HTTP requests, so add a server-side endpoint that can handle these queries in a single request.
-
Add back dimension records to QueryDatasets
It turns out the QueryDatasets class is shared by multiple CLI scripts, some of which need dimension records included. So add back `with_dimension_records` to the internal implementation of query_all_datasets.
-
-
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.