DM-45993: Optimize DirectButlerCollections.query_info to avoid too many queries

[andy-slac]

Direct butler reimplements query_info method to avoid multiple queries, which makes it significantly faster. This patch also adds two optional parameters to query_info to allow further optimizations. There is still an inefficiency in fetch_summaries method when the number of potential collections is very large (when collections are *). Further optimization would probably need more work (and I think that we'll have to optimize it as the number of collections grows every day).

@dhirving, I added the same parameters to remote butler interface, but they are not used for now. I know you are working on DM-46129, maybe you can add forwarding of those parameters to remote server?

Checklist

• ran Jenkins
• added a release note for user-visible changes to doc/changes
• (if changing dimensions.yaml) make a copy of dimensions.yaml in configs/old_dimensions

[dhirving]

Sounds good. I'll rebase #1074 after this merges and add the new parameters.

[codecov]

Codecov Report

Attention: Patch coverage is 87.83784% with 9 lines in your changes missing coverage. Please review.

Project coverage is 89.66%. Comparing base (1d1bf7b) to head (76d34d6).
Report is 6 commits behind head on main.

Files with missing lines	Patch %	Lines
python/lsst/daf/butler/script/queryDatasets.py	60.00%	2 Missing and 2 partials ⚠️
python/lsst/daf/butler/_butler_collections.py	81.81%	1 Missing and 1 partial ⚠️
...butler/direct_butler/_direct_butler_collections.py	91.66%	1 Missing and 1 partial ⚠️
...on/lsst/daf/butler/script/queryDimensionRecords.py	85.71%	0 Missing and 1 partial ⚠️

Additional details and impacted files

@@           Coverage Diff           @@
##             main    #1075   +/-   ##
=======================================
  Coverage   89.65%   89.66%           
=======================================
  Files         359      359           
  Lines       46885    46925   +40     
  Branches     9637     9650   +13     
=======================================
+ Hits        42036    42073   +37     
- Misses       3482     3485    +3     
  Partials     1367     1367           

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[andy-slac]

@timj, I have added a new private method to Butler.collections and changed query-data-ids, query-datasets, and query-dimension-records to use it.

[timj]

Looks great. Thanks for unifying the logic in query-dimension-record and data-ids as well.
It makes sense not to include the doc strings in queries by default.