Skip to content

Commit

Permalink
Add QGraph dependencies to dimension clustering.
Browse files Browse the repository at this point in the history
  • Loading branch information
MichelleGower committed Oct 7, 2024
1 parent b2f8419 commit 5b1014e
Show file tree
Hide file tree
Showing 6 changed files with 580 additions and 120 deletions.
1 change: 1 addition & 0 deletions doc/changes/DM-46513.misc.rst
Original file line number Diff line number Diff line change
@@ -1,2 +1,3 @@
Fixed test_clustered_quantum_graph.testClusters.
Moved validation of ClusteredQuantumGraph from tests to class definition.
Added following QuantumGraph dependencies to dimension clustering to enable clustering when dimension values aren't equal (e.g., group vs visit).
37 changes: 37 additions & 0 deletions doc/lsst.ctrl.bps/quickstart.rst
Original file line number Diff line number Diff line change
Expand Up @@ -1099,6 +1099,43 @@ Relevant Config Entries:
# requestCpus: N # Overrides for jobs in this cluster
# requestMemory: NNNN # MB, Overrides for jobs in this cluster
.. _bps-dimension_dependency:

User-defined Dimension Clustering with QuantumGraph Dependencies
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

There are instances where the dimensions aren't the same for quanta
that we want to put in the same cluster. In many cases, we can use
``equalDimensions`` to solve this problem. However that only works if
the values are equal, but just different dimension names (e.g., visit
and exposure). In the case of group and visit, the values aren't the
same. The QuantumGraph has dependencies between those quanta that
can be used instead of comparing dimension values.

Using the dependencies is an option per cluster definition. To enable it,
define ``find_dependency_method``. A subgraph of the pipeline graph is
made (i.e., a directed graph of the pipeline task labels specified for
the cluster). A value of ``sink`` says to use the dimensions of the sink
nodes in the subgraph and then find ancestors in the ``QuantumGraph`` to
complete the cluster. A value of ``source`` says to use the dimensions
of the source nodes in the subgraph and then find descendants in the
``QuantumGraph`` to complete the cluster. Generally, it doesn't matter
which direction is used, but the direction determines which dimension
values appear in the cluster names and thus job names.

.. code-block:: YAML
clusterAlgorithm: lsst.ctrl.bps.quantum_clustering_funcs.dimension_clustering
cluster:
# Repeat cluster subsection for however many clusters there are
# with or without find_dependency_method
clusterLabel1:
dimensions: visit, detector
pipetasks: getRegionTimeFromVisit, loadDiaCatalogs, diaPipe
find_dependency_method: sink
# requestCpus: N # Overrides for jobs in this cluster
# requestMemory: NNNN # MB, Overrides for jobs in this cluster
.. _bps-softlink:

WMS-id softlink
Expand Down
Loading

0 comments on commit 5b1014e

Please sign in to comment.