Code Quirks

Code Quirks

Some elements in the code can be difficult to understand. The complications arose from some trade-off, usually for better performance or requiring less GPU memory. Here we'll try to explain some of the details that might take longer to acquire by simply looking through the code.

ClusterData

Background

It's common practice with a view (i.e., Kokkos::View) to copy into a lambda for a device kernel or pass by value into a function. This is because a shallow copy of a view is cheap. This idea extends (to a point) to a struct holding multiple views. This is what the ClusterData class is. A problem arises because we want every reaction to have access to the cluster data. Since ClusterData has at least ten views, it is no longer a trivial amount of memory having a shallow copy exist in each reaction. In order to use a reference in reactions, we needed a copy of the ClusterData object (not just the data its views own) to exist in a permanent location on the GPU.

Design

Reaction

The Reaction class has a const pointer to ClusterData member variable which gets set at construction. This can be updated if and when the central ClusterData object is replaced using the updateData function.

ReactionNetwork

ReactionNetwork "owns" the cluster data and is responsible for managing synchronization between device and host memory as well as notifying reactions when changes are made.

ReactionNetwork::_clusterData

The data on the device is owned by two instances of ClusterData, the permanent instance on the device and another on the host. These are managed using a Kokkos::DualView named _clusterData. Note that what the DualView is mirroring between host and device is an instance of the ClusterData struct, not the actual data referenced from the views within the struct. Therefore, for this member, there exists only one instance of the data collectively referenced by the views in ClusterData. This data lives on-device and both instances (both host and device) of _clusterData (.h_view() and .d_view()) hold views referencing data on-device. When an actual data member of _clusterData is modified (e.g., gridSize) on the host, the device instance must be updated. This can be done constistently by calling ReactionNetwork::copyClusterDataView().

ReactionNetwork::_clusterDataMirror

Several places in the code need access to the actual data from ClusterData in host memory. This is the purpose of ReactionNetwork's _clusterDataMirror member.

DO NOT use _clusterDataMirror directly. Always access this by calling ReactionNetwork::getClusterDataMirror().

The _clusterDataMirror member is a std::optional<ClusterDataMirror> where ClusterDataMirror aliases a template instantiation of ClusterData whose deep data lives on the host. The getClusterDataMirror function will make a deep copy from the device if necessary (i.e., if the std::optional does not have a value) so that you get the most up-to-date data. Any time the device data is modified, there should be a call to ReactionNetwork::invalidateDataMirror() which resets the std::optional. This way, the next time the host mirror is accessed, a new deep copy will be performed.