[MRG] Updates to network read and write (#704)

* Added initial read/write method for network object

* Added __eq__ method for cell, section and network classes

* Added support for attribute gid_ranges

* Added support for pos_dict

* Attempt to remove circle ci errors

* Added support for cell response

* Added support for external drives

* Added test for net after simulation and resolved bug in cell response

* Refactored code

* Added support for external biases and connectivity

* Added support for extracellular arrays, threshold and delay

* Resolve mechs issue - Attempt 1

* Added overwrite option and resolved params issue

* Added read/write for simulated/unsimulated network

* Resolved errors in connectivity

* Added support for saving object type

* Added tests for different models

* Added smoke tests and refactored test code

* Refactored code

* Documentation attempt 1 and removing load mechanisms from other place

* Added authors, removed params, simplified __eq__ and shortened test simulations

* Documentation attempt 2 - Added support for fill doc and copy doc

* Shifted mne code to externals and finished documentation

* Resolved all conflicts and made sure tests are passing

* Replaced tmpdir with tmp_path

* Revorked documentation

* Minor fixes

* Updated documentation

* Added tests, refactored code and changed function signatures

* Refactored __eq__ functions

* Added initial read/write method for network object

* Added __eq__ method for cell, section and network classes

* Added support for attribute gid_ranges

* Added support for pos_dict

* Attempt to remove circle ci errors

* Added support for cell response

* Added support for external drives

* Added test for net after simulation and resolved bug in cell response

* Refactored code

* Added support for external biases and connectivity

* Added support for extracellular arrays, threshold and delay

* Resolve mechs issue - Attempt 1

* Added overwrite option and resolved params issue

* Added read/write for simulated/unsimulated network

* Resolved errors in connectivity

* Added support for saving object type

* Added tests for different models

* Added smoke tests and refactored test code

* Refactored code

* Documentation attempt 1 and removing load mechanisms from other place

* Added authors, removed params, simplified __eq__ and shortened test simulations

* Documentation attempt 2 - Added support for fill doc and copy doc

* Shifted mne code to externals and finished documentation

* Resolved all conflicts and made sure tests are passing

* Replaced tmpdir with tmp_path

* Revorked documentation

* Minor fixes

* Updated documentation

* Added tests, refactored code and changed function signatures

* Refactored __eq__ functions

* refactor: refactor _create_cell_coords function

* refactor: refactor write_network function for clarity and performance

* refactor: refactor write_connectivity function

* refactor: updated pathing and imports to test_io.py

* refactor: rename helper functions from write to to_dict

* fix: rename module to hnn_io because io conflicts with python built-in module and causes odd debugging behavior.

* refactor: rename functions

* refactor: reorganize functions

* refactor: rename _connectivity_to_dict to _connectivity_to_list_of_dicts because it generates a list of dictionaries.

* test: refactor to smaller tests and fixtures

* test: test_external_drive_to_dict

* test: test_str_to_node

* fix: update Network eq method to check for the connectivity contents

* test: update eq test for connectivity and drive changes

* refactor: remove redundant test

* test: add tests_write_network_no_output

* fix: update connectivity logic in Network eq

* tests: read tests

* tests: update tests and read functions with new mesh_shape Network argument

* fix: read connectivity from HDF5 by directly supplying source and target gids from gid_pairs dict

* fix: update _gather_trial_data to access mesh size from attributes _N_pyr_x and _N_pyr_y

* tests: added tests for connectivity and drive weight equivalency

* chore: flake8 formatting fixes

* fix:  Update explicit pathing for test assets to work on github runner windows and linux builds

* chore:  Update whats_new.rst and authorships

* fix: Update Network eq method to detect connectivity edge case where there is the same number of connections but different replicate connections

---------

Co-authored-by: raj1701 <rajatpartani@gmail.com>

doc/api.rst

@@ -105,6 +105,7 @@ Input and Output:
    read_params
    read_dipole
    read_spikes
+   read_network
 
 GUI (:py:mod:`hnn_core.gui`):
 -----------------------------

doc/network_file_desc.rst

@@ -0,0 +1,200 @@
+:orphan:
+
+.. _network_file_content:
+
+====================
+Network File Content
+====================
+
+File Content Description
+------------------------
+hdf5 is the file format used for storing the Network object. The network is stored in a layered format. The first layer consists of the network attributes.
+The attributes of the network are then broken down until the network can be representated as a collection of key value pairs. For example - cell_types is a network
+attribute therefore in the first layer. The description of each cell type is in layer 2. Each cell has various sections. The description of a section is in layer 3.
+Below are the attributes of the network.
+
+object_type : str
+    Type of object (Network) saved.
+N_pyr_x : int
+    Nr of cells (x).
+N_pyr_y : int
+    Nr of cells (y).
+threshold : float
+    Firing threshold of all cells.
+celsius : float
+    Temperature in degree celsius.
+cell_types : dict of Cell Object
+    key : name of cell type.
+    value : Cell object.
+gid_ranges : dict of dict
+    key : cell name or drive name.
+    value : dict.
+pos_dict : dict
+    key : cell type name.
+    value : All co-ordintes of the cell types.
+cell_response : Instance of Cell Response Object
+    The Cell Response object.
+external_drives : dict of dict
+    key : external drive name.
+    value : dict.
+external_biases : dict of dict
+    key : external bias name.
+    value : dict.
+connectivity : list of dict
+    Contains connection info between cells and
+    cells and external drives.
+rec_arrays : dict of Extracellular Arrays
+    key : extracellular array name.
+    value : Instance of Extracellular Array object.
+delay : float
+    Synaptic delay in ms.
+
+Cell Type Description
+---------------------
+
+cell_name : str
+    Name of the cell.
+pos : tuple
+    The (x, y, z) coordinates.
+sections : dict of Section
+    Dictionary with keys as section name.
+synapses : dict of dict
+    Keys are name of synaptic mechanism. Each synaptic mechanism
+    has keys for parameters of the mechanism, e.g., 'e', 'tau1',
+    'tau2'.
+cell_tree : dict of list
+    Stores the tree representation of a cell.
+sect_loc : dict of list
+    Can have keys 'proximal' or 'distal' each containing
+    names of section locations that are proximal or distal.
+gid : int
+    GID of the cell in a network (or None if not yet assigned).
+dipole_pp : list of h.Dipole()
+    The Dipole objects (see dipole.mod).
+vsec : dict
+    Recording of section specific voltage. Must be enabled
+    by running simulate_dipole(net, record_vsec=True) or
+    simulate_dipole(net, record_vsoma=True).
+isec : dict
+    Contains recording of section specific currents indexed
+    by synapse type (keys can be soma_gabaa, soma_gabab etc.).
+    Must be enabled by running simulate_dipole(net, record_isec=True)
+    or simulate_dipole(net, record_isoma=True).
+tonic_biases : list of h.IClamp
+    The current clamps inserted at each section of the cell
+    for tonic biasing inputs.
+
+Section Description
+-------------------
+
+L : float
+    length of a section in microns.
+diam : float
+    diameter of a section in microns.
+cm : float
+    membrane capacitance in micro-Farads.
+Ra : float
+    axial resistivity in ohm-cm.
+end_pts : list of [x, y, z]
+    The start and stop points of the section.
+syns : list of str
+    The synaptic mechanisms to add in this section.
+mechs : dict
+    Mechanisms to insert in this section. The keys
+    are the names of the mechanisms and values
+    are the properties.
+
+Gid Range Description
+---------------------
+
+start : int
+    Start of the gid_range.
+stop : int
+    End of the gid_range.
+
+External Drive Description
+--------------------------
+
+name : str
+    Unique name for the drive.
+dynamics : dict
+    Parameters describing how the temporal dynamics of spike trains in the
+    drive. The keys are specific to the type of drive ('evoked', 'bursty',
+    etc.).
+location : str
+    Target location of synapses.
+cell_specific : bool
+    Whether each artifical drive cell has 1-to-1 (True, default) or
+    all-to-all (False) connection parameters.
+weights_ampa : dict or None
+    Synaptic weights (in uS) of AMPA receptors on each targeted cell
+    type (dict keys).
+weights_nmda : dict or None
+    Synaptic weights (in uS) of NMDA receptors on each targeted cell
+    type (dict keys).
+probability : dict or float
+    Probability of connection between any src-target pair.
+synaptic_delays : dict or float
+    Synaptic delay (in ms) at the column origin, dispersed laterally as
+    a function of the space_constant. If float, applies to all target
+    cell types.
+event_seed : int
+    Optional initial seed for random number generator.
+conn_seed : int
+    Optional initial seed for random number generator.
+n_drive_cells : int | 'n_cells'
+    The number of drive cells that each contribute an independently
+    sampled synaptic spike to the network according to the Gaussian
+    time distribution (mu, sigma).
+events : list
+    Contains the spike times of exogeneous inputs.
+
+External Bias Description
+-------------------------
+
+cell_type : str
+    The cell type whose cells will get the tonic input.
+amplitude : float
+    The amplitude of the input.
+t0 : float
+    The start time of tonic input (in ms).
+tstop : float
+    The end time of tonic input (in ms).
+
+Connection Description
+----------------------
+
+target_types : str
+    Cell type of target gids.
+target_gids : list of int
+    Identifer for targets of source cells.
+num_targets : int
+    Number of unique target gids.
+src_type : str
+    Cell type of source gids.
+src_gids : list of int
+    Identifier for source cells.
+num_srcs : int
+    Number of unique source gids.
+gid_pairs : dict
+    dict indexed by src gids.
+loc : str
+    Target location of synapses.
+receptor : str
+    Synaptic receptor of connection.
+nc_dict : dict
+    Contains information about delay, weight, lamtha etc.
+allow_autapses : bool
+    If True, allow connecting neuron to itself.
+probability : float
+    Probability of connection between any src-target pair.
+
+Extracellular Array Description
+-------------------------------
+
+positions : tuple | list of tuple
+    The (x, y, z) coordinates (in um) of the extracellular electrodes.
+conductivity : float
+    Extracellular conductivity, in S/m.
+method : str
+    Approximation to use.

doc/whats_new.rst

@@ -30,6 +30,9 @@ Changelog
 - Add ability to specify number of cells in :class:`~hnn_core.Network`,
   by `Nick Tolley`_ in :gh:`705`
 
+- Add feature to read/write :class:`~hnn_core.Network` from/to HDF5 format,
+  by `Rajat Partani`_ and `George Dang`_ in :gh:`704`
+
 Bug
 ~~~
 - Fix inconsistent connection mapping from drive gids to cell gids, by

hnn_core/__init__.py

@@ -6,5 +6,6 @@
 from .cell_response import CellResponse, read_spikes
 from .cells_default import pyramidal, basket
 from .parallel_backends import MPIBackend, JoblibBackend
+from .hnn_io import read_network
 
 __version__ = '0.4.dev0'

hnn_core/cell.py

@@ -95,6 +95,10 @@ def _get_gaussian_connection(src_pos, target_pos, nc_dict,
     return weight, delay
 
 
+def node_to_str(node):
+    return node[0] + "," + str(node[1])
+
+
 class _ArtificialCell:
     """The ArtificialCell class for initializing a NEURON feed source.
 
@@ -220,6 +224,56 @@ def __init__(self, L, diam, Ra, cm, end_pts=None):
     def __repr__(self):
         return f'L={self.L}, diam={self.diam}, cm={self.cm}, Ra={self.Ra}'
 
+    def __eq__(self, other):
+        if not isinstance(other, Section):
+            return NotImplemented
+
+        # Check equality for mechs
+        for mech_name in self.mechs.keys():
+            self_mech = self.mechs[mech_name]
+            other_mech = other.mechs[mech_name]
+            for attr in self_mech.keys():
+                if self_mech[attr] != other_mech[attr]:
+                    return False
+
+        # Check end_pts
+        for self_end_pt, other_end_pt in zip(self.end_pts, other.end_pts):
+            if np.testing.assert_almost_equal(self_end_pt,
+                                              other_end_pt, 5) is not None:
+                return False
+
+        all_attrs = dir(self)
+        attrs_to_ignore = [x for x in all_attrs if x.startswith('_')]
+        attrs_to_ignore.extend(['end_pts', 'mechs', 'to_dict'])
+        attrs_to_check = [x for x in all_attrs if x not in attrs_to_ignore]
+
+        # Check all other attributes
+        for attr in attrs_to_check:
+            if getattr(self, attr) != getattr(other, attr):
+                return False
+
+        return True
+
+    def to_dict(self):
+        """Converts an object of Section class to a dictionary.
+
+        Returns
+        -------
+        dictionary form of an object of Section class.
+        """
+        section_data = dict()
+        section_data['L'] = self.L
+        section_data['diam'] = self.diam
+        section_data['cm'] = self.cm
+        section_data['Ra'] = self.Ra
+        section_data['end_pts'] = self.end_pts
+        section_data['nseg'] = self.nseg
+        # Need to solve the partial function problem
+        # in mechs
+        section_data['mechs'] = self.mechs
+        section_data['syns'] = self.syns
+        return section_data
+
     @property
     def L(self):
         return self._L
@@ -355,6 +409,68 @@ def __repr__(self):
         class_name = self.__class__.__name__
         return f'<{class_name} | gid={self._gid}>'
 
+    def __eq__(self, other):
+        if not isinstance(other, Cell):
+            return NotImplemented
+
+        all_attrs = dir(self)
+        attrs_to_ignore = [x for x in all_attrs if x.startswith('_')]
+        attrs_to_ignore.extend(['build', 'copy', 'create_tonic_bias',
+                                'define_shape', 'distance_section', 'gid',
+                                'list_IClamp', 'modify_section',
+                                'parconnect_from_src', 'plot_morphology',
+                                'record', 'sections', 'setup_source_netcon',
+                                'syn_create', 'to_dict'])
+        attrs_to_check = [x for x in all_attrs if x not in attrs_to_ignore]
+
+        # Check all other attributes
+        for attr in attrs_to_check:
+            if getattr(self, attr) != getattr(other, attr):
+                return False
+
+        if not (self.sections.keys() == other.sections.keys()):
+            return False
+
+        for key in self.sections.keys():
+            if self.sections[key] != other.sections[key]:
+                return False
+
+        return True
+
+    def to_dict(self):
+        """Converts an object of Cell class to a dictionary.
+
+        Returns
+        -------
+        dictionary form of an object of Cell class.
+        """
+        cell_data = dict()
+        cell_data['name'] = self.name
+        cell_data['pos'] = self.pos
+        cell_data['sections'] = dict()
+        for key in self.sections:
+            cell_data['sections'][key] = self.sections[key].to_dict()
+        cell_data['synapses'] = self.synapses
+        # cell_data['cell_tree'] = self.cell_tree
+        if self.cell_tree is None:
+            cell_data['cell_tree'] = None
+        else:
+            cell_tree_dict = dict()
+            for parent, children in self.cell_tree.items():
+                key = node_to_str(parent)
+                value = list()
+                for child in children:
+                    value.append(node_to_str(child))
+                cell_tree_dict[key] = value
+            cell_data['cell_tree'] = cell_tree_dict
+        cell_data['sect_loc'] = self.sect_loc
+        cell_data['gid'] = self.gid
+        cell_data['dipole_pp'] = self.dipole_pp
+        cell_data['vsec'] = self.vsec
+        cell_data['isec'] = self.isec
+        cell_data['tonic_biases'] = self.tonic_biases
+        return cell_data
+
     @property
     def gid(self):
         return self._gid
@@ -444,8 +560,6 @@ def _set_biophysics(self, sections):
                     if isinstance(val, list):
                         seg_xs, seg_vals = val[0], val[1]
                         for seg, seg_x, seg_val in zip(sec, seg_xs, seg_vals):
-                            # Checking equality till 5 decimal places
-                            np.testing.assert_almost_equal(seg.x, seg_x, 5)
                             setattr(seg, attr, seg_val)
                     else:
                         setattr(sec, attr, val)

hnn_core/cell_response.py

@@ -136,7 +136,9 @@ def __eq__(self, other):
                        for trial in other._spike_times]
         return (times_self == times_other and
                 self._spike_gids == other._spike_gids and
-                self._spike_types == other._spike_types)
+                self._spike_types == other._spike_types and
+                self.vsec == other.vsec and
+                self.isec == other.isec)
 
     def __getitem__(self, gid_item):
         """Returns a CellResponse object with a copied subset filtered by gid.