Hotfix - direct messaging between nodes & multihop messages (#99)

Addresses #93 - allows messages to be sent directly between net nodes with less fuss, as well as allows messages to be addressed with a list, see the appropriate tests and the discussion in the linked issue In addition, fixed a number of things en passant - added a few tests, including the testing framework to allow GUI tests to be run on the CI, and added tests to the docs experimentally - send full continuous sound through queue rather than in frames, which was causing the continous sound to become truncated - pop a dialogue if terminal is launched without any pilots prompting to add a pilot, addressing #27 - better identification of objects in log creation, looking first for an `id` field that will eventually be the uniform way of identifying objects - split networking into its own module - more type hinting - continued work on removing ad-hoc logic, failing loudly instead of trying to do things implicitly - added favicon to docs <3 - refactored docs into subfolders rather than long period-delimited filenames. - bumped zmq and tornado dep versions Squashed commits: * travis basics * travis test 2 * travis test 3 - no enc setup scripts in setup.py * travis test 4 - get the command right * named test file right * reversing test of event invoker * jesuz * drop all chunks at once * debug flag * handling emptiness and making logger from process * fixing multihop messages... just the send part, now need to fix the 'handle listen' in both the Station and the Net_Node also some type hinting and docstring fixes and cleanup as we go * refactoring networking to own module refactoring docs to use folders instead of extremely long paths * image links * api links * favicon * somehow still missing networking docs * favicon * actually adding scripts * draft of direct messaging between net_nodes * networking docs * clean up station object - absolutely had no idea what class attributes were when i made this i guess - fixing docstring - type hinting - spawning thread in the process by putting it in run rather than in init * correct type hints * maybe get scripts to render? * node-to-node test * multihop test - station objects and node objects count how many messages they receive - let station objects be instantiated programmatically - fixing programmatic station release - loggers check id first before name * fixing prefs test defaults * coverage of mp? * coverage of mp? * coverage of mp? * coverage of mp? * coverage of mp? * coverage of mp? * testing delay in sound trigger * testing delay in sound trigger * set agent clear method for prefs * the godforsaken managers module * the godforsaken managers module * the godforsaken managers module * - terminal test - get app from QApplication.instance() instead of implicitly - close ioloop on release of net nodes - 'float' type in setup - don't save prefs if in tests - bumping requirements to support some features we rely on * asyncio debug mode * do it except not in different build configs * do it except not in different build configs * close event? * istg * again istg * again istg * graspin at straws * graspin at straws * good heavens finally. reverting exploratory attempts at fixes * adding tests to the docs * prompt to add pilots on blank screen init
auto-pi-lot · Jun 1, 2021 · cc86611 · cc86611
1 parent c08d3ed
commit cc86611
Show file tree

Hide file tree

Showing 100 changed files with 1,779 additions and 1,291 deletions.
diff --git a/.coveragerc b/.coveragerc
@@ -0,0 +1,8 @@
+[run]
+concurrency = multiprocessing
+parallel = 1
+omit =
+    */site-packages/_pytest/*
+    */site-packages/py/*
+    */site-packages/pytest/*
+    */site-packages/pytest_cov/*
diff --git a/.travis.yml b/.travis.yml
@@ -19,6 +19,13 @@ addons:
       - herbstluftwm # https://pytest-qt.readthedocs.io/en/latest/troubleshooting.html#xvfb-assertionerror-timeouterror-when-using-waituntil-waitexposed-and-ui-events
       - qt5-default
       - qttools5-dev-tools
+      - libxcb-icccm4 # https://pytest-qt.readthedocs.io/en/latest/troubleshooting.html#github-actions
+      - libxcb-image0
+      - libxcb-keysyms1
+      - libxcb-randr0
+      - libxcb-render-util0
+      - libxcb-xinerama0
+      - libxcb-xfixes0
 
 # enable virtual framebuffer for testing GUI
 services:
@@ -34,9 +41,10 @@ install:
   - pip install -U pytest-cov
   - pip install -U pylint
   - pip install -U coveralls
+  - pip install -U pytest-qt
   - pip install -e .
   - "/sbin/start-stop-daemon --start --quiet --pidfile /tmp/custom_xvfb_99.pid --make-pidfile --background --exec /usr/bin/Xvfb -- :99 -screen 0 1920x1200x24 -ac +extension GLX +render -noreset"
-  - sleep 2
+  - sleep 3
 
 # make directories?
 before_script:
@@ -46,9 +54,10 @@ before_script:
   # - pylint autopilot
 
 script:
-  - pytest --cov=autopilot --cov-report term-missing tests
+  - pytest --cov=autopilot --cov-config=.coveragerc --cov-report term-missing tests
 
 after_script:
+  - coverage combine --append
   - coveralls
 
 before_deploy:

diff --git a/autopilot/core/gui.py b/autopilot/core/gui.py
@@ -38,7 +38,7 @@
 from autopilot.core.subject import Subject
 from autopilot import tasks, prefs
 from autopilot.stim.sound import sounds
-from autopilot.core.networking import Net_Node
+from autopilot.networking import Net_Node
 from functools import wraps
 from autopilot.core.utils import InvokeEvent
 from autopilot.core import styles
@@ -120,7 +120,7 @@ class Control_Panel(QtWidgets.QWidget):
     # Hosts two nested tab widgets to select pilot and subject,
     # set params, run subjects, etc.
 
-    def __init__(self, subjects, start_fn, ping_fn, pilots=None):
+    def __init__(self, subjects, start_fn, ping_fn, pilots):
         """
 
         """
@@ -134,20 +134,8 @@ def __init__(self, subjects, start_fn, ping_fn, pilots=None):
         # We get the Terminal's send_message function so we can communicate directly from here
         self.start_fn = start_fn
         self.ping_fn = ping_fn
+        self.pilots = pilots
 
-        if pilots:
-            self.pilots = pilots
-        else:
-            try:
-                # Try finding prefs in the encapsulating namespaces
-                with open(prefs.get('PILOT_DB')) as pilot_file:
-                    self.pilots = json.load(pilot_file, object_pairs_hook=odict)
-            except NameError:
-                try:
-                    with open('/usr/autopilot/pilot_db.json') as pilot_file:
-                        self.pilots = json.load(pilot_file, object_pairs_hook=odict)
-                except IOError:
-                    Exception('Couldnt find pilot directory!')
 
         # Make dict to store handles to subjects lists
         self.subject_lists = {}
@@ -180,6 +168,7 @@ def init_ui(self):
         for pilot_id, pilot_params in self.pilots.items():
             self.add_pilot(pilot_id, pilot_params.get('subjects', []))
 
+    @gui_event
     def add_pilot(self, pilot_id:str, subjects:typing.Optional[list]=None):
         """
         Add a :class:`.Pilot_Panel` for a new pilot, and populate a :class:`.Subject_List` for it
@@ -2880,21 +2869,28 @@ def plot_params(self):
         return _plot_params
 
 
-
-
-
-
 def pop_dialog(message:str,
                details:str="",
                buttons:tuple=("Ok",),
                modality:str="nonmodal",
-               msg_type:str="info",):
+               msg_type:str="info",) -> QtWidgets.QMessageBox:
     """Convenience function to pop a :class:`.QtGui.QDialog window to display a message.
 
     .. note::
 
         This function does *not* call `.exec_` on the dialog so that it can be managed by the caller.
 
+    Examples:
+        box = pop_dialog(
+            message='Hey what up',
+            details='i got something to tell you',
+            buttons = ('Ok', 'Cancel'))
+        ret = box.exec_()
+        if ret == box.Ok:
+            print("user answered 'Ok'")
+        else:
+            print("user answered 'Cancel'")
+
     Args:
         message (str): message to be displayed
         details (str): Additional detailed to be added to the displayed message

diff --git a/autopilot/core/loggers.py b/autopilot/core/loggers.py
@@ -75,18 +75,17 @@ def init_logger(instance=None, module_name=None, class_name=None, object_name=No
             p_num = 2
             if module_name in globals()['_LOGGERS']:
                 for existing_mod in globals()['_LOGGERS']:
-                    if module_name in existing_mod and re.match('\d$', existing_mod):
+                    if module_name in existing_mod and re.match(r'\d$', existing_mod):
                         p_num += 1
 
             module_name += f"_{str(p_num).zfill(2)}"
 
 
-
         # get name of object if it has one
-        if hasattr(instance, 'name'):
-            object_name = str(instance.name)
-        elif hasattr(instance, 'id'):
+        if hasattr(instance, 'id'):
             object_name = str(instance.id)
+        elif hasattr(instance, 'name'):
+            object_name = str(instance.name)
         else:
             object_name = None
 

diff --git a/autopilot/core/pilot.py b/autopilot/core/pilot.py
@@ -52,7 +52,7 @@
         else:
             from autopilot.stim.sound import jackclient
 
-from autopilot.core.networking import Pilot_Station, Net_Node, Message
+from autopilot.networking import Message, Net_Node, Pilot_Station
 from autopilot import external
 from autopilot import tasks
 from autopilot.hardware import gpio

diff --git a/autopilot/core/plots.py b/autopilot/core/plots.py
@@ -38,7 +38,7 @@
 from autopilot.core import styles
 from autopilot.core.utils import get_invoker
 from .utils import InvokeEvent, Invoker
-from autopilot.core.networking import Net_Node
+from autopilot.networking import Net_Node
 from autopilot.core.loggers import init_logger
 
 

diff --git a/autopilot/core/terminal.py b/autopilot/core/terminal.py
@@ -8,6 +8,7 @@
 import datetime
 import logging
 import threading
+from pathlib import Path
 from collections import OrderedDict as odict
 import numpy as np
 from PySide2 import QtCore, QtGui, QtSvg, QtWidgets
@@ -37,9 +38,9 @@
 
 from autopilot.core.subject import Subject
 from autopilot.core.plots import Plot_Widget
-from autopilot.core.networking import Terminal_Station, Net_Node
+from autopilot.networking import Net_Node, Terminal_Station
 from autopilot.core.utils import InvokeEvent, Invoker, get_invoker
-from autopilot.core.gui import Control_Panel, Protocol_Wizard, Weights, Reassign, Calibrate_Water, Bandwidth_Test
+from autopilot.core.gui import Control_Panel, Protocol_Wizard, Weights, Reassign, Calibrate_Water, Bandwidth_Test, pop_dialog
 from autopilot.core.loggers import init_logger
 
 # Try to import viz, but continue if that doesn't work
@@ -143,8 +144,13 @@ def __init__(self):
         self.logger = init_logger(self)
 
         # Load pilots db as ordered dictionary
-        with open(prefs.get('PILOT_DB')) as pilot_file:
-            self.pilots = json.load(pilot_file, object_pairs_hook=odict)
+        pilot_db_fn = Path(prefs.get('PILOT_DB'))
+        if pilot_db_fn.exists():
+            with open(pilot_db_fn) as pilot_file:
+                self.pilots = json.load(pilot_file, object_pairs_hook=odict)
+        else:
+            self.logger.warning(f'Pilot DB file not found at! {pilot_db_fn}')
+            self.pilots = {}
 
         # Listen dictionary - which methods to call for different messages
         # Methods are spawned in new threads using handle_message
@@ -171,15 +177,15 @@ def __init__(self):
         # The split is so the external networking can run in another process, do potentially time-consuming tasks
         # like resending & confirming message delivery without blocking or missing messages
 
-        self.node = Net_Node(id="_T", upstream='T', port=prefs.get('MSGPORT'), listens=self.listens)
-        self.logger.info("Net Node Initialized")
-
         # Start external communications in own process
         # Has to be after init_network so it makes a new context
         self.networking = Terminal_Station(self.pilots)
         self.networking.start()
         self.logger.info("Station object Initialized")
 
+        self.node = Net_Node(id="_T", upstream='T', port=prefs.get('MSGPORT'), listens=self.listens, instance=False)
+        self.logger.info("Net Node Initialized")
+
         # send an initial ping looking for our pilots
         self.node.send('T', 'INIT')
 
@@ -190,6 +196,18 @@ def __init__(self):
         #self.heartbeat(once=True)
         self.logger.info('Terminal Initialized')
 
+        # if we don't have any pilots, pop a dialogue to declare one
+        if len(self.pilots) == 0:
+            box = pop_dialog(
+                'No Pilots', 'No pilots were found in the pilot_db, add one now?',
+                buttons=('Yes', 'No')
+            )
+            ret = box.exec_()
+            if ret == box.Yes:
+                self.new_pilot()
+
+
+
     def initUI(self):
         """
         Initializes graphical elements of Terminal.
@@ -214,22 +232,32 @@ def initUI(self):
         self.setWindowTitle('Terminal')
         #self.menuBar().setFixedHeight(40)
 
-        # This is the pixel resolution of the entire screen 
-        screensize = app.primaryScreen().size()
-
-        # This is the available geometry of the primary screen, excluding
-        # window manager reserved areas such as task bars and system menus.
-        primary_display = app.primaryScreen().availableGeometry()
-
-
+        # This is the pixel resolution of the entire screen
+        if 'pytest' in sys.modules:
+            primary_display = None
+            terminal_winsize_behavior = 'custom'
+            custom_size=[0,0,1000,480]
+        else:
+            terminal_winsize_behavior = prefs.get('TERMINAL_WINSIZE_BEHAVIOR')
+            custom_size = prefs.get('TERMINAL_CUSTOM_SIZE')
+            app = QtWidgets.QApplication.instance()
+            screensize = app.primaryScreen().size()
+
+            # This is the available geometry of the primary screen, excluding
+            # window manager reserved areas such as task bars and system menus.
+            primary_display = app.primaryScreen().availableGeometry()
+
         ## Initalize the menuBar
         # Linux: Set the menuBar to a fixed height
         # Darwin: Don't worry about menuBar
         if sys.platform == 'darwin':
             bar_height = 0
         else:
-            bar_height = (primary_display.height()/30)+5
-            self.menuBar().setFixedHeight(bar_height)
+            if primary_display is None:
+                self.menuBar().setFixedHeight(30)
+            else:
+                bar_height = (primary_display.height()/30)+5
+                self.menuBar().setFixedHeight(bar_height)
 
         # Create a File menu
         self.file_menu = self.menuBar().addMenu("&File")
@@ -280,9 +308,9 @@ def initUI(self):
         self.data_panel.init_plots(self.pilots.keys())
 
         # Set logo to corner widget
-        if sys.platform != 'darwin':
-            self.menuBar().setCornerWidget(self.logo, QtCore.Qt.TopRightCorner)
-            self.menuBar().adjustSize()
+        # if sys.platform != 'darwin':
+        #     self.menuBar().setCornerWidget(self.logo, QtCore.Qt.TopRightCorner)
+        #     self.menuBar().adjustSize()
 
         # Add Control Panel and Data Panel to main layout
         #self.layout.addWidget(self.logo, 0,0,1,2)
@@ -297,8 +325,7 @@ def initUI(self):
         # If 'remember': restore to the geometry from the last close
         # If 'maximum': restore to fill the entire screen
         # If 'moderate': restore to a reasonable size of (1000, 400) pixels
-        terminal_winsize_behavior = prefs.get('TERMINAL_WINSIZE_BEHAVIOR')
-
+
         # Set geometry according to pref
         if terminal_winsize_behavior == 'maximum':
             # Set geometry to available geometry
@@ -328,7 +355,6 @@ def initUI(self):
                 self.restoreGeometry(self.settings.value("geometry"))
 
         elif terminal_winsize_behavior == "custom":
-            custom_size = prefs.get('TERMINAL_CUSTOM_SIZE')
             self.move(custom_size[0], custom_size[1])
             self.resize(custom_size[2], custom_size[3])
         else:

diff --git a/autopilot/hardware/__init__.py b/autopilot/hardware/__init__.py
@@ -38,7 +38,7 @@
 
 
 from autopilot import prefs
-from autopilot.core.networking import Net_Node
+from autopilot.networking import Net_Node
 from autopilot.core.loggers import init_logger
 
 # FIXME: Hardcoding names of metaclasses, should have some better system of denoting which classes can be instantiated

diff --git a/autopilot/hardware/i2c.py b/autopilot/hardware/i2c.py
@@ -1,7 +1,7 @@
 import os
 import sys
 from autopilot import prefs
-from autopilot.core.networking import Net_Node
+from autopilot.networking import Net_Node
 from autopilot.hardware import Hardware
 from autopilot.hardware.cameras import Camera
 

diff --git a/autopilot/hardware/usb.py b/autopilot/hardware/usb.py
@@ -11,7 +11,7 @@
 from inputs import devices
 
 from autopilot import prefs
-from autopilot.core.networking import Net_Node
+from autopilot.networking import Net_Node
 from autopilot.hardware import Hardware
 
 

diff --git a/autopilot/networking/__init__.py b/autopilot/networking/__init__.py
@@ -0,0 +1,50 @@
+"""
+Classes for network communication.
+
+There are two general types of network objects -
+
+* :class:`autopilot.networking.Station` and its children are independent processes that should only be instantiated once
+    per piece of hardware. They are used to distribute messages between :class:`.Net_Node` s,
+    forward messages up the networking tree, and responding to messages that don't need any input from
+    the :class:`~.pilot.Pilot` or :class:`~.terminal.Terminal`.
+* :class:`.Net_Node` is a pop-in networking class that can be given to any other object that
+    wants to send or receive messages.
+
+The :class:`~autopilot.networking.Message` object is used to serialize and pass
+messages. When sent, messages are ``JSON`` serialized (with some special magic
+to compress/encode numpy arrays) and sent as ``zmq`` multipart messages.
+
+Each serialized message, when sent, can have ``n`` frames of the format::
+
+    [hop_0, hop_1, ... hop_n, final_recipient, serialized_message]
+
+Or, messages can have multiple "hops" (a typical message will have one 'hop' specified
+by the ``to`` field), the second to last frame is always the final intended recipient,
+and the final frame is the serialized message. Note that the ``to`` field of a
+:class:`~autopilot.networking.Message` object will always be the final recipient
+even if a list is passed for ``to`` when sending. This lets :class:`~.networking.Station`
+objects efficiently forward messages without deserializing them at every hop.
+"""
+
+
+import base64
+
+import blosc
+
+from autopilot.networking.station import Station, Terminal_Station, Pilot_Station
+from autopilot.networking.node import Net_Node
+from autopilot.networking.message import Message
+
+
+def serialize_array(array):
+    """
+    Pack an array with :func:`blosc.pack_array` and serialize with :func:`base64.b64encode`
+
+    Args:
+        array (:class:`numpy.ndarray`): Array to serialize
+
+    Returns:
+        dict: {'NUMPY_ARRAY': base-64 encoded, blosc-compressed array.}
+    """
+    compressed = base64.b64encode(blosc.pack_array(array)).decode('ascii')
+    return {'NUMPY_ARRAY': compressed}