Merge branch 'main' of https://github.com/mehta-lab/recOrder into tscope

Author: ieivanov

.github/workflows/pytests.yml

@@ -11,7 +11,7 @@ jobs:
     runs-on: ubuntu-latest
     strategy:
       matrix:
-        python-version: ['3.8', '3.9', '3.10']
+        python-version: ['3.9', '3.10']
 
     steps:
     - name: Checkout repo

.github/workflows/test_and_deploy.yml

@@ -11,7 +11,7 @@ on:
       - "*"
   pull_request:
     branches:
-      - main
+      - "*"
   workflow_dispatch:
 
 jobs:
@@ -21,13 +21,13 @@ jobs:
     strategy:
       matrix:
         platform: [ubuntu-latest, windows-latest, macos-latest]
-        python-version: ['3.8', '3.9', '3.10']
+        python-version: ['3.9', '3.10']
 
     steps:
-      - uses: actions/checkout@v2
+      - uses: actions/checkout@v3
 
       - name: Set up Python ${{ matrix.python-version }}
-        uses: actions/setup-python@v2
+        uses: actions/setup-python@v4
         with:
           python-version: ${{ matrix.python-version }}
 
@@ -50,16 +50,14 @@ jobs:
           python -m pip install --upgrade pip
           python -m pip install setuptools tox tox-gh-actions
 
-      # this runs the platform-specific tests declared in tox.ini
-      - name: Test with tox
-        uses: GabrielBB/xvfb-action@v1
+      # https://github.com/napari/cookiecutter-napari-plugin/commit/cb9a8c152b68473e8beabf44e7ab11fc46483b5d
+      - name: Test
+        uses: aganders3/headless-gui@v1
         with:
           run: python -m tox
-        env:
-          PLATFORM: ${{ matrix.platform }}
 
       - name: Coverage
-        uses: codecov/codecov-action@v2
+        uses: codecov/codecov-action@v3
 
   deploy:
     # this will run when you have tagged a commit with a version number

.gitignore

@@ -145,3 +145,6 @@ dmypy.json
 */_version.py
 recOrder/_version.py
 *.autosave
+
+# example data
+/examples/data_temp/*

README.md

@@ -14,7 +14,7 @@ The acquisition, calibration, background correction, reconstruction, and applica
 Syuan-Ming Guo, Li-Hao Yeh, Jenny Folkesson, Ivan E Ivanov, Anitha P Krishnan, Matthew G Keefe, Ezzat Hashemi, David Shin, Bryant B Chhun, Nathan H Cho, Manuel D Leonetti, May H Han, Tomasz J Nowakowski, Shalin B Mehta, "Revealing architectural order with quantitative label-free imaging and deep learning," eLife 2020;9:e55502 DOI: 10.7554/eLife.55502 (2020).
 ```
 
-`recOrder` is to be used alongside a conventional widefield microscope fitted with a universal polarizer (Panel A below).  The universal polarizer allows for the collection of label-free information including the intrinsic anisotropy of the sample and its relative phase (density). These measurements are collected by acquiring data under calibrated, polarization-diverse illumination followed by a computational reconstruction.  The overall structure of `recOrder` is shown in Panel B, highlighting the two different usage modes and their features: graphical user interface (GUI) through a napari plugin, and a command line interface (CLI).
+`recOrder` is to be used alongside a conventional widefield microscope fitted with a universal polarizer (Panel A below).  The universal polarizer allows for the collection of label-free information including the intrinsic anisotropy of the sample and its relative phase (density). These measurements are collected by acquiring data under calibrated, polarization-diverse illumination followed by a computational reconstruction. The overall structure of `recOrder` is shown in Panel B, highlighting the structure of the graphical user interface (GUI) through a napari plugin.
 
 ![Flow Chart](https://github.com/mehta-lab/recOrder/blob/main/docs/images/recOrder_Fig1_Overview.png?raw=true)
 
@@ -34,13 +34,6 @@ conda create -y -n recOrder python=3.9
 conda activate recOrder
 ```
 
-> *Apple Silicon users please use*:
->
-> ```sh
-> CONDA_SUBDIR=osx-64 conda create -y -n recOrder python=3.9
-> conda activate recOrder
-> ```
-
 Install `recOrder-napari`:
 
 ```sh
@@ -53,10 +46,5 @@ Open `napari` with `recOrder-napari`:
 napari -w recOrder-napari
 ```
 
-View command-line help by running
-
-```sh
-recOrder.help
-```
-
-For more help, see [`recOrder`s documentation](./docs).
+For more help, see [`recOrder`'s documentation](https://github.com/mehta-lab/recOrder/tree/main/docs). To install `recOrder` 
+on a microscope, see the [microscope installation guide](https://github.com/mehta-lab/recOrder/blob/main/docs/microscope-installation-guide.md).

docs/development-guide.md

@@ -9,16 +9,6 @@
     conda activate recOrder
     ```
 
-    > *Apple Silicon users please use*:
-    >
-    > ```sh
-    > CONDA_SUBDIR=osx-64 conda create -y -n recOrder python=3.9
-    > conda activate recOrder
-    > ```
-    >
-    > Reason: `napari` requires `PyQt5` which is not available for `arm64` from PyPI wheels.
-    > Specifying `CONDA_SUBDIR=osx-64` will install an `x86_64` version of `python` which has `PyQt5` wheels available.
-
 2. Clone the `recOrder` directory:
 
     ```sh

docs/images/cli_structure.png

@@ -7,6 +7,22 @@ This guide will walk through a complete recOrder installation consisting of:
 
 Before you start you will need a machine with Windows 10, a Meadowlark DS5020 connected to a liquid crystal device, and a microscope system compatible with `Micromanager`. 
 
+## Install Meadowlark DS5020 and liquid crystals
+
+Start by installing the Meadowlark DS5020 and liquid crystals using the software on the USB stick provided by Meadowlark. You will need to install the USB drivers and CellDrive5000.
+
+**Check your installation versions** by opening CellDrive5000 and double clicking the Meadowlark Optics logo. **We have tested `recOrder == 0.4.0` with "PC software version 1.08" and "Controller firmware version 1.04",** and you will need to upgrade if your software and firmware versions are older. 
+
+To upgrade your "PC software version" use these steps:
+
+- From "Add and remove programs", remove CellDrive5000 and "National Instruments Software".
+- From "Device manager", open the "Meadowlark Optics" group, right click `mlousb`, click "Uninstall device", check "Delete the driver software for this device", and click "Uninstall". Uninstall `Meadowlark Optics D5020 LC Driver` following the same steps.
+- Using the USB stick provided by Meadowlark, reinstall the USB drivers and CellDrive5000. 
+- Confirm that "PC software version" == 1.08
+- **Upgrading users:** you will need to reinstall the Meadowlark device to your micromanager configuration file, because the device driver's name has changed to `MeadowlarkLC`. 
+
+To upgrade your DS5020's firmware, use Meadowlark's "Firmware Updater".
+
 ## Install recOrder software
 
 (Optional but recommended) install [anaconda](https://www.anaconda.com/products/distribution) and create a virtual environment  
@@ -27,11 +43,11 @@ should launch napari (may take 15 seconds on a fresh installation) with the recO
 
 ## Install and configure `Micromanager`
 
-Install `Micromanager 2.0` nightly build `20220920` (https://micro-manager.org/Micro-Manager_Nightly_Builds). 
+Download and install [`Micromanager 2.0` nightly build `20230426` (~150 MB link).](https://download.micro-manager.org/nightly/2.0/Windows/MMSetup_64bit_2.0.1_20230426.exe)
 
-**Note:** We have tested recOrder with `20220920`, but most features will work with newer builds. We recommend testing a minimal installation with `20220920` before testing with a different nightly build or additional device drivers. 
+**Note:** We have tested recOrder with `20230426`, but most features will work with newer builds. We recommend testing a minimal installation with `20230426` before testing with a different nightly build or additional device drivers. 
 
-Before launching `Micromanager`, download the Meadowlark device adapters and calibration files from the [release page](https://github.com/mehta-lab/recOrder/releases/) and place these three unzipped files into your `Micromanager` folder (likely `C:\Program Files\Micro-Manager` or similar). 
+Before launching `Micromanager`, download the USB driver dll from the [release page](https://github.com/mehta-lab/recOrder/releases/) and place this pair of unzipped files into your `Micromanager` folder (likely `C:\Program Files\Micro-Manager` or similar). 
 
 Launch `Micromanager`, open `Devices > Hardware Configuration Wizard...`, and add the `MeadowlarkLcOpenSource` device to your configuration. Confirm your installation by opening `Devices > Device Property Browser...` and confirming that `MeadowlarkLCOpenSource` properties appear. 
 
@@ -72,4 +88,4 @@ Finally, enable port access so that `Micromanager` can communicate with recOrder
 
 From the `recOrder` window, click `Switch to Online`. If you see `Success`, your installation is complete and you can [proceed to the napari plugin guide](./napari-plugin-guide.md). 
 
-If you you see `Failed`, check that `Micromanager` is open, check that you've enabled `Run server on port 4827`. If the connection continues to fail, report an issue with you stack trace for support. 
+If you you see `Failed`, check that `Micromanager` is open, check that you've enabled `Run server on port 4827`. If the connection continues to fail, report an issue with your stack trace for support. 

docs/images/recOrder_Fig1_Overview.png

@@ -1,13 +1,25 @@
 # Automating reconstructions
 
-`recOrder` is undergoing changes to the way that it handles automated reconstructions.
+`recOrder` uses a configuration-file-based command-line interface (CLI) to perform all reconstructions.
 
-`recOrder==0.2.0` had an offline mode with `config.yml` files that were used to configure reconstructions. These config files could be generated and saved via the GUI, and the reconstructions could be run via GUI or CLI. 
+## How can I use `recOrder`'s CLI to perform reconstructions?
+`recOrder`'s CLI is summarized in the following figure:
+<img src="./images/cli_structure.png" align="center">
 
-Although the offline mode had many valuable and convenient features, we found that it had diverged from the online mode and it was difficult to recreate results between online and offline modes. These design limitations led us to the following plan for our upcoming releases:
+The main command `reconstruct` command is composed of two subcommands: `compute-tf` and `apply-inv-tf`. 
 
-`recOrder==0.3.0` (release candidate in the coming weeks) will remove the offline mode and use a set of scripts instead. We recommend modifying the reconstruction scripts in `recOrder/examples/` to recreate and automate reconstructions. 
+A reconstruction can be performed with a single `reconstruct` call. For example:
+```
+recorder reconstruct data.zarr/0/0/0 -c config.yml -o reconstruction.zarr
+```
+Equivalently, a reconstruction can be performed with a `compute-tf` call followed by an `apply-inv-tf` call. For example:
+```
+recorder compute-tf data.zarr/0/0/0 -c config.yml -o tf.zarr
+recorder apply-inv-tf data.zarr/0/0/0 tf.zarr -c config.yml -o reconstruction.zarr
+```
+Computing the transfer function is typically the most expensive part of the reconstruction, so saving a transfer function then applying it to many datasets can save time. 
 
-Although these scripts are not as user friendly as a GUI+CLI solution, we are preparing for a much cleaner solution in `1.0.0`, and we appreciate your patience as we go through this change. We ask users who are having any difficulty with the scripts to [open an issue](https://github.com/mehta-lab/recOrder/issues/new/choose) or [send us an email](mailto:shalin.mehta@czbiohub.org,talon.chandler@czbiohub.org).
+## What types of reconstructions are supported?
+See `/recOrder/examples/` for a list of example configuration files. 
 
-`recOrder==1.0.0` will use a single mode to acquire and reconstruct the data. We are currently planning a refactor that will enable an "acquire once, quickly iterate your reconstruction" workflow, an "acquire now, reconstruct later" workflow, and a "live acquisition, live reconstruction" workflow, among others.  
+TODO: Expand this documentation...need docs for each reconstruction type and parameter.