WIP Update development.rst

README.rst

@@ -13,16 +13,3 @@ Build/Test:
 Test Coverage:
 
 .. image:: ../../coverage.svg
-
-#### Docker
-
-The Dockerfile is currently using,\
-- bioformats2raw version: 0.7.0.
-- imod version: 4.11.24
-
-In order to build the docker image, use `--platform linux/amd64` option.
-Explanation can be found [here](https://teams.microsoft.com/l/entity/com.microsoft.teamspace.tab.wiki/tab::5f55363b-bb53-4e5b-9564-8bed5289fdd5?context=%7B%22subEntityId%22%3A%22%7B%5C%22pageId%5C%22%3A15%2C%5C%22sectionId%5C%22%3A17%2C%5C%22origin%5C%22%3A2%7D%22%2C%22channelId%22%3A%2219%3A869be6677ee54848bc13f2066d847cc0%40thread.skype%22%7D&tenantId=14b77578-9773-42d5-8507-251ca2dc2b06)
-
-```bash
-$ docker build . -t hedwig_pipelines --platform linux/amd64
-```

docs/source/development.rst

@@ -50,15 +50,21 @@ pre-commit
 format compliance at commit time. It is configured via a top-level ``.pre-commit-config.yaml`` file that runs ``black``,
 ``Flake8`` and other checks.
 
+Docker
+======
+
+`Docker <https://www.docker.com/products/docker-desktop/>`_ is a platform that provides OS-level virtualization so that softwares are developed and packaged in containers. This helps consistent development in different environments.
+
 *******
 Testing
 *******
 
-There are currently four ``pytest`` files in the `test` directory:
+All ``pytest`` files reside in the `test` directory:
 
 - ``test_brt``: end-to-end test of batchruntomo pipeline
 - ``test_dm``: 2D end-to-end pipeline test
 - ``test_sem``: end-to-end test of FIBSEM pipeline
+- ``test_lrg_2d``: Large 2d pipeline test
 - ``test_utils``: unit tests of utils/utils.py module
 
 There is test data for `test_dm` in the Git repo, but not for the others. These files need to be
@@ -125,6 +131,32 @@ not have binaries built for the M1. Need to install using an x86_64 version of P
 are issues installing `biofomats2raw` as there is no OpenCV package for arm64 chip. There currently
 is no fix or workaround for this issue.
 
+Docker
+======
+
+You can also choose to use Docker for local development and testing.
+
+In order to build the docker image, use `--platform linux/amd64` option.
+Explanation can be found `here <https://teams.microsoft.com/l/entity/com.microsoft.teamspace.tab.wiki/tab::5f55363b-bb53-4e5b-9564-8bed5289fdd5?context=%7B%22subEntityId%22%3A%22%7B%5C%22pageId%5C%22%3A15%2C%5C%22sectionId%5C%22%3A17%2C%5C%22origin%5C%22%3A2%7D%22%2C%22channelId%22%3A%2219%3A869be6677ee54848bc13f2066d847cc0%40thread.skype%22%7D&tenantId=14b77578-9773-42d5-8507-251ca2dc2b06>`_
+
+The basic usage for testing would look like such:
+
+.. code-block:: sh
+
+   # To build the image
+   docker build . -t hedwig_pipelines --platform linux/amd64
+
+
+In order to boot up a container with pipeline image, you can run the command below. The command assumes that you are running the container from the project directory.
+
+.. code-block:: sh
+
+   # To run the container of that image
+   docker run -v "$(pwd):/image_portal_workflows" -e USER=root -it --rm hedwig_pipelines:latest
+
+
+Once you are in the container, you can run the commands you want to. For example: `pytest`.
+
 HPC Set up
 ==========
 

test/copy_test_data.sh

@@ -7,12 +7,12 @@
 #  - cd to top-level "image_portal_workflows" dir of your repo
 
 # test_brt - one 970MB file
-mkdir -p test/input_files/brt_inputs
-rsync --progress --stats -ave ssh ai-rmlsbatch1.niaid.nih.gov:/mnt/ai-fas12/RMLEMHedwigQA/test/input_files/brt_inputs/Projects/2013-1220-dA30_5-BSC-1_10.mrc test/input_files/brt_inputs/
+mkdir -p test/input_files/brt_inputs/Projects
+rsync --progress --stats -ave ssh ai-rmlsbatch1.niaid.nih.gov:/mnt/ai-fas12/RMLEMHedwigQA/test/input_files/brt_inputs/Projects/2013-1220-dA30_5-BSC-1_10.mrc test/input_files/brt_inputs/Projects
 
 # test_sem - directory of smallish files, but 1.1GB total
-mkdir -p test/input_files/sem_inputs
-rsync --progress --stats -ave ssh ai-rmlsbatch1.niaid.nih.gov:/mnt/ai-fas12/RMLEMHedwigDev/Projects/BCBB_TEST/test/input_files/sem_inputs/ test/input_files/sem_inputs/
+mkdir -p test/input_files/sem_inputs/Projects
+rsync --progress --stats -ave ssh ai-rmlsbatch1.niaid.nih.gov:/mnt/ai-fas12/RMLEMHedwigDev/Projects/BCBB_TEST/test/input_files/sem_inputs/1/ test/input_files/sem_inputs/Projects/
 
 # test_dm - These were in the repo as of March 2023, but uncomment to copy
 #rsync -ave ssh ai-rmlsbatch1.niaid.nih.gov:/mnt/ai-fas12/RMLEMHedwigQA/test/input_files/dm_inputs/Projects/Lab/PI/  test/input_files/dm_inputs/

test/test_utils.py

@@ -237,7 +237,6 @@ def test_mrc_to_movie(mock_nfs_mount):
     input_dir = "test/input_files/sem_inputs/Projects/mrc_movie_test"
     input_path = Path(os.path.join(proj_dir, input_dir))
     # FIXME input directory `sem_inputs` in test/input_files is missing
-    print(input_path)
     assert os.path.exists(input_path)
     # FIXME adjusted.mrc is missing
     image_path = Path(os.path.join(proj_dir, input_dir, "adjusted.mrc"))