Merge branch 'main' into fix/extra-parc-dim-in-merge

docs/api/nilearn.rst

@@ -1,8 +1,6 @@
 Nilearn
 =======
 
-This package provides re-implementations of some of the functions in `nilearn`_.
-
 .. automodule:: junifer.external.nilearn
    :members:
    :imported-members:

docs/api/onthefly.rst

@@ -1,8 +1,6 @@
 On-the-fly
 ==========
 
-This package provides functions for quick transform operations on stored data.
-
 .. automodule:: junifer.onthefly
    :members:
    :imported-members:

docs/builtin.rst

@@ -729,14 +729,6 @@ Available
      - | Compute a brain mask for the images by guessing the value of the
        | background from the border of the image.
        | See :func:`nilearn.masking.compute_background_mask`
-   * - ``nilearn``'s ICBM152 template gray-matter mask
-     - | ``fetch_icbm152_brain_gm_mask``
-     - ``MNI152NLin2009aAsym``
-     - 0.0.2
-     - | Compute a gray-matter mask from the asymmetrical ICBM152 2009 template,
-       | release a.
-       | See :func:`nilearn.datasets.fetch_icbm152_brain_gm_mask`
-
 
 ..
    Planned

docs/changes/newsfragments/115.doc

@@ -0,0 +1 @@
+Improve sub-package level docstrings to better reflect their purposes by `Synchon Mandal`_

docs/changes/newsfragments/323.feature

@@ -0,0 +1 @@
+Add ``junifer list-elements`` to list out available elements for a DataGrabber based on filtering via ``--element`` by `Synchon Mandal`_

docs/changes/newsfragments/330.doc

@@ -0,0 +1 @@
+Add missing DataGrabber ``types`` options in respective docstrings of :class:`.DataladAOMICID1000`, :class:`.DataladAOMICPIOP1`, :class:`.DataladAOMICPIOP2` and :class:`.DMCC13Benchmark` by `Synchon Mandal`_

docs/changes/newsfragments/332.bugfix

@@ -0,0 +1 @@
+Fix ``junifer reset`` to properly delete storage directory and handle ``junifer_jobs`` deletion if not empty by `Synchon Mandal`_

docs/changes/newsfragments/336.removal

@@ -0,0 +1 @@
+Remove ``Power`` coordinates, ``fetch_icbm152_brain_gm_mask`` mask, ``BOLDWarper`` Preprocessor by `Synchon Mandal`_

docs/whats_new.rst

@@ -79,7 +79,7 @@ API Changes
 - Add a positional argument ``using`` for Markers and Preprocessors having
   implementation-based variations, in particular :class:`.ReHoParcels`,
   :class:`.ReHoSpheres`, :class:`.ALFFParcels`, :class:`.ALFFSpheres` and
-  :class:`.BOLDWarper` by `Synchon Mandal`_ (:gh:`311`)
+  ``BOLDWarper`` by `Synchon Mandal`_ (:gh:`311`)
 - Change all ``probseg_`` types to ``VBM_`` types by `Fede Raimondo`_
   (:gh:`320`)
 - Change the subject and session patterns for :class:`.DataladAOMICID1000`,
@@ -120,13 +120,13 @@ Enhancements
   coordinates by `Synchon Mandal`_ and `Fede Raimondo`_ (:gh:`268`)
 - Add ``mode`` as an aggregation function option in
   :func:`.get_aggfunc_by_name` by `Synchon Mandal`_ (:gh:`287`)
-- Adapt :class:`.BOLDWarper` to use FSL or ANTs depending on warp file
+- Adapt ``BOLDWarper`` to use FSL or ANTs depending on warp file
   extension by `Synchon Mandal`_ (:gh:`293`)
 - Rewrite :func:`.compute_brain_mask` to allow variable template fetching via
   templateflow, according to target data by `Synchon Mandal`_ (:gh:`299`)
 - Replace ``requests`` with ``httpx`` for fetching parcellations by `Synchon
   Mandal`_ (:gh:`300`)
-- Allow :class:`.BOLDWarper` to warp BOLD data to other MNI spaces by `Synchon
+- Allow ``BOLDWarper`` to warp BOLD data to other MNI spaces by `Synchon
   Mandal`_ (:gh:`302`)
 - Add support for local ``junifer queue`` via GNU Parallel by `Synchon Mandal`_
   (:gh:`306`)
@@ -168,7 +168,7 @@ Features
   by `Synchon Mandal`_ (:gh:`265`)
 - Introduce ``junifer.preprocess.fsl.apply_warper._ApplyWarper`` to wrap FSL's
   ``applywarp`` by `Synchon Mandal`_ (:gh:`266`)
-- Introduce :class:`.BOLDWarper` for warping BOLD data via FSL's ``applywarp``
+- Introduce ``BOLDWarper`` for warping BOLD data via FSL's ``applywarp``
   by `Synchon Mandal`_ (:gh:`267`)
 - Introduce :class:`.DMCC13Benchmark` to access `DMCC13benchmark dataset
   <https://openneuro.org/datasets/ds003452/versions/1.0.1>`_ by `Synchon
@@ -216,7 +216,7 @@ Miscellaneous
   `Synchon Mandal`_ (:gh:`279`)
 - Add support for accessing FSL ``img2imgcoord`` via Docker wrapper command by
   `Synchon Mandal`_ (:gh:`281`)
-- Make :class:`.BOLDWarper` tool-agnostic by moving it from
+- Make ``BOLDWarper`` tool-agnostic by moving it from
   ``junifer.preprocess.fsl`` to :mod:`junifer.preprocess` by `Synchon Mandal`_
   (:gh:`288`)
 - Add support for accessing ANTs' ``ResampleImage`` via Docker wrapper by
@@ -232,7 +232,7 @@ Miscellaneous
 Deprecations and Removals
 ^^^^^^^^^^^^^^^^^^^^^^^^^
 
-- Deprecate :class:`.BOLDWarper` and mark for removal in v0.0.4 by `Synchon
+- Deprecate ``BOLDWarper`` and mark for removal in v0.0.4 by `Synchon
   Mandal`_ (:gh:`301`)
 
 

junifer/api/__init__.py

@@ -1,4 +1,4 @@
-"""Provide imports for api sub-package."""
+"""Public API and CLI components."""
 
 # Authors: Federico Raimondo <f.raimondo@fz-juelich.de>
 #          Synchon Mandal <s.mandal@fz-juelich.de>

junifer/api/cli.py

@@ -8,7 +8,7 @@
 import subprocess
 import sys
 from pathlib import Path
-from typing import Dict, List, Tuple, Union
+from typing import Dict, List, Optional, Tuple, Union
 
 import click
 import pandas as pd
@@ -20,6 +20,7 @@
     warn_with_log,
 )
 from .functions import collect as api_collect
+from .functions import list_elements as api_list_elements
 from .functions import queue as api_queue
 from .functions import reset as api_reset
 from .functions import run as api_run
@@ -451,6 +452,69 @@ def reset(
     api_reset(config)
 
 
+@cli.command()
+@click.argument(
+    "filepath",
+    type=click.Path(
+        exists=True, readable=True, dir_okay=False, path_type=pathlib.Path
+    ),
+)
+@click.option("--element", type=str, multiple=True)
+@click.option(
+    "-o",
+    "--output-file",
+    type=click.Path(dir_okay=False, writable=True, path_type=pathlib.Path),
+)
+@click.option(
+    "-v",
+    "--verbose",
+    type=click.UNPROCESSED,
+    callback=_validate_verbose,
+    default="info",
+)
+def list_elements(
+    filepath: click.Path,
+    element: Tuple[str],
+    output_file: Optional[click.Path],
+    verbose: Union[str, int],
+) -> None:
+    """Element listing command for CLI.
+
+    \f
+
+    Parameters
+    ----------
+    filepath : click.Path
+        The filepath to the configuration file.
+    element : tuple of str
+        The element to operate on.
+    output_file : click.Path or None
+        The path to write the output to. If not None, writing to
+        stdout is not performed.
+    verbose : click.Choice
+        The verbosity level: warning, info or debug (default "info").
+
+    """
+    configure_logging(level=verbose)
+    # Parse YAML
+    config = parse_yaml(filepath)  # type: ignore
+    # Fetch datagrabber
+    datagrabber = config["datagrabber"]
+    # Parse elements
+    elements = _parse_elements(element, config)
+    # Perform operation
+    listed_elements = api_list_elements(
+        datagrabber=datagrabber,
+        elements=elements,
+    )
+    # Check if output file is provided
+    if output_file is not None:
+        output_file.touch()
+        output_file.write_text(listed_elements)
+    else:
+        click.secho(listed_elements, fg="blue")
+
+
 @cli.group()
 def setup() -> None:  # pragma: no cover
     """Configure commands for Junifer."""

junifer/api/functions.py

@@ -5,6 +5,7 @@
 #          Synchon Mandal <s.mandal@fz-juelich.de>
 # License: AGPL
 
+import os
 import shutil
 import typing
 from pathlib import Path
@@ -339,12 +340,12 @@ def reset(config: Dict) -> None:
     storage = config["storage"]
     storage_uri = Path(storage["uri"])
     logger.info(f"Deleting {storage_uri.resolve()!s}")
-    # Delete storage; will be str
+    # Delete storage
     if storage_uri.exists():
-        # Delete files in the directory
-        for file in storage_uri.iterdir():
+        # Delete files in the job storage directory
+        for file in storage_uri.parent.iterdir():
             file.unlink(missing_ok=True)
-        # Remove directory
+        # Remove job storage directory
         storage_uri.parent.rmdir()
 
     # Fetch job name (if present)
@@ -359,5 +360,47 @@ def reset(config: Dict) -> None:
         if job_dir.exists():
             # Remove files and directories
             shutil.rmtree(job_dir)
-            # Remove directory
-            job_dir.parent.rmdir()
+            # Remove parent directory (if empty)
+            if not next(os.scandir(job_dir.parent), None):
+                job_dir.parent.rmdir()        
+
+
+def list_elements(
+    datagrabber: Dict,
+    elements: Union[str, List[Union[str, Tuple]], Tuple, None] = None,
+) -> str:
+    """List elements of the datagrabber filtered using `elements`.
+
+    Parameters
+    ----------
+    datagrabber : dict
+        DataGrabber to index. Must have a key ``kind`` with the kind of
+        DataGrabber to use. All other keys are passed to the DataGrabber
+        constructor.
+    elements : str or tuple or list of str or tuple, optional
+        Element(s) to filter using. Will be used to index the DataGrabber
+        (default None).
+
+    """
+    # Get datagrabber to use
+    datagrabber_object = _get_datagrabber(datagrabber)
+
+    # Fetch elements
+    raw_elements_to_list = []
+    with datagrabber_object:
+        if elements is not None:
+            for element in datagrabber_object.filter(elements):
+                raw_elements_to_list.append(element)
+        else:
+            for element in datagrabber_object:
+                raw_elements_to_list.append(element)
+
+    elements_to_list = []
+    for element in raw_elements_to_list:
+        # Stringify elements if tuple for operation
+        str_element = (
+            ",".join(element) if isinstance(element, tuple) else element
+        )
+        elements_to_list.append(str_element)
+
+    return "\n".join(elements_to_list)

junifer/api/queue_context/__init__.py

@@ -1,4 +1,4 @@
-"""Provide imports for queue context sub-package."""
+"""Context adapters for queueing."""
 
 # Authors: Synchon Mandal <s.mandal@fz-juelich.de>
 # License: AGPL

junifer/api/tests/test_cli.py

@@ -14,6 +14,7 @@
 from junifer.api.cli import (
     _parse_elements_file,
     collect,
+    list_elements,
     queue,
     reset,
     run,
@@ -297,6 +298,88 @@ def test_reset(
         assert reset_result.exit_code == 0
 
 
+@pytest.mark.parametrize(
+    "elements",
+    [
+        ("sub-01", "sub-02"),
+        ("sub-03", "sub-04"),
+    ],
+)
+def test_list_elements_stdout(
+    elements: Tuple[str, ...],
+) -> None:
+    """Test elements listing to stdout.
+
+    Parameters
+    ----------
+    elements : tuple of str
+        The parametrized elements for filtering.
+
+    """
+    # Get test config
+    infile = Path(__file__).parent / "data" / "partly_cloudy_agg_mean_tian.yml"
+    # List elements command arguments
+    list_elements_args = [
+        str(infile.absolute()),
+        "--verbose",
+        "debug",
+        "--element",
+        elements[0],
+        "--element",
+        elements[1],
+    ]
+    # Invoke list elements command
+    list_elements_result = runner.invoke(list_elements, list_elements_args)
+    # Check
+    assert list_elements_result.exit_code == 0
+    assert f"{elements[0]}\n{elements[1]}" in list_elements_result.stdout
+
+
+@pytest.mark.parametrize(
+    "elements",
+    [
+        ("sub-01", "sub-02"),
+        ("sub-03", "sub-04"),
+    ],
+)
+def test_list_elements_output_file(
+    tmp_path: Path,
+    elements: Tuple[str, ...],
+) -> None:
+    """Test elements listing to output file.
+
+    Parameters
+    ----------
+    tmp_path : pathlib.Path
+        The path to the test directory.
+    elements : tuple of str
+        The parametrized elements for filtering.
+
+    """
+    # Get test config
+    infile = Path(__file__).parent / "data" / "partly_cloudy_agg_mean_tian.yml"
+    # Output file
+    output_file = tmp_path / "elements.txt"
+    # List elements command arguments
+    list_elements_args = [
+        str(infile.absolute()),
+        "--verbose",
+        "debug",
+        "--element",
+        elements[0],
+        "--element",
+        elements[1],
+        "--output-file",
+        str(output_file.resolve()),
+    ]
+    # Invoke list elements command
+    list_elements_result = runner.invoke(list_elements, list_elements_args)
+    # Check
+    assert list_elements_result.exit_code == 0
+    with open(output_file) as f:
+        assert f"{elements[0]}\n{elements[1]}" == f.read()
+
+
 def test_wtf_short() -> None:
     """Test short version of wtf command."""
     # Invoke wtf command