Spring cleaning for dependencies, Pt. 2: The Tree-Chopping

.github/workflows/integration.yml

@@ -214,7 +214,8 @@ jobs:
       matrix:
         install-mode:
           - pip-default
-          - pip-optional
+          - pip-complete
+          - pip-all
           - conda-like
         os:
           - linux
@@ -223,8 +224,10 @@ jobs:
             runner-image: ubuntu-20.04
           - install-mode: pip-default
             pip-install-target: '.'
-          - install-mode: pip-optional
-            pip-install-target: .[optional]
+          - install-mode: pip-complete
+            pip-install-target: .[ui,dmf,grid]
+          - install-mode: pip-all
+            pip-install-target: .[ui,dmf,grid,omlt,coolprop]
           - install-mode: conda-like
             pip-install-target: '.'
             dependencies-to-uninstall: omlt

README.md

@@ -33,11 +33,17 @@ Now, in that "idaes-pse" environment, install the IDAES Toolkit using either `pi
 
 ```bash
 # install latest stable release
-pip install idaes_pse
+pip install idaes-pse
+# install latest stable release with one set of optional dependencies, e.g. `ui` for the user interface
+pip install "idaes-pse[ui]"
+# install latest stable release with multiple sets of optional dependencies
+pip install "idaes-pse[ui,dmf,omlt,grid,coolprop]"
 # install latest version from the main branch of this repository
-pip install 'idaes-pse[prerelease] @ https://github.com/IDAES/idaes-pse/archive/main.zip'
+pip install "idaes-pse @ git+https://github.com/IDAES/idaes-pse@main"
+# install from the `mybranch` branch of the fork belonging to `myuser`
+pip install "idaes-pse @ git+https://github.com/myuser/idaes-pse@mybranch"
 ```
-.
+
 You can check the version installed with the command:
 
 ```bash

docs/explanations/modeling_extensions/surrogate/api/omlt-keras/index.rst

@@ -1,3 +1,5 @@
+.. _omlt:
+
 OMLT: Optimization and Machine Learning Toolkit
 ===============================================
 

docs/how_to_guides/index.rst

@@ -8,4 +8,5 @@ How-To-Guides
     vis/index
     workflow/index
     data_management_framework/index
+    opt_dependencies
     versioned_idaes_install

docs/how_to_guides/opt_dependencies.rst

@@ -0,0 +1,52 @@
+Installing optional dependencies
+================================
+
+Depending on the installation method (Conda, pip) and the IDAES version, not all of IDAES's dependencies will be installed by default.
+
+Installing these *optional dependencies* might require additional steps, described below, in addition to the default installation steps described elsewhere in the documentation.
+
+For pip installations
+^^^^^^^^^^^^^^^^^^^^^
+
+.. important:: Users who installed IDAES by running ``pip install idaes-pse`` should follow these steps.
+
+When installing IDAES using pip, optional dependencies can be installed by specifying one or more "``extras_require`` targets":
+
+.. code-block:: bash
+
+   # install the `ui` target
+   pip install "idaes-pse[ui]"
+
+   # install the `ui` and `dmf` targets
+   pip install "idaes-pse[ui,dmf]"
+
+.. important:: The ``pip install`` argument should be wrapped in double quotes (``"``) when it contains square brackets. Otherwise, it might cause an error to be reported by the shell/command interface used to invoke the command.
+
+Available optional dependencies targets
+---------------------------------------
+
+As of IDAES 2.1, the following ``extras_require`` targets are available:
+
+* ``ui``: for the :ref:`IDAES Flowsheet Visualizer <IFV>`
+* ``dmf``: for the :ref:`Data Management Framework <dmf-overview>`
+* ``grid``: for the :ref:`IDAES Grid integration <idaes-grid>`
+* ``omlt``: for the :ref:`OMLT integration <omlt>`
+* ``coolprop``: for the :py:mod:`idaes.models.properties.modular_properties.coolprop` property package
+
+Specifying optional dependencies when installing a prerelease version of IDAES
+------------------------------------------------------------------------------
+
+.. code-block:: bash
+
+   pip install "idaes-pse[ui,dmf] @ git+https://github.com/IDAES/idaes-pse@main"
+
+For Conda installations
+^^^^^^^^^^^^^^^^^^^^^^^
+
+.. important:: These steps apply to users who installed IDAES by running ``conda install <...>``. They **do not apply** if IDAES was installed using pip.
+
+As of IDAES 2.1, the IDAES Conda package available already includes several dependencies that are optional for pip installations.
+This difference is due to a variety of reasons, including the fact that not all optional dependencies are available as Conda packages (e.g. OMLT); or that Conda does not have an equivalent of ``extras_require`` targets to specify optional dependencies at install-time.
+The consequence of this is that some optional dependencies might be unavailable if IDAES was installed using Conda.
+
+Therefore, we encourage users who encounter issues after a Conda installation of IDAES to **try installing IDAES again using pip instead** and see if the issues are resolved.

docs/how_to_guides/versioned_idaes_install.rst

@@ -37,16 +37,16 @@ Installing IDAES
 ----------------
 .. _updating_install:
 
-* To get a previous `IDAES release <https://github.com/IDAES/idaes-pse/releases>`_, for example v1.13 ::
+* To get a previous `IDAES release <https://github.com/IDAES/idaes-pse/releases>`_, for example 2.0 ::
 
-   pip install idaes-pse==1.13
+   pip install idaes-pse==2.0
 
 * To get the latest version from the GitHub main branch ::
 
-   pip install 'idaes-pse[prerelease] @ https://github.com/IDAES/idaes-pse/archive/main.zip'
+   pip install "idaes-pse @ git+https://github.com/IDAES/idaes-pse@main"
 
-* To get a specific fork or branch, for example myfork (of idaes-pse) and mybranch ::
+* To get a specific fork or branch, for example for branch ``mybranch`` of the fork belonging to ``myuser`` ::
 
-   pip install 'idaes-pse[prerelease] @ https://github.com/myfork/idaes-pse/archive/mybranch.zip'
+   pip install "idaes-pse @ git+https://github.com/myuser/idaes-pse@mybranch"
 
 * **For IDAES Contributors**: follow the :ref:`advanced user installation<tutorials/advanced_install/index:Advanced User Installation>`.

docs/reference_guides/apps/grid_integration/index.rst

@@ -1,3 +1,5 @@
+.. _idaes-grid:
+
 .. grid_integration documentation master file, created by
    sphinx-quickstart on Wed Dec 15 17:14:39 2021.
    You can adapt this file completely to your liking, but it should at least

docs/tutorials/getting_started/index.rst

@@ -28,7 +28,6 @@ OS Specific Instructions
     windows
     mac_osx
     binaries
-    opt_dependencies
 
 .. list-table::
    :header-rows: 1

idaes/commands/__init__.py

@@ -23,10 +23,13 @@
 from idaes.commands.base import command_base as cb
 
 # import all the commands
-for loader, module_name, is_pkg in pkgutil.walk_packages(__path__):
-    if module_name == "base":
+for loader, dotted_module_name, is_pkg in pkgutil.walk_packages(__path__):
+    module_name_parts = dotted_module_name.split(".")
+    module_name = module_name_parts[-1]
+    is_test_module = module_name.startswith("test_")
+    if dotted_module_name == "base":
         pass
-    elif module_name.startswith("test"):
+    elif is_test_module:
         pass
     else:
         loader.find_module(module_name).load_module(module_name)

idaes/conftest.py

@@ -14,7 +14,13 @@
 # pylint: disable=missing-module-docstring
 # pylint: disable=missing-function-docstring
 
+import importlib.abc
+import importlib.machinery
 import sys
+from typing import Dict
+from typing import Iterable
+from typing import List
+
 import pytest
 
 ####
@@ -133,4 +139,106 @@ def pytest_configure(config):
         setattr(config.option, "markexpr", "performance")
 
 
+ModuleName = str
+
+
+class ImportorskipLoader(importlib.abc.Loader):
+    """
+    A wrapper class around a concrete Loader instance. If a ModuleNotFoundError is raised
+    during module execution and the module name matches one of the registered modules,
+    it is replaced with a module-level call to :func:`pyest.skip()`.
+    """
+
+    def __init__(
+        self, wrapped: importlib.abc.Loader, skip_if_not_found: Iterable[ModuleName]
+    ):
+        self._wrapped = wrapped
+        self.skip_if_not_found = list(skip_if_not_found)
+
+    def module_repr(self, module) -> str:
+        return self._wrapped.module_repr(module)
+
+    def create_module(self, spec):
+        return self._wrapped.create_module(spec)
+
+    def exec_module(self, module):
+        try:
+            return self._wrapped.exec_module(module)
+        except ModuleNotFoundError as e:
+            if e.name in self.skip_if_not_found:
+                pytest.skip(allow_module_level=True)
+            raise e
+
+
+class ImportorskipFinder(importlib.abc.MetaPathFinder):
+    """
+    Custom Finder class to modify import behavior for registered modules.
+
+    If inserted in sys.meta_path before the default finders, it will cause
+    a custom Loader to be used for registered modules.
+    """
+
+    def __init__(self, registry: Dict[ModuleName, List[ModuleName]]):
+        self._registry = registry
+
+    def find_spec(self, *args, **kwargs):
+        spec = importlib.machinery.PathFinder.find_spec(*args, **kwargs)
+        if spec is None:
+            return
+        registered_for_skipping = self._registry.get(spec.name, None)
+        if registered_for_skipping:
+            spec.loader = ImportorskipLoader(spec.loader, registered_for_skipping)
+        return spec
+
+
+class Importorskipper:
+    """
+    A pytest plugin that allows automatically skipping test modules if they attempt to import
+    modules distributed in optional dependencies.
+
+    The functionality is similar to pytest.importorskip(), but using a centralized registry
+    instead of having to call importorskip() in each test module for each possibly missing module.
+    """
+
+    def __init__(self, registry: Dict[ModuleName, List[ModuleName]]):
+        self._registry = dict(registry)
+        self._finder = ImportorskipFinder(self._registry)
+
+    def pytest_configure(self):
+        sys.meta_path.insert(0, self._finder)
+
+    def pytest_sessionfinish(self):
+        sys.meta_path.remove(self._finder)
+
+    def pytest_report_collectionfinish(self) -> List[str]:
+        preamble = [
+            "The following modules are registered in the importorskipper plugin",
+            " and will cause tests to be skipped if any of the registered modules is not found: ",
+        ]
+        lines = []
+        for importing_mod, mods in self._registry.items():
+            lines.append(f"- {importing_mod}:\t{mods}")
+        if lines:
+            lines = preamble + lines
+        return lines
+
+
+def pytest_addhooks(pluginmanager: pytest.PytestPluginManager):
+    skipper_plugin = Importorskipper(
+        {
+            "idaes.core.dmf": ["traitlets", "tinydb"],
+            # idaes.tests.test_import must be specified instead of idaes.core.dmf.util
+            # since colorama is not imported at the module level in idaes.core.dmf.util,
+            # but it is imported at the module level in idaes.tests.test_import
+            # when instantiating ColorTerm()
+            "idaes.tests.test_import": ["colorama"],
+            "idaes.core.surrogate.keras_surrogate": ["omlt"],
+            "idaes.core.ui.fsvis.tests.test_fsvis": ["requests"],
+            "idaes.core.ui.fsvis.tests.test_model_server": ["requests"],
+        }
+    )
+
+    pluginmanager.register(skipper_plugin, name="importorskipper")
+
+
 ####

idaes/core/dmf/util.py

@@ -32,8 +32,6 @@
 from typing import Union
 import yaml
 
-# third-party
-import colorama
 
 __author__ = "Dan Gunter"
 
@@ -268,6 +266,8 @@ def __getattr__(self, a):
     def __init__(self, enabled=True):
         self._width = None
         if enabled:
+            import colorama  # pylint: disable=import-outside-toplevel
+
             colorama.init(autoreset=True)
             # Colorama colors and styles
             F = self.Fore = colorama.Fore

idaes/core/ui/fsvis/tests/test_model_server.py

@@ -18,6 +18,8 @@
 import pytest
 from pyomo.environ import ConcreteModel
 
+import requests
+
 # pkg
 from idaes.core.ui.fsvis import model_server, errors, persist
 from idaes.core import FlowsheetBlock
@@ -86,7 +88,6 @@ def flash_model():
 
 @pytest.mark.component
 def test_flowsheet_server_run(flash_model):
-    import requests
 
     srv = model_server.FlowsheetServer()
     srv.start()

idaes/core/util/convergence/convergence_base.py

@@ -82,7 +82,6 @@ class from ConvergenceEvaluation, and implement three methods:
 
 # idaes
 import idaes.core.util.convergence.mpi_utils as mpiu
-from idaes.core.dmf import resource
 import idaes.logger as idaeslog
 from idaes.core.solvers import get_solver
 
@@ -934,7 +933,15 @@ def to_dict(self):
     def to_json(self, fp):
         json.dump(self.to_dict(), fp, indent=4)
 
-    def to_dmf(self, dmf):
+    def to_dmf(self, dmf) -> None:
+        try:
+            # pylint: disable-next=import-outside-toplevel
+            from idaes.core.dmf import resource
+        except ImportError as err:
+            _log.error(
+                "Stats.to_dmf() failed because DMF is not available: %s", str(err)
+            )
+            return None
         # PYLINT-TODO-FIX fix error due to undefined variable "stats"
         rsrc = resource.Resource(
             value={

idaes/tests/test_headers.py

@@ -19,8 +19,8 @@
 
 # third-party
 import pytest
-import yaml
 
+yaml = pytest.importorskip("yaml", reason="`yaml` package is not available")
 
 addheader_add = pytest.importorskip(
     "addheader.add", reason="`addheader` package is not available"

pytest.ini

@@ -1,7 +1,7 @@
 [pytest]
-addopts = -W ignore
+addopts = --pyargs idaes
           --durations=100
-testpaths = idaes
+          -W ignore
 log_file = pytest.log
 log_file_date_format = %Y-%m-%dT%H:%M:%S
 log_file_format = %(asctime)s %(levelname)-7s <%(filename)s:%(lineno)d> %(message)s

requirements-dev.txt

@@ -31,6 +31,6 @@ jupyter_contrib_nbextensions
 snowballstemmer==1.2.1
 addheader>=0.2.2
 
-# this will install IDAES in editable mode using the dependencies defined under the `prerelease` tag of `extras_require` in `setup.py`
-# to customize this (e.g. to install a local clone of the Pyomo git repository), install IDAES without the `prerelease` tag and then install the dependencies separately
---editable .[prerelease,optional]
+# this will install IDAES in editable mode using the dependencies defined under the `extras_require` tags defined in `setup.py`
+--editable .[ui,dmf,grid,omlt,coolprop]
+# to customize this (e.g. to install a local clone of the Pyomo git repository), add the desired alternate requirements below