Merge pull request #38 from eX-Mech/enh/xarray-backend

Xarray Backend

.github/workflows/tests.yml

@@ -2,9 +2,9 @@ name: tests
 
 on:
   push:
-    branches: [ master ]
+    branches: [ main ]
   pull_request:
-    branches: [ master ]
+    branches: [ main ]
 
 env:
   PIP_CACHE_DIR: ~/.cache/pip
@@ -21,6 +21,8 @@ jobs:
     steps:
     - name: Checkout
       uses: actions/checkout@v2
+      with:
+        submodules: recursive
 
     - name: Set up Python ${{ matrix.python-version }}
       uses: actions/setup-python@v2
@@ -46,7 +48,7 @@ jobs:
         pip install -e .[tests]
 
     - name: Run tests
-      run: pytest -s -v -n 4
+      run: pytest -s -v -n auto
 
     - name: Upload coverage
       if: ${{ success() && matrix.python-version == '3.7' }}
@@ -75,6 +77,8 @@ jobs:
             miniforge-variant: Mambaforge
     steps:
     - uses: actions/checkout@v2
+      with:
+        submodules: recursive
 
     - name: Cache conda
       uses: actions/cache@v2
@@ -107,4 +111,4 @@ jobs:
     - name: Run tests
       shell: bash -l {0}
       run: |
-        python -m pytest -s -v -n 4
+        python -m pytest -s -v -n auto

.gitmodules

@@ -0,0 +1,3 @@
+[submodule "data"]
+	path = tests/data
+	url = https://github.com/eX-Mech/pymech-test-data.git

CHANGELOG.md

@@ -32,17 +32,23 @@ Security in case of vulnerabilities.
 - Dataclass {class}`pymech.neksuite.Header` using [`pydantic`](https://pydantic-docs.helpmanual.io/) for reading file headers
 - Function {func}`pymech.neksuite.read_header`
 - Module {mod}`pymech.meshtools` for manipulating meshes
+- Xarray backend ``pymech``. See {ref}`dataset`
 
 ### Changed
 
 - Attributes `lr1` and `var` {class}`pymech.core.Elem` are tuples instead of lists
-- Attributes of {class}`pymech.core.DataLims` are now immutable tuples.
+- Attributes of {class}`pymech.core.DataLims` are now immutable tuples
+- {func}`pymech.dataset.open_mfdataset` is now a partial function. This change should be fully backwards compatible
 
 ### Deprecated
 
 - Module {ref}`exadata` is deprecated in favour of {mod}`pymech.core`.
   See {ref}`exadata` for more information on migrating your code.
 
+### Removed
+
+- Test data files from the ``pymech`` git repository. The files are now available at <https://github.com/eX-Mech/pymech-test-data/>
+
 ## [1.4.1] - 2021-05-07
 
 Backwards compatible release with a lot of housekeeping and some usability

README.md

@@ -17,7 +17,7 @@ Read the full documentation at [Pymech doc](http://pymech.readthedocs.io/en/stab
 For some quick wins, download some sample data
 
 ```sh
-curl -LO https://github.com/eX-Mech/pymech/raw/master/tests/nek/channel3D_0.f00001
+curl -fLO https://raw.githubusercontent.com/eX-Mech/pymech-test-data/main/nek/channel3D_0.f00001
 ```
 
 Fire up a Python / IPython console and execute:

docs/Makefile

@@ -19,5 +19,14 @@ help:
 %: Makefile
 	@$(SPHINXBUILD) -M $@ "$(SOURCEDIR)" "$(BUILDDIR)" $(SPHINXOPTS) $(O)
 
+intersphinx-python:
+	@python -msphinx.ext.intersphinx https://docs.python.org/3/objects.inv
+
+intersphinx-nek:
+	@python -msphinx.ext.intersphinx https://nek5000.github.io/NekDoc/objects.inv
+
+intersphinx-xr:
+	@python -msphinx.ext.intersphinx https://xarray.pydata.org/en/stable/objects.inv
+
 autobuild:
 	@sphinx-autobuild "$(SOURCEDIR)" "$(BUILDDIR)/html"

docs/conf.py

@@ -56,6 +56,7 @@
     "sphinx.ext.autodoc",
     # 'sphinx.ext.autosummary',
     # "sphinx.ext.doctest",
+    "sphinx.ext.intersphinx",
     "sphinx.ext.todo",
     #  "sphinx.ext.coverage",
     #  "sphinx.ext.mathjax",
@@ -105,6 +106,14 @@
 
 autodoc_mock_imports = ["tvtk", "pymech._version"]
 
+# -- Options for Intersphinx -------------------------------------------------
+
+intersphinx_mapping = dict(
+    python=("https://docs.python.org/3", None),
+    nek=("https://nek5000.github.io/NekDoc", None),
+    xr=("https://xarray.pydata.org/en/stable/", None),
+)
+
 # -- MyST options ------------------------------------------------------------
 
 myst_heading_anchors = 2

docs/core.rst

@@ -23,10 +23,12 @@ element(s).  The general information is constituted by:
 ``time``     float                          simulation time of the file
 ``istep``    integer                        simulation time step of the file
 ``wdsz``     integer                        word size, i.e. double or single precision file
-``endian``   string                         endianness of the file (little/big)
+``endian``   string                         endianness_ of the file (little/big)
 ``lims``     :ref:`datalims`                extrema of all quantities stored
 ==========   ============================   ===============================================
 
+.. _endianness: https://wizardzines.com/comics/little-endian/
+
 The element by element data is stored in:
 
 ==========   ============================   ===============================================

docs/dataset.rst

@@ -3,6 +3,35 @@
 pymech.dataset
 ==============
 
+Interface for reading files as xarray_ datasets.
+
+.. _xarray: https://xarray.pydata.org
+
+Installing ``pymech`` also registers as a ``xarray`` backend. This means
+in addition to :func:`pymech.dataset.open_dataset`, a file can be directly opened via
+:func:`xarray.open_dataset` as follows:
+
+>>> import xarray as xr
+>>> xr.open_dataset("case0.f00001")  # let xarray choose the backend / engine
+>>> xr.open_dataset("case0.f00001", engine="pymech")  # or explicitly mention the *engine*
+
+.. seealso::
+
+    `The backend API of xarray
+    <https://xarray.pydata.org/en/stable/internals/how-to-add-new-backend.html>`__
+    and the implementation :class:`PymechXarrayBackend` (:ref:`internals`)
+
+.. todo::
+
+    Opening as a object is not supported by :func:`pymech.neksuite.readnek`
+
+
+The module also provides :func:`pymech.dataset.open_mfdataset` to open multiple
+files and merge into a single dataset. This is a wrapper around
+:func:`xarray.open_mfdataset`, with sane defaults such as merge along ``time``
+dimension and combine using :func:`xarray.combine_nested`.
+
+
 Contents of dataset.py
 ----------------------
 

docs/dev.md

@@ -9,10 +9,10 @@ Ready to contribute? Here's how to set up Pymech for local development.
 1. Fork the `pymech` repo on GitHub.
 2. Clone your fork locally:
     ```{tab} HTTPS
-        git clone https://github.com/your_name_here/pymech.git
+        git clone --recursive https://github.com/your_name_here/pymech.git
     ```
     ```{tab} SSH
-        git clone git@github.com:your_name_here/pymech.git
+        git clone --recursive git@github.com:your_name_here/pymech.git
     ```
 3. To set up for local development, first, create a [virtual
    environment](https://packaging.python.org/guides/installing-using-pip-and-virtual-environments):

docs/internals.rst

@@ -11,7 +11,7 @@ Internals
 
 
 :mod:`pymech.neksuite`
----------------------
+----------------------
 
 .. automodule:: pymech.neksuite
    :noindex:
@@ -24,5 +24,5 @@ Internals
 .. automodule:: pymech.dataset
    :noindex:
    :no-members:
-   :private-members: _open_nek_dataset, _NekDataStore
+   :private-members:  can_open_nek_dataset, _open_nek_dataset, _NekDataStore, PymechXarrayBackend
 

docs/release.rst

@@ -12,7 +12,7 @@ Release checklist for maintainers
 
 - Ensure tests pass locally and on CI::
 
-      pytest --cov --numprocesses=4
+      pytest --numprocesses=auto
 
 - Compile documentation::
 
@@ -49,7 +49,7 @@ Release checklist for maintainers
 
 - Upload to repository::
 
-      git push --tags && git push
+      git push --atomic origin main $VERSION
 
 .. _twine: https://twine.readthedocs.io/en/latest/
 .. _TestPyPI: https://packaging.python.org/guides/using-testpypi/

docs/tests.rst

@@ -10,6 +10,9 @@ the tests::
     pip install -e '.[tests]'
     pytest -s -v
 
+The directory `tests/data` is a git submodule repository
+(https://github.com/eX-Mech/pymech-test-data.git), which houses all the data
+files required to run the test suite.
 
 Continuous Integration
 ----------------------

docs/usage.ipynb

@@ -36,13 +36,13 @@
       "remote: Total 1322 (delta 187), reused 253 (delta 132), pack-reused 1001\u001b[K\n",
       "Receiving objects: 100% (1322/1322), 12.38 MiB | 8.30 MiB/s, done.\n",
       "Resolving deltas: 100% (806/806), done.\n",
-      "/home/avmo/src/snek5000/pymech/docs/pymech/tests/nek\n"
+      "/home/avmo/src/snek5000/pymech/docs/pymech/tests/data/nek\n"
      ]
     }
    ],
    "source": [
-    "!git clone https://github.com/eX-Mech/pymech.git\n",
-    "%cd pymech/tests/nek"
+    "!git clone --recursive https://github.com/eX-Mech/pymech.git\n",
+    "%cd pymech/tests/data/nek"
    ]
   },
   {
@@ -158,18 +158,7 @@
    "cell_type": "markdown",
    "metadata": {},
    "source": [
-    "Majority of the attributes store metadata, such as:\n",
-    "\n",
-    "- endian\n",
-    "- iteration\n",
-    "- element shape / polynomial order\n",
-    "- boundary conditions\n",
-    "- curvature\n",
-    "- number of dimensions\n",
-    "- number of elements\n",
-    "- time\n",
-    "- number of variables in (X: positions, U: velocities, P: pressure, T: temperature, S: scalars)\n",
-    "- word size / precision"
+    "Here `check_connectivity` and `merge` are methods, and `elem` and `elmap` are data objects. The rest of the attributes store metadata. See {ref}`core` to understand what they imply."
    ]
   },
   {
@@ -2059,7 +2048,7 @@
    "name": "python",
    "nbconvert_exporter": "python",
    "pygments_lexer": "ipython3",
-   "version": "3.8.5"
+   "version": "3.8.9"
   }
  },
  "nbformat": 4,