Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

resolve problems with unit test workflow #317

Merged
merged 40 commits into from
Jan 12, 2024
Merged
Show file tree
Hide file tree
Changes from 36 commits
Commits
Show all changes
40 commits
Select commit Hold shift + click to select a range
4af04bb
ENV #316 more suitcases
prjemian Dec 19, 2023
b007cee
MNT #316 convert ResultTuple to ordinary tuple
prjemian Dec 19, 2023
ce219d5
MNT #316 can JSON serialize a tuple
prjemian Dec 19, 2023
d863efc
MNT #303 type annotations for py38
prjemian Dec 19, 2023
8241083
CI #316 hints
prjemian Dec 19, 2023
e0ba082
DOC #316
prjemian Dec 19, 2023
7fe8b47
TST #316 more diagnostic tools
prjemian Dec 22, 2023
9ad8e04
TST #316
prjemian Dec 22, 2023
dfb7f18
MNT #316 refactor to use a list
prjemian Dec 22, 2023
470ed92
Merge branch 'main' into 316-CI-and-testing-problems
prjemian Dec 23, 2023
8b60e8b
CI #316 databroker version in test matrix
prjemian Dec 23, 2023
a5fa404
STY #316 trailing whitespace
prjemian Dec 23, 2023
ec99c7e
CI #316 syntax
prjemian Dec 23, 2023
f53235d
CI #316 explicit versions
prjemian Dec 23, 2023
1d3a91a
CI #316 syntax
prjemian Dec 23, 2023
ae2028c
ENV #316 require tiled
prjemian Dec 23, 2023
4da2327
CI #316 pick tiled version
prjemian Dec 23, 2023
c4fdb73
CI #316 typo
prjemian Dec 23, 2023
8f745c7
CI #316 simpler
prjemian Dec 23, 2023
0a5af3c
MNT #316 revert changes in commit c4fdb7379da
prjemian Dec 23, 2023
535c74b
CI #316
prjemian Dec 23, 2023
aa1e0c5
CI #316
prjemian Dec 23, 2023
6a4ceaa
MNT #316 error for databroker v2 now
prjemian Dec 23, 2023
861cf5a
DOC #316
prjemian Dec 23, 2023
1f4b462
PKG #316 pin databroker version
prjemian Dec 23, 2023
e6014fd
PKG #316 syntax
prjemian Dec 23, 2023
68de748
DOC #316
prjemian Dec 23, 2023
2bd334d
MNT #318 enable use of databroker v2+
prjemian Dec 23, 2023
274ebd0
Merge pull request #319 from bluesky/318-databroker-2
prjemian Dec 23, 2023
1fc55ef
CI #317 require chardet package for sphinx
prjemian Dec 23, 2023
f9b18ca
CI #318 verbose logging
prjemian Dec 23, 2023
97d8a27
CI #317 build docs with py311
prjemian Dec 23, 2023
f6000bb
DOC #317 cite each dataclass separately
prjemian Dec 23, 2023
154accf
CI #317 pin trio package version
prjemian Dec 23, 2023
496e25f
CI #317 drop the verbosity now
prjemian Dec 23, 2023
32e3043
Update hkl/util.py
prjemian Jan 6, 2024
9671da7
Update hkl/util.py
prjemian Jan 12, 2024
2fb010f
MNT #317 as suggested (for py38)
prjemian Jan 12, 2024
e10f49b
MNT #317 revert
prjemian Jan 12, 2024
ae13efe
Update .github/workflows/conda_unit_test.yml
prjemian Jan 12, 2024
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 2 additions & 2 deletions .github/workflows/conda_unit_test.yml
Original file line number Diff line number Diff line change
Expand Up @@ -21,13 +21,13 @@ jobs:
runs-on: ubuntu-latest
# needs: lint
strategy:
max-parallel: 5
matrix:
python-version:
- "3.8"
- "3.9"
- "3.10"
- "3.11"
max-parallel: 5

steps:
- uses: actions/checkout@v4
Expand Down Expand Up @@ -71,6 +71,6 @@ jobs:
- name: Test with coverage and pytest
shell: bash -l {0}
run: |
coverage run --concurrency=thread --parallel-mode -m pytest -vvv
coverage run --concurrency=thread --parallel-mode -m pytest -vvv # --exitfirst .
prjemian marked this conversation as resolved.
Show resolved Hide resolved
coverage combine
coverage report
2 changes: 1 addition & 1 deletion .github/workflows/publish-docs.yml
Original file line number Diff line number Diff line change
Expand Up @@ -26,7 +26,7 @@ jobs:
runs-on: ubuntu-latest
strategy:
matrix:
python-version: [3.9]
python-version: [3.11]

steps:
- name: Set env vars
Expand Down
3 changes: 3 additions & 0 deletions RELEASE_NOTES.rst
Original file line number Diff line number Diff line change
Expand Up @@ -39,6 +39,9 @@ Fixes
-----

* ``diffract.forward()`` should pick solution consistent with ``diffract.forward_solution_table()``, if it can. Otherwise, fall back to previous iterative method.
* Make ``util.list_orientation_runs()`` work with databroker v1.2 or v2+.
* Make ``util.run_orientation_info()`` work with databroker v1.2 or v2+.
* Resolved under-reported problems in CI unit tests.
* ``util.restore_reflections()`` use renamed motor axes if so defined.

Maintenance
Expand Down
62 changes: 61 additions & 1 deletion docs/source/configuration.rst
Original file line number Diff line number Diff line change
Expand Up @@ -169,7 +169,67 @@ API
.. automodule:: hkl.configuration
:members:
DiffractometerConfiguration,
DCConstraint, DCLattice, DCReflection, DCSample, DCConfiguration,
_check_key, _check_not_value, _check_range, _check_type, _check_value
:private-members:
:undoc-members:

.. autoclass:: hkl.configuration.DCConfiguration

.. autoattribute:: geometry
.. autoattribute:: engine
.. autoattribute:: library
.. autoattribute:: mode
.. autoattribute:: canonical_axes
.. autoattribute:: real_axes
.. autoattribute:: reciprocal_axes
.. autoattribute:: samples
.. autoattribute:: samples
.. autoattribute:: name
.. autoattribute:: datetime
.. autoattribute:: wavelength_angstrom
.. autoattribute:: energy_keV
.. autoattribute:: hklpy_version
.. autoattribute:: library_version
.. autoattribute:: python_class
.. autoattribute:: other
.. automethod:: validate
.. automethod:: write

.. autoclass:: hkl.configuration.DCConstraint

.. autoattribute:: low_limit
.. autoattribute:: high_limit
.. autoattribute:: value
.. autoattribute:: fit
.. automethod:: validate
.. autoproperty:: values

.. autoclass:: hkl.configuration.DCLattice

.. autoattribute:: a
.. autoattribute:: b
.. autoattribute:: c
.. autoattribute:: alpha
.. autoattribute:: beta
.. autoattribute:: gamma
.. automethod:: validate
.. autoproperty:: values

.. autoclass:: hkl.configuration.DCReflection

.. autoattribute:: reflection
.. autoattribute:: position
.. autoattribute:: wavelength
.. autoattribute:: orientation_reflection
.. autoattribute:: flag
.. automethod:: validate

.. autoclass:: hkl.configuration.DCSample

.. autoattribute:: name
.. autoattribute:: lattice
.. autoattribute:: reflections
.. autoattribute:: UB
.. autoattribute:: U
.. automethod:: validate
.. automethod:: write
2 changes: 2 additions & 0 deletions env-dev.yml
Original file line number Diff line number Diff line change
Expand Up @@ -16,6 +16,7 @@ dependencies:
- apischema
- black
- bluesky
- chardet
- databroker
- flake8
- hkl
Expand All @@ -35,6 +36,7 @@ dependencies:
- spec2nexus
- sphinx
- tqdm
- trio >=0.22
- pip:
- coveralls
- pytest
Expand Down
5 changes: 5 additions & 0 deletions environment.yml
Original file line number Diff line number Diff line change
Expand Up @@ -15,6 +15,7 @@ dependencies:
- python >=3.8,<3.12
- apischema
- bluesky
- databroker
- hkl
- numpy
- ophyd
Expand All @@ -25,6 +26,10 @@ dependencies:
- pygobject
- pyRestTable
- spec2nexus
- suitcase-jsonl
- suitcase-mongo
- suitcase-msgpack
- suitcase-utils
- tqdm
- pip:
- coveralls
Expand Down
5 changes: 5 additions & 0 deletions hkl/calc.py
Original file line number Diff line number Diff line change
Expand Up @@ -800,6 +800,11 @@ def __repr__(self):
def __str__(self):
return repr(self)

@property
def _cfg_reciprocal(self):
"""Return reciprocal lattice to save as configuration."""
return tuple(list(self.sample.reciprocal))


class CalcE4CH(CalcRecip):
"""Geometry: E4CH"""
Expand Down
25 changes: 14 additions & 11 deletions hkl/configuration.py
Original file line number Diff line number Diff line change
Expand Up @@ -38,6 +38,8 @@
from dataclasses import asdict
from dataclasses import dataclass
from dataclasses import field
from typing import Dict
from typing import List

import numpy
import pyRestTable
Expand Down Expand Up @@ -187,13 +189,13 @@ def values(self):
class DCReflection:
"""(internal) Configuration of one orientation reflection."""

reflection: dict[str, float]
reflection: Dict[str, float]
"""
Reciprocal-space axis positions. Keys must match in the list of
``reciprocal_axes``.
"""

position: dict[str, float]
position: Dict[str, float]
"""
Real-space axis positions. Keys must match in the list of
``canonical_axes``.
Expand Down Expand Up @@ -233,17 +235,18 @@ class DCSample:
lattice: DCLattice
"""Crystal lattice parameters (angstroms and degrees)"""

reflections: list[DCReflection]
reflections: List[DCReflection]
"""List of orientation reflections."""

UB: list[list[float]]
UB: List[List[float]]
"""
Orientation matrix (3 x 3). U is the crystal orientation matrix relative
to the diffractometer and B is the transition matrix of a non-orthonormal
(the reciprocal of the crystal) in an orthonormal system.
"""

U: list[list[float]] = field(default_factory=list[list[float]])
# TODO: Once py38 is dropped, re-enable the default value setting
U: List[List[float]] # = field(default_factory=list[list[float]])
Comment on lines +248 to +249
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is py38 the issue, or is it because DCReflection is not a dataclass?

I believe the default_factory field should be a zero-argument callable, such as...

default_factory=lambda: [[]]  # list containing an empty list

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks, I can check that.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Unfortunately, a change to

    U: field(default_factory=lambda: [[]])  # list containing an empty list

produces many versions of this exception. This example pytest -vvv --lf hkl/tests/test_configuration.py::test_restore tests the first such CI failure:

platform linux -- Python 3.11.6, pytest-7.4.3, pluggy-1.3.0 -- /home/prjemian/.conda/envs/bluesky_2024_1/bin/python3.11
cachedir: .pytest_cache
rootdir: /home/prjemian/Documents/projects/Bluesky/hklpy
configfile: pytest.ini
plugins: pytest_notebook-0.9.0, anyio-4.0.0
collected 1 item                                                                                                                                     
run-last-failure: 4 known failures not in selected tests

hkl/tests/test_configuration.py::test_restore FAILED                                                                                           [100%]

====================================================================== FAILURES ======================================================================
____________________________________________________________________ test_restore ____________________________________________________________________

e4cv_renamed = CustomFourCircle(prefix='', name='e4cv_renamed', settle_time=0.0, timeout=None, egu='', limits=(0, 0), source='compute... '_hklpy_version', '_pseudos', '_reals', '_constraints', '_mode', 'orientation_attrs', 'h', 'k', 'l'], concurrent=True)
k4cv = SimulatedK4CV(prefix='', name='k4cv', settle_time=0.0, timeout=None, egu='', limits=(0, 0), source='computed', read_at... '_hklpy_version', '_pseudos', '_reals', '_constraints', '_mode', 'orientation_attrs', 'h', 'k', 'l'], concurrent=True)

    def test_restore(e4cv_renamed, k4cv):
        config = DiffractometerConfiguration(e4cv_renamed)
>       before = config.export("dict")

hkl/tests/test_configuration.py:37: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
hkl/configuration.py:540: in export
    data = getattr(self, f"to_{fmt}")()
hkl/configuration.py:804: in to_dict
    return serialize(DCConfiguration, self.model)
hkl/configuration.py:745: in model
    obj = deserialize(DCConfiguration, data)  # also validates structure
../../../../.conda/envs/bluesky_2024_1/lib/python3.11/site-packages/apischema/deserialization/__init__.py:887: in deserialize
    return deserialization_method(
../../../../.conda/envs/bluesky_2024_1/lib/python3.11/site-packages/apischema/deserialization/__init__.py:817: in deserialization_method
    deserialization_method_factory(
../../../../.conda/envs/bluesky_2024_1/lib/python3.11/site-packages/apischema/deserialization/__init__.py:753: in deserialization_method_factory
    ).visit_with_conv(tp, conversion)
...
../../../../.conda/envs/bluesky_2024_1/lib/python3.11/site-packages/apischema/objects/visitor.py:129: in unsupported
    return super().unsupported(tp) if fields is dummy else self._object(tp, fields)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

self = <apischema.deserialization.DeserializationMethodVisitor object at 0x7f47556da810>
tp = Field(name=None,type=None,default=<dataclasses._MISSING_TYPE object at 0x7f47d0d2fd50>,default_factory=<function DCSam...e,compare=True,metadata=mappingproxy({}),kw_only=<dataclasses._MISSING_TYPE object at 0x7f47d0d2fd50>,_field_type=None)

    def unsupported(self, tp: AnyType) -> Result:
>       raise Unsupported(tp)
E       apischema.visitor.Unsupported: Field(name=None,type=None,default=<dataclasses._MISSING_TYPE object at 0x7f47d0d2fd50>,default_factory=<function DCSample.<lambda> at 0x7f476f358860>,init=True,repr=True,hash=None,compare=True,metadata=mappingproxy({}),kw_only=<dataclasses._MISSING_TYPE object at 0x7f47d0d2fd50>,_field_type=None)

../../../../.conda/envs/bluesky_2024_1/lib/python3.11/site-packages/apischema/visitor.py:151: Unsupported
================================================================== warnings summary ==================================================================
../../../../.conda/envs/bluesky_2024_1/lib/python3.11/site-packages/pkg_resources/__init__.py:2871
../../../../.conda/envs/bluesky_2024_1/lib/python3.11/site-packages/pkg_resources/__init__.py:2871
  /home/prjemian/.conda/envs/bluesky_2024_1/lib/python3.11/site-packages/pkg_resources/__init__.py:2871: DeprecationWarning: Deprecated call to `pkg_resources.declare_namespace('sphinxcontrib')`.
  Implementing implicit namespace packages (as specified in PEP 420) is preferred to `pkg_resources.declare_namespace`. See https://setuptools.pypa.io/en/latest/references/keywords.html#keyword-namespace-packages
    declare_namespace(pkg)

-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html
============================================================== short test summary info ===============================================================
FAILED hkl/tests/test_configuration.py::test_restore - apischema.visitor.Unsupported: Field(name=None,type=None,default=<dataclasses._MISSING_TYPE object at 0x7f47d0d2fd50>,default_factory=<function D...
=========================================================== 1 failed, 2 warnings in 0.23s ============================================================

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Note this example used Py3.11

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Created new issue for this.

"""
Orientation matrix (3 x 3) of the crystal relative to the diffractometer.
(optional)
Expand Down Expand Up @@ -337,36 +340,36 @@ class DCConfiguration:
diffractometer to restore.
"""

canonical_axes: list[str]
canonical_axes: List[str]
"""
List of the diffractometer real-space axis names. Both the exact spelling
and order are defined by the back-end computation library. MUST match
diffractometer to restore.
"""

real_axes: list[str]
real_axes: List[str]
"""
User-defined real-space axis names. MUST match diffractometer to restore.
The length and order of this list must be the same as the
``canonical_axes``. It is used to resolve any (real-space) ``positioner``
names in this file.
"""

reciprocal_axes: list[str]
reciprocal_axes: List[str]
"""
List of names of the diffractometer reciprocal-space (pseudo) axes. Both
the exact spelling and order are defined by the back-end computation
library ``engine``.
MUST match diffractometer to restore.
"""

constraints: dict[str, DCConstraint]
constraints: Dict[str, DCConstraint]
"""
Limits to be imposed on the real-space axes for operations and
computations. Keys must match in the list of ``canonical_axes``.
"""

samples: dict[str, DCSample]
samples: Dict[str, DCSample]
"""
Crystalline samples (lattice and orientation reflections).
The sample name is used as the key in the dictionary.
Expand Down Expand Up @@ -410,7 +413,7 @@ class DCConfiguration:
Name of the Python class that defines this diffractometer. (optional)
"""

other: dict[str, typing.Any] = field(default_factory=dict)
other: Dict[str, typing.Any] = field(default_factory=dict)
"""
*Any* other content goes into this dictionary (comments, unanticipated
keys, ...) (optional)
Expand Down
11 changes: 8 additions & 3 deletions hkl/diffract.py
Original file line number Diff line number Diff line change
Expand Up @@ -168,7 +168,7 @@ class (using `calc_kw`) to instantiate a new one.
lattice_reciprocal = Cpt(
# fmt: off
AttributeSignal,
attr="calc.sample.reciprocal",
attr="calc._cfg_reciprocal",
doc="Reciprocal lattice",
# fmt: on
)
Expand All @@ -177,8 +177,8 @@ class (using `calc_kw`) to instantiate a new one.
UB = Cpt(AttributeSignal, attr="calc.sample.UB", doc="UB matrix")
# fmt: off
reflections = Cpt(
ArrayAttributeSignal,
attr="calc.sample.reflections",
AttributeSignal,
attr="_reflections",
doc="Reflections",
)
reflections_details = Cpt(
Expand Down Expand Up @@ -449,6 +449,11 @@ def engine(self):
# -- it becomes a problem when someone uses these functions
# outside of move()

@property
def _reflections(self):
"""Return the list of reflections as a [[float]]"""
return [list(r) for r in self.calc.sample.reflections]

@pseudo_position_argument
def forward(self, pseudo):
"""
Expand Down
72 changes: 53 additions & 19 deletions hkl/tests/test_save_restore_UB.py
Original file line number Diff line number Diff line change
@@ -1,3 +1,5 @@
import time

import bluesky.plans as bp
import databroker
import numpy.testing
Expand Down Expand Up @@ -84,18 +86,16 @@ def test_fourc_orientation_save(cat, RE, fourc):
# this run will not save orientation information
_uids = RE(bp.count([det]))
assert len(_uids) == 1
time.sleep(1)
assert len(cat) == 1
assert "fourc" not in cat[1].primary.config

# this run _will_ save orientation information
_uids = RE(bp.count([det, fourc]))
assert len(_uids) == 1
time.sleep(1)
assert len(cat) == 2
xarray_data = cat[2].primary.config["fourc"].read()
assert not isinstance(xarray_data, dict)
descriptors = xarray_data.to_dict()
assert isinstance(descriptors, dict)
assert list(descriptors.keys()) == "coords attrs dims data_vars".split()

key_list = """
_pseudos
_reals
Expand All @@ -108,20 +108,54 @@ def test_fourc_orientation_save(cat, RE, fourc):
sample_name
UB
""".split()
for key in key_list:
key_name = f"fourc_{key}"
assert hasattr(xarray_data, key_name)
assert key_name in descriptors["data_vars"]

assert xarray_data.fourc_class_name == "Fourc"
assert xarray_data.fourc_geometry_name == "E4CV"
assert xarray_data.fourc_diffractometer_name == "fourc"
assert xarray_data.fourc_sample_name == "Si"

assert len(xarray_data.fourc__pseudos) == 1
assert xarray_data.fourc__pseudos[0].values.tolist() == "h k l".split()
assert len(xarray_data.fourc__reals) == 1
assert xarray_data.fourc__reals.values[0].tolist() == "omega chi phi tth".split()

if databroker.__version__ < "2.0":
conf = cat[_uids[0]].primary.config["fourc"].read()
assert not isinstance(conf, dict)
descriptors = conf.to_dict()
assert isinstance(descriptors, dict)
assert list(descriptors.keys()) == "coords attrs dims data_vars".split()
for key in key_list:
key_name = f"fourc_{key}"
assert hasattr(conf, key_name)
assert key_name in descriptors["data_vars"]

assert conf.fourc_class_name == "Fourc"
assert conf.fourc_geometry_name == "E4CV"
assert conf.fourc_diffractometer_name == "fourc"
assert conf.fourc_sample_name == "Si"

assert len(conf.fourc__pseudos) == 1
assert conf.fourc__pseudos[0].values.tolist() == "h k l".split()
assert len(conf.fourc__reals) == 1
assert conf.fourc__reals.values[0].tolist() == "omega chi phi tth".split()

else:
descriptors = cat[_uids[0]].primary.descriptors
assert isinstance(descriptors, list)
assert len(descriptors) > 0
for descriptor in descriptors:
for device, configuration in descriptor.get("configuration", {}).items():
assert isinstance(device, str)
assert isinstance(configuration, dict)
conf = configuration.get("data", {})
assert isinstance(conf, dict)

if device != "fourc":
continue
for key in key_list:
key_name = f"fourc_{key}"
assert key_name in conf

assert conf["fourc_class_name"] == "Fourc"
assert conf["fourc_geometry_name"] == "E4CV"
assert conf["fourc_diffractometer_name"] == "fourc"
assert conf["fourc_sample_name"] == "Si"

assert len(conf["fourc__pseudos"]) == 3
assert conf["fourc__pseudos"] == "h k l".split()
assert len(conf["fourc__reals"]) == 4
assert conf["fourc__reals"] == "omega chi phi tth".split()


def test_fourc_run_orientation_info(cat, RE, fourc):
Expand Down
Loading
Loading