resolve problems with unit test workflow #317

prjemian · 2023-12-19T06:58:43Z

prjemian · 2023-12-19T07:30:44Z

Caught up with #314

prjemian · 2023-12-19T07:44:38Z

The remaining problems do not happen with databroker 1.2. The CI uses databroker v2.0.0b31 (at this time). All of these problems are related to storing the UB matrix (and other information) in the run as a detector.

FAILED hkl/tests/test_save_restore_UB.py::test_fourc_orientation_save - assert 0 == 1
FAILED hkl/tests/test_save_restore_UB.py::test_fourc_run_orientation_info - AssertionError: assert 'fourc' in {}
FAILED hkl/tests/test_save_restore_UB.py::test_list_orientation_runs - AttributeError: 'DataFrame' object has no attribute 'scan_id'
FAILED hkl/tests/test_save_restore_UB.py::test_no_primary_stream - AttributeError: 'DataFrame' object has no attribute 'scan_id'
FAILED hkl/tests/test_save_restore_UB.py::test_missing_energy_key - assert 0 == 1
FAILED hkl/tests/test_save_restore_UB.py::test_restore_orientation - KeyError: 'fourc'
FAILED hkl/tests/test_save_restore_UB.py::test_restore_sample - KeyError: 'fourc'
FAILED hkl/tests/test_util.py::test_issue215 - assert 0 == 1

prjemian · 2023-12-23T16:43:14Z

It's clear that the hkl.util.restore_orientation_info() function needs some serious attention before it can find the information known to be available from a run that was saved before this branch was started. That's a new problem with databroker v2+. Reverting the workflow to databroker 1.2.5 and tiled 0.1.0a96.

prjemian · 2023-12-23T21:53:38Z

Still:

2023-12-23T21:48:26.2424512Z [autodoc] from hkl.configuration import DCConfiguration.canonical_axes
2023-12-23T21:48:26.2427010Z [autodoc] import hkl.configuration => <module 'hkl.configuration' from '/home/runner/work/hklpy/hklpy/hkl/configuration.py'>
2023-12-23T21:48:26.2428908Z [autodoc] getattr(_, 'DCConfiguration')
2023-12-23T21:48:26.2430174Z [autodoc] => <class 'hkl.configuration.DCConfiguration'>
2023-12-23T21:48:26.2431422Z [autodoc] getattr(_, 'canonical_axes')
2023-12-23T21:48:26.2433439Z [app] emitting event: 'build-finished'(TypeError("traceback_exception_init() got an unexpected keyword argument 'max_group_width'"),)

Switch to Py311 for this build

prjemian · 2023-12-23T22:05:04Z

Locally, docs fail 30 times on lines such as these examples:

autodoc: failed to import attribute 'DCConfiguration.canonical_axes' from module 'hkl.configuration'; the following exception was raised:
autodoc: failed to import attribute 'DCConfiguration.constraints' from module 'hkl.configuration'; the following exception was raised:
autodoc: failed to import attribute 'DCConfiguration.energy_keV' from module 'hkl.configuration'; the following exception was raised:
autodoc: failed to import attribute 'DCConfiguration.engine' from module 'hkl.configuration'; the following exception was raised:

All are classes using the @dataclass property.

prjemian · 2023-12-23T22:30:02Z

This SO post is helpful. When using type annotations and @dataclass property, need to help Sphinx out by being more explicit about how to describe the docs.

prjemian · 2023-12-23T22:52:43Z

Issue in trio repository related to this persistent error. This is trio 0.19.0. Our error starts with:

2023-12-23T22:42:31.9288016Z [autodoc] => <class 'hkl.configuration.DCConfiguration'>
2023-12-23T22:42:31.9289251Z [autodoc] getattr(_, 'geometry')
2023-12-23T22:42:31.9298270Z [app] emitting event: 'build-finished'(TypeError("traceback_exception_init() got an unexpected keyword argument 'max_group_width'"),)
2023-12-23T22:42:31.9300360Z 
2023-12-23T22:42:31.9338927Z Error in sys.excepthook:
2023-12-23T22:42:31.9339203Z Traceback (most recent call last):
2023-12-23T22:42:31.9340648Z   File "/home/runner/micromamba/envs/documentation/lib/python3.11/site-packages/dask/base.py", line 71, in wrapper
2023-12-23T22:42:31.9341217Z     return func(exc_type, exc.with_traceback(tb), tb)
2023-12-23T22:42:31.9342102Z            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2023-12-23T22:42:31.9343323Z   File "/home/runner/micromamba/envs/documentation/lib/python3.11/site-packages/trio/_core/_multierror.py", line 447, in trio_excepthook
2023-12-23T22:42:31.9344046Z     for chunk in traceback.format_exception(etype, value, tb):
2023-12-23T22:42:31.9346975Z                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2023-12-23T22:42:31.9348181Z   File "/home/runner/micromamba/envs/documentation/lib/python3.11/traceback.py", line 139, in format_exception
2023-12-23T22:42:31.9348894Z     te = TracebackException(type(value), value, tb, limit=limit, compact=True)
2023-12-23T22:42:31.9351416Z          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2023-12-23T22:42:31.9353171Z   File "/home/runner/micromamba/envs/documentation/lib/python3.11/site-packages/trio/_core/_multierror.py", line 393, in traceback_exception_init

prjemian · 2023-12-23T22:54:04Z

Locally, trio 0.23.2, which is latest release.

prjemian · 2023-12-23T23:09:31Z

@padraic-shafer @mrakitin This is ready now.

mrakitin

Looks good! I have a clarification question about the logic in the run_orientation_info.

hkl/util.py

I like this. Co-authored-by: Padraic Shafer <76011594+padraic-shafer@users.noreply.github.com>

prjemian · 2024-01-08T20:52:36Z

@mrakitin @padraic-shafer - Anything else?

prjemian · 2024-01-10T04:16:42Z

Absent additional comments to be addressed, I plan to merge this PR by the end of 2024-01-10.

mrakitin

Looks good to me now!

padraic-shafer

Looks good. Some suggestions were added in the comments.

.github/workflows/conda_unit_test.yml

padraic-shafer · 2024-01-10T18:11:49Z

hkl/configuration.py

+    # TODO: Once py38 is dropped, re-enable the default value setting
+    U: List[List[float]]  # = field(default_factory=list[list[float]])


Is py38 the issue, or is it because DCReflection is not a dataclass?

I believe the default_factory field should be a zero-argument callable, such as...

default_factory=lambda: [[]] # list containing an empty list

Thanks, I can check that.

Unfortunately, a change to

U: field(default_factory=lambda: [[]]) # list containing an empty list

produces many versions of this exception. This example pytest -vvv --lf hkl/tests/test_configuration.py::test_restore tests the first such CI failure:

platform linux -- Python 3.11.6, pytest-7.4.3, pluggy-1.3.0 -- /home/prjemian/.conda/envs/bluesky_2024_1/bin/python3.11 cachedir: .pytest_cache rootdir: /home/prjemian/Documents/projects/Bluesky/hklpy configfile: pytest.ini plugins: pytest_notebook-0.9.0, anyio-4.0.0 collected 1 item run-last-failure: 4 known failures not in selected tests hkl/tests/test_configuration.py::test_restore FAILED [100%] ====================================================================== FAILURES ====================================================================== ____________________________________________________________________ test_restore ____________________________________________________________________ e4cv_renamed = CustomFourCircle(prefix='', name='e4cv_renamed', settle_time=0.0, timeout=None, egu='', limits=(0, 0), source='compute... '_hklpy_version', '_pseudos', '_reals', '_constraints', '_mode', 'orientation_attrs', 'h', 'k', 'l'], concurrent=True) k4cv = SimulatedK4CV(prefix='', name='k4cv', settle_time=0.0, timeout=None, egu='', limits=(0, 0), source='computed', read_at... '_hklpy_version', '_pseudos', '_reals', '_constraints', '_mode', 'orientation_attrs', 'h', 'k', 'l'], concurrent=True) def test_restore(e4cv_renamed, k4cv): config = DiffractometerConfiguration(e4cv_renamed) > before = config.export("dict") hkl/tests/test_configuration.py:37: _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ hkl/configuration.py:540: in export data = getattr(self, f"to_{fmt}")() hkl/configuration.py:804: in to_dict return serialize(DCConfiguration, self.model) hkl/configuration.py:745: in model obj = deserialize(DCConfiguration, data) # also validates structure ../../../../.conda/envs/bluesky_2024_1/lib/python3.11/site-packages/apischema/deserialization/__init__.py:887: in deserialize return deserialization_method( ../../../../.conda/envs/bluesky_2024_1/lib/python3.11/site-packages/apischema/deserialization/__init__.py:817: in deserialization_method deserialization_method_factory( ../../../../.conda/envs/bluesky_2024_1/lib/python3.11/site-packages/apischema/deserialization/__init__.py:753: in deserialization_method_factory ).visit_with_conv(tp, conversion) ... ../../../../.conda/envs/bluesky_2024_1/lib/python3.11/site-packages/apischema/objects/visitor.py:129: in unsupported return super().unsupported(tp) if fields is dummy else self._object(tp, fields) _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ self = <apischema.deserialization.DeserializationMethodVisitor object at 0x7f47556da810> tp = Field(name=None,type=None,default=<dataclasses._MISSING_TYPE object at 0x7f47d0d2fd50>,default_factory=<function DCSam...e,compare=True,metadata=mappingproxy({}),kw_only=<dataclasses._MISSING_TYPE object at 0x7f47d0d2fd50>,_field_type=None) def unsupported(self, tp: AnyType) -> Result: > raise Unsupported(tp) E apischema.visitor.Unsupported: Field(name=None,type=None,default=<dataclasses._MISSING_TYPE object at 0x7f47d0d2fd50>,default_factory=<function DCSample.<lambda> at 0x7f476f358860>,init=True,repr=True,hash=None,compare=True,metadata=mappingproxy({}),kw_only=<dataclasses._MISSING_TYPE object at 0x7f47d0d2fd50>,_field_type=None) ../../../../.conda/envs/bluesky_2024_1/lib/python3.11/site-packages/apischema/visitor.py:151: Unsupported ================================================================== warnings summary ================================================================== ../../../../.conda/envs/bluesky_2024_1/lib/python3.11/site-packages/pkg_resources/__init__.py:2871 ../../../../.conda/envs/bluesky_2024_1/lib/python3.11/site-packages/pkg_resources/__init__.py:2871 /home/prjemian/.conda/envs/bluesky_2024_1/lib/python3.11/site-packages/pkg_resources/__init__.py:2871: DeprecationWarning: Deprecated call to `pkg_resources.declare_namespace('sphinxcontrib')`. Implementing implicit namespace packages (as specified in PEP 420) is preferred to `pkg_resources.declare_namespace`. See https://setuptools.pypa.io/en/latest/references/keywords.html#keyword-namespace-packages declare_namespace(pkg) -- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html ============================================================== short test summary info =============================================================== FAILED hkl/tests/test_configuration.py::test_restore - apischema.visitor.Unsupported: Field(name=None,type=None,default=<dataclasses._MISSING_TYPE object at 0x7f47d0d2fd50>,default_factory=<function D... =========================================================== 1 failed, 2 warnings in 0.23s ============================================================

Note this example used Py3.11

Created new issue for this.

hkl/util.py

Co-authored-by: Padraic Shafer <76011594+padraic-shafer@users.noreply.github.com>

prjemian · 2024-01-12T15:40:03Z

@padraic-shafer @mrakitin : added the pipefail to the CI -- all tests pass at this time

padraic-shafer

Looks good to go

mrakitin

Looks good to me!

ENV #316 more suitcases

4af04bb

prjemian added bug task labels Dec 19, 2023

prjemian added this to the v1.1 milestone Dec 19, 2023

prjemian self-assigned this Dec 19, 2023

prjemian added 3 commits December 19, 2023 01:10

MNT #316 convert ResultTuple to ordinary tuple

b007cee

MNT #316 can JSON serialize a tuple

ce219d5

MNT #303 type annotations for py38

d863efc

prjemian mentioned this pull request Dec 19, 2023

xx forward() should pick answer consistent with forward_solutions_table() and default decision function #314

Closed

prjemian added 2 commits December 19, 2023 01:32

CI #316 hints

8241083

DOC #316

e0ba082

prjemian added 15 commits December 22, 2023 17:08

TST #316 more diagnostic tools

7fe8b47

TST #316

9ad8e04

MNT #316 refactor to use a list

dfb7f18

Merge branch 'main' into 316-CI-and-testing-problems

470ed92

CI #316 databroker version in test matrix

8b60e8b

STY #316 trailing whitespace

a5fa404

CI #316 syntax

ec99c7e

CI #316 explicit versions

f53235d

CI #316 syntax

1d3a91a

ENV #316 require tiled

ae2028c

CI #316 pick tiled version

4da2327

CI #316 typo

c4fdb73

CI #316 simpler

8f745c7

MNT #316 revert changes in commit c4fdb73

0a5af3c

CI #316

535c74b

CI #317 build docs with py311

97d8a27

prjemian marked this pull request as draft December 23, 2023 22:03

DOC #317 cite each dataclass separately

f6000bb

prjemian added 2 commits December 23, 2023 16:56

CI #317 pin trio package version

154accf

CI #317 drop the verbosity now

496e25f

prjemian marked this pull request as ready for review December 23, 2023 23:08

mrakitin reviewed Jan 5, 2024

View reviewed changes

hkl/util.py Outdated Show resolved Hide resolved

Update hkl/util.py

32e3043

I like this. Co-authored-by: Padraic Shafer <76011594+padraic-shafer@users.noreply.github.com>

mrakitin approved these changes Jan 10, 2024

View reviewed changes

padraic-shafer approved these changes Jan 10, 2024

View reviewed changes

prjemian and others added 4 commits January 12, 2024 09:13

Update hkl/util.py

9671da7

Co-authored-by: Padraic Shafer <76011594+padraic-shafer@users.noreply.github.com>

MNT #317 as suggested (for py38)

2fb010f

MNT #317 revert

e10f49b

Update .github/workflows/conda_unit_test.yml

ae13efe

Co-authored-by: Padraic Shafer <76011594+padraic-shafer@users.noreply.github.com>

prjemian mentioned this pull request Jan 12, 2024

type annotation in DCReflection class #321

Open

padraic-shafer approved these changes Jan 12, 2024

View reviewed changes

mrakitin approved these changes Jan 12, 2024

View reviewed changes

prjemian merged commit a3f9718 into main Jan 12, 2024
8 checks passed

prjemian deleted the 316-CI-and-testing-problems branch January 12, 2024 20:16

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

resolve problems with unit test workflow #317

resolve problems with unit test workflow #317

prjemian commented Dec 19, 2023 •

edited

Loading

prjemian commented Dec 19, 2023

prjemian commented Dec 19, 2023

prjemian commented Dec 23, 2023

prjemian commented Dec 23, 2023

prjemian commented Dec 23, 2023

prjemian commented Dec 23, 2023

prjemian commented Dec 23, 2023 •

edited

Loading

prjemian commented Dec 23, 2023

prjemian commented Dec 23, 2023

mrakitin left a comment

prjemian commented Jan 8, 2024

prjemian commented Jan 10, 2024

mrakitin left a comment

padraic-shafer left a comment

padraic-shafer Jan 10, 2024

prjemian Jan 10, 2024

prjemian Jan 12, 2024

prjemian Jan 12, 2024

prjemian Jan 12, 2024

prjemian commented Jan 12, 2024

padraic-shafer left a comment

mrakitin left a comment

		# TODO: Once py38 is dropped, re-enable the default value setting
		U: List[List[float]] # = field(default_factory=list[list[float]])

resolve problems with unit test workflow #317

resolve problems with unit test workflow #317

Conversation

prjemian commented Dec 19, 2023 • edited Loading

prjemian commented Dec 19, 2023

prjemian commented Dec 19, 2023

prjemian commented Dec 23, 2023

prjemian commented Dec 23, 2023

prjemian commented Dec 23, 2023

prjemian commented Dec 23, 2023

prjemian commented Dec 23, 2023 • edited Loading

prjemian commented Dec 23, 2023

prjemian commented Dec 23, 2023

mrakitin left a comment

Choose a reason for hiding this comment

prjemian commented Jan 8, 2024

prjemian commented Jan 10, 2024

mrakitin left a comment

Choose a reason for hiding this comment

padraic-shafer left a comment

Choose a reason for hiding this comment

padraic-shafer Jan 10, 2024

Choose a reason for hiding this comment

prjemian Jan 10, 2024

Choose a reason for hiding this comment

prjemian Jan 12, 2024

Choose a reason for hiding this comment

prjemian Jan 12, 2024

Choose a reason for hiding this comment

prjemian Jan 12, 2024

Choose a reason for hiding this comment

prjemian commented Jan 12, 2024

padraic-shafer left a comment

Choose a reason for hiding this comment

mrakitin left a comment

Choose a reason for hiding this comment

prjemian commented Dec 19, 2023 •

edited

Loading

prjemian commented Dec 23, 2023 •

edited

Loading