Test fails / test procedure is not documented

[ktahar]

Part of JOSS review: openjournals/joss-reviews#6371

I ran test included in the repository and two of them failed: test_gradient_descent and test_simple_sweep_with_plot_and_fit in protocol_test.

• I ran it on Ubuntu 22.04. Done pip install nomad-camels in a new virtual environment (created using virtualenv).
• I could not find how to run test. I guessed that I need pytest and pytest-qt and installed them. I think this should be documented somewhere (for possible developers/contributors).
• The cause of error looks like ImportError regarding matplotlib backend. Full pytest output is pasted below:

============================= test session starts ==============================
platform linux -- Python 3.10.12, pytest-8.0.1, pluggy-1.4.0
PySide6 6.6.2 -- Qt runtime 6.6.2 -- Qt compiled 6.6.2
rootdir: /home/ktaha/dev/NOMAD-CAMELS/nomad_camels/tests
plugins: qt-4.4.0
collected 15 items

instrument_management_test.py ..                                         [ 13%]
protocol_test.py ..F.....F...                                            [ 93%]
startup_test.py .                                                        [100%]

=================================== FAILURES ===================================
____________________________ test_gradient_descent _____________________________

qtbot = <pytestqt.qtbot.QtBot object at 0x7f3ea835fe80>
tmp_path = PosixPath('/tmp/pytest-of-ktaha/pytest-5/test_gradient_descent0')

    def test_gradient_descent(qtbot, tmp_path):
        """Opens the config for "Gradient Descent" tries to configure it for the
        demo instrument and tries to run a protocol with this step."""
        ensure_demo_in_devices()
        from nomad_camels.loop_steps import gradient_descent
        conf = protocol_config.Protocol_Config()
        qtbot.addWidget(conf)
        action = get_action_from_name(conf.add_actions, 'Gradient Descent')
        action.trigger()
        conf_widge = conf.loop_step_configuration_widget
        assert isinstance(conf_widge,
                          gradient_descent.Gradient_Descent_Config)
        conf_widge.sub_widget.lineEdit_momentum.setText('1')
        conf_widge.sub_widget.lineEdit_threshold.setText('1e-3')
        conf_widge.sub_widget.lineEdit_max_val.setText('1')
        conf_widge.sub_widget.lineEdit_min_val.setText('1')
        conf_widge.sub_widget.lineEdit_opt_func.setText('demo_instrument_detectorY')
        conf_widge.sub_widget.lineEdit_max_n_steps.setText('25')
        conf_widge.sub_widget.lineEdit_largest_step.setText('1')
        conf_widge.sub_widget.lineEdit_smallest_step.setText('0.01')
        conf_widge.sub_widget.lineEdit_learning_rate.setText('1')
        conf_widge.sub_widget.lineEdit_starting_val.setText('0')
        conf_widge.sub_widget.comboBox_extremum_type.setCurrentText('Maximum')
        conf_widge.sub_widget.comboBox_output_channel.setCurrentText('demo_instrument_motorY')
        with qtbot.waitSignal(conf.accepted) as blocker:
            conf.accept()
        prot = conf.protocol
        prot.loop_steps[0].read_channels.append('demo_instrument_detectorY')
        prot.name = 'test_gradient_descent_protocol'
        assert 'Gradient Descent (Gradient_Descent)' in prot.loop_step_dict
        catalog_maker(tmp_path)
>       run_test_protocol(tmp_path, prot)

protocol_test.py:116: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
protocol_test.py:547: in run_test_protocol
    py_package.main()
/tmp/pytest-of-ktaha/pytest-5/test_gradient_descent0/test_gradient_descent_protocol.py:153: in main
    additional_step_data = steps_add_main(RE, devs)
/tmp/pytest-of-ktaha/pytest-5/test_gradient_descent0/test_gradient_descent_protocol.py:91: in steps_add_main
    plots, subs, _ = create_plots_Gradient_Descent(RE, "Gradient_Descent")
/tmp/pytest-of-ktaha/pytest-5/test_gradient_descent0/test_gradient_descent_protocol.py:52: in create_plots_Gradient_Descent
    plot_0 = plot_widget.PlotWidget(x_name="demo_instrument_motorY", y_names=['demo_instrument_detectorY'], ylabel="demo_instrument_detectorY", xlabel="demo_instrument_motorY", title="gradient-descent", stream_name=stream, namespace=namespace, fits=fits, multi_stream=False, y_axes={'demo_instrument_detectorY': 1}, ylabel2="", logX=False, logY=False, logY2=False, maxlen="inf")
../main_classes/plot_widget.py:139: in __init__
    canvas = MPLwidget()
../main_classes/plot_widget.py:57: in __init__
    fig, ax = plt.subplots()
/home/ktaha/virtualenv/camels/lib/python3.10/site-packages/matplotlib/pyplot.py:1613: in subplots
    fig = figure(**fig_kw)
/home/ktaha/virtualenv/camels/lib/python3.10/site-packages/matplotlib/pyplot.py:934: in figure
    manager = new_figure_manager(
/home/ktaha/virtualenv/camels/lib/python3.10/site-packages/matplotlib/pyplot.py:464: in new_figure_manager
    _warn_if_gui_out_of_main_thread()
/home/ktaha/virtualenv/camels/lib/python3.10/site-packages/matplotlib/pyplot.py:441: in _warn_if_gui_out_of_main_thread
    canvas_class = cast(type[FigureCanvasBase], _get_backend_mod().FigureCanvas)
/home/ktaha/virtualenv/camels/lib/python3.10/site-packages/matplotlib/pyplot.py:280: in _get_backend_mod
    switch_backend(rcParams._get("backend"))  # type: ignore[attr-defined]
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

newbackend = 'TkAgg'

    def switch_backend(newbackend: str) -> None:
        """
        Set the pyplot backend.
    
        Switching to an interactive backend is possible only if no event loop for
        another interactive backend has started.  Switching to and from
        non-interactive backends is always possible.
    
        If the new backend is different than the current backend then all open
        Figures will be closed via ``plt.close('all')``.
    
        Parameters
        ----------
        newbackend : str
            The case-insensitive name of the backend to use.
    
        """
        global _backend_mod
        # make sure the init is pulled up so we can assign to it later
        import matplotlib.backends
    
        if newbackend is rcsetup._auto_backend_sentinel:
            current_framework = cbook._get_running_interactive_framework()
            mapping = {'qt': 'qtagg',
                       'gtk3': 'gtk3agg',
                       'gtk4': 'gtk4agg',
                       'wx': 'wxagg',
                       'tk': 'tkagg',
                       'macosx': 'macosx',
                       'headless': 'agg'}
    
            if current_framework in mapping:
                candidates = [mapping[current_framework]]
            else:
                candidates = []
            candidates += [
                "macosx", "qtagg", "gtk4agg", "gtk3agg", "tkagg", "wxagg"]
    
            # Don't try to fallback on the cairo-based backends as they each have
            # an additional dependency (pycairo) over the agg-based backend, and
            # are of worse quality.
            for candidate in candidates:
                try:
                    switch_backend(candidate)
                except ImportError:
                    continue
                else:
                    rcParamsOrig['backend'] = candidate
                    return
            else:
                # Switching to Agg should always succeed; if it doesn't, let the
                # exception propagate out.
                switch_backend("agg")
                rcParamsOrig["backend"] = "agg"
                return
        # have to escape the switch on access logic
        old_backend = dict.__getitem__(rcParams, 'backend')
    
        module = importlib.import_module(cbook._backend_module_name(newbackend))
        canvas_class = module.FigureCanvas
    
        required_framework = canvas_class.required_interactive_framework
        if required_framework is not None:
            current_framework = cbook._get_running_interactive_framework()
            if (current_framework and required_framework
                    and current_framework != required_framework):
>               raise ImportError(
                    "Cannot load backend {!r} which requires the {!r} interactive "
                    "framework, as {!r} is currently running".format(
                        newbackend, required_framework, current_framework))
E               ImportError: Cannot load backend 'TkAgg' which requires the 'tk' interactive framework, as 'qt' is currently running

/home/ktaha/virtualenv/camels/lib/python3.10/site-packages/matplotlib/pyplot.py:350: ImportError
----------------------------- Captured stdout call -----------------------------
connecting demo_instrument
devices connected
demo_instrument is being closed
----------------------------- Captured stderr call -----------------------------



  0%|          | 0/1 [00:00<?, ?it/s]�[A�[A
_____________________ test_simple_sweep_with_plot_and_fit ______________________

qtbot = <pytestqt.qtbot.QtBot object at 0x7f3ea82dbb20>
tmp_path = PosixPath('/tmp/pytest-of-ktaha/pytest-5/test_simple_sweep_with_plot_an0')

    def test_simple_sweep_with_plot_and_fit(qtbot, tmp_path):
        """Opens the config for "Simple Sweep" tries to configure it for the
        demo instrument. Further it adds a plot and a fit to the sweep and tries to
        run a protocol with this step."""
        ensure_demo_in_devices()
        from nomad_camels.loop_steps import simple_sweep
        from nomad_camels.frontpanels import plot_definer
        conf = protocol_config.Protocol_Config()
        qtbot.addWidget(conf)
        action = get_action_from_name(conf.add_actions, 'Simple Sweep')
        action.trigger()
        conf_widge = conf.loop_step_configuration_widget
        assert isinstance(conf_widge,
                          simple_sweep.Simple_Sweep_Config)
        conf_widge.sweep_widget.lineEdit_start.setText('-10')
        conf_widge.sweep_widget.lineEdit_stop.setText('10')
        conf_widge.sweep_widget.lineEdit_n_points.setText('21')
        conf_widge.comboBox_sweep_channel.setCurrentText('demo_instrument_motorY')
    
        table = conf_widge.read_table.tableWidget_channels
        row = get_row_from_channel_table('demo_instrument_detectorY', table)
        table.item(row, 0).setCheckState(Qt.CheckState.Checked)
        row = get_row_from_channel_table('demo_instrument_motorY', table)
        table.item(row, 0).setCheckState(Qt.CheckState.Checked)
    
        fit = plot_definer.Fit_Info(True, 'Gaussian', x='demo_instrument_motorY',
                                    y='demo_instrument_detectorY')
        plot = plot_definer.Plot_Info(x_axis='demo_instrument_motorY',
                                      y_axes={'formula': ["demo_instrument_detectorY"],
                                              'axis': ['left']},
                                      fits=[fit])
        conf_widge.plot_widge.plot_data = [plot]
    
        with qtbot.waitSignal(conf.accepted) as blocker:
            conf.accept()
        prot = conf.protocol
        prot.name = 'test_simple_sweep_protocol'
        assert 'Simple Sweep (Simple_Sweep)' in prot.loop_step_dict
        catalog_maker(tmp_path)
>       run_test_protocol(tmp_path, prot)

protocol_test.py:252: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
protocol_test.py:547: in run_test_protocol
    py_package.main()
/tmp/pytest-of-ktaha/pytest-5/test_simple_sweep_with_plot_an0/test_simple_sweep_protocol.py:163: in main
    additional_step_data = steps_add_main(RE, devs)
/tmp/pytest-of-ktaha/pytest-5/test_simple_sweep_with_plot_an0/test_simple_sweep_protocol.py:101: in steps_add_main
    plots, subs, _ = create_plots_Simple_Sweep(RE, "Simple_Sweep")
/tmp/pytest-of-ktaha/pytest-5/test_simple_sweep_with_plot_an0/test_simple_sweep_protocol.py:57: in create_plots_Simple_Sweep
    plot_0 = plot_widget.PlotWidget(x_name="demo_instrument_motorY", y_names=['demo_instrument_detectorY'], ylabel="demo_instrument_detectorY", xlabel="demo_instrument_motorY", title="", stream_name=stream, namespace=namespace, fits=fits, multi_stream=False, y_axes={'demo_instrument_detectorY': 1}, ylabel2="", logX=False, logY=False, logY2=False, maxlen="inf")
../main_classes/plot_widget.py:139: in __init__
    canvas = MPLwidget()
../main_classes/plot_widget.py:57: in __init__
    fig, ax = plt.subplots()
/home/ktaha/virtualenv/camels/lib/python3.10/site-packages/matplotlib/pyplot.py:1613: in subplots
    fig = figure(**fig_kw)
/home/ktaha/virtualenv/camels/lib/python3.10/site-packages/matplotlib/pyplot.py:934: in figure
    manager = new_figure_manager(
/home/ktaha/virtualenv/camels/lib/python3.10/site-packages/matplotlib/pyplot.py:464: in new_figure_manager
    _warn_if_gui_out_of_main_thread()
/home/ktaha/virtualenv/camels/lib/python3.10/site-packages/matplotlib/pyplot.py:441: in _warn_if_gui_out_of_main_thread
    canvas_class = cast(type[FigureCanvasBase], _get_backend_mod().FigureCanvas)
/home/ktaha/virtualenv/camels/lib/python3.10/site-packages/matplotlib/pyplot.py:280: in _get_backend_mod
    switch_backend(rcParams._get("backend"))  # type: ignore[attr-defined]
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

newbackend = 'TkAgg'

    def switch_backend(newbackend: str) -> None:
        """
        Set the pyplot backend.
    
        Switching to an interactive backend is possible only if no event loop for
        another interactive backend has started.  Switching to and from
        non-interactive backends is always possible.
    
        If the new backend is different than the current backend then all open
        Figures will be closed via ``plt.close('all')``.
    
        Parameters
        ----------
        newbackend : str
            The case-insensitive name of the backend to use.
    
        """
        global _backend_mod
        # make sure the init is pulled up so we can assign to it later
        import matplotlib.backends
    
        if newbackend is rcsetup._auto_backend_sentinel:
            current_framework = cbook._get_running_interactive_framework()
            mapping = {'qt': 'qtagg',
                       'gtk3': 'gtk3agg',
                       'gtk4': 'gtk4agg',
                       'wx': 'wxagg',
                       'tk': 'tkagg',
                       'macosx': 'macosx',
                       'headless': 'agg'}
    
            if current_framework in mapping:
                candidates = [mapping[current_framework]]
            else:
                candidates = []
            candidates += [
                "macosx", "qtagg", "gtk4agg", "gtk3agg", "tkagg", "wxagg"]
    
            # Don't try to fallback on the cairo-based backends as they each have
            # an additional dependency (pycairo) over the agg-based backend, and
            # are of worse quality.
            for candidate in candidates:
                try:
                    switch_backend(candidate)
                except ImportError:
                    continue
                else:
                    rcParamsOrig['backend'] = candidate
                    return
            else:
                # Switching to Agg should always succeed; if it doesn't, let the
                # exception propagate out.
                switch_backend("agg")
                rcParamsOrig["backend"] = "agg"
                return
        # have to escape the switch on access logic
        old_backend = dict.__getitem__(rcParams, 'backend')
    
        module = importlib.import_module(cbook._backend_module_name(newbackend))
        canvas_class = module.FigureCanvas
    
        required_framework = canvas_class.required_interactive_framework
        if required_framework is not None:
            current_framework = cbook._get_running_interactive_framework()
            if (current_framework and required_framework
                    and current_framework != required_framework):
>               raise ImportError(
                    "Cannot load backend {!r} which requires the {!r} interactive "
                    "framework, as {!r} is currently running".format(
                        newbackend, required_framework, current_framework))
E               ImportError: Cannot load backend 'TkAgg' which requires the 'tk' interactive framework, as 'qt' is currently running

/home/ktaha/virtualenv/camels/lib/python3.10/site-packages/matplotlib/pyplot.py:350: ImportError
----------------------------- Captured stdout call -----------------------------
connecting demo_instrument
devices connected
demo_instrument is being closed
----------------------------- Captured stderr call -----------------------------







  0%|          | 0/1 [00:00<?, ?it/s]�[A�[A�[A�[A�[A�[A
=============================== warnings summary ===============================
../CAMELS_start.py:10
  /home/ktaha/dev/NOMAD-CAMELS/nomad_camels/CAMELS_start.py:10: DeprecationWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html
    from pkg_resources import resource_filename

../../../../virtualenv/camels/lib/python3.10/site-packages/databroker/v1.py:4
  /home/ktaha/virtualenv/camels/lib/python3.10/site-packages/databroker/v1.py:4: DeprecationWarning: 
  Pyarrow will become a required dependency of pandas in the next major release of pandas (pandas 3.0),
  (to allow more performant data types, such as the Arrow string type, and better interoperability with other libraries)
  but was not found to be installed on your system.
  If this would cause problems for you,
  please provide us feedback at https://github.com/pandas-dev/pandas/issues/54466
          
    import pandas

protocol_test.py: 20 warnings
  /home/ktaha/virtualenv/camels/lib/python3.10/site-packages/databroker/v1.py:965: PendingDeprecationWarning: The method Broker.insert may be removed in a future release of databroker.
    warnings.warn(

protocol_test.py::test_read_channels
protocol_test.py::test_trigger_and_read_channels
  /home/ktaha/virtualenv/camels/lib/python3.10/site-packages/databroker/intake_xarray_core/base.py:23: FutureWarning: The return type of `Dataset.dims` will be changed to return a set of dimension names in future, in order to be more consistent with `DataArray.dims`. To access a mapping from dimension names to lengths, please use `Dataset.sizes`.
    'dims': dict(self._ds.dims),

-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html
=========================== short test summary info ============================
FAILED protocol_test.py::test_gradient_descent - ImportError: Cannot load bac...
FAILED protocol_test.py::test_simple_sweep_with_plot_and_fit - ImportError: C...
================== 2 failed, 13 passed, 24 warnings in 6.87s ===================

[ktahar]

And many warnings are raised during test.

[NicolasCARPi]

and they must run at each push/release on a github action. I recommend using "black" for linting.

[ktahar]

and they must run at each push/release on a github action. I recommend using "black" for linting.

I agree. These may not be the requirements for JOSS publication but I also recommend to do them.

[A-D-Fuchs]

We already have automated tests running for every push, see here.
We updated all Python files using "black".

[Jo-Leh]

Thanks for posting the full output.
Unfortunately I cannot reproduce the error you encountered, also on GitHub Actions we never encountered that error. It might be an issue with matplotlib not knowing, which backend to use, so we now tell it to use qt. This will be added in 1.0.1 which we plan to release later today.
The warnings are quite a lot, you are right. All I could find are DeprecationWarnings or similar, coming from packages using other packages. We will keep an eye on those and set restraints on the required package versions if necessary.

[Jo-Leh]

I am still unsure about the error you encountered, @ktahar.
It might be connected to the Qt libraries and some necessary packages for it. Some libraries are now listed at https://fau-lap.github.io/NOMAD-CAMELS/doc/installation/installation_custom_unix.html#troubleshooting

Also, we uploaded a minor update (also to pypi, v1.0.1).

[ktahar]

Hi @Jo-Leh , Sorry for slow response.
After updating nomad-camels, error has gone.
Both main and develop branches are fine now on my environment.

And I found you have already added "how to run test" page in the docs. Thank you.