Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Intermittent test failure on Wayland testbed tests #2807

Closed
freakboy3742 opened this issue Sep 2, 2024 · 4 comments · Fixed by #2808
Closed

Intermittent test failure on Wayland testbed tests #2807

freakboy3742 opened this issue Sep 2, 2024 · 4 comments · Fixed by #2808
Labels
bug A crash or error in behavior. linux The issue relates Linux support.

Comments

@freakboy3742
Copy link
Member

Describe the bug

There is an intermittent error on the linux-wayland testbed backend test.

Steps to reproduce

Difficult to reproduce; it occurs in CI on the linux-wayland testbed target:

=================================== FAILURES ===================================
____________________________ test_save_as_document _____________________________
Traceback (most recent call last):
  File "/home/runner/work/toga/toga/testbed/build/testbed/ubuntu/jammy/testbed-0.0.1/usr/lib/testbed/app_packages/_pytest/runner.py", line 341, in from_call
    result: TResult | None = func()
  File "/home/runner/work/toga/toga/testbed/build/testbed/ubuntu/jammy/testbed-0.0.1/usr/lib/testbed/app_packages/_pytest/runner.py", line 242, in <lambda>
    lambda: runtest_hook(item=item, **kwds), when=when, reraise=reraise
  File "/home/runner/work/toga/toga/testbed/build/testbed/ubuntu/jammy/testbed-0.0.1/usr/lib/testbed/app_packages/pluggy/_hooks.py", line 513, in __call__
    return self._hookexec(self.name, self._hookimpls.copy(), kwargs, firstresult)
  File "/home/runner/work/toga/toga/testbed/build/testbed/ubuntu/jammy/testbed-0.0.1/usr/lib/testbed/app_packages/pluggy/_manager.py", line 120, in _hookexec
    return self._inner_hookexec(hook_name, methods, kwargs, firstresult)
  File "/home/runner/work/toga/toga/testbed/build/testbed/ubuntu/jammy/testbed-0.0.1/usr/lib/testbed/app_packages/pluggy/_callers.py", line 139, in _multicall
    raise exception.with_traceback(exception.__traceback__)
  File "/home/runner/work/toga/toga/testbed/build/testbed/ubuntu/jammy/testbed-0.0.1/usr/lib/testbed/app_packages/pluggy/_callers.py", line 122, in _multicall
    teardown.throw(exception)  # type: ignore[union-attr]
  File "/home/runner/work/toga/toga/testbed/build/testbed/ubuntu/jammy/testbed-0.0.1/usr/lib/testbed/app_packages/_pytest/threadexception.py", line 92, in pytest_runtest_call
    yield from thread_exception_runtest_hook()
  File "/home/runner/work/toga/toga/testbed/build/testbed/ubuntu/jammy/testbed-0.0.1/usr/lib/testbed/app_packages/_pytest/threadexception.py", line 68, in thread_exception_runtest_hook
    yield
  File "/home/runner/work/toga/toga/testbed/build/testbed/ubuntu/jammy/testbed-0.0.1/usr/lib/testbed/app_packages/pluggy/_callers.py", line 122, in _multicall
    teardown.throw(exception)  # type: ignore[union-attr]
  File "/home/runner/work/toga/toga/testbed/build/testbed/ubuntu/jammy/testbed-0.0.1/usr/lib/testbed/app_packages/_pytest/unraisableexception.py", line 95, in pytest_runtest_call
    yield from unraisable_exception_runtest_hook()
  File "/home/runner/work/toga/toga/testbed/build/testbed/ubuntu/jammy/testbed-0.0.1/usr/lib/testbed/app_packages/_pytest/unraisableexception.py", line 70, in unraisable_exception_runtest_hook
    yield
  File "/home/runner/work/toga/toga/testbed/build/testbed/ubuntu/jammy/testbed-0.0.1/usr/lib/testbed/app_packages/pluggy/_callers.py", line 122, in _multicall
    teardown.throw(exception)  # type: ignore[union-attr]
  File "/home/runner/work/toga/toga/testbed/build/testbed/ubuntu/jammy/testbed-0.0.1/usr/lib/testbed/app_packages/_pytest/logging.py", line 848, in pytest_runtest_call
    yield from self._runtest_for(item, "call")
  File "/home/runner/work/toga/toga/testbed/build/testbed/ubuntu/jammy/testbed-0.0.1/usr/lib/testbed/app_packages/_pytest/logging.py", line 831, in _runtest_for
    yield
  File "/home/runner/work/toga/toga/testbed/build/testbed/ubuntu/jammy/testbed-0.0.1/usr/lib/testbed/app_packages/pluggy/_callers.py", line 122, in _multicall
    teardown.throw(exception)  # type: ignore[union-attr]
  File "/home/runner/work/toga/toga/testbed/build/testbed/ubuntu/jammy/testbed-0.0.1/usr/lib/testbed/app_packages/_pytest/capture.py", line 879, in pytest_runtest_call
    return (yield)
  File "/home/runner/work/toga/toga/testbed/build/testbed/ubuntu/jammy/testbed-0.0.1/usr/lib/testbed/app_packages/pluggy/_callers.py", line 122, in _multicall
    teardown.throw(exception)  # type: ignore[union-attr]
  File "/home/runner/work/toga/toga/testbed/build/testbed/ubuntu/jammy/testbed-0.0.1/usr/lib/testbed/app_packages/_pytest/skipping.py", line 257, in pytest_runtest_call
    return (yield)
  File "/home/runner/work/toga/toga/testbed/build/testbed/ubuntu/jammy/testbed-0.0.1/usr/lib/testbed/app_packages/pluggy/_callers.py", line 103, in _multicall
    res = hook_impl.function(*args)
  File "/home/runner/work/toga/toga/testbed/build/testbed/ubuntu/jammy/testbed-0.0.1/usr/lib/testbed/app_packages/_pytest/runner.py", line 174, in pytest_runtest_call
    item.runtest()
  File "/home/runner/work/toga/toga/testbed/build/testbed/ubuntu/jammy/testbed-0.0.1/usr/lib/testbed/app_packages/pytest_asyncio/plugin.py", line 457, in runtest
    super().runtest()
  File "/home/runner/work/toga/toga/testbed/build/testbed/ubuntu/jammy/testbed-0.0.1/usr/lib/testbed/app_packages/_pytest/python.py", line 1627, in runtest
    self.ihook.pytest_pyfunc_call(pyfuncitem=self)
  File "/home/runner/work/toga/toga/testbed/build/testbed/ubuntu/jammy/testbed-0.0.1/usr/lib/testbed/app_packages/pluggy/_hooks.py", line 513, in __call__
    return self._hookexec(self.name, self._hookimpls.copy(), kwargs, firstresult)
  File "/home/runner/work/toga/toga/testbed/build/testbed/ubuntu/jammy/testbed-0.0.1/usr/lib/testbed/app_packages/pluggy/_manager.py", line 120, in _hookexec
    return self._inner_hookexec(hook_name, methods, kwargs, firstresult)
  File "/home/runner/work/toga/toga/testbed/build/testbed/ubuntu/jammy/testbed-0.0.1/usr/lib/testbed/app_packages/pluggy/_callers.py", line 182, in _multicall
    return outcome.get_result()
  File "/home/runner/work/toga/toga/testbed/build/testbed/ubuntu/jammy/testbed-0.0.1/usr/lib/testbed/app_packages/pluggy/_result.py", line 100, in get_result
    raise exc.with_traceback(exc.__traceback__)
  File "/home/runner/work/toga/toga/testbed/build/testbed/ubuntu/jammy/testbed-0.0.1/usr/lib/testbed/app_packages/pluggy/_callers.py", line 103, in _multicall
    res = hook_impl.function(*args)
  File "/home/runner/work/toga/toga/testbed/build/testbed/ubuntu/jammy/testbed-0.0.1/usr/lib/testbed/app_packages/_pytest/python.py", line 159, in pytest_pyfunc_call
    result = testfunction(**testargs)
  File "/home/runner/work/toga/toga/testbed/build/testbed/ubuntu/jammy/testbed-0.0.1/usr/lib/testbed/app_packages/pytest_asyncio/plugin.py", line 929, in inner
    _loop.run_until_complete(task)
  File "/home/runner/work/toga/toga/testbed/build/testbed/ubuntu/jammy/testbed-0.0.1/usr/lib/testbed/app/tests/conftest.py", line 133, in run_until_complete
    return asyncio.run_coroutine_threadsafe(coro, self.loop).result()
  File "/usr/lib/python3.10/concurrent/futures/_base.py", line 458, in result
    return self.__get_result()
  File "/usr/lib/python3.10/concurrent/futures/_base.py", line 403, in __get_result
    raise self._exception
  File "/home/runner/work/toga/toga/testbed/build/testbed/ubuntu/jammy/testbed-0.0.1/usr/lib/testbed/app/tests/app/test_document_app.py", line 154, in test_save_as_document
    app.documents[0]._content.write.assert_called_with(
  File "/usr/lib/python3.10/unittest/mock.py", line 920, in assert_called_with
    raise AssertionError(error_message)
AssertionError: expected call not found.
Expected: write(PosixPath('/tmp/pytest-of-runner/pytest-0/test_save_as_document0/new_filename.testbed'))
Actual: not called.
=========================== short test summary info ============================
FAILED tests/app/test_document_app.py::test_save_as_document - AssertionError: expected call not found.
Expected: write(PosixPath('/tmp/pytest-of-runner/pytest-0/test_save_as_document0/new_filename.testbed'))
Actual: not called.
======= 1 failed, 444 passed, 59 skipped, 7 xfailed in 105.96s (0:01:45) =======

[testbed] Test suite failed!

Expected behavior

The test should pass. It usually does pass if you re-run the Wayland test.

Screenshots

No response

Environment

  • Operating System: CI
  • Python version: 3.10
  • Software versions:
    • Briefcase: 0.3.19+
    • Toga: 0.4.6+
    • ...

Logs

Additional context

The error is presumably timing related; something is happening too quickly (or too slowly) under CI conditions. However, I've been unable to replicate the error locally.
@freakboy3742 freakboy3742 added bug A crash or error in behavior. linux The issue relates Linux support. labels Sep 2, 2024
@rmartin16
Copy link
Member

The error is occurring because this hasattr() call is returning False:

toga/core/src/toga/documents.py

Lines 373 to 380 in 069557c

async def save_as(self):
"""Save the current content of an app under a different filename.
If there isn't a current window, or the current window hasn't defined a
``save_as()`` method, the save-as request will be ignored.
"""
if hasattr(self.app.current_window, "save_as"):
await self.app.current_window.save_as()

@rmartin16
Copy link
Member

Whoops...ctrl+enter submits your post apparently.

Anyway, I was able to replicate this with:

while true; do WAYLAND_DISPLAY=toga briefcase run linux --log --test -- tests/app/test_document_app.py; sleep 1; done

While I haven't fully digested all of the Document code, I presume that the wrong window has focus when the test runs.

@freakboy3742
Copy link
Member Author

Oh sure - the question is why that test fails sometimes, and only on the save-as test - the save and save-all tests do almost exactly the same thing.

I guess adding deliberate delay=0.1 or similar to this pause should serve as a workaround; I'm just not clear why only on Wayland, and only sometimes, the default window creation delay isn't sufficient.

I guess there's precedent in wait_for_window()... so we should probably add a 0.1s delay to all the "document has been opened" redraws, just in case.

@rmartin16
Copy link
Member

In the case of test_save_all_documents, it calls app.documents.save_all() which iterates all windows and calls save() for each one; so, the test should be insensitive to which window has focus.

As for test_save_document, it seems this may be an order of events thing; I was able to create the issue for test_save_document by running test_save_as_document first.

AssertionError: expected call not found.
Expected: write(PosixPath('/home/russell/github/beeware/toga/testbed/build/testbed/pop/jammy/testbed-0.0.1/usr/lib/testbed/app/tests/app/docs/example.testbed'))
Actual: not called.
=========================== short test summary info ============================
FAILED tests/app/test_document_app.py::test_save_document - AssertionError: expected call not found.

@rmartin16 rmartin16 mentioned this issue Sep 2, 2024
Merged
4 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
bug A crash or error in behavior. linux The issue relates Linux support.
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Avoid Testbed failures for test_save_as_document rmartin16/toga
2 participants
@freakboy3742 @rmartin16