Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[BUGFIX] Fixing integration test failures. #3959

Conversation

alexsherstinsky
Copy link
Collaborator

@alexsherstinsky alexsherstinsky commented Mar 8, 2024

Scope

We have had consistent failures from two tests involving image processing. Through troubleshooting, the cause was determined to be the running out of storage space on the test host Linux machine in GitHub actions. The fixes are:

  1. Improve error handling in Carton utilities module.
  2. Add GitHub Action steps to clean out unneeded storage space.

Code Pull Requests

Please provide the following:

  • a clear explanation of what your code does
  • if applicable, a reference to an issue
  • a reproducible test for your PR (code, config and data sample)

Documentation Pull Requests

Note that the documentation HTML files are in docs/ while the Markdown sources are in mkdocs/docs.

If you are proposing a modification to the documentation you should change only the Markdown files.

api.md is automatically generated from the docstrings in the code, so if you want to change something in that file, first modify ludwig/api.py docstring, then run mkdocs/code_docs_autogen.py, which will create mkdocs/docs/api.md .

Copy link

github-actions bot commented Mar 8, 2024

Unit Test Results

  6 files  ±0    6 suites  ±0   14m 29s ⏱️ ±0s
12 tests ±0    9 ✔️ ±0    3 💤 ±0  0 ±0 
60 runs  ±0  42 ✔️ ±0  18 💤 ±0  0 ±0 

Results for commit 9e5b5cd. ± Comparison against base commit e0f112c.

♻️ This comment has been updated with latest results.

@alexsherstinsky alexsherstinsky changed the title [BUGFIX] Troubleshooting test failures. <DO_NOT_MERGE>[BUGFIX] Troubleshooting test failures.</DO_NOT_MERGE> Mar 8, 2024
@alexsherstinsky alexsherstinsky changed the title <DO_NOT_MERGE>[BUGFIX] Troubleshooting test failures.</DO_NOT_MERGE> <DO_NOT_MERGE_TROUBLESHOOTING>[BUGFIX] Troubleshooting test failures.</DO_NOT_MERGE_TROUBLESHOOTING> Mar 8, 2024
@alexsherstinsky alexsherstinsky changed the title <DO_NOT_MERGE_TROUBLESHOOTING>[BUGFIX] Troubleshooting test failures.</DO_NOT_MERGE_TROUBLESHOOTING> [BUGFIX] Fixing integration test failures. Mar 12, 2024
@alexsherstinsky alexsherstinsky marked this pull request as ready for review March 12, 2024 16:25
uses: actions/setup-python@v2
with:
python-version: 3.9
python-version: "3.10"
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is the python version bump strictly necessary?

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@arnavgarg1 It is not; I just felt that testing on the latest -- if all tests pass -- would be better. What do you think?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fair enough!

Copy link
Contributor

@justinxzhao justinxzhao left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is a lovely change -- thank you!

exception_message: str = 'An Exception inside "pack()" occurred.\n'
exception_traceback: str = traceback.format_exc()
exception_message += f'{type(e).__name__}: "{str(e)}". Traceback: "{exception_traceback}".'
sys.stderr.write(exception_message)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is it possible to use the logging directory here? Or do we actually need to write to sys stderr when we're raising it through the ValueError anyway?

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@arnavgarg1 I was thinking of showing these messages on the screen, so that in GitHub log review, we can see exactly what went wrong. We could create a directory, but that might hide the failure error messages. It was very challenging to troubleshoot even this one, because the code is wrapping Rust. Happy to discuss, of course. Thanks!

@alexsherstinsky alexsherstinsky merged commit 606c732 into master Mar 12, 2024
18 checks passed
@alexsherstinsky alexsherstinsky deleted the bugfix/alexsherstinsky/integration_tests/fix_tests_f_intermittent_failures_in_images_preprocessing_and_carton_export-2024_03_08-33 branch March 12, 2024 17:53
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants