Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Add dataset fixture to ease fetching DataFrames for tests #847

Merged
merged 35 commits into from
Apr 14, 2023
Merged

Add dataset fixture to ease fetching DataFrames for tests #847

merged 35 commits into from
Apr 14, 2023

Conversation

dagardner-nv
Copy link
Contributor

@dagardner-nv dagardner-nv commented Apr 5, 2023 •
edited
Loading

Description

  • Adds new DatasetLoader class which loads and caches DataFrames of test data.
  • Move DataFrame utility methods specific to tests to the new DatasetLoader class
  • Add dataset, dataset_pandas and dataset_cudf fixtures.
  • Replace usage of the use_pandas marker as a way to get test data in Pandas format which had the unintentional impact of disabling C++ execution for a test. Resulting in a total number of 873 test iterations (when using both --run_slow and --run_kafka)

fixes #785

Checklist

  • I am familiar with the Contributing Guidelines.
  • New or existing tests cover these changes.
  • The documentation is up to date with these changes.

@dagardner-nv dagardner-nv added non-breaking Non-breaking change improvement Improvement to existing functionality 3 - Ready for Review labels Apr 5, 2023
@dagardner-nv dagardner-nv requested a review from a team as a code owner April 5, 2023 18:41
@dagardner-nv dagardner-nv mentioned this pull request Apr 13, 2023
Merged
3 tasks
mdemoret-nv
mdemoret-nv approved these changes Apr 13, 2023
View reviewed changes
Copy link
Contributor

@mdemoret-nv mdemoret-nv left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Approved. Merge once comments are addressed

tests/dataset_loader.py Outdated Show resolved Hide resolved
tests/dataset_loader.py Outdated Show resolved Hide resolved
tests/dataset_loader.py Outdated Show resolved Hide resolved
tests/dataset_loader.py Outdated Show resolved Hide resolved
tests/utils.py Show resolved Hide resolved
tests/conftest.py Outdated Show resolved Hide resolved
tests/conftest.py Outdated Show resolved Hide resolved
tests/dataset_loader.py Outdated Show resolved Hide resolved
mdemoret-nv
mdemoret-nv reviewed Apr 13, 2023
View reviewed changes
tests/test_conftest.py Outdated Show resolved Hide resolved
@dagardner-nv
Copy link
Contributor Author

/merge

@rapids-bot rapids-bot bot merged commit 5f85d5f into nv-morpheus:branch-23.07 Apr 14, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@mdemoret-nv mdemoret-nv mdemoret-nv approved these changes

Assignees
No one assigned
Labels
improvement Improvement to existing functionality non-breaking Non-breaking change
Projects
Morpheus Boards
Archived in project
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

[FEA]: Implement a dataset fixture for loading, modifying and organizing test datasets
2 participants
@dagardner-nv @mdemoret-nv