Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Basic statistics allow computation on sparse data and add test #2095

Draft
wants to merge 14 commits into
base: main
Choose a base branch
from
Draft

Basic statistics allow computation on sparse data and add test #2095

wants to merge 14 commits into from

Conversation

md-shafiul-alam
Copy link
Contributor

Description

Add a comprehensive description of proposed changes

List associated issue number(s) if exist(s): #6 (for example)

Documentation PR (if needed): #1340 (for example)

Benchmarks PR (if needed): IntelPython/scikit-learn_bench#155 (for example)

Checklist to comply with before moving PR from draft:

PR completeness and readability

  • I have reviewed my changes thoroughly before submitting this pull request.
  • I have commented my code, particularly in hard-to-understand areas.
  • I have updated the documentation to reflect the changes or created a separate PR with update and provided its number in the description, if necessary.
  • Git commit message contains an appropriate signed-off-by string (see CONTRIBUTING.md for details).
  • I have added a respective label(s) to PR if I have a permission for that.
  • I have resolved any merge conflicts that might occur with the base branch.

Testing

  • I have run it locally and tested the changes extensively.
  • All CI jobs are green or I have provided justification why they aren't.
  • I have extended testing suite if new functionality was introduced in this PR.

Performance

  • I have measured performance for affected algorithms using scikit-learn_bench and provided at least summary table with measured data, if performance change is expected.
  • I have provided justification why performance has changed or why changes are not expected.
  • I have provided justification why quality metrics have changed or why changes are not expected.
  • I have extended benchmarking suite and provided corresponding scikit-learn_bench PR if new measurable functionality was introduced in this PR.

@md-shafiul-alam md-shafiul-alam added bug Something isn't working enhancement New feature or request and removed bug Something isn't working labels Oct 8, 2024
@md-shafiul-alam md-shafiul-alam changed the title Basic statistics sparse fix and adding test Basic statistics allow computation on sparse data and add test Oct 8, 2024
Vika-F
Vika-F reviewed Oct 9, 2024
View reviewed changes
sklearnex/basic_statistics/tests/test_basic_statistics.py Outdated
if weighted:
weights = gen.uniform(low=-0.5, high=1.0, size=row_count)
weights = weights.astype(dtype=dtype)
basicstat = BasicStatistics(result_options=["mean", "max", "sum"])
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

According to onedal tests, need to exclude "max" at it contains bugs:
https://github.com/intel/scikit-learn-intelex/blob/main/onedal/basic_statistics/tests/test_basic_statistics.py#L273

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

According to onedal tests, need to exclude "max" at it contains bugs: https://github.com/intel/scikit-learn-intelex/blob/main/onedal/basic_statistics/tests/test_basic_statistics.py#L273

@olegkkruglov please message out a link to the ticket associated with this error (just to make sure it wasn't lost)

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm not sure if I have it. This skip was added not by me.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sorry, my mistake, I didn't dig deep enough in the git blame. Turns out it was introduced here #1846 by @Vika-F . Do you know if there was any follow-up work after #1846 on the max issues/ any memory on what was going on?

Copy link
Contributor Author

@md-shafiul-alam md-shafiul-alam Oct 10, 2024 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I observed that issue as well, temporarily removed the "max" in tests for this PR.

@icfaust
Copy link
Contributor

icfaust commented Oct 9, 2024

Please add this to run_to_run_stability sparse stability testing.

samir-nasibli
samir-nasibli reviewed Oct 11, 2024
View reviewed changes
sklearnex/basic_statistics/tests/test_basic_statistics.py
@@ -178,6 +181,53 @@ def test_multiple_options_on_random_data(
assert_allclose(gtr_sum, res_sum, atol=tol)


@pytest.mark.parametrize("queue", get_queues())
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

please use _get_dataframes_and_queues instead

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sparse data can't work with dataframes

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@olegkkruglov olegkkruglov olegkkruglov left review comments

@Vika-F Vika-F Vika-F left review comments

@samir-nasibli samir-nasibli samir-nasibli left review comments

@icfaust icfaust icfaust left review comments

@Alexsandruss Alexsandruss Awaiting requested review from Alexsandruss Alexsandruss will be requested when the pull request is marked ready for review Alexsandruss is a code owner

At least 1 approving review is required to merge this pull request.

Assignees
No one assigned
Labels
enhancement New feature or request
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
5 participants
@md-shafiul-alam @icfaust @samir-nasibli @Vika-F @olegkkruglov