Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

FEAT-#7331: Initial Polars API #7332

Merged
merged 20 commits into from
Jul 24, 2024
Merged

FEAT-#7331: Initial Polars API #7332

merged 20 commits into from
Jul 24, 2024

Conversation

devin-petersohn
Copy link
Collaborator

@devin-petersohn devin-petersohn commented Jun 26, 2024 •
edited
Loading

This commit adds a polars namespace to Modin, and the DataFrame and Series objects and their respective APIs.

This PR adds the following to Modin:

  • modin.polars.DataFrame
  • modin.polars.Series
  • modin.polars.GroupBy

This PR does NOT include matching error handling and is still missing several polars features:

  • LazyFrame
  • Expressions
  • String, Temporal, Struct, and other Series accessors
  • Several parameters
  • Operators that we don't have query compiler methods for
    • e.g. sin, cos, tan, etc.

These features will be added in a future PR.

What do these changes do?

  • first commit message and PR title follow format outlined here

    NOTE: If you edit the PR title to match this format, you need to add another commit (even if it's empty) or amend your last commit for the CI job that checks the PR title to pick up the new PR title.

  • passes flake8 modin/ asv_bench/benchmarks scripts/doc_checker.py
  • passes black --check modin/ asv_bench/benchmarks scripts/doc_checker.py
  • signed commit with git commit -s
  • Resolves Polars API #7331
  • tests added and passing
  • module layout described at docs/development/architecture.rst is up-to-date

@sfc-gh-dpetersohn
FEAT-modin-project#7331: Initial Polars API
cb4f9aa 
This commit adds a polars namespace to Modin, and the DataFrame and
Series objects and their respective APIs. This doesn't include error
handling and is still missing several polars features:

* LazyFrame
* Expressions
* String, Temporal, Struct, and other Series accessors
* Several parameters
* Operators that we don't have query compiler methods for
   * e.g. sin, cos, tan, etc.

Those will be handled in a future PR.

Signed-off-by: Devin Petersohn <devin.petersohn@snowflake.com>
@sfc-gh-dpetersohn
Lint
b95ac9c 
Signed-off-by: Devin Petersohn <devin.petersohn@snowflake.com>
github-advanced-security[bot]
github-advanced-security bot found potential problems Jun 26, 2024
View reviewed changes
modin/polars/dataframe.py Dismissed Show dismissed Hide dismissed
modin/polars/series.py Fixed Show fixed Hide fixed
modin/polars/dataframe.py Fixed Show fixed Hide fixed
modin/polars/groupby.py Fixed Show fixed Hide fixed
modin/polars/base.py Fixed Show fixed Hide fixed
modin/polars/base.py Fixed Show fixed Hide fixed
modin/polars/dataframe.py Fixed Show fixed Hide fixed
modin/polars/series.py Fixed Show fixed Hide fixed
@sfc-gh-dpetersohn
flake8
cf284dc 
Signed-off-by: Devin Petersohn <devin.petersohn@snowflake.com>
@sfc-gh-dpetersohn
isort
7e8cabb 
Signed-off-by: Devin Petersohn <devin.petersohn@snowflake.com>
github-advanced-security[bot]
github-advanced-security bot found potential problems Jun 27, 2024
View reviewed changes
modin/polars/series.py Dismissed Show dismissed Hide dismissed
@sfc-gh-dpetersohn
headers
1f92673 
Signed-off-by: Devin Petersohn <devin.petersohn@snowflake.com>
@sfc-gh-dpetersohn
forgot one
fb21f30 
Signed-off-by: Devin Petersohn <devin.petersohn@snowflake.com>
@sfc-gh-dpetersohn
Add test
331d7bc 
Signed-off-by: Devin Petersohn <devin.petersohn@snowflake.com>
@sfc-gh-dpetersohn
header
41532f1 
Signed-off-by: Devin Petersohn <devin.petersohn@snowflake.com>
@sfc-gh-dpetersohn
isort
3fd3c3c 
Signed-off-by: Devin Petersohn <devin.petersohn@snowflake.com>
@sfc-gh-dpetersohn
Add to CI
df07fb4 
Signed-off-by: Devin Petersohn <devin.petersohn@snowflake.com>
@sfc-gh-dpetersohn
fix name
b9d554b 
Signed-off-by: Devin Petersohn <devin.petersohn@snowflake.com>
sfc-gh-mvashishtha
sfc-gh-mvashishtha reviewed Jul 2, 2024
View reviewed changes
Copy link
Contributor

@sfc-gh-mvashishtha sfc-gh-mvashishtha left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I reviewed the files alphabetically through item in dataframe.py . I'll make another round of comments later today.

modin/polars/base.py Outdated Show resolved Hide resolved
modin/polars/base.py Show resolved Hide resolved
modin/polars/base.py Outdated Show resolved Hide resolved
modin/polars/dataframe.py Outdated Show resolved Hide resolved
modin/polars/dataframe.py Outdated Show resolved Hide resolved
modin/polars/dataframe.py Show resolved Hide resolved
modin/polars/dataframe.py Show resolved Hide resolved
sfc-gh-dpetersohn and others added 3 commits July 3, 2024 09:58
@sfc-gh-dpetersohn @sfc-gh-mvashishtha
Update modin/polars/base.py
15eec3f 
Co-authored-by: Mahesh Vashishtha <mahesh.vashishtha@snowflake.com>
@sfc-gh-dpetersohn
address comments
58d226f 
Signed-off-by: Devin Petersohn <devin.petersohn@snowflake.com>
@sfc-gh-dpetersohn
polars 1
64a91d7 
Signed-off-by: Devin Petersohn <devin.petersohn@snowflake.com>
sfc-gh-mvashishtha
sfc-gh-mvashishtha reviewed Jul 9, 2024
View reviewed changes
Copy link
Contributor

@sfc-gh-mvashishtha sfc-gh-mvashishtha left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I've made it through len in groupby.py

modin/polars/base.py Show resolved Hide resolved
@sfc-gh-dpetersohn
Update for polars 1.x and fix some hacks
0019245 
Signed-off-by: Devin Petersohn <devin.petersohn@snowflake.com>
@sfc-gh-dpetersohn
Remove hax
5800e3e 
Signed-off-by: Devin Petersohn <devin.petersohn@snowflake.com>
@sfc-gh-dpetersohn
Black
267bc9f 
Signed-off-by: Devin Petersohn <devin.petersohn@snowflake.com>
sfc-gh-mvashishtha
sfc-gh-mvashishtha reviewed Jul 19, 2024
View reviewed changes
Copy link
Contributor

@sfc-gh-mvashishtha sfc-gh-mvashishtha left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I've gone through all the files now!

modin/polars/groupby.py Show resolved Hide resolved
modin/polars/series.py Show resolved Hide resolved
modin/polars/series.py Show resolved Hide resolved
@sfc-gh-dpetersohn
Address comments
4244e64 
Signed-off-by: Devin Petersohn <devin.petersohn@snowflake.com>
@sfc-gh-dpetersohn
Lint
f89c66a 
Signed-off-by: Devin Petersohn <devin.petersohn@snowflake.com>
@sfc-gh-dpetersohn
Address comment
5a072ea 
Signed-off-by: Devin Petersohn <devin.petersohn@snowflake.com>
@devin-petersohn devin-petersohn merged commit 7c1dde0 into modin-project:main Jul 24, 2024
38 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@sfc-gh-dpetersohn sfc-gh-dpetersohn sfc-gh-dpetersohn left review comments

@sfc-gh-mvashishtha sfc-gh-mvashishtha sfc-gh-mvashishtha approved these changes

@mvashishtha mvashishtha Awaiting requested review from mvashishtha mvashishtha is a code owner

@RehanSD RehanSD Awaiting requested review from RehanSD RehanSD is a code owner

@YarShev YarShev Awaiting requested review from YarShev YarShev is a code owner

@vnlitvinov vnlitvinov Awaiting requested review from vnlitvinov vnlitvinov is a code owner

@anmyachev anmyachev Awaiting requested review from anmyachev anmyachev is a code owner

@dchigarev dchigarev Awaiting requested review from dchigarev dchigarev is a code owner

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

Polars API
3 participants
@devin-petersohn @sfc-gh-dpetersohn @sfc-gh-mvashishtha