feat: make ColumnDropper dataframe-agnostic

[MarcoGorelli]

Hey 👋 I discussed a bit with Francesco, who discussed it with Vincent. So, I was keen to get the ball rolling

Description

Making a start towards dataframe-agnosticism. For now, this only makes ColumnDropped dataframe-agnostic. It requires adding an extremely lightweight dependecy (Narwhals), but the result is that computation can happen natively for pandas/Polars/modin/cuDF (and any other library which may want to become Narwhals-compliant), without any data conversion.

pandas is still a required dependency, as it's required for the rest of scikit-lego. But, it doesn't seem that far-fetched to be able to make all of scikit-lego dataframe-agnostic. If you're open to this, we could do it in stages?

There is no impact on current users, other than that Narwhals would be a required dependency (though it's really lightweight, and without any dependencies itself, so there's no risk for conflicts)

Type of change

• Bug fix (non-breaking change which fixes an issue)
• New feature (non-breaking change which adds functionality)
• Breaking change (fix or feature that would cause existing functionality to not work as expected)

Checklist:

• My code follows the style guidelines (ruff)
• I have commented my code, particularly in hard-to-understand areas
• I have made corresponding changes to the documentation (also to the readme.md)
• I have added tests that prove my fix is effective or that my feature works
• I have added tests to check whether the new feature adheres to the sklearn convention
• New and existing unit tests pass locally with my changes

Not sure about some of these items so I haven't checked them.

---

Demo

pandas users, they can keep using scikit-lego, without needing Polars to be installed

$ pip uninstall polars -y
Found existing installation: polars 0.20.23
Uninstalling polars-0.20.23:
  Successfully uninstalled polars-0.20.23
$ cat t.py 
import pandas as pd
from sklego.preprocessing import ColumnDropper

df = pd.DataFrame({
    "name": ["Swen", "Victor", "Alex"],
    "length": [1.82, 1.85, 1.80],
    "shoesize": [42, 44, 45]
})
print(ColumnDropper(["name"]).fit_transform(df))
$ python t.py 
   length  shoesize
0    1.82        42
1    1.85        44
2    1.80        45

But, Polars users can use this, without it doing any conversion to pandas. In particular, Polars LazyFrame is supported (and stays lazy, but here's I'm calling .collect to show you the output):

$ cat t.py 
import polars as pl
from sklego.preprocessing import ColumnDropper

df = pl.LazyFrame({
    "name": ["Swen", "Victor", "Alex"],
    "length": [1.82, 1.85, 1.80],
    "shoesize": [42, 44, 45]
})
result = ColumnDropper(["name"]).fit_transform(df)
print(result.explain())
print(result.collect())
$ python t.py 
DF ["name", "length", "shoesize"]; PROJECT 2/3 COLUMNS; SELECTION: "None"
shape: (3, 2)
┌────────┬──────────┐
│ length ┆ shoesize │
│ ---    ┆ ---      │
│ f64    ┆ i64      │
╞════════╪══════════╡
│ 1.82   ┆ 42       │
│ 1.85   ┆ 44       │
│ 1.8    ┆ 45       │
└────────┴──────────┘

[MarcoGorelli]

I'm removing this here, as nw.from_native already raises a similar error. For example, if you tried passing in a numpy array here, you'd get:

TypeError: Expected pandas-like dataframe, Polars dataframe, or Polars lazyframe, got: <class 'numpy.ndarray'>

[FBruzzesi]

Honestly it looks quite neat! I left some considerations for better sync with narwhals in the long term.

Edit: Regarding macos tests, we are getting a lot of fails since latest upgraded to macos-14. We definitely need to get a closer look at those, but completely unrelated to the PR

[koaning]

@FBruzzesi notice those failing test runs, I have a creeping suspicion that these may be caused by uv and they feel similar to the failing cronjobs. Just to check, did you have a look?

[FBruzzesi]

@koaning I finally took a look at the CI...and you are right 😁

• Moving back to pip doesn't break when building wheel (proof on my fork)
• uv has some related issue when building wheels from scratch and qdldl has no built distribution for macos 10+ and arm architecture.

As a side note: even with pip, the --pre flag break as of scikit-learn==1.5.0rc1 (proof on my fork).

[FBruzzesi]

@koaning another food for thought is that... all these transformers are in a ../pandastransformers.py module, however this should be transparent to the user as they are import-able from sklego.preprocessing directly. I would feel fairly safe to change the filename once all of them are dataframe agnostic

[koaning]

@FBruzzesi any objections to moving back to pip for now? I prefer to wait until uv is a bit more reliable before relying our CI on it.

I made an issue for that v1.5rc01 test failure on the sklearn side, feels like it might be breaking so upstream should know about it.

I would feel fairly safe to change the filename once all of them are dataframe agnostic.

Totally!

[FBruzzesi]

I am biased as I added a few commits myself 😂

[koaning]

Do we want to release early with a micro release by the way or would we prefer to make a bigger splash? There's something to be said for both, but given that we are introducing a dependency ... might be good to make a larger release?

[anopsy]

If everybody is fine with it, I could finish pandastransformers.py, or at least have a shot at ColumnSelector and see how it goes

[FBruzzesi]

@koaning I would personally opt for one somewhat big release. We can wait to merge the PRs related to Narwhals migration to have some space for bug fixes if needed.

@anopsy Of course feel free to develop that. I will create an issue to track progress 😇

[MarcoGorelli]

Just got a question about

We can wait to merge the PRs related to Narwhals migration to have some space for bug fixes if needed.

No objections to waiting for a bigger release to include this, I'm just worried that it's going to become quite hard to review if everything is done in a single PR, or if PRs build on top of other open PRs

May I suggest that you (i.e. a scikit-lego committer) make a narwhals branch in scikit-lego, we make PRs to that branch, and then if/when you're ready, you merge the narwhals branch into main and include it in the release?

[FBruzzesi]

@MarcoGorelli that's very reasonable! Here you have it: narwhals-development branch.

[koaning]

FWIW I think pushing to main is fine too. It's more of a "oh, we have one new feature, usually that means deploy". But we can also collect a bunch of things to main ... this is the main thing we're going to work on short-term.