feat: Added `struct` namespace with `field` method. #2146

osoucy · 2025-03-04T17:23:09Z

What type of PR is this? (check all applicable)

Related issues

Related issue #<issue number>
Closes #<issue number>

Checklist

Code follows style guide (ruff)
Tests added
Documented the changes

If you have comments or can explain your changes, please do so below

Introduction of the strut namespace for applicable expressions and series with an initial field function that returns a field of a struct.

# Conflicts: # narwhals/_pandas_like/expr.py

MarcoGorelli · 2025-03-04T18:15:17Z

narwhals/_pandas_like/series_struct.py

+
+    def field(self: Self, name: str) -> PandasLikeSeries:
+        return self._compliant_series._from_native_series(
+            self._compliant_series._native_series.apply(lambda x: x[name]).rename(name),


Instead of using apply can we use https://pandas.pydata.org/pandas-docs/dev/reference/api/pandas.Series.struct.field.html and only support this for pyarrow-backed dtypes?

I feel the majority of Pandas users don't leverage pyarrow-backed dtypes. I added a switch to check and use the struct namespace if available and fallbacks on the apply function if it's not the case.

EdAbati

This looks great! Thank you for doing this 🙌🏼

I've added a couple of comments 'on the go' (I'll check the rest later)

narwhals/_polars/expr.py

docs/api-completeness/expr_struct.md

docs/api-completeness/series_struct.md

FBruzzesi

Thanks @osoucy - it generally looks very promising. Left an additional comment to those of Edo and Marco

FBruzzesi · 2025-03-04T18:31:56Z

narwhals/expr_struct.py

+            >>> df.with_columns(name=nw.col("user").struct.field("name"))
+        """


Output is missing and doctest check would fail without it

The output has to be manually generated? There is no command that I can use to generate it?

narwhals/series_struct.py

Co-authored-by: Edoardo Abati <29585319+EdAbati@users.noreply.github.com>

…/struct_namespace

narwhals/_pandas_like/series_struct.py

# Conflicts: # narwhals/_duckdb/expr.py # narwhals/_spark_like/expr.py

# Conflicts: # narwhals/_expression_parsing.py

Co-authored-by: Edoardo Abati <29585319+EdAbati@users.noreply.github.com>

osoucy · 2025-03-07T06:08:20Z

Are we good to merge? Anything else to add?

MarcoGorelli

thanks @osoucy !

i'll check the evaluate_output_names / alias_output_names more closely, I'm not 100% sure about those

docs/api-completeness/expr_struct.md

MarcoGorelli · 2025-03-09T10:24:57Z

narwhals/_arrow/expr_struct.py

+            "struct",
+            "field",
+            name=name,
+            evaluate_output_names=lambda _col: [name],


For all implementations, instead of overwriting evaluate output names, can we just use .alias?

A good test would be nw.col('a').struct.field('b').name.keep(). I think for polars the resulting column name would be 'a' we should check that we do the same

It's odd. That's what I tried initially, but it caused some errors when evaluating (mismatching expected and actual something). I must have forgotten something. I removed the changes made to the _from_call and reuse_series_namespace_implementation functions.

MarcoGorelli · 2025-03-09T15:06:53Z

narwhals/_arrow/series_struct.py

@@ -17,5 +17,5 @@ def __init__(self: Self, series: ArrowSeries) -> None:
    def field(self: Self, name: str) -> ArrowSeries:
        self._compliant_series._name = name


you can't mutate self._compliant_series, you'll need alias here too

For the Arrow* stuff I'd suggest using .compliant and .native

refactor: adds _Stores(Native|Compliant) #2130

They weren't available when you started the PR @osoucy

Some more context #2130 (comment)

this is fine as a follow-up

At least the current implementation

def field(self: Self, name: str) -> ArrowSeries: return self._compliant_series._from_native_series( pc.struct_field(self._compliant_series.alias(name)._native_series, name), )

avoids mutating the compliant series. Maybe I can look into @dangotbanned in a future PR?

All good @osoucy 😊

So we are good to merge? :)

MarcoGorelli · 2025-03-09T16:43:42Z

there's some unnecessary alias calls - I tried making a commit and pushing to your branch, but was denied access

could you check that the "allow edits by maintainers" checkbox is checked? you should see something like this

osoucy · 2025-03-09T17:40:38Z

there's some unnecessary alias calls - I tried making a commit and pushing to your branch, but was denied access

could you check that the "allow edits by maintainers" checkbox is checked? you should see something like this

This option is not available because the fork is created inside of my organization instead of inside my personnal account. I added you as a contributor to the organization. You should be able to push to the branch now.

MarcoGorelli · 2025-03-09T21:16:46Z

narwhals/_pandas_like/series_list.py

@@ -16,6 +16,9 @@

 class PandasLikeSeriesListNamespace:
    def __init__(self: Self, series: PandasLikeSeries) -> None:
+        if not hasattr(series._native_series, "list"):
+            msg = "Series must be of PyArrow List type to support struct namespace."


MarcoGorelli · 2025-03-09T21:16:58Z

narwhals/_expression_parsing.py

+        evaluate_output_names: Output names function.
+        alias_output_names: Alias output names function.


MarcoGorelli

thanks @osoucy ! great feature 🙌

osoucy · 2025-03-09T21:40:38Z

Awesome! Thanks for the guidance and review!

osoucy added 8 commits March 3, 2025 13:59

Update

0d18f96

Update

496cbd0

Update

9592277

Update

aaebbbe

Update

273f331

Update

2d95d3d

Update

decba02

Merge branch 'main' into feat/struct_namespace

f89f7d3

# Conflicts: # narwhals/_pandas_like/expr.py

MarcoGorelli reviewed Mar 4, 2025

View reviewed changes

EdAbati reviewed Mar 4, 2025

View reviewed changes

narwhals/_polars/expr.py Outdated Show resolved Hide resolved

docs/api-completeness/expr_struct.md Outdated Show resolved Hide resolved

docs/api-completeness/series_struct.md Outdated Show resolved Hide resolved

FBruzzesi reviewed Mar 4, 2025

View reviewed changes

osoucy and others added 13 commits March 4, 2025 23:48

Addressed PR comment

e75da43

Update docs/api-completeness/expr_struct.md

ef8b863

Co-authored-by: Edoardo Abati <29585319+EdAbati@users.noreply.github.com>

Merge remote-tracking branch 'origin/feat/struct_namespace' into feat…

fa802a4

…/struct_namespace

Addressed PR comment

5e04a46

Addressed PR comment

47345f5

Merge branch 'main' into feat/struct_namespace

9517b91

Addressed PR comment

e8623e2

Addressed PR comment

2d4ce6b

Addressed PR comment

825d29e

Addressed PR comment

6f867ef

Addressed PR comment

e54663c

Addressed PR comment

8fe86ae

Addressed PR comment

37dc7fc

MarcoGorelli reviewed Mar 5, 2025

View reviewed changes

narwhals/_pandas_like/series_struct.py Outdated Show resolved Hide resolved

dangotbanned mentioned this pull request Mar 5, 2025

feat: Spec'd-out CompliantExpr #2119

Merged

10 tasks

osoucy added 4 commits March 5, 2025 10:25

Addressed PR comment

104d40d

Merge branch 'main' into feat/struct_namespace

9ead00a

# Conflicts: # narwhals/_duckdb/expr.py # narwhals/_spark_like/expr.py

Merge branch 'main' into feat/struct_namespace

9e2a24d

# Conflicts: # narwhals/_duckdb/expr.py # narwhals/_spark_like/expr.py

Merge branch 'main' into feat/struct_namespace

20ec4eb

# Conflicts: # narwhals/_duckdb/expr.py # narwhals/_spark_like/expr.py

osoucy and others added 5 commits March 5, 2025 21:28

PR Comments

6c5597b

Merge branch 'main' into feat/struct_namespace

99a78e3

# Conflicts: # narwhals/_expression_parsing.py

Update docs/api-completeness/series_struct.md

105d3f9

Co-authored-by: Edoardo Abati <29585319+EdAbati@users.noreply.github.com>

PR Comments

d21bdc2

Merge branch 'main' into feat/struct_namespace

9db179a

Merge branch 'main' into feat/struct_namespace

865bedd

MarcoGorelli reviewed Mar 8, 2025

View reviewed changes

docs/api-completeness/expr_struct.md Outdated Show resolved Hide resolved

osoucy added 5 commits March 8, 2025 13:56

PR Comments

3ad7ae9

PR Comments

7e5d813

PR Comments

c5cf993

PR Comments

4d36295

Merge branch 'main' into feat/struct_namespace

8701e43

MarcoGorelli reviewed Mar 9, 2025

View reviewed changes

PR Comments

4ed080c

MarcoGorelli reviewed Mar 9, 2025

View reviewed changes

osoucy and others added 4 commits March 9, 2025 11:08

PR Comments

195088c

Merge branch 'main' into feat/struct_namespace

71df89d

remove unnecessary aliases

ebe4d6b

remove unnecessary aliases

59fb567

extra test

736599e

dangotbanned added the enhancement New feature or request label Mar 9, 2025

MarcoGorelli reviewed Mar 9, 2025

View reviewed changes

remove outdated

f71e677

MarcoGorelli approved these changes Mar 9, 2025

View reviewed changes

MarcoGorelli merged commit ea213c4 into narwhals-dev:main Mar 9, 2025
29 checks passed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat: Added `struct` namespace with `field` method. #2146

feat: Added `struct` namespace with `field` method. #2146

osoucy commented Mar 4, 2025

MarcoGorelli Mar 4, 2025

osoucy Mar 5, 2025

EdAbati left a comment

FBruzzesi left a comment

FBruzzesi Mar 4, 2025

osoucy Mar 5, 2025

osoucy commented Mar 7, 2025

MarcoGorelli left a comment

MarcoGorelli Mar 9, 2025

osoucy Mar 9, 2025

MarcoGorelli Mar 9, 2025

osoucy Mar 9, 2025

dangotbanned Mar 9, 2025

dangotbanned Mar 9, 2025

MarcoGorelli Mar 9, 2025

osoucy Mar 9, 2025

dangotbanned Mar 9, 2025

osoucy Mar 9, 2025

MarcoGorelli commented Mar 9, 2025

osoucy commented Mar 9, 2025

MarcoGorelli Mar 9, 2025

MarcoGorelli Mar 9, 2025

MarcoGorelli left a comment

osoucy commented Mar 9, 2025

		>>> df.with_columns(name=nw.col("user").struct.field("name"))
		"""

		@@ -17,5 +17,5 @@ def __init__(self: Self, series: ArrowSeries) -> None:
		def field(self: Self, name: str) -> ArrowSeries:
		self._compliant_series._name = name

		evaluate_output_names: Output names function.
		alias_output_names: Alias output names function.

feat: Added struct namespace with field method. #2146

feat: Added struct namespace with field method. #2146

Conversation

osoucy commented Mar 4, 2025

What type of PR is this? (check all applicable)

Related issues

Checklist

If you have comments or can explain your changes, please do so below

Choose a reason for hiding this comment

Choose a reason for hiding this comment

EdAbati left a comment

Choose a reason for hiding this comment

FBruzzesi left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

osoucy commented Mar 7, 2025

MarcoGorelli left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

MarcoGorelli commented Mar 9, 2025

osoucy commented Mar 9, 2025

Choose a reason for hiding this comment

Choose a reason for hiding this comment

MarcoGorelli left a comment

Choose a reason for hiding this comment

osoucy commented Mar 9, 2025

feat: Added `struct` namespace with `field` method. #2146

feat: Added `struct` namespace with `field` method. #2146