[Done] Retrieving the length of a DataFrame

[jaychia]

Summary

We need an easy way for users to pull down the number of rows in a DataFrame as an integer

Please upvote this discussion and share your use-cases if this is a feature you would like to see implemented!

Proposal

df = DataFrame.read_*(...)

# Throws an error: dataframe not yet materialized
len(df)

# Executes a plan, performs a global aggregation and pulls the result to the driver as an integer
df.count_rows()

# After materialization of a DataFrame, len now works.
# It merely queries each partition for its size metadata, and returns
# the sum of all sizes (does not run any computation on partitions)
df.collect()
len(df)

[xdssio]

In my oinion, just for naming sake, if already implementing count_rows, df.count will do, why introduce a new function name? What else would the count of a dataframe mean?

I do still believe that len(df) is a very common use case, which you almost always have to run to validate your splits, filters, "where()" and so on, and even though it is expensive, there is no way around it beside an efficient implementation and cache on the dataframe level. And if it is very common to use, then using a standard pythonic len is nicer than teaching the count, which comes with a small friction from Polars/pandas.

Maybe moving the cache around when using with_column and cleaning it when running a filter can help with some of the pain behind the scenes.

[jaychia]

What else would the count of a dataframe mean?

In PostgreSQL, COUNT(column) actually does not count null values. COUNT(*) is a special case, which counts the total number of rows.

Our df.count method currently returns another DataFrame (after applying a COUNT(column) aggregation on the specified columns). If possible, I'd prefer not to override the method signature to return an int instead if no arguments are supplied - that seems a little unintuitive.

---

Here is an example that might be unexpected with len:

df = DataFrame.read_*(...)
df = df.where(expensive_func(df["A"]) > 1)

# This is going to take a while to run, but could be unexpected since
# in most libraries, `len` does not have a side-effect like triggering computation
for i in range(len(df)):
    print(f"Hello {i}")

# This takes a while to run, but is expected
num_rows = df.count_rows()

# Now this runs immediately, which is also expected because num_rows is a simple int
for i in range(num_rows):
    print(f"Hello {i}")