pyo3_runtime.PanicException: Polars' maximum length reached. Consider installing 'polars-u64-idx' thrown on list of length

[kszlim]

Checks

• I have checked that this issue has not already been reported.
• I have confirmed this bug exists on the latest version of Polars.

Reproducible example

import polars as pl
import numpy as np

array = np.random.rand(1000, 10303488)
df = pl.DataFrame({"a": array})
# Throws: pyo3_runtime.PanicException: Polars' maximum length reached. Consider installing 'polars-u64-idx'

The actual issue I encountered is parquet reading from a parquet file with a column of that shape. My theory is that when there are more than u32 values in total (row_count * array width) > u32::max that this occurs. So i'm not 100% sure this is a bug.

If you care to produce such a parquet file:

use arrow::array::{FixedSizeListBuilder, UInt8Builder};
use arrow::datatypes::{DataType, Field, Schema};
use arrow::record_batch::RecordBatch;
use parquet::arrow::ArrowWriter;
use parquet::file::properties::WriterProperties;
use rand::Rng;
use std::fs::File;
use std::sync::Arc;

fn main() -> Result<(), Box<dyn std::error::Error>> {
    // Define the field and schema for a single column that is a fixed-size list of floats.
    let list_length = 10_303_488;
    let field = Field::new(
        "mylist",
        DataType::FixedSizeList(Arc::new(Field::new("item", DataType::UInt8, true)), list_length),
        true,
    );
    let schema = Arc::new(Schema::new(vec![field]));

    // Create a writer for the Parquet file
    let file = File::create("output_randomized.parquet")?;
    let props = WriterProperties::builder().build();
    let mut writer = ArrowWriter::try_new(file, schema.clone(), Some(props))?;

    let iterations = 1;
    let values_per_batch = list_length;

    let mut list_arr_builder = FixedSizeListBuilder::new(UInt8Builder::new(), list_length);
    for i in 0..iterations {
        // Generate random data for the values array
        let mut rng = rand::thread_rng();
        let values: Vec<u8> = (0..values_per_batch)
            .map(|_| rng.gen())
            .collect();

        list_arr_builder.values().append_slice(&values);
        list_arr_builder.append(true);
        let output = list_arr_builder.finish();
        let batch = RecordBatch::try_new(schema.clone(), vec![Arc::new(output)])?;
        writer.write(&batch)?;
        println!("Wrote iteration: {i}");
    }

    writer.close()?;
    Ok(())
}

With deps:

[package]
name = "repro"
version = "0.1.0"
edition = "2021"

[dependencies]
arrow = "53.3.0"
parquet = "53.3.0"
rand = "0.8.5"

The doing:

import polars as pl
pl.read_parquet("output_randomized.parquet")['mylist'].to_arrow()
# Throws: pyo3_runtime.PanicException: Polars' maximum length reached. Consider installing 'polars-u64-idx'

Log output

No response

Issue description

As described.

Expected behavior

Shouldn't fail? (not 100% sure)

Installed versions

>>> pl.show_versions()
--------Version info---------
Polars:              1.16.0
Index type:          UInt32
Platform:            macOS-15.1-arm64-arm-64bit
Python:              3.12.7 (main, Oct 16 2024, 07:12:08) [Clang 18.1.8 ]
LTS CPU:             False

----Optional dependencies----
adbc_driver_manager  <not installed>
altair               <not installed>
boto3                <not installed>
cloudpickle          <not installed>
connectorx           <not installed>
deltalake            <not installed>
fastexcel            <not installed>
fsspec               <not installed>
gevent               <not installed>
google.auth          <not installed>
great_tables         <not installed>
matplotlib           <not installed>
nest_asyncio         <not installed>
numpy                2.1.3
openpyxl             <not installed>
pandas               <not installed>
pyarrow              18.1.0
pydantic             <not installed>
pyiceberg            <not installed>
sqlalchemy           <not installed>
torch                <not installed>
xlsx2csv             <not installed>
xlsxwriter           <not installed>