Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Panic when reading empty pyarrow.Table #575

Closed
jwimberl opened this issue Jan 29, 2024 · 0 comments · Fixed by #613
Closed

Panic when reading empty pyarrow.Table #575

jwimberl opened this issue Jan 29, 2024 · 0 comments · Fixed by #613
Labels
bug Something isn't working

Comments

@jwimberl
Copy link

Describe the bug
When trying to create a DataFrame from a pyarrow.Table object with a nonzero number of columns, but zero rows, I encounter a panic in src/context.rs:294.

To Reproduce

>>> import datafusion as df
>>> import pyarrow as pa
>>> ctx = df.SessionContext()
>>> import pandas as pd
>>> df = pd.DataFrame({'col': []})
>>> import pyarrow as pa
>>> emptyTable = pa.Table.from_pandas(df)
>>> emptyTable
pyarrow.Table
col: double
----
col: [[]]
>>> ctx.from_arrow_table(emptyTable)
thread '<unnamed>' panicked at src/context.rs:294:37:
index out of bounds: the len is 0 but the index is 0
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
pyo3_runtime.PanicException: index out of bounds: the len is 0 but the index is 0

Expected behavior
I expect this to create a DataFrame with zero rows, such as the following (created via .limit(0) from a non-empty DataFrame):

>>> empty
DataFrame()
++
++
>>> empty.describe()
DataFrame()
+------------+-----+
| describe   | col |
+------------+-----+
| count      | 0.0 |
| null_count | 0.0 |
| mean       |     |
| std        |     |
| min        |     |
| max        |     |
| median     |     |
+------------+-----+

Additional context

  • Operating system: Rocky 8
  • Python version: 3.10.4
  • Python module versions used:
>>> df.__version__
'34.0.0'
>>> pa.__version__
'15.0.0'
>>> pd.__version__
'2.2.0'
@jwimberl jwimberl added the bug Something isn't working label Jan 29, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working
Projects
None yet
Development

Successfully merging a pull request may close this issue.

1 participant