Are pyarrow and pandas required or not?

[matrey]

Looking at pyproject.toml it seems like both pyarrow and pandas are meant to be optional:

[project.optional-dependencies]
dataframe = ["pandas>=2.0.0", "pyarrow>=13.0.0"]

But the wheel "METADATA" file reads:

Requires-Dist: pyarrow >=13.0.0
Requires-Dist: pandas >=2.0.0

And if I use pip install --no-deps chdb to force install the chdb wheel without the rest, then it seems like at least pyarrow is not optional:

# python3 -m chdb "SELECT 1,'abc'" Pretty
ImportError: No module named 'pyarrow'
Please install pyarrow via "pip install pyarrow"
Traceback (most recent call last):
  File "<frozen runpy>", line 189, in _run_module_as_main
  File "<frozen runpy>", line 148, in _get_module_details
  File "<frozen runpy>", line 112, in _get_module_details
  File "/usr/local/lib/python3.12/site-packages/chdb/__init__.py", line 116, in <module>
    from . import dbapi, session, udf, utils  # noqa: E402
    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/chdb/dbapi/__init__.py", line 2, in <module>
    from . import connections as _orig_conn
  File "/usr/local/lib/python3.12/site-packages/chdb/dbapi/connections.py", line 4, in <module>
    from ..state import sqlitelike as chdb_stateful
  File "/usr/local/lib/python3.12/site-packages/chdb/state/__init__.py", line 1, in <module>
    from .sqlitelike import connect
  File "/usr/local/lib/python3.12/site-packages/chdb/state/sqlitelike.py", line 10, in <module>
    raise ImportError("Failed to import pyarrow") from None
ImportError: Failed to import pyarrow

After adding pyarrow but not pandas:

Installing collected packages: pyarrow
ERROR: pip's dependency resolver does not currently take into account all the packages that are installed. This behaviour is the source of the following dependency conflicts.
chdb 3.6.0 requires pandas>=2.0.0, which is not installed.
Successfully installed pyarrow-21.0.0

Now I can run the sample:

root@bf740873d4fd:/usr/local/lib/python3.12/site-packages/chdb/state# python3 -m chdb "SELECT 1,'abc'" Pretty
   ┏━━━┳━━━━━━━┓
   ┃ 1 ┃ 'abc' ┃
   ┡━━━╇━━━━━━━┩
1. │ 1 │ abc   │
   └───┴───────┘

[wudidapaopao]

Hi @matrey, thanks so much for your question! chDB depends on pandas and pyarrow because it supports direct interaction with Python objects—such as pandas DataFrames and pyarrow Tables—and can also convert SQL query results into pandas DataFrames or pyarrow Tables.

However, for simple queries like SELECT 1, pandas and pyarrow are actually unnecessary. Due to the current structure of our code, pyarrow is still imported regardless. This means that if pyarrow isn’t installed, even a basic query like SELECT 1 will throw an error. We will adjust the code structure later to fix this issue.