Releases: redivis/redivis-python
Releases · redivis/redivis-python
0.15.0
0.14.9
0.14.8
0.14.7
0.14.0
0.13.0
0.12.6
0.12.5
- Fix file path resolution on windows (#34, thanks @jbiggsets!)
- Fix an issue with the
dataset.update()
method - Added support for
REDIVIS_TMPDIR
environment variable, which can be used to specify the temp directory used by this library (if not specified, the system default is used)
0.12.1
0.12.0
This is a major update to the redivis-python client library, complete with significant performance improvements, new mechanisms for working with larger tables, and new integrations with pyarrow, polars, and dask. It also includes updates that allow for the generation of output tables within a redivis notebook environment.
New features 🚀
- New methods on Table, Query, and Upload classes:
- to_dask_dataframe(): read a redivis table as a dask dataframe
- to_polars_lazyframe(): read a redivis table as a polars lazyframe
- to_arrow_table(): read a redivis table as a pyarrow table
- to_arrow_dataset(): read a redivis table as a pyarrow dataset
- to_arrow_batch_iterator(): iterate over a table via pyarrow RecordBatches
- to_geopandas_dataframe(): read a redivis table with a geography variable as a geopandas dataframe
- to_pandas_dataframe(): read a redivis table as a pandas dataframe for a table with a geography variable, using the new pyarrow dtypes by default (supercedes the old
to_dataframe()
method, which has been deprecated)
- Added
batch_preprocessor
argument to all above functions to efficiently pre-filter data as it is loaded - Added
dtype_backend
argument toto_[geo]pandas_dataframe()
methods. Specify eitherpyarrow
,numpy
, ornumpy_nullable
- Add
date_as_object
boolean argument to_[geo]pandas_dataframe()` methods. Configure how dates should be stored in when using the legacy numpy dtypes. - Add
redivis.current_notebook()
, returning an instance of the current notebook within a Redivis notebook environment (returns null in other environments)- Add
Notebook.create_output_table(df)
method to generate an output table from a redivis notebook
- Add
Performance ⚡
- Multi-threaded data streaming, alongside performance improvements in the underlying transfer protocol, should lead to at least a 3x improvement in data transfer speeds
- Reduced memory overhead significantly, with new mechanisms to load larger-than-memory data
Deprecations
- The
to_dataframe
method has been deprecated, in favor of the more explicitto_pandas_dataframe()
andto_geopandas_dataframe()
methods. The new methods default to the "pyarrow" dtype_backend — to continue using numpy as the pandas backend, you must specifydtype_backend="numpy"
for the new methods. However, pyarrow is significantly faster, and supports nullable data types. It is recommended if numpy isn't required, and will generally work with existing code.
Dependencies
Bumped the following dependencies:
pandas >= 2.0.0
pyarrow >= 13.0.0
pyarrow >= 0.14.0
Added the following dependencies:
dask >= 2023.9.3
polars >= 0.19
shapely >= 2.0.1