[FR][PERF] Arrow, Parquet, SIMD

[westurner]

Proposal Summary

• Apache Arrow, Parquet, and things that do SIMD are fast but not yet supported?
  • https://github.com/apache/arrow & SIMD:
    • https://github.com/pola-rs/polars
    • https://github.com/apache/arrow-datafusion #known-uses
      • https://github.com/apache/arrow-datafusion-python
    • https://github.com/apache/arrow-ballista
      • Integrate Ballista scheduler with DataFusion apache/datafusion#64
        • https://arrow.apache.org/datafusion/contributor-guide/roadmap.html#move-query-scheduler-into-datafusion
    • https://duckdb.org/docs/guides/python/sql_on_arrow.html
    • https://github.com/simdjson/simdjson
• Apache Arrow: https://en.wikipedia.org/wiki/Apache_Arrow
• SIMD: https://en.wikipedia.org/wiki/Single_instruction,_multiple_data#Advantages

Motivation

• What is the use case for this feature?
  Data reading, writing, transformation, interchange

• Why is this use case valuable to support for FiftyOne users in general?
  Performance

• Why is this use case valuable to support for your project(s) or organization?
  Speed, Energy efficiency

• Why is it currently difficult to achieve this use case?

  Are there existing pyarrow.dataset datasets stored as Parquet files that are useful fiftyone.core.dataset datasets for fiftyone?

  • https://arrow.apache.org/docs/python/dataset.html
  • https://arrow.apache.org/docs/python/api/dataset.html
  • https://voxel51.com/docs/fiftyone/api/fiftyone.core.dataset.html
  • https://voxel51.com/docs/fiftyone/api/fiftyone.core.odm.dataset.html

What areas of FiftyOne does this feature affect?

• App: FiftyOne application
• Core: Core fiftyone Python library
• Server: FiftyOne server

Details

• Read/Write Parquet with Arrow
• Allow other queries over Parquet files w/ e.g. DataFusion (& Ballista as the Scheduler (which is sort of like dask w/ distributed scheduler https://docs.dask.org/en/stable/scheduling.html#dask-distributed-cluster))
• JSON-LD RDF linked data metadata w/ Arrow datasets somehow

Willingness to contribute

The FiftyOne Community welcomes contributions! Would you or another member of your organization be willing to contribute an implementation of this feature?

• Thought I'd mention before you

[swheaton]

Currently not possible, we use MongoDB database backing which stores documents in json format