bids2table is a library for efficiently indexing and querying large-scale BIDS neuroimaging datasets and derivatives. It aims to improve upon the efficiency of PyBIDS by leveraging modern data science tools.
bids2table represents a BIDS dataset index as a single table with columns for BIDS entities and file metadata. The index is constructed using Arrow and stored in Parquet format, a binary tabular file format optimized for efficient storage and retrieval.
A pre-release version of bids2table can be installed with
pip install bids2tableThe latest development version can be installed with
pip install git+https://github.com/childmindresearch/bids2table.gitOur documentation is here.
import pandas as pd
from bids2table import bids2table
# Load in memory as pandas dataframe
df = bids2table("/path/to/dataset")
# Load in parallel and stream to disk as a Parquet dataset
df = bids2table("/path/to/dataset", persistent=True, workers=8)See here for a more complete example.
bids2table significantly outperforms both PyBIDS and ancpBIDS in terms of indexing run time, index size on disk, and query run time.
Indexing run time and index size on disk for the NKI Rockland Sample dataset. See the indexing benchmark for more details.
| Index | Num workers | Run time (s) | Index size (MB) |
|---|---|---|---|
| PyBIDS | 1 | 1618 | 448 |
| ancpBIDS | 1 | 465 | -- |
| bids2table | 1 | 402 | 4.02 |
| bids2table | 8 | 53.2 | 3.84 |
| bids2table | 64 | 10.7 | 4.82 |
Query run times for the Chinese Color Nest Project dataset. See the query benchmark for more details.
| Index | Get subjects (ms) | Get BOLD (ms) | Query metadata (ms) | Get morning scans (ms) |
|---|---|---|---|---|
| PyBIDS | 1350 | 12.3 | 6.53 | 34.3 |
| ancpBIDS | 30.6 | 19.2 | -- | -- |
| bids2table | 0.046 | 0.346 | 0.312 | 0.352 |