Faiss to Parquet Conversion

[valamuri2020]

get_faiss_indexes.sh - downloads BEIR collections in faiss format

faiss_to_parquet - converts a collection in faiss to parquet format

Usage:

python src/main/python/parquet/faiss_to_parquet.py --input /path/to/faiss/collection/ --output /path/to/output/ --overwrite. Use --overwrite to overwrite the output directory. Each file in the output directory will have up to 1M vectors per file.

run_conversions.sh - Script to convert collections to parquet, run indexing, search, and eval with the newly converted Parquet data. This was part of validation to ensure embedding quality matched JSON to Parquet conversion - ended up keeping the script after.

Usage:

To use the default base directory:
./run_conversions.sh --all
To specify a custom base directory:
./run_conversions.sh /path/to/custom/dir --all
To process specific subdirectories in a custom directory:
./run_conversions.sh /path/to/custom/dir subdir1 subdir2

[lintool]

Please update submodule tools/; your current PR is going backwards to an earlier commit state.