[Question] Treating disk index shards as dynamic indices?

[jaredcthomas]

Hello!
I'm wondering whether it is viable to send new points to the appropriate shards of an existing disk index, apply the streaming procedure within each shard, and stitch everything back together. Are there any logistical/performance penalties for this strategy?

[harsha-simhadri]

If you can load the shards into memory, apply the delta and write back to disk, you would save a lot of small random writes.

Could you elaborate on your scenario a bit more? what is your machine size, dataset size and the number of shards?

[jaredcthomas]

@harsha-simhadri Certainly, thanks for responding.

I have ~80M 768d vectors and a build DRAM budget of 150 GB, so I wound up with 5 shards. Really, I am just looking for the most efficient way of handling bulk insertions/deletions (say several million vectors at a time) without having to rebuild, if possible.

[harsha-simhadri]

@jaredcthomas. I assume you meant 768d floats. So the overall index size is about 80M * (3KB vec + 100degX4) ~ 250GB?
Loading one shard (~50GB) at a time into DRAM, batch updating and writing to SSD seems reasonable.
We have a long overdue (and out of date) PR#11 that can merge batch updates to disk as in https://arxiv.org/abs/2105.09613. Would appreciate help with that PR. In any case, we plan to get to that in the next few months.