Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

FAISS GPU Baseline script issue with MSSPACEV dataset #289

Open
karthik86248 opened this issue Apr 15, 2024 · 1 comment
Open

FAISS GPU Baseline script issue with MSSPACEV dataset #289

karthik86248 opened this issue Apr 15, 2024 · 1 comment

Comments

@karthik86248
Copy link

I'm trying to repro the faiss GPU baseline results with MSSPACEV dataset. I'm following the instructions using the [FAISS T3 Baseline] (https://github.com/harsha-simhadri/big-ann-benchmarks/tree/main/neurips21/track3_baseline_faiss).

Built FAISS using source from the main branch.

Then executed the faiss baseline script and running into the below issue:

/big-ann-benchmarks/neurips21/track3_baseline_faiss$ python3 gpu_baseline_faiss.py --dataset msspacev-1M --indexkey IVF65536,SQ8 --train_on_gpu --build --quantize
r_on_gpu_add --add_splits 30 --search --searchparams nprobe={1,4,16,64,256} --parallel_mode 3  --quantizer_on_gpu_search
nb processors 64
model name      : Intel(R) Xeon(R) Gold 6326 CPU @ 2.90GHz
Dataset MSSPACEV1B in dimension 100, with distance euclidean, search_type knn, size: Q 29316 B 1000000
build index, key= IVF65536,SQ8
Build-time number of threads: 64
Update add-time parameters
setting maxtrain to 3276800
getting first 3276800 dataset vectors for training
data/MSSPACEV1B/spacev1b_base.i8bin
train, size (1000000, 100)
add a training index on GPU
Training level-1 quantizer
Training level-1 quantizer on 1000000 vectors in 100D
WARNING clustering 1000000 points to 65536 centroids: please provide at least 2555904 training points
Clustering 1000000 points in 100D to 65536 clusters, redo 1 times, 10 iterations
  Preprocessing in 0.28 s
  Iteration 9 (6.14 s, search 5.43 s): objective=6.03294e+09 imbalance=1.691 nsplit=0
Training IVF residual
  Input training set too big (max size is 100000), sampling 100000 / 1000000 vectors
  Total train time 50.826 s
adding
============== SPLIT 0/30
data/MSSPACEV1B/spacev1b_base.i8bin
  adding 0:33333 / 1000000 [0.223 s, RSS 3757956 kiB]
Traceback (most recent call last):
  File "/mnt/ssd_volume/big-ann-benchmarks/neurips21/track3_baseline_faiss/gpu_baseline_faiss.py", line 591, in <module>
    main()
  File "/mnt/ssd_volume/big-ann-benchmarks/neurips21/track3_baseline_faiss/gpu_baseline_faiss.py", line 543, in main
    index = build_index(args, ds)
  File "/mnt/ssd_volume/big-ann-benchmarks/neurips21/track3_baseline_faiss/gpu_baseline_faiss.py", line 210, in build_index
    index.add_core(
  File "/mnt/ssd_volume/mumba1/lib/python3.10/site-packages/faiss-1.8.0-py3.10.egg/faiss/swigfaiss.py", line 6956, in add_core
    return _swigfaiss.IndexIVFScalarQuantizer_add_core(self, n, x, xids, precomputed_idx, inverted_list_context)
TypeError: Wrong number or type of arguments for overloaded function 'IndexIVFScalarQuantizer_add_core'.
  Possible C/C++ prototypes are:
    faiss::IndexIVFScalarQuantizer::add_core(faiss::idx_t,float const *,faiss::idx_t const *,faiss::idx_t const *,void *)
    faiss::IndexIVFScalarQuantizer::add_core(faiss::idx_t,float const *,faiss::idx_t const *,faiss::idx_t const *)

this appeared to be an issue with the faiss swig interface not being able to properly cast from the numpy datatype passed to the faiss library. Hence, tried to manually cast them to satisfy the compiler.

+                temp = xblock.astype('f4')
+                temp1 = assign.astype('i8')
                 index.add_core(
                     len(xblock),
-                    faiss.swig_ptr(xblock),
+                    faiss.swig_ptr(temp),
                     None,
-                    faiss.swig_ptr(assign)
+                    faiss.swig_ptr(temp1)
                 )

The compiler errors are gone, but running into the below runtime exception.

/big-ann-benchmarks/neurips21/track3_baseline_faiss$ (base) cs21resch14001@gpuserver:/mnt/ssd_volume/big-ann-benchmarks/neurips21/track3_baseline_faiss$ vi gpu_bapython3 gpu_baseline_faiss.py --dataset msspacev-1M --indexkey IVF65536,SQ8 --train_on_gpu --build --quantizer_on_gpu_add --add_splits 30 --search --searchparams nprobe={1,4,16,64,256} --parallel_mode 3  --quantizer_on_gpu_search
nb processors 64
model name      : Intel(R) Xeon(R) Gold 6326 CPU @ 2.90GHz
Dataset MSSPACEV1B in dimension 100, with distance euclidean, search_type knn, size: Q 29316 B 1000000
build index, key= IVF65536,SQ8
Build-time number of threads: 64
Update add-time parameters
setting maxtrain to 3276800
getting first 3276800 dataset vectors for training
data/MSSPACEV1B/spacev1b_base.i8bin
train, size (1000000, 100)
add a training index on GPU
Training level-1 quantizer
Training level-1 quantizer on 1000000 vectors in 100D
WARNING clustering 1000000 points to 65536 centroids: please provide at least 2555904 training points
Clustering 1000000 points in 100D to 65536 clusters, redo 1 times, 10 iterations
  Preprocessing in 0.29 s
  Iteration 9 (6.83 s, search 5.99 s): objective=6.03294e+09 imbalance=1.691 nsplit=0
Training IVF residual
  Input training set too big (max size is 100000), sampling 100000 / 1000000 vectors
  Total train time 55.841 s
adding
============== SPLIT 0/30
data/MSSPACEV1B/spacev1b_base.i8bin
  adding 0:33333 / 1000000 [0.272 s, RSS 3744696 kiB]
============== SPLIT 1/30
data/MSSPACEV1B/spacev1b_base.i8bin
  adding 33333:66666 / 1000000 [0.332 s, RSS 3759812 kiB]
============== SPLIT 2/30
data/MSSPACEV1B/spacev1b_base.i8bin
  adding 66666:100000 / 1000000 [0.421 s, RSS 3787152 kiB]
============== SPLIT 3/30
data/MSSPACEV1B/spacev1b_base.i8bin
  adding 100000:133333 / 1000000 [0.477 s, RSS 3787208 kiB]
============== SPLIT 4/30
data/MSSPACEV1B/spacev1b_base.i8bin
  adding 133333:166666 / 1000000 [0.533 s, RSS 3792648 kiB]
============== SPLIT 5/30
data/MSSPACEV1B/spacev1b_base.i8bin
  adding 166666:200000 / 1000000 [0.589 s, RSS 3796876 kiB]
============== SPLIT 6/30
data/MSSPACEV1B/spacev1b_base.i8bin
  adding 200000:233333 / 1000000 [0.622 s, RSS 3801216 kiB]
============== SPLIT 7/30
data/MSSPACEV1B/spacev1b_base.i8bin
  adding 233333:266666 / 1000000 [0.672 s, RSS 3824248 kiB]
============== SPLIT 8/30
data/MSSPACEV1B/spacev1b_base.i8bin
  adding 266666:300000 / 1000000 [0.747 s, RSS 3827172 kiB]
============== SPLIT 9/30
data/MSSPACEV1B/spacev1b_base.i8bin
  adding 300000:333333 / 1000000 [0.817 s, RSS 3830356 kiB]
============== SPLIT 10/30
data/MSSPACEV1B/spacev1b_base.i8bin
  adding 333333:366666 / 1000000 [0.857 s, RSS 3836412 kiB]
============== SPLIT 11/30
data/MSSPACEV1B/spacev1b_base.i8bin
  adding 366666:400000 / 1000000 [0.895 s, RSS 3842488 kiB]
============== SPLIT 12/30
data/MSSPACEV1B/spacev1b_base.i8bin
  adding 400000:433333 / 1000000 [0.962 s, RSS 3846732 kiB]
============== SPLIT 13/30
data/MSSPACEV1B/spacev1b_base.i8bin
  adding 433333:466666 / 1000000 [1.010 s, RSS 3851480 kiB]
============== SPLIT 14/30
data/MSSPACEV1B/spacev1b_base.i8bin
  adding 466666:500000 / 1000000 [1.055 s, RSS 3858196 kiB]
============== SPLIT 15/30
data/MSSPACEV1B/spacev1b_base.i8bin
  adding 500000:533333 / 1000000 [1.102 s, RSS 3861880 kiB]
============== SPLIT 16/30
data/MSSPACEV1B/spacev1b_base.i8bin
  adding 533333:566666 / 1000000 [1.166 s, RSS 3883036 kiB]
============== SPLIT 17/30
data/MSSPACEV1B/spacev1b_base.i8bin
  adding 566666:600000 / 1000000 [1.195 s, RSS 3886160 kiB]
============== SPLIT 18/30
data/MSSPACEV1B/spacev1b_base.i8bin
  adding 600000:633333 / 1000000 [1.225 s, RSS 3891904 kiB]
============== SPLIT 19/30
data/MSSPACEV1B/spacev1b_base.i8bin
  adding 633333:666666 / 1000000 [1.254 s, RSS 3895884 kiB]
============== SPLIT 20/30
data/MSSPACEV1B/spacev1b_base.i8bin
  adding 666666:700000 / 1000000 [1.292 s, RSS 3902764 kiB]
============== SPLIT 21/30
data/MSSPACEV1B/spacev1b_base.i8bin
  adding 700000:733333 / 1000000 [1.499 s, RSS 3920764 kiB]
============== SPLIT 22/30
data/MSSPACEV1B/spacev1b_base.i8bin
  adding 733333:766666 / 1000000 [1.539 s, RSS 3924992 kiB]
============== SPLIT 23/30
data/MSSPACEV1B/spacev1b_base.i8bin
  adding 766666:800000 / 1000000 [1.682 s, RSS 3947948 kiB]
============== SPLIT 24/30
data/MSSPACEV1B/spacev1b_base.i8bin
  adding 800000:833333 / 1000000 [1.754 s, RSS 3949752 kiB]
============== SPLIT 25/30
data/MSSPACEV1B/spacev1b_base.i8bin
  adding 833333:866666 / 1000000 [1.795 s, RSS 3954252 kiB]
============== SPLIT 26/30
data/MSSPACEV1B/spacev1b_base.i8bin
  adding 866666:900000 / 1000000 [1.828 s, RSS 3959776 kiB]
============== SPLIT 27/30
data/MSSPACEV1B/spacev1b_base.i8bin
  adding 900000:933333 / 1000000 [1.877 s, RSS 3966388 kiB]
============== SPLIT 28/30
data/MSSPACEV1B/spacev1b_base.i8bin
  adding 933333:966666 / 1000000 [1.921 s, RSS 3969824 kiB]
============== SPLIT 29/30
data/MSSPACEV1B/spacev1b_base.i8bin
  adding 966666:1000000 / 1000000 [1.973 s, RSS 3974576 kiB]
  add in 2.092 s
imbalance_factor= 1.690308313088
current RSS: 3439468544
precomputed tables size: 0
Search threads: 64
setting IVF parallel mode to 3
Optimize for intersection @  10
Running evaluation on 5 searchparams
parameters                                   inter@ 10 time(ms/q)   nb distances %quantization #runs
nprobe=1                                 Traceback (most recent call last):
  File "/mnt/ssd_volume/big-ann-benchmarks/neurips21/track3_baseline_faiss/gpu_baseline_faiss.py", line 591, in <module>
    main()
  File "/mnt/ssd_volume/big-ann-benchmarks/neurips21/track3_baseline_faiss/gpu_baseline_faiss.py", line 588, in main
    run_experiments_searchparams(ds, index, args)
  File "/mnt/ssd_volume/big-ann-benchmarks/neurips21/track3_baseline_faiss/gpu_baseline_faiss.py", line 434, in run_experiments_searchparams
    eval_setting_knn(
  File "/mnt/ssd_volume/big-ann-benchmarks/neurips21/track3_baseline_faiss/gpu_baseline_faiss.py", line 248, in eval_setting_knn
    D, I = index.search(xq, k)
  File "/mnt/ssd_volume/big-ann-benchmarks/neurips21/track3_baseline_faiss/gpu_baseline_faiss.py", line 336, in search
    self.index_ivf.search_preassigned(
TypeError: handle_Index.<locals>.replacement_search_preassigned() takes 5 positional arguments but 9 were given
FATAL: exception not rethrown
FATAL: exception not rethrown
Aborted (core dumped)
@mdouze
Copy link
Collaborator

mdouze commented May 10, 2024

The way to call search_preassigned on the current Faiss is

D[i0:i0+ni], I[i0:i0+ni] = self.index_ivf.search_preassigned(xblock, k, Ic, Dc)

@karthik86248 karthik86248 mentioned this issue Aug 4, 2024
Open
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@mdouze @karthik86248