Updated readme with Big ANN conversion programs

aerospike/README.md

@@ -348,12 +348,11 @@ The Vector's Search Params (HnswParams) as a Json value used to obtain the neigh
 
 The Bin Name that represents the Primary Key for a record. If not provided the Aerospike PK will try to be used, if the PK value is returned.
 
-If the Aerospike PK is not a value (digest), PK array will not be part of the HDF dataset. (default: \_proximus_uk_)
+If the Aerospike PK is not a value (digest), PK array will not be part of the HDF dataset. (default: \_avs_uk_)
 
 \--records LIMIT
 
-The number of records (PKs) returned from the application Aerospike Set. If -1, all records are returned from the set. 
-(default: -1)
+The number of records (PKs) returned from the application Aerospike Set. If -1, all records are returned from the set. (default: -1)
 
 \-n NEIGHBORS, --neighbors NEIGHBORS
 
@@ -430,6 +429,82 @@ The Logging level
 -   vector_distance -- The Aerospike distance type (VectorDistanceMetric)
 -   hnsw -- The hnsw parameters defined when the index was created
 
+# Big ANN Datasets
+
+You can create a HDF5 file that can be used by the ANN framework or “Standalone ANN”, The Big ANN datasets are in a different format than the “standard” ANN HDF5 file.
+
+To convert from a Big ANN format to an ANN HDF5 file you must perform these two steps:
+
+1.  Call the bigann_download.py program to download the Big ANN files
+2.  Call bigann_convert_hdf.py that will convert from the Big ANN files to the HDF5 file
+
+## bigann_download
+
+This will download the Big ANN dataset files and place the files into the default “data” folder where the dataset name will be a sub-folder within this folder. **This requires the following applications installed in your environment:**
+
+-   **axel**
+-   **azcopy**
+
+Below are the arguments:
+
+-   \--dataset name  
+    You can find more information about the Big ANN datasets [here](https://big-ann-benchmarks.com/neurips23.html). Name is:
+    -   'bigann-1B'
+    -   'bigann-100M'
+    -   'bigann-10M'
+    -   'deep-1B'
+    -   'deep-100M'
+    -   'deep-10M'
+    -   'ssnpp-1B'
+    -   'ssnpp-10M'
+    -   'ssnpp-100M'
+    -   'ssnpp-1M'
+    -   'text2image-1B'
+    -   'text2image-1M'
+    -   'text2image-10M'
+    -   'text2image-100M'
+    -   'msturing-1B'
+    -   'msturing-100M'
+    -   'msturing-10M'
+    -   'msturing-1M'
+    -   'msturing-10M-clustered'
+    -   'msturing-30M-clustered'
+    -   'msspacev-1B' 'msspacev-100M'
+    -   'msspacev-10M' 'msspacev-1M'
+    -   'yfcc-10M' 'yfcc-10M-unfiltered'
+    -   'yfcc-10M-dummy'
+    -   'yfcc-10M-dummy-unfiltered'
+    -   'sparse-small'
+    -   'sparse-1M'
+    -   'sparse-full'
+    -   'random-xs'
+    -   'random-s'
+    -   'random-xs-clustered'
+    -   'random-range-xs'
+    -   'random-range-s'
+    -   'random-filter-s'
+    -   'openai-embedding-1M'
+
+## bigann_convert_hdf
+
+This will convert a previously downloaded Big ANN dataset into the standard ANN HDF5 file.
+
+The arguments are:
+
+-   \--dataset name – The Big ANN dataset name as defined and download from [bigann_download](#bigann_download).
+-   \--hdf HDFFILE – The name of the HDF5 file that will be created in the “data” folder. This file will have the following attributes:
+    -   dimension – The dimensions of the training vector as defined by the Big ANN dataset
+    -   distance – The distance type
+    -   nbrneighbors -- The number of neighbors returned
+    -   point_type – The point type
+    -   searchtype – The distance search type
+    -   sourcedataset – The Big ANN dataset name
+    -   type – Always "dense"
+    -   distances – A collection of vectors and shape of the test results distances as defined in the Big ANN dataset
+    -   neighbors -- A collection of vectors and shape of the test results neighbors as defined in the Big ANN dataset
+    -   test -- A vector and shape of the test vectors used to obtain the results as defined in the Big ANN dataset
+    -   train -- A vector and shape of the training vectors used to by training to query the DB
+
 # Prometheus
 
 The module outputs certain meters to Prometheus. They are:
@@ -455,12 +530,28 @@ The module outputs certain meters to Prometheus. They are:
         -   `Paused – paused due to resource exhausted`
         -   `Running - Populating`
         -   `Idle – before population, after, or after waiting…`
+        -   Collecting HDF – Obtaining information from the HDF file
+        -   AVS Status – Connecting to the AVS server and obtaining index information
+        -   Query – Performing a Query action
         -   `Done – Population/Wait done`
     -   `"action” – If importing (populating) or querying`
     -   "remainingRecs" – The current number of records that have not been populated
     -   "remainingquerynbrs” - The total number of queries (includes all runs) that have not been executed
     -   "hnswparams" -- The Vector Index's HNSW params
-    -   "queryef" -- The query's "ef" value 
+    -   "queryef" -- The query's "ef" value
+    -   “popresrcevt" – during population, the resource exhausted event handling. Values are:
+        -   Wait Idx – Wait for Index to Complete before proceeding
+        -   Sleep N – Sleep N seconds until proceeding
+        -   Exception – Rethrow the resource exhausted exception
+    -   "popconcurrent” – during population the concurrency action of the upserts. Values are:
+        -   All – All records are upserted, concurrently
+        -   Concurrent N – Records are upseted N records concurrently, at a time. Once the records are submitted, the application waits for server to respond before the next N records are submitted.
+        -   Single – One record is submitted and waits for the sever upsert response
+        -   Disabled – Population is disabled
+    -   "popwait" – The action taken once all records are upserted into the DB. Values are:
+        -   Wait for Completion – Wait for the server to acknowledge that all upsets are committed to the database.
+        -   No Wait – The application doesn’t wait for acknowledgement and eventually the upserts will be committed.
+    -   "idxstate" – Provides if the index already existed, newly created, or dropped and re-created.
 -   `aerospike.hdf.populate Current record rate that have been upserted. Defined as a counter. Attributes:`
     -   `"type" -- upsert`
     -   `"ns" -- Namespace`