Server Disconnects at Concurrency Level 85 with PgVector and Pinecone

[jimin0]

I’m encountering an error where the server disconnects at a concurrency level of 85. (+break searching)
Is there a solution for this? (Pinecone, PgVector)

Is there a possible workaround or configuration setting to prevent the server from disconnecting at concurrency level 85?
or how to modify CONCURRENCY num.?
Thank you

log

.myenv03119) [root@localhost vectordb]# free -h
	      total        used        free      shared  buff/cache   available
Mem:           7.5Gi       1.3Gi       5.8Gi       7.0Mi       696Mi       6.2Gi
Swap:          7.9Gi       412Mi       7.5Gi

(.myenv03119) [root@localhost vectordb]# init_bench
2024-10-31 02:14:43,251 | INFO: all configs: [('ALIYUN_OSS_URL', 'assets.zilliz.com.cn/benchmark/'), ('AWS_S3_URL', 'assets.zilliz.com/benchmark/'), ('CONCURRENCY_DURATION', 30), ('CONFIG_LOCAL_DIR', PosixPath('/home/jimin/vectordb/VectorDBBench/vectordb_bench/config-files')), ('CUSTOM_CONFIG_DIR', PosixPath('/home/jimin/vectordb/VectorDBBench/vectordb_bench/custom/custom_case.json')), ('DATASET_LOCAL_DIR', PosixPath('/tmp/vectordb_bench/dataset')), ('DEFAULT_DATASET_URL', 'assets.zilliz.com/benchmark/'), ('DROP_OLD', True), ('K_DEFAULT', 100), ('LOG_LEVEL', 'INFO'), ('NUM_CONCURRENCY', [1, 5, 10, 15, 20, 25, 30, 35, 40, 45, 50, 55, 60, 65, 70, 75, 80, 85, 90, 95, 100]), ('NUM_PER_BATCH', 5000), ('RESULTS_LOCAL_DIR', PosixPath('/home/jimin/vectordb/VectorDBBench/vectordb_bench/results')), ('USE_SHUFFLED_DATA', True)] (__main__.py:11) (204651)

Collecting usage statistics. To deactivate, set browser.gatherUsageStats to false.


 You can now view your Streamlit app in your browser.
 Local URL: http://localhost:8501

2024-10-31 19:25:07,374 | INFO: generated uuid for the tasks: 786896f2a4094455ab98c0c0c727041d (interface.py:66) (208540)
2024-10-31 19:25:07,445 | INFO | DB             | CaseType     Dataset               Filter | task_label (task_runner.py:338)
2024-10-31 19:25:07,445 | INFO | -----------    | ------------ -------------------- ------- | -------    (task_runner.py:338)
2024-10-31 19:25:07,445 | INFO | PgVector       | Performance  OpenAI-SMALL-50K        None | 2024103119 (task_runner.py:338)
2024-10-31 19:25:07,445 | INFO: task submitted: id=786896f2a4094455ab98c0c0c727041d, 2024103119, case number: 1 (interface.py:231) (208540)
2024-10-31 19:25:07.722 WARNING streamlit.runtime.scriptrunner_utils.script_run_context: Thread 'MainThread': missing ScriptRunContext! This warning can be ignored when running in bare mode.
2024-10-31 19:25:08,228 | INFO: [1/1] start case: {'label': <CaseLabel.Performance: 2>, 'dataset': {'data': {'name': 'OpenAI', 'size': 50000, 'dim': 1536, 'metric_type': <MetricType.COSINE: 'COSINE'>}}, 'db': 'PgVector'}, drop_old=True (interface.py:164) (208629)
2024-10-31 19:25:08,228 | INFO: Starting run (task_runner.py:100) (208629)
2024-10-31 19:25:08,356 | INFO: PgVector config values: {'host': 'localhost', 'port': 5432, 'dbname': 'mydatabase', 'user': 'postgres', 'password': 'imeidpaw'}
metric_type=<MetricType.COSINE: 'COSINE'> create_index_before_load=False create_index_after_load=True lists=10 probes=1 index=<IndexType.ES_IVFFlat: 'ivfflat'> maintenance_work_mem='4GB' max_parallel_workers=7 quantization_type='none' (pgvector.py:54) (208629)
2024-10-31 19:25:08,357 | INFO: PgVector client drop index : pgvector_index (pgvector.py:208) (208629)
2024-10-31 19:25:08,419 | INFO: PgVector client drop table : pg_vector_collection (pgvector.py:184) (208629)
2024-10-31 19:25:08,479 | INFO: PgVector client create table : pg_vector_collection (pgvector.py:346) (208629)

2024-10-31 19:25:08,479 | INFO: PgVector client create table : pg_vector_collection (pgvector.py:346) (208629)
2024-10-31 19:25:17,886 | INFO: Read the entire file into memory: test.parquet (dataset.py:229) (208629)
2024-10-31 19:25:17,954 | INFO: Read the entire file into memory: neighbors.parquet (dataset.py:229) (208629)
2024-10-31 19:25:17,981 | INFO: Start performance case (task_runner.py:158) (208629)
2024-10-31 19:25:18.231 WARNING streamlit.runtime.scriptrunner_utils.script_run_context: Thread 'MainThread': missing ScriptRunContext! This warning can be ignored when running in bare mode.
2024-10-31 19:25:18,886 | INFO: (SpawnProcess-1:1) Start inserting embeddings in batch 5000 (serial_runner.py:35) (208702)
2024-10-31 19:25:18,887 | INFO: Get iterator for shuffle_train.parquet (dataset.py:247) (208702)
2024-10-31 19:25:27,640 | INFO: (SpawnProcess-1:1) Finish loading all dataset into VectorDB, dur=8.753800899954513 (serial_runner.py:61) (208702)
2024-10-31 19:25:28.167 WARNING streamlit.runtime.scriptrunner_utils.script_run_context: Thread 'MainThread': missing ScriptRunContext! This warning can be ignored when running in bare mode.
2024-10-31 19:25:28,821 | INFO: PgVector post insert before optimize (pgvector.py:200) (208768)
2024-10-31 19:25:28,821 | INFO: PgVector client drop index : pgvector_index (pgvector.py:208) (208768)
2024-10-31 19:25:28,823 | INFO: PgVector client create index : pgvector_index (pgvector.py:288) (208768)
2024-10-31 19:25:28,838 | INFO: PgVector parallel index creation parameters: [('7',), ('7',), ('4GB',)] (pgvector.py:283) (208768)
2024-10-31 19:25:32,768 | INFO: Finish loading the entire dataset into VectorDB, insert_duration=9.923500600038096, optimize_duration=3.8101974999299273 load_duration(insert + optimize) = 13.7337 (task_runner.py:167) (208629)
2024-10-31 19:25:33.105 WARNING streamlit.runtime.scriptrunner_utils.script_run_context: Thread 'MainThread': missing ScriptRunContext! This warning can be ignored when running in bare mode.
2024-10-31 19:25:33,746 | INFO: SpawnProcess-1:3 start search the entire test_data to get recall and latency (serial_runner.py:171) (208812)

2024-10-31 19:26:34,986 | INFO: SpawnProcess-1:3 search entire test_data: cost=15.3295s, queries=1000, avg_recall=0.5961, avg_ndcg=0.6078,avg_latency=0.0153, p99=0.0326 (serial_runner.py:209) (208812)
2024-10-31 19:26:35.401 WARNING streamlit.runtime.scriptrunner_utils.script_run_context: Thread 'MainThread': missing ScriptRunContext! This warning can be ignored when running in bare mode.
2024-10-31 19:26:35,899 | INFO: Start search 30s in concurrency 1, filters: None (mp_runner.py:100) (208629)
2024-10-31 19:26:36.182 WARNING streamlit.runtime.scriptrunner_utils.script_run_context: Thread 'MainThread': missing ScriptRunContext! This warning can be ignored when running in bare mode.
2024-10-31 19:26:36,926 | INFO: Syncing all process and start concurrency search, concurrency=1 (mp_runner.py:109) (208629)
.
.2024-10-31 19:27:08,009 | INFO: Start search 30s in concurrency 5, filters: None (mp_runner.py:100) (208629)
2024-10-31 19:27:08.274 WARNING streamlit.runtime.scriptrunner_utils.script_run_context: Thread 'MainThread': missing ScriptRunContext! This warning can be ignored when running in bare mode.
2024-10-31 19:27:08.328 WARNING streamlit.runtime.scriptrunner_utils.script_run_context: Thread 'MainThread': missing ScriptRunContext! This warning can be ignored when running in bare mode.
2024-10-31 19:27:08.362 WARNING streamlit.runtime.scriptrunner_utils.script_run_context: Thread 'MainThread': missing ScriptRunContext! This warning can be ignored when running in bare mode.
2024-10-31 19:27:08.411 WARNING streamlit.runtime.scriptrunner_utils.script_run_context: Thread 'MainThread': missing ScriptRunContext! This warning can be ignored when running in bare mode.
2024-10-31 19:27:08.569 WARNING streamlit.runtime.scriptrunner_utils.script_run_context: Thread 'MainThread': missing ScriptRunContext! This warning can be ignored when running in bare mode.
2024-10-31 19:27:13,078 | INFO: Syncing all process and start concurrency search, concurrency=5 (mp_runner.py:109) (208629)
.
.
.
.
.



2024-10-30 21:36:17,877 | INFO: SpawnProcess-2:306 search 30s: actual_dur=30.2232s, count=205, qps in this process: 6.7829 (mp_runner.py:77) (181745)
2024-10-30 21:36:17,879 | INFO: SpawnProcess-2:319 search 30s: actual_dur=30.2254s, count=216, qps in this process: 7.1463 (mp_runner.py:77) (181758)
2024-10-30 21:36:17,888 | INFO: SpawnProcess-2:325 search 30s: actual_dur=30.2307s, count=201, qps in this process: 6.6489 (mp_runner.py:77) (181764)
2024-10-30 21:36:17,879 | INFO: SpawnProcess-2:317 search 30s: actual_dur=30.2166s, count=208, qps in this process: 6.8836 (mp_runner.py:77) (181756)
2024-10-30 21:36:17,888 | INFO: SpawnProcess-2:309 search 30s: actual_dur=30.2262s, count=206, qps in this process: 6.8153 (mp_runner.py:77) (181748)
2024-10-30 21:36:17,877 | INFO: SpawnProcess-2:333 search 30s: actual_dur=30.2001s, count=209, qps in this process: 6.9205 (mp_runner.py:77) (181779)
2024-10-30 21:36:17,864 | INFO: SpawnProcess-2:324 search 30s: actual_dur=30.1626s, count=207, qps in this process: 6.8628 (mp_runner.py:77) (181763)



2024-10-30 21:36:17,821 | INFO: SpawnProcess-2:339 search 30s: actual_dur=30.1474s, count=209, qps in this process: 6.9326 (mp_runner.py:77) (181786)
2024-10-30 21:36:17,903 | INFO: SpawnProcess-2:315 search 30s: actual_dur=30.0579s, count=211, qps in this process: 7.0198 (mp_runner.py:77) (181754)
2024-10-30 21:36:17,915 | INFO: SpawnProcess-2:294 search 30s: actual_dur=30.2031s, count=194, qps in this process: 6.4232 (mp_runner.py:77) (181733)
2024-10-30 21:36:17,941 | INFO: SpawnProcess-2:323 search 30s: actual_dur=30.2966s, count=208, qps in this process: 6.8655 (mp_runner.py:77) (181762)
2024-10-30 21:40:54.100 WARNING streamlit.runtime.scriptrunner_utils.script_run_context: Thread 'MainThread': missing ScriptRunContext! This warning can be ignored when running in bare mode.
2024-10-30 21:40:54.133 WARNING streamlit.runtime.scriptrunner_utils.script_run_context: Thread 'MainThread': missing ScriptRunContext! This warning can be ignored when running in bare mode.
2024-10-30 21:40:54.156 WARNING streamlit.runtime.scriptrunner_utils.script_run_context: Thread 'MainThread': missing ScriptRunContext! This warning can be ignored when running in bare mode.
2024-10-30 21:40:54.242 WARNING streamlit.runtime.scriptrunner_utils.script_run_context: Thread 'MainThread': missing ScriptRunContext! This warning can be ignored when running in bare mode.
2024-10-30 21:40:54.307 WARNING streamlit.runtime.scriptrunner_utils.script_run_context: Thread 'MainThread': missing ScriptRunContext! This warning can be ignored when running in bare mode.
2024-10-30 21:40:54.474 WARNING streamlit.runtime.scriptrunner_utils.script_run_context: Thread 'MainThread': missing ScriptRunContext! This warning can be ignored when running in bare mode.
2024-10-30 21:40:54.585 WARNING streamlit.runtime.scriptrunner_utils.script_run_context: Thread 'MainThread': missing ScriptRunContext! This warning can be ignored when running in bare mode.
2024-10-30 21:40:54.587 WARNING streamlit.runtime.scriptrunner_utils.script_run_context: Thread 'MainThread': missing ScriptRunContext! This warning can be ignored when running in bare mode.
2024-10-30 21:40:54.637 WARNING streamlit.runtime.scriptrunner_utils.script_run_context: Thread 'MainThread': missing ScriptRunContext! This warning can be ignored when running in bare mode.
2024-10-30 21:40:54.656 WARNING streamlit.runtime.scriptrunner_utils.script_run_context: Thread 'MainThread': missing ScriptRunContext! This warning can be ignored when running in bare mode.
2024-10-30 21:40:54.620 WARNING streamlit.runtime.scriptrunner_utils.script_run_context: Thread 'MainThread': missing ScriptRunContext! This warning can be ignored when running in bare mode.
2024-10-30 21:40:54.678 WARNING streamlit.runtime.scriptrunner_utils.script_run_context: Thread 'MainThread': missing ScriptRunContext! This warning can be ignored when running in bare mode.
2024-10-30 21:40:54.800 WARNING streamlit.runtime.scriptrunner_utils.script_run_context: Thread 'MainThread': missing ScriptRunContext! This warning can be ignored when running in bare mode.
2024-10-30 21:40:54.820 WARNING streamlit.runtime.scriptrunner_utils.script_run_context: Thread 'MainThread': missing ScriptRunContext! This warning can be ignored when running in bare mode.
2024-10-30 21:40:54.831 WARNING streamlit.runtime.scriptrunner_utils.script_run_context: Thread 'MainThread': missing ScriptRunContext! This warning can be ignored when running in bare mode.
2024-10-30 21:40:54.924 WARNING streamlit.runtime.scriptrunner_utils.script_run_context: Thread 'MainThread': missing ScriptRunContext! This warning can be ignored when running in bare mode.
2024-10-30 21:40:54.909 WARNING streamlit.runtime.scriptrunner_utils.script_run_context: Thread 'MainThread': missing ScriptRunContext! This warning can be ignored when running in bare mode.
2024-10-30 21:40:54.933 WARNING streamlit.runtime.scriptrunner_utils.script_run_context: Thread 'MainThread': missing ScriptRunContext! This warning can be ignored when running in bare mode.
2024-10-30 21:40:54.946 WARNING streamlit.runtime.scriptrunner_utils.script_run_context: Thread 'MainThread': missing ScriptRunContext! This warning can be ignored when running in bare mode.
2024-10-30 21:40:55.126 WARNING streamlit.runtime.scriptrunner_utils.script_run_context: Thread 'MainThread': missing ScriptRunContext! This warning can be ignored when running in bare mode.
2024-10-30 21:40:55.216 WARNING streamlit.runtime.scriptrunner_utils.script_run_context: Thread 'MainThread': missing ScriptRunContext! This warning can be ignored when running in bare mode.
2024-10-30 21:40:55.283 WARNING streamlit.runtime.scriptrunner_utils.script_run_context: Thread 'MainThread': missing ScriptRunContext! This warning can be ignored when running in bare mode.
2024-10-30 21:40:55.330 WARNING streamlit.runtime.scriptrunner_utils.script_run_context: Thread 'MainThread': missing ScriptRunContext! This warning can be ignored when running in bare mode.
2024-10-30 21:40:55.341 WARNING streamlit.runtime.scriptrunner_utils.script_run_context: Thread 'MainThread': missing ScriptRunContext! This warning can be ignored when running in bare mode.
2024-10-30 21:40:55.463 WARNING streamlit.runtime.scriptrunner_utils.script_run_context: Thread 'MainThread': missing ScriptRunContext! This warning can be ignored when running in bare mode.
2024-10-30 21:40:55.568 WARNING streamlit.runtime.scriptrunner_utils.script_run_context: Thread 'MainThread': missing ScriptRunContext! This warning can be ignored when running in bare mode.
2024-10-30 21:40:55.752 WARNING streamlit.runtime.scriptrunner_utils.script_run_context: Thread 'MainThread': missing ScriptRunContext! This warning can be ignored when running in bare mode.
2024-10-30 21:40:56.356 WARNING streamlit.runtime.scriptrunner_utils.script_run_context: Thread 'MainThread': missing ScriptRunContext! This warning can be ignored when running in bare mode.


.
.
.
.
.


2024-10-30 21:44:44,553 | INFO: Start search 30s in concurrency 85, filters: None (mp_runner.py:100) (172408)

---

[alwayslove2013]

@jimin0 It appears that the client machine where VDBB is located may be experiencing insufficient CPU or memory resources, which prevents it from creating the necessary number of processes for testing.

VDBBench has a default concurrency setting defined in the init file. You can find it here:

VectorDBBench/vectordb_bench/__init__.py

Line 25 in 20c513d

NUM_CONCURRENCY = env.list("NUM_CONCURRENCY", [1, 5, 10, 15, 20, 25, 30, 35, 40, 45, 50, 55, 60, 65, 70, 75, 80, 85, 90, 95, 100], subcast=int )

You can modify this setting by injecting an environment variable as follows:

$ export NUM_CONCURRENCY=1,5,10,20
$ init_bench