DRIVERS-2768 [Vector Search GA] Add support for types in search index creation #1541

NoahStapp · 2024-03-06T22:06:47Z

Please complete the following before merging:

Update changelog.
Make sure there are generated JSON files from the YAML test files.
Test changes in at least one language driver.
Test these changes against all server versions and topologies (including standalone, replica set, sharded
clusters, and serverless).

blink1073 · 2024-03-07T00:43:25Z

source/index-management/index-management.rst

@@ -685,6 +685,16 @@ Common API Components
 */
 name: String;

+ /**


Note: we just switched this spec to Markdown, so you'll have to move these changes to that file.

blink1073 · 2024-03-07T00:44:25Z

source/index-management/tests/README.rst

@@ -221,3 +221,88 @@ Case 6: Driver can successfully create and list search indexes with non-default
 - An index with the ``name`` of ``test-search-index-case6`` is present and the index has a field ``queryable`` with a value of ``true``.

 #. Assert that ``index`` has a property ``latestDefinition`` whose value is ``{ 'mappings': { 'dynamic': false } }``
+
+Case 7: Driver can successfully handle search index types when creating indexes


Same for the README file (actually I should have removed it in #1537)

blink1073 · 2024-03-07T00:46:50Z

Can you also add new spec tests that use type:vector to ensure they fail in a non-Atlas environment?

blink1073 · 2024-03-07T00:48:15Z

We should also either update one of the existing prose tests or add a new one that explicitly uses type: search.

NoahStapp · 2024-03-07T17:54:03Z

We should also either update one of the existing prose tests or add a new one that explicitly uses type: search.

The new prose test does explicitly create an index using type: search.

NoahStapp · 2024-03-07T17:57:10Z

Can you also add new spec tests that use type:vector to ensure they fail in a non-Atlas environment?

The existing spec tests aren't run against an Atlas cluster, they're used as unit tests to verify the driver sends the correct command document. They expect an error telling the user to use Atlas for search support. I'm not sure adding more of the same tests that have the vectorSearch type makes sense in that context.

blink1073 · 2024-03-07T19:14:50Z

I'm not sure adding more of the same tests that have the vectorSearch type makes sense in that context.

Perhaps just one extra test to make sure the field is handled appropriately?

…en creating search indexes

blink1073 · 2024-03-11T15:48:48Z

source/index-management/tests/README.md

+
+ - An index with the ``name`` of ``test-search-index-case7-implicit`` is present and the index has a field ``queryable`` with a value of ``true``.
+
+5. Assert that ``index1`` has a property ``type`` whose value is ``search``


Suggested change

5. Assert that ``index1`` has a property ``type`` whose value is ``search``

5. Assert that ``index1`` has a property ``type`` whose value is ``search``.

blink1073 · 2024-03-11T15:49:02Z

source/index-management/tests/README.md

+
+ - An index with the ``name`` of ``test-search-index-case7-explicit`` is present and the index has a field ``queryable`` with a value of ``true``.
+
+9. Assert that ``index2`` has a property ``type`` whose value is ``search``


Suggested change

9. Assert that ``index2`` has a property ``type`` whose value is ``search``

9. Assert that ``index2`` has a property ``type`` whose value is ``search``.

blink1073 · 2024-03-11T15:49:24Z

source/index-management/tests/README.md

+
+ - An index with the ``name`` of ``test-search-index-case7-vector`` is present and the index has a field ``queryable`` with a value of ``true``.
+
+13. Assert that ``index3`` has a property ``type`` whose value is ``vectorSearch``


Suggested change

13. Assert that ``index3`` has a property ``type`` whose value is ``vectorSearch``

13. Assert that ``index3`` has a property ``type`` whose value is ``vectorSearch``.

blink1073 · 2024-03-11T15:50:33Z

source/index-management/tests/createSearchIndexes.json

+ ]
+ },
+ {
+ "description": "create a vector search index",


This test is not in the yml file.

Good catch, fixed!

blink1073 · 2024-03-11T15:51:22Z

Can you please install and run pre-commit run --all-files locally?

blink1073

LGTM! Let's hold off on merging until the first two implementations are done.

ShaneHarvey · 2024-03-25T20:54:28Z

source/index-management/index-management.md

@@ -664,6 +664,15 @@ interface IndexOptions {
 * @example For an index of name: 1, age: -1, the generated name would be "name_1_age_-1".
 */
 name: String;
+
+ /**
+ * Optionally specify a type for the index. Defaults to "search" if not provided.


Many other driver options are implemented such that we only send a value if the user explicitly provides one. How do you feel about mandating that driver MUST NOT send this field if not provided?

Makes sense to me if that's a consistent pattern among drivers. The server should default to search implicitly if type is not provided, so the behavior is consistent.

Should this change be made to the CreateSearchIndexOptions instead of IndexOptions? https://github.com/mongodb/specifications/blob/master/source/index-management/index-management.md#common-interfaces

CreateSearchIndexOptions is currently unused. If drivers support adding type to IndexOptions, that seems to be a better fit as then the top-level fields of type and name are present in the same interface.

Actually, looking more closely it probably belongs in the SearchIndexModel?

IndexOptions is used for regular indexes, not search indexes.

Oh good catch! Yes, SearchIndexModel seems like the correct place for type.

baileympearson · 2024-03-27T19:49:49Z

source/index-management/index-management.md

@@ -664,6 +664,15 @@ interface IndexOptions {
 * @example For an index of name: 1, age: -1, the generated name would be "name_1_age_-1".
 */
 name: String;
+
+ /**
+ * Optionally specify a type for the index. Defaults to "search" if not provided.


Should this change be made to the CreateSearchIndexOptions instead of IndexOptions? https://github.com/mongodb/specifications/blob/master/source/index-management/index-management.md#common-interfaces

source/index-management/tests/README.md

baileympearson

POC passing in Node: mongodb/node-mongodb-native#4060

NoahStapp requested a review from blink1073 March 6, 2024 22:06

NoahStapp requested a review from a team as a code owner March 6, 2024 22:06

blink1073 requested changes Mar 7, 2024

View reviewed changes

NoahStapp added 3 commits March 7, 2024 15:20

DRIVERS-2768 [Vector Search GA] Add support for the new type field wh…

0de1a9a

…en creating search indexes

Update changelog

dc8f737

Add spec test for vectorSearch

4336d00

NoahStapp force-pushed the DRIVERS-2768 branch from ef1f17d to 4336d00 Compare March 8, 2024 00:00

NoahStapp requested a review from blink1073 March 8, 2024 00:00

blink1073 requested changes Mar 11, 2024

View reviewed changes

Add missing tests to yml

3b8ee0a

NoahStapp requested a review from blink1073 March 11, 2024 16:32

Fix schemas

850f12f

blink1073 approved these changes Mar 11, 2024

View reviewed changes

ShaneHarvey mentioned this pull request Mar 25, 2024

PYTHON-4297 Allow passing arbitrary index options to create_search_index mongodb/mongo-python-driver#1561

Merged

ShaneHarvey reviewed Mar 25, 2024

View reviewed changes

baileympearson requested changes Mar 27, 2024

View reviewed changes

NoahStapp added 2 commits April 5, 2024 10:55

Move type to SearchIndexModel, uncouple tests

c8918f9

Remove default for search index type

8a2f326

baileympearson reviewed Apr 15, 2024

View reviewed changes

source/index-management/tests/README.md Show resolved Hide resolved

baileympearson approved these changes Apr 19, 2024

View reviewed changes

blink1073 approved these changes Apr 22, 2024

View reviewed changes

NoahStapp merged commit b746fcc into mongodb:master Apr 22, 2024
4 checks passed

kevinAlbs mentioned this pull request Aug 22, 2024

DRIVERS-2768 skip search index tests on 7.1 #1639

Merged

3 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

DRIVERS-2768 [Vector Search GA] Add support for types in search index creation #1541

DRIVERS-2768 [Vector Search GA] Add support for types in search index creation #1541

NoahStapp commented Mar 6, 2024

blink1073 Mar 7, 2024

blink1073 Mar 7, 2024

blink1073 commented Mar 7, 2024

blink1073 commented Mar 7, 2024

NoahStapp commented Mar 7, 2024

NoahStapp commented Mar 7, 2024

blink1073 commented Mar 7, 2024

blink1073 Mar 11, 2024

blink1073 Mar 11, 2024

blink1073 Mar 11, 2024

blink1073 Mar 11, 2024

NoahStapp Mar 11, 2024

blink1073 commented Mar 11, 2024

blink1073 left a comment

ShaneHarvey Mar 25, 2024

NoahStapp Mar 26, 2024

baileympearson Mar 27, 2024

NoahStapp Mar 27, 2024 •

edited

Loading

baileympearson Mar 27, 2024

NoahStapp Mar 27, 2024

baileympearson Mar 27, 2024

baileympearson left a comment


		- An index with the ``name`` of ``test-search-index-case7-implicit`` is present and the index has a field ``queryable`` with a value of ``true``.

		5. Assert that ``index1`` has a property ``type`` whose value is ``search``


		- An index with the ``name`` of ``test-search-index-case7-explicit`` is present and the index has a field ``queryable`` with a value of ``true``.

		9. Assert that ``index2`` has a property ``type`` whose value is ``search``


		- An index with the ``name`` of ``test-search-index-case7-vector`` is present and the index has a field ``queryable`` with a value of ``true``.

		13. Assert that ``index3`` has a property ``type`` whose value is ``vectorSearch``

DRIVERS-2768 [Vector Search GA] Add support for types in search index creation #1541

DRIVERS-2768 [Vector Search GA] Add support for types in search index creation #1541

Conversation

NoahStapp commented Mar 6, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

blink1073 commented Mar 7, 2024

blink1073 commented Mar 7, 2024

NoahStapp commented Mar 7, 2024

NoahStapp commented Mar 7, 2024

blink1073 commented Mar 7, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

blink1073 commented Mar 11, 2024

blink1073 left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

NoahStapp Mar 27, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

baileympearson left a comment

Choose a reason for hiding this comment

NoahStapp Mar 27, 2024 •

edited

Loading