Vector Search: Adds support for Partitioned DiskANN

[aayush3011]

Description

This PR adds optional attributes in the VectorIndexDefinition class to support partitioned DiskANN. A typical index definition would be something like the below:

• quantizationByteSize the number of bytes used in product quantization of the vectors. A larger value may result in better recall for vector searches at the expense of latency. This applies to index types DiskANN and quantizedFlat.
• indexingSearchListSize which represents the size of the candidate list of approximate neighbors stored while building the DiskANN index as part of the optimization processes.
• vectorIndexShardKey the list of string containing the shard keys used for partitioning the vector indexes. This applies to index types DiskANN and quantizedFlat.

{
    "indexingPolicy": {
        "automatic": true,
        "indexingMode": "Consistent",
        "includedPaths": [
            {
                "path": "/*",
                "indexes": []
            }
        ],
        "excludedPaths": [],
        "compositeIndexes": [],
        "spatialIndexes": [],
        "vectorIndexes": [
            {
                "path": "/vector1",
                "type": "flat"
            },
            {
                "path": "/vector2",
                "type": "quantizedFlat",
                "quantizationByteSize": 3,
                "vectorIndexShardKey": [
                    "/Country"
                ]
            },
            {
                "path": "/vector3",
                "type": "diskANN",
                "quantizationByteSize": 2,
                "indexingSearchListSize": 100,
                "vectorIndexShardKey": [
                    "/ZipCode"
                ]
            }
        ]
    },
    "vectorEmbeddingPolicy": {
        "vectorEmbeddings": [
            {
                "path": "/vector1",
                "dataType": "int8",
                "dimensions": 1200,
                "distanceFunction": "dotproduct"
            },
            {
                "path": "/vector2",
                "dataType": "uint8",
                "dimensions": 3,
                "distanceFunction": "cosine"
            },
            {
                "path": "/vector3",
                "dataType": "float32",
                "dimensions": 400,
                "distanceFunction": "euclidean"
            }
        ]
    },
    "id": "test_binary_vector_container_6",
    "partitionKey": {
        "paths": [
            "/pk"
        ],
        "kind": "Hash"
    }
}

If an SDK is being regenerated based on a new swagger spec, a link to the pull request containing these swagger spec changes has been included above.

All SDK Contribution checklist:

• The pull request does not introduce [breaking changes]
• CHANGELOG is updated for new features, bug fixes or other significant changes.
• I have read the contribution guidelines.

General Guidelines and Best Practices

• Title of the pull request is clear and informative.
• There are a small number of commits, each of which have an informative message. This means that previously merged commits do not appear in the history of the PR. For more information on cleaning up the commits in your PR, see this page.

Testing Guidelines

• Pull request includes test coverage for the included changes.

[Pilchie]

👀

[azure-sdk]

API change check

APIView has identified API level changes in this PR and created following API reviews.

com.azure:azure-cosmos

[Pilchie]

Do we want to do this validation client side, or just allow the server to fail the call?

[Pilchie]

I think this would be clearer as something like:

Suggested change

	private Boolean validateIndexType(boolean isIndexingSearchListSize) {
	private Boolean validatePropertySupportedByIndexType(IndexProperty property) {

along with

enum IndexProperty {
    SearchListSize,
    QuantizationByteSize,
    ShardKey,
}

[Pilchie]

Do you need to think about a way to set it back to null if it's already been set?

[Pilchie]

We've confirmed that these changes are expected?