[Internal] ContainerProperties: Adds Vector Embedding and Indexing Policy

[kundadebdatta]

Pull Request Template

Description

This PR adds a new VectorEmbeddingPolicy in ContainerProperties and a new VectorIndexes in the IndexingPolicy to enable Vector Similarity Search in Cosmos DB.

• Vector Embedding Policy is the part of the Container level policy, which has the details about embedding dimensions, data type of the embeddings, and the distance function to be used to calculate the similarity between the embedding vectors.

• Vector Indexes in Indexing policy is very similar to that of the composite or spatial indexes, in terms of the specs. It specifies the index path, as well as the new index type.

Below is a sample for the new ContainerProperties with Indexing Policy and Vector Embedding Policy highlighted.

{
    "vectorEmbeddingPolicy": {
        "vectorEmbeddings": [
            {
                "path": "/vector1",
                "dataType": "float32",
                "dimensions": 1200,
                "distanceFunction": "cosine"
            },
            {
                "path": "/vector2",
                "dataType": "int8",
                "dimensions": 3,
                "distanceFunction": "dotproduct"
            },
            {
                "path": "/vector3",
                "dataType": "uint8",
                "dimensions": 400,
                "distanceFunction": "euclidean"
            }
        ]
    },
    "partitionKey": {
        "kind": "Hash",
        "paths": [
            "/pk"
        ]
    },
    "indexingPolicy": {
        "automatic": true,
        "indexingMode": "Consistent",
        "vectorIndexes": [
            {
                "path": "/vector1",
                "type": "flat"
            },
            {
                "path": "/vector2",
                "type": "quantizedFlat"
            },
            {
                "path": "/vector3",
                "type": "diskANN"
            }
        ],
        "includedPaths": [
            {
                "path": "/name/?"
            },
            {
                "path": "/description/?"
            }
        ],
        "excludedPaths": [
            {
                "path": "/*"
            }
        ]
    },
    "uniqueKeyPolicy": {}
}

[Note: Since the public emulator is not ready to support the vector embeddings and indexes yet, some of the emulator tests has been marked as skipped for now. These will be enabled in the future as soon as the emulator changes are available publicly.]

Type of change

Please delete options that are not relevant.

• New feature (non-breaking change which adds functionality)

Closing issues

To automatically close an issue: closes #4364

[FabianMeiswinkel]

LGTM

[FabianMeiswinkel]

LGTM