Skip to content

Cosmos full-text search "what's new" docs #5009

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 1 commit into from
May 16, 2025
Merged

Cosmos full-text search "what's new" docs #5009

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
131 changes: 131 additions & 0 deletions entity-framework/core/providers/cosmos/full-text-search.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,131 @@
---
title: Full Text Search - Azure Cosmos DB Provider - EF Core
description: Full text search with the Azure Cosmos DB EF Core Provider
author: maumar
ms.date: 04/19/2025
uid: core/providers/cosmos/full-text-search
---
# Full text search

Azure Cosmos DB now offers support for [full-text search](/azure/cosmos-db/gen-ai/full-text-search). It enables efficient and effective text searches using advanced techniques like stemming, as well as evaluating the relevance of documents to a given search query. It can be used in combination with vector search (i.e. hybrid search) to improve the accuracy of responses in some AI scenarios.
EF Core allows for modeling the database with full-text search enabled properties and using full-text search functions inside queries targeting Azure Cosmos DB.

## Model configuration

A property can be configured inside `OnModelCreating` to use full-text search by enabling it for the property and defining a full-text index:

```c#
public class Blog
{
...

public string Contents { get; set; }
}

public class BloggingContext
{
...

protected override void OnModelCreating(ModelBuilder modelBuilder)
{
modelBuilder.Entity<Blog>(b =>
{
b.Property(x => x.Contents).EnableFullTextSearch();
b.HasIndex(x => x.Contents).IsFullTextIndex();
});
}
}
```

> [!NOTE]
> Configuring the index is not mandatory, but it is recommended as it greatly improves performance of full-text search queries.

Full-text search operations are language specific, using American English (`en-US`) by default. You can customize the language for individual properties as part of `EnableFullTextSearch` call:

```c#
protected override void OnModelCreating(ModelBuilder modelBuilder)
{
modelBuilder.Entity<Blog>(b =>
{
b.Property(x => x.Contents).EnableFullTextSearch();
b.HasIndex(x => x.Contents).IsFullTextIndex();
b.Property(x => x.ContentsGerman).EnableFullTextSearch("de-DE");
b.HasIndex(x => x.ContentsGerman).IsFullTextIndex();
});
}
```

You can also set a default language for the container - unless overridden in the `EnableFullTextSearch` method, all full-text properties inside the container will use that language.

```c#
protected override void OnModelCreating(ModelBuilder modelBuilder)
{
modelBuilder.Entity<Blog>(b =>
{
b.HasDefaultFullTextLanguage("de-DE");
b.Property(x => x.ContentsEnglish).EnableFullTextSearch("en-US");
b.HasIndex(x => x.ContentsEnglish).IsFullTextIndex();
b.Property(x => x.ContentsGerman).EnableFullTextSearch();
b.HasIndex(x => x.ContentsGerman).IsFullTextIndex();
b.Property(x => x.TagsGerman).EnableFullTextSearch();
b.HasIndex(x => x.TagsGerman).IsFullTextIndex();
});
}
```

## Querying

As part of the full-text search feature, Azure Cosmos DB introduced several built-in functions which allow for efficient querying of content inside the full-text search enabled properties. These functions are: [`FullTextContains`](/azure/cosmos-db/nosql/query/fulltextcontains), [`FullTextContainsAll`](/azure/cosmos-db/nosql/query/fulltextcontainsall), [`FullTextContainsAny`](/azure/cosmos-db/nosql/query/fulltextcontainsany), which look for specific keyword or keywords and [`FullTextScore`](/azure/cosmos-db/nosql/query/fulltextscore), which returns [BM25 score](https://en.wikipedia.org/wiki/Okapi_BM25) based on provided keywords.

> [!NOTE]
> `FullTextScore` can only be used inside `OrderBy` to rank the documents based on the score.

EF Core exposes these functions as part of `EF.Functions` so they can be used in queries:

```c#
var cosmosBlogs = await context.Blogs.Where(x => EF.Functions.FullTextContainsAll(x.Contents, "database", "cosmos")).ToListAsync();

var keywords = new string[] { "AI", "agent", "breakthrough" };
var mostInteresting = await context.Blogs.OrderBy(x => EF.Functions.FullTextScore(x.Contents, keywords)).Take(5).ToListAsync();
```

## Hybrid search

Full-text search can be used with vector search in the same query (i.e. hybrid search), by combining results of `FullTextScore` and `VectorDistance` functions. It can be done using the [`RRF`](/azure/cosmos-db/nosql/query/rrf) (Reciprocal Rank Fusion) function, which EF Core also provides inside `EF.Functions`:

```c#
public class Blog
{
...

public float[] Vector { get; set; }
public string Contents { get; set; }
}

public class BloggingContext
{
...

protected override void OnModelCreating(ModelBuilder modelBuilder)
{
modelBuilder.Entity<Blog>(b =>
{
b.Property(x => x.Contents).EnableFullTextSearch();
b.HasIndex(x => x.Contents).IsFullTextIndex();

b.Property(x => x.Vector).IsVectorProperty(DistanceFunction.Cosine, dimensions: 1536);
b.HasIndex(x => x.Vector).IsVectorIndex(VectorIndexType.Flat);
});
}
}

float[] myVector = /* generate vector data from text, image, etc. */
var hybrid = await context.Blogs.OrderBy(x => EF.Functions.Rrf(
EF.Functions.FullTextScore(x.Contents, "database"),
EF.Functions.VectorDistance(x.Vector, myVector)))
.Take(10)
.ToListAsync();
```

> [!TIP]
> You can combine more than two scoring functions inside `Rrf` call, as well as using only `FullTextScore`, or only `VectorDistance`.
23 changes: 13 additions & 10 deletions entity-framework/core/providers/cosmos/vector-search.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -7,14 +7,9 @@ uid: core/providers/cosmos/vector-search
---
# Vector search

> [!WARNING]
> Azure Cosmos DB vector search is currently in preview. As a result, using EF's vector search APIs will generate an "experimental API" warning (`EF9103`) which must be suppressed. The APIs and capabilities may change in breaking ways in the future.
Azure Cosmos DB now offers support for vector similarity search. Vector search is a fundamental part of some application types, including AI, semantic search and others. Azure Cosmos DB allows you to store vectors directly in your documents alongside the rest of your data, meaning you can perform all of your queries against a single database. This can considerably simplify your architecture and remove the need for an additional, dedicated vector database solution in your stack. To learn more about Azure Cosmos DB vector search, [see the documentation](/azure/cosmos-db/nosql/vector-search).

Azure Cosmos DB now offers preview support for vector similarity search. Vector search is a fundamental part of some application types, including AI, semantic search and others. Azure Cosmos DB allows you to store vectors directly in your documents alongside the rest of your data, meaning you can perform all of your queries against a single database. This can considerably simplify your architecture and remove the need for an additional, dedicated vector database solution in your stack. To learn more about Azure Cosmos DB vector search, [see the documentation](/azure/cosmos-db/nosql/vector-search).

To use vector search, you must first [enroll in the preview feature](/azure/cosmos-db/nosql/vector-search#enroll-in-the-vector-search-preview-feature). Then, [define vector policies on your container](/azure/cosmos-db/nosql/vector-search#container-vector-policies) to identify which JSON properties in your documents contain vectors and vector-related information for those properties (dimensions, data type, distance function).

Once your container is properly set up, add a vector property to your model in the path you defined in the container policy, and configure it with EF as a vector:
Vector property can be configured inside `OnModelCreating`:

```c#
public class Blog
Expand All @@ -30,9 +25,11 @@ public class BloggingContext

protected override void OnModelCreating(ModelBuilder modelBuilder)
{
modelBuilder.Entity<Blog>()
.Property(b => b.Embeddings)
.IsVector(DistanceFunction.Cosine, dimensions: 1536);
modelBuilder.Entity<Blog>(b =>
{
b.Property(b => b.Vector).IsVectorProperty(DistanceFunction.Cosine, dimensions: 1536);
b.HasIndex(x => x.Vector).IsVectorIndex(VectorIndexType.Flat);
});
}
}
```
Expand All @@ -56,3 +53,9 @@ var blogs = await context.Blogs
```

This will returns the top five Blogs, based on the similarity of their `Vector` property and the externally-provided `anotherVector` data.

## Hybrid search

Vector similarity search can be used with full-text search in the same query (i.e. hybrid search), by combining results of `VectorDistance` and `FullTextScore` functions using the [`RRF`](/azure/cosmos-db/nosql/query/rrf) (Reciprocal Rank Fusion) function.

See [documentation](xref:core/providers/cosmos/full-text-search?#hybrid-search) to learn how to enable full-text search support in EF model and how to use hybrid search in queries.
86 changes: 76 additions & 10 deletions entity-framework/core/what-is-new/ef-core-10.0/whatsnew.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -21,6 +21,72 @@ EF10 requires the .NET 10 SDK to build and requires the .NET 10 runtime to run.

## Azure Cosmos DB for NoSQL

<a name="full-text-search-support"></a>

### Full-text search support

Azure Cosmos DB now offers support for [full-text search](/azure/cosmos-db/gen-ai/full-text-search). It enables efficient and effective text searches, as well as evaluating the relevance of documents to a given search query. It can be used in combination with vector search to improve the accuracy of responses in some AI scenarios.
EF Core 10 is adding support for this feature allowing for modeling the database with full-text search enabled properties and using full-text search functions inside queries targeting Azure Cosmos DB.

Here is a basic EF model configuration enabling full-text search on one of the properties:

```c#
public class Blog
{
...

public string Contents { get; set; }
}

public class BloggingContext
{
...

protected override void OnModelCreating(ModelBuilder modelBuilder)
{
modelBuilder.Entity<Blog>(b =>
{
b.Property(x => x.Contents).EnableFullTextSearch();
b.HasIndex(x => x.Contents).IsFullTextIndex();
});
}
}
```

Once the model is configured, we can use full-text search operations in queries using methods provided in `EF.Functions`:

```c#
var cosmosBlogs = await context.Blogs.Where(x => EF.Functions.FullTextContains(x.Contents, "cosmos")).ToListAsync();
```

The following full-text operations are currently supported: [`FullTextContains`](/azure/cosmos-db/nosql/query/fulltextcontains), [`FullTextContainsAll`](/azure/cosmos-db/nosql/query/fulltextcontainsall), [`FullTextContainsAny`](/azure/cosmos-db/nosql/query/fulltextcontainsany), [`FullTextScore`](/azure/cosmos-db/nosql/query/fulltextscore).

For more information on Cosmos full-text search, see the [docs](xref:core/providers/cosmos/full-text-search).

### Hybrid search

EF Core now supports [`RRF`](/azure/cosmos-db/nosql/query/rrf) (Reciprocal Rank Fusion) function, which combines vector similarity search and full-text search (i.e. hybrid search). Here is an example query using hybrid search:

```c#
float[] myVector = /* generate vector data from text, image, etc. */
var hybrid = await context.Blogs.OrderBy(x => EF.Functions.Rrf(
EF.Functions.FullTextScore(x.Contents, "database"),
EF.Functions.VectorDistance(x.Vector, myVector)))
.Take(10)
.ToListAsync();
```

For more information on Cosmos hybrid search, see the [docs](xref:core/providers/cosmos/full-text-search?#hybrid-search).

### Vector similarity search exits preview

In EF9 we added experimental support for [vector similarity search](xref:core/what-is-new/ef-core-9.0/whatsnew#vector-similarity-search-preview). In EF Core 10, vector similarity search support is no longer experimental. We have also made some improvements to the feature:

- EF Core can now generate containers with vector properties defined on owned reference entities. Containers with vector properties defined on owned collections still have to be created by other means. However, they can be used in queries.
- Model building APIs have been renamed. A vector property can now be configured using the `IsVectorProperty` method, and vector index can be configured using the `IsVectorIndex` method.

For more information on Cosmos vector search, see the [docs](xref:core/providers/cosmos/vector-search).

<a name="improved-model-evolution"></a>

### Improved experience when evolving the model
Expand Down Expand Up @@ -64,12 +130,12 @@ See [#12793](https://github.com/dotnet/efcore/issues/12793) and [#35367](https:/

### Other query improvements

* Translation for DateOnly.ToDateTime(timeOnly) ([#35194](https://github.com/dotnet/efcore/pull/35194), contributed by [@mseada94](https://github.com/mseada94)).
* Optimization for multiple consecutive `LIMIT`s ([#35384](https://github.com/dotnet/efcore/pull/35384), contributed by [@ranma42](https://github.com/ranma42)).
* Optimization for use of `Count` operation on `ICollection<T>` ([#35381](https://github.com/dotnet/efcore/pull/35381), contributed by [@ChrisJollyAU](https://github.com/ChrisJollyAU)).
* Optimization for `MIN`/`MAX` over `DISTINCT` ([#34699](https://github.com/dotnet/efcore/pull/34699), contributed by [@ranma42](https://github.com/ranma42)).
* Translation for date/time functions using `DatePart.Microsecond` and `DatePart.Nanosecond` arguments ([#34861](https://github.com/dotnet/efcore/pull/34861)).
* Simplifying parameter names (e.g. from `@__city_0` to `city`) ([#35200](https://github.com/dotnet/efcore/pull/35200)).
- Translation for DateOnly.ToDateTime(timeOnly) ([#35194](https://github.com/dotnet/efcore/pull/35194), contributed by [@mseada94](https://github.com/mseada94)).
- Optimization for multiple consecutive `LIMIT`s ([#35384](https://github.com/dotnet/efcore/pull/35384), contributed by [@ranma42](https://github.com/ranma42)).
- Optimization for use of `Count` operation on `ICollection<T>` ([#35381](https://github.com/dotnet/efcore/pull/35381), contributed by [@ChrisJollyAU](https://github.com/ChrisJollyAU)).
- Optimization for `MIN`/`MAX` over `DISTINCT` ([#34699](https://github.com/dotnet/efcore/pull/34699), contributed by [@ranma42](https://github.com/ranma42)).
- Translation for date/time functions using `DatePart.Microsecond` and `DatePart.Nanosecond` arguments ([#34861](https://github.com/dotnet/efcore/pull/34861)).
- Simplifying parameter names (e.g. from `@__city_0` to `city`) ([#35200](https://github.com/dotnet/efcore/pull/35200)).

## ExecuteUpdateAsync now accepts a regular, non-expression lambda

Expand Down Expand Up @@ -120,7 +186,7 @@ Thanks to [@aradalvand](https://github.com/aradalvand) for proposing and pushing

## Other improvements

* Make SQL Server scaffolding compatible with Azure Data Explorer ([#34832](https://github.com/dotnet/efcore/pull/34832), contributed by [@barnuri](https://github.com/barnuri)).
* Associate the DatabaseRoot with the scoped options instance and not the singleton options ([#34477](https://github.com/dotnet/efcore/pull/34477), contributed by [@koenigst](https://github.com/koenigst)).
* Redact inlined constants from log when sensitive logging is off ([#35724](https://github.com/dotnet/efcore/pull/35724)).
* Improve LoadExtension to work correctly with dotnet run and lib* named libs ([#35617](https://github.com/dotnet/efcore/pull/35617), contributed by [@krwq](https://github.com/krwq)).
- Make SQL Server scaffolding compatible with Azure Data Explorer ([#34832](https://github.com/dotnet/efcore/pull/34832), contributed by [@barnuri](https://github.com/barnuri)).
- Associate the DatabaseRoot with the scoped options instance and not the singleton options ([#34477](https://github.com/dotnet/efcore/pull/34477), contributed by [@koenigst](https://github.com/koenigst)).
- Redact inlined constants from log when sensitive logging is off ([#35724](https://github.com/dotnet/efcore/pull/35724)).
- Improve LoadExtension to work correctly with dotnet run and lib* named libs ([#35617](https://github.com/dotnet/efcore/pull/35617), contributed by [@krwq](https://github.com/krwq)).