Skip to content

Commit 3948017

Browse files
maumarmaumar
committed
full-text search docs
Also added note about vector search going out of preview
1 parent 63752a7 commit 3948017

File tree

1 file changed

+162
-10
lines changed
  • entity-framework/core/what-is-new/ef-core-10.0

1 file changed

+162
-10
lines changed

‎entity-framework/core/what-is-new/ef-core-10.0/whatsnew.md

Lines changed: 162 additions & 10 deletions
Original file line numberDiff line numberDiff line change
@@ -21,6 +21,158 @@ EF10 requires the .NET 10 SDK to build and requires the .NET 10 runtime to run.
2121

2222
## Azure Cosmos DB for NoSQL
2323

24+
<a name="full-text-search-support"></a>
25+
26+
### Full-text search support
27+
28+
Azure Cosmos DB now offers support for [full-text search](/azure/cosmos-db/gen-ai/full-text-search). It enables efficient and effective text searches, as well as evaluating the relevance of documents to a given search query. It can be used in combination with vector search to improve the accuracy of responses in some AI scenarios.
29+
EF Core 10 is adding support for this feature allowing for modeling the database with full-text search enabled properties and using full-text search functions inside queries targeting Azure Cosmos DB.
30+
31+
<a name="full-text-search-model-configuration"></a>
32+
33+
#### Model configuration
34+
35+
A property can be configured inside `OnModelCreating` to use full-text search by enabling it for the property and defining a full-text index:
36+
37+
```c#
38+
public class Blog
39+
{
40+
...
41+
42+
public string Contents { get; set; }
43+
}
44+
45+
public class BloggingContext
46+
{
47+
...
48+
49+
protected override void OnModelCreating(ModelBuilder modelBuilder)
50+
{
51+
modelBuilder.Entity<Blog>(b =>
52+
{
53+
b.Property(x => x.Contents).EnableFullTextSearch();
54+
b.HasIndex(x => x.Contents).IsFullTextIndex();
55+
});
56+
}
57+
}
58+
```
59+
60+
> [!NOTE]
61+
> Configuring the index is not mandatory, but it is recommended as it greatly improves performance of full-text search queries.
62+
63+
Full-text search operations are language specific, using American English (`en-US`) by default. You can customize the language for individual properties as part of `EnableFullTextSearch` call:
64+
65+
```c#
66+
protected override void OnModelCreating(ModelBuilder modelBuilder)
67+
{
68+
modelBuilder.Entity<Blog>(b =>
69+
{
70+
b.Property(x => x.Contents).EnableFullTextSearch();
71+
b.HasIndex(x => x.Contents).IsFullTextIndex();
72+
b.Property(x => x.ContentsGerman).EnableFullTextSearch("de-DE");
73+
b.HasIndex(x => x.ContentsGerman).IsFullTextIndex();
74+
});
75+
}
76+
```
77+
78+
You can also set a default language for the container - unless overridden in the `EnableFullTextSearch` method, all full-text properties inside the container will use that language.
79+
80+
```c#
81+
protected override void OnModelCreating(ModelBuilder modelBuilder)
82+
{
83+
modelBuilder.Entity<Blog>(b =>
84+
{
85+
b.HasDefaultFullTextLanguage("de-DE");
86+
b.Property(x => x.ContentsEnglish).EnableFullTextSearch("en-US");
87+
b.HasIndex(x => x.ContentsEnglish).IsFullTextIndex();
88+
b.Property(x => x.ContentsGerman).EnableFullTextSearch();
89+
b.HasIndex(x => x.ContentsGerman).IsFullTextIndex();
90+
b.Property(x => x.TagsGerman).EnableFullTextSearch();
91+
b.HasIndex(x => x.TagsGerman).IsFullTextIndex();
92+
});
93+
}
94+
```
95+
96+
<a name="full-text-search-querying"></a>
97+
98+
#### Querying
99+
100+
As part of the full-text search feature, Azure Cosmos DB introduced several built-in functions which allow for efficient querying of content inside the full-text search enabled properties. These functions are: `FullTextContains`, `FullTextContainsAll`, `FullTextContainsAny`, which look for specific keyword or keywords and `FullTextScore`, which returns [BM25 score](https://en.wikipedia.org/wiki/Okapi_BM25) based on provided keywords.
101+
102+
> [!NOTE]
103+
> `FullTextScore` can only be used inside `OrderBy` to rank the documents based on the score.
104+
105+
EF Core exposes these functions as part of `EF.Functions` so they can be used in queries:
106+
107+
```c#
108+
var cosmosBlogs = await context.Blogs.Where(x => EF.Functions.FullTextContainsAll(x.Contents, "database", "cosmos")).ToListAsync();
109+
110+
var keywords = new string[] { "AI", "agent", "breakthrough" };
111+
var mostInteresting = await context.Blogs.OrderBy(x => EF.Functions.FullTextScore(x.Contents, keywords)).Take(5).ToListAsync();
112+
```
113+
114+
<a name="full-text-search-hybrid-search"></a>
115+
116+
#### Hybrid search
117+
118+
Full-text search can be used with vector search in the same query (i.e. hybrid search), by combining results of `FullTextScore` and `VectorDistance` functions. It can be done using the `RRF` function (Reciprocal Rank Fusion), which EF Core also provides inside `EF.Functions`:
119+
120+
```c#
121+
public class Blog
122+
{
123+
...
124+
125+
public float[] Vector { get; set; }
126+
public string Contents { get; set; }
127+
}
128+
129+
public class BloggingContext
130+
{
131+
...
132+
133+
protected override void OnModelCreating(ModelBuilder modelBuilder)
134+
{
135+
modelBuilder.Entity<Blog>(b =>
136+
{
137+
b.Property(x => x.Contents).EnableFullTextSearch();
138+
b.HasIndex(x => x.Contents).IsFullTextIndex();
139+
140+
b.Property(x => x.Vector).IsVectorProperty(DistanceFunction.Cosine, dimensions: 1536);
141+
b.HasIndex(x => x.Vector).IsVectorIndex(VectorIndexType.Flat);
142+
});
143+
}
144+
}
145+
146+
float[] myVector = ...
147+
var hybrid = await context.Blogs.OrderBy(x => EF.Functions.Rrf(
148+
EF.Functions.FullTextScore(x.Contents, "database"),
149+
EF.Functions.VectorDistance(x.Vector, myVector)))
150+
.Take(10)
151+
.ToListAsync();
152+
```
153+
154+
> [!TIP]
155+
> You can combine more than two scoring functions inside `Rrf` call, as well as using only `FullTextScore`, or only `VectorDistance`.
156+
157+
<a name="full-text-search-limitations"></a>
158+
159+
#### Limitations
160+
161+
Here are some current limitations of the full-text search using EF Core:
162+
163+
- `FullTextScore` and `RRF` functions can only be used inside `OrderBy` and are mutually exclusive with other forms of ordering.
164+
- EF Core can't create a container with full-text search or vector properties defined inside collection navigation - you can use those properties in queries, but the database has to be generated by other means (e.g. Microsoft.Azure.Cosmos SDK or using UI on Azure Portal).
165+
- Keyword arguments for `FullTextContainsAll`, `FullTextContainsAny` and `FullTextScore` have to be either constants or parameters. `FullTextContains` allows more complex expressions.
166+
167+
<a name="vector-search-exits-preview"></a>
168+
169+
### Vector similarity search exits preview
170+
171+
In EF9 we added experimental support for [vector similarity search](xref:core/what-is-new/ef-core-9.0/whatsnew#vector-similarity-search-preview). In EF Core 10, vector similarity search support is no longer experimental. We have also made some improvements to the feature:
172+
173+
- EF Core can now generate containers with vector properties defined on owned reference entities. Containers with vector properties defined on owned collections still have to be created by other means. However, they can be used in queries.
174+
- Model building APIs have been renamed. A vector property can now be configured using the `IsVectorProperty` method, and vector index can be configured using the `IsVectorIndex` method.
175+
24176
<a name="improved-model-evolution"></a>
25177

26178
### Improved experience when evolving the model
@@ -64,12 +216,12 @@ See [#12793](https://github.com/dotnet/efcore/issues/12793) and [#35367](https:/
64216

65217
### Other query improvements
66218

67-
* Translation for DateOnly.ToDateTime(timeOnly) ([#35194](https://github.com/dotnet/efcore/pull/35194), contributed by [@mseada94](https://github.com/mseada94)).
68-
* Optimization for multiple consecutive `LIMIT`s ([#35384](https://github.com/dotnet/efcore/pull/35384), contributed by [@ranma42](https://github.com/ranma42)).
69-
* Optimization for use of `Count` operation on `ICollection<T>` ([#35381](https://github.com/dotnet/efcore/pull/35381), contributed by [@ChrisJollyAU](https://github.com/ChrisJollyAU)).
70-
* Optimization for `MIN`/`MAX` over `DISTINCT` ([#34699](https://github.com/dotnet/efcore/pull/34699), contributed by [@ranma42](https://github.com/ranma42)).
71-
* Translation for date/time functions using `DatePart.Microsecond` and `DatePart.Nanosecond` arguments ([#34861](https://github.com/dotnet/efcore/pull/34861)).
72-
* Simplifying parameter names (e.g. from `@__city_0` to `city`) ([#35200](https://github.com/dotnet/efcore/pull/35200)).
219+
- Translation for DateOnly.ToDateTime(timeOnly) ([#35194](https://github.com/dotnet/efcore/pull/35194), contributed by [@mseada94](https://github.com/mseada94)).
220+
- Optimization for multiple consecutive `LIMIT`s ([#35384](https://github.com/dotnet/efcore/pull/35384), contributed by [@ranma42](https://github.com/ranma42)).
221+
- Optimization for use of `Count` operation on `ICollection<T>` ([#35381](https://github.com/dotnet/efcore/pull/35381), contributed by [@ChrisJollyAU](https://github.com/ChrisJollyAU)).
222+
- Optimization for `MIN`/`MAX` over `DISTINCT` ([#34699](https://github.com/dotnet/efcore/pull/34699), contributed by [@ranma42](https://github.com/ranma42)).
223+
- Translation for date/time functions using `DatePart.Microsecond` and `DatePart.Nanosecond` arguments ([#34861](https://github.com/dotnet/efcore/pull/34861)).
224+
- Simplifying parameter names (e.g. from `@__city_0` to `city`) ([#35200](https://github.com/dotnet/efcore/pull/35200)).
73225

74226
## ExecuteUpdateAsync now accepts a regular, non-expression lambda
75227

@@ -120,7 +272,7 @@ Thanks to [@aradalvand](https://github.com/aradalvand) for proposing and pushing
120272

121273
## Other improvements
122274

123-
* Make SQL Server scaffolding compatible with Azure Data Explorer ([#34832](https://github.com/dotnet/efcore/pull/34832), contributed by [@barnuri](https://github.com/barnuri)).
124-
* Associate the DatabaseRoot with the scoped options instance and not the singleton options ([#34477](https://github.com/dotnet/efcore/pull/34477), contributed by [@koenigst](https://github.com/koenigst)).
125-
* Redact inlined constants from log when sensitive logging is off ([#35724](https://github.com/dotnet/efcore/pull/35724)).
126-
* Improve LoadExtension to work correctly with dotnet run and lib* named libs ([#35617](https://github.com/dotnet/efcore/pull/35617), contributed by [@krwq](https://github.com/krwq)).
275+
- Make SQL Server scaffolding compatible with Azure Data Explorer ([#34832](https://github.com/dotnet/efcore/pull/34832), contributed by [@barnuri](https://github.com/barnuri)).
276+
- Associate the DatabaseRoot with the scoped options instance and not the singleton options ([#34477](https://github.com/dotnet/efcore/pull/34477), contributed by [@koenigst](https://github.com/koenigst)).
277+
- Redact inlined constants from log when sensitive logging is off ([#35724](https://github.com/dotnet/efcore/pull/35724)).
278+
- Improve LoadExtension to work correctly with dotnet run and lib* named libs ([#35617](https://github.com/dotnet/efcore/pull/35617), contributed by [@krwq](https://github.com/krwq)).

0 commit comments

Comments
 (0)