.Net: Agent memory explore v2

docs/decisions/00NN-agents-with-memory.md

@@ -0,0 +1,206 @@
+---
+# These are optional elements. Feel free to remove any of them.
+status: accepted
+contact: westey-m
+date: 2025-04-17
+deciders: westey-m, markwallace-microsoft, alliscode, TaoChenOSU, moonbox3, crickman
+consulted: westey-m, markwallace-microsoft, alliscode, TaoChenOSU, moonbox3, crickman
+informed: westey-m, markwallace-microsoft, alliscode, TaoChenOSU, moonbox3, crickman
+---
+
+# Agents with Memory
+
+## What do we mean by Memory?
+
+By memory we mean the capability to remember information and skills that are learned during
+a conversation and re-use those later in the same conversation or later in a subsequent conversation.
+
+## Context and Problem Statement
+
+Today we support multiple agent types with different characteristics:
+
+1. In process vs remote.
+1. Remote agents that store and maintain conversation state in the service vs those that require the caller to provide conversation state on each invocation.
+
+We need to support advanced memory capabilities across this range of agent types.
+
+### Memory Scope
+
+Another aspect of memory that is important to consider is the scope of different memory types.
+Most agent implementations have instructions and skills but the agent is not tied to a single conversation.
+On each invocation of the agent, the agent is told which conversation to participate in, during that invocation.
+
+Memories about a user or about a conversation with a user is therefore extracted from one of these conversation and recalled
+during the same or another conversation with the same user.
+These memories will typically contain information that the user would not like to share with other users of the system.
+
+Other types of memories also exist which are not tied to a specific user or conversation.
+E.g. an Agent may learn how to do something and be able to do that in many conversations with different users.
+With these type of memories there is of cousrse risk in leaking personal information between different users which is important to guard against.
+
+### Packaging memory capabilities
+
+All of the above memory types can be supported for any agent by attaching software components to conversation threads.
+This is achieved via a simple mechanism of:
+
+1. Inspecting and using messages as they are passed to and from the agent.
+1. Passing additional context to the agent per invocation.
+
+With our current `AgentThread` implementation, when an agent is invoked, all input and output messages are already passed to the `AgentThread`
+and can be made available to any components attached to the `AgentThread`.
+Where agents are remote/external and manage conversation state in the service, passing the messages to the `AgentThread` may not have any
+affect on the thread in the service. This is OK, since the service will have already updated the thread during the remote invocation.
+It does however, still allow us to subscribe to messages in any attached components.
+
+For the second requirement of getting additional context per invocation, the agent may ask the thread passed to it, to in turn ask
+each of the components attached to it, to provide context to pass to the Agent.
+This enables the component to provide memories that it contains to the Agent as needed.
+
+Different memory capabilities can be built using separate components. Each component would have the following characteristics:
+
+1. May store some context that can be provided to the agent per invocation.
+1. May inspect messages from the conversation to learn from the conversation and build its context.
+1. May register plugins to allow the agent to directly store, retrieve, update or clear memories.
+
+### Suspend / Resume
+
+Building a service to host an agent comes with challenges.
+It's hard to build a stateful service, but service consumers expect an experience that looks stateful from the outside.
+E.g. on each invocation, the user expects that the service can continue a conversation they are having.
+
+This means that where the the service is exposing a local agent with local conversation state management (e.g. via `ChatHistory`)
+that conversation state needs to be loaded and persisted for each invocation of the service.
+
+It also means that any memory components that may have some in-memory state will need to be loaded and persisted too.
+
+For cases like this, the `OnSuspend` and `OnResume` methods allow notification of the components that they need to save or reload their state.
+It is up to each of these components to decide how and where to save state to or load state from.
+
+## Proposed interface for Memory Components
+
+The types of events that Memory Components require are not unique to memory, and can be used to package up other capabilities too.
+The suggestion is therefore to create a more generally named type that can be used for other scenarios as well and can even
+be used for non-agent scenarios too.
+
+This type should live in the `Microsoft.SemanticKernel.Abstractions` nuget, since these components can be used by systems other than just agents.
+
+```csharp
+namespace Microsoft.SemanticKernel;
+
+public abstract class ConversationStateExtension
+{
+    public virtual IReadOnlyCollection<AIFunction> AIFunctions => Array.Empty<AIFunction>();
+
+    public virtual Task OnThreadCreatedAsync(string? threadId, CancellationToken cancellationToken = default);
+    public virtual Task OnThreadDeleteAsync(string? threadId, CancellationToken cancellationToken = default);
+
+    // OnThreadCheckpointAsync not included in initial release, maybe in future.
+    public virtual Task OnThreadCheckpointAsync(string? threadId, CancellationToken cancellationToken = default);
+
+    public virtual Task OnNewMessageAsync(string? threadId, ChatMessage newMessage, CancellationToken cancellationToken = default);
+    public abstract Task<string> OnModelInvokeAsync(ICollection<ChatMessage> newMessages, CancellationToken cancellationToken = default);
+
+    public virtual Task OnSuspendAsync(string? threadId, CancellationToken cancellationToken = default);
+    public virtual Task OnResumeAsync(string? threadId, CancellationToken cancellationToken = default);
+}
+```
+
+> TODO: Decide about the correct namespace for `ConversationStateExtension`
+
+## Managing multiple components
+
+To manage multiple components I propose that we have a `ConversationStateExtensionsManager`.
+This class allows registering components and delegating new message notifications, ai invocation calls, etc. to the contained components.
+
+## Integrating with agents
+
+I propose to add a `ConversationStateExtensionsManager` to the `AgentThread` class, allowing us to attach components to any `AgentThread`.
+
+When an `Agent` is invoked, we will call `OnModelInvokeAsync` on each component via the `ConversationStateExtensionsManager` to get
+a combined set of context to pass to the agent for this invocation. This will be internal to the `Agent` class and transparent to the user.
+
+```csharp
+var additionalInstructions = await currentAgentThread.OnModelInvokeAsync(messages, cancellationToken).ConfigureAwait(false);
+```
+
+## Usage examples
+
+### Multiple threads using the same memory component
+
+```csharp
+// Create a vector store for storing memories.
+var vectorStore = new InMemoryVectorStore();
+// Create a memory store that is tired to a "Memories" collection in the vector store and stores memories under the "user/12345" namespace.
+using var textMemoryStore = new VectorDataTextMemoryStore<string>(vectorStore, textEmbeddingService, "Memories", "user/12345", 1536);
+
+// Create a memory component to will pull user facts from the conversation, store them in the vector store
+// and pass them to the agent as additional instructions.
+var userFacts = new UserFactsMemoryComponent(this.Fixture.Agent.Kernel, textMemoryStore);
+
+// Create a thread and attach a Memory Component.
+var agentThread1 = new ChatHistoryAgentThread();
+agentThread1.ThreadExtensionsManager.Add(userFacts);
+var asyncResults1 = agent.InvokeAsync("Hello, my name is Caoimhe.", agentThread1);
+
+// Create a second thread and attach a Memory Component.
+var agentThread2 = new ChatHistoryAgentThread();
+agentThread2.ThreadExtensionsManager.Add(userFacts);
+var asyncResults2 = agent.InvokeAsync("What is my name?.", agentThread2);
+// Expected response contains Caoimhe.
+```
+
+### Using a RAG component
+
+```csharp
+// Create Vector Store and Rag Store/Component
+var vectorStore = new InMemoryVectorStore();
+using var ragStore = new TextRagStore<string>(vectorStore, textEmbeddingService, "Memories", 1536, "group/g2");
+var ragComponent = new TextRagComponent(ragStore, new TextRagComponentOptions());
+
+// Upsert docs into vector store.
+await ragStore.UpsertDocumentsAsync(
+[
+    new TextRagDocument("The financial results of Contoso Corp for 2023 is as follows:\nIncome EUR 174 000 000\nExpenses EUR 152 000 000")
+    {
+        SourceName = "Contoso 2023 Financial Report",
+        SourceReference = "https://www.consoso.com/reports/2023.pdf",
+        Namespaces = ["group/g2"]
+    }
+]);
+
+// Create a new agent thread and register the Rag component
+var agentThread = new ChatHistoryAgentThread();
+agentThread.ThreadExtensionsManager.RegisterThreadExtension(ragComponent);
+
+// Inovke the agent.
+var asyncResults1 = agent.InvokeAsync("What was the income of Contoso for 2023", agentThread);
+// Expected response contains the 174M income from the document.
+```
+
+## Decisions to make
+
+### Extension base class name
+
+1. ConversationStateExtension
+    1. Long
+1. MemoryComponent
+    1. Too specific
+
+Chose ConversationStateExtension.
+
+### Location for abstractions
+
+1. Microsoft.SemanticKernel.<baseclass>
+1. Microsoft.SemanticKernel.Memory.<baseclass>
+1. Microsoft.SemanticKernel.Memory.<baseclass> (in separate nuget)
+
+Chose Microsoft.SemanticKernel.<baseclass>.
+
+### Location for memory components
+
+1. A nuget for each component
+1. Microsoft.SemanticKernel.Core nuget
+1. Microsoft.SemanticKernel.Memory nuget
+1. Microsoft.SemanticKernel.ConversationStateExtensions nuget
+
+Chose Microsoft.SemanticKernel.Core nuget

dotnet/SK-dotnet.sln

@@ -524,6 +524,11 @@ Project("{FAE04EC0-301F-11D3-BF4B-00C04F79EFBC}") = "MCPClient", "samples\Demos\
 		{12C7E0C7-A7DF-3BC3-0D4B-1A706BCE6981} = {12C7E0C7-A7DF-3BC3-0D4B-1A706BCE6981}
 	EndProjectSection
 EndProject
+Project("{2150E333-8FDC-42A3-9474-1A3956D46DE8}") = "memory", "memory", "{4B850B93-46D6-4F25-9DB1-90D1E6E4AB70}"
+EndProject
+Project("{FAE04EC0-301F-11D3-BF4B-00C04F79EFBC}") = "Memory.Abstractions", "src\Memory\Memory.Abstractions\Memory.Abstractions.csproj", "{F5124057-1DA1-4799-9357-D9A635047678}"
+EndProject
+Project("{FAE04EC0-301F-11D3-BF4B-00C04F79EFBC}") = "Memory", "src\Memory\Memory\Memory.csproj", "{A0538079-AB6F-4C7D-9138-A15258583F80}"
 Project("{2150E333-8FDC-42A3-9474-1A3956D46DE8}") = "ProcessWithCloudEvents", "ProcessWithCloudEvents", "{7C092DD9-9985-4D18-A817-15317D984149}"
 	ProjectSection(SolutionItems) = preProject
 		samples\Demos\ProcessWithCloudEvents\README.md = samples\Demos\ProcessWithCloudEvents\README.md
@@ -1460,6 +1465,18 @@ Global
 		{B06770D5-2F3E-4271-9F6B-3AA9E716176F}.Publish|Any CPU.Build.0 = Release|Any CPU
 		{B06770D5-2F3E-4271-9F6B-3AA9E716176F}.Release|Any CPU.ActiveCfg = Release|Any CPU
 		{B06770D5-2F3E-4271-9F6B-3AA9E716176F}.Release|Any CPU.Build.0 = Release|Any CPU
+		{F5124057-1DA1-4799-9357-D9A635047678}.Debug|Any CPU.ActiveCfg = Debug|Any CPU
+		{F5124057-1DA1-4799-9357-D9A635047678}.Debug|Any CPU.Build.0 = Debug|Any CPU
+		{F5124057-1DA1-4799-9357-D9A635047678}.Publish|Any CPU.ActiveCfg = Release|Any CPU
+		{F5124057-1DA1-4799-9357-D9A635047678}.Publish|Any CPU.Build.0 = Release|Any CPU
+		{F5124057-1DA1-4799-9357-D9A635047678}.Release|Any CPU.ActiveCfg = Release|Any CPU
+		{F5124057-1DA1-4799-9357-D9A635047678}.Release|Any CPU.Build.0 = Release|Any CPU
+		{A0538079-AB6F-4C7D-9138-A15258583F80}.Debug|Any CPU.ActiveCfg = Debug|Any CPU
+		{A0538079-AB6F-4C7D-9138-A15258583F80}.Debug|Any CPU.Build.0 = Debug|Any CPU
+		{A0538079-AB6F-4C7D-9138-A15258583F80}.Publish|Any CPU.ActiveCfg = Release|Any CPU
+		{A0538079-AB6F-4C7D-9138-A15258583F80}.Publish|Any CPU.Build.0 = Release|Any CPU
+		{A0538079-AB6F-4C7D-9138-A15258583F80}.Release|Any CPU.ActiveCfg = Release|Any CPU
+		{A0538079-AB6F-4C7D-9138-A15258583F80}.Release|Any CPU.Build.0 = Release|Any CPU
 		{31F6608A-FD36-F529-A5FC-C954A0B5E29E}.Debug|Any CPU.ActiveCfg = Debug|Any CPU
 		{31F6608A-FD36-F529-A5FC-C954A0B5E29E}.Debug|Any CPU.Build.0 = Debug|Any CPU
 		{31F6608A-FD36-F529-A5FC-C954A0B5E29E}.Publish|Any CPU.ActiveCfg = Release|Any CPU
@@ -1703,6 +1720,9 @@ Global
 		{879545ED-D429-49B1-96F1-2EC55FFED31D} = {5D4C0700-BBB5-418F-A7B2-F392B9A18263}
 		{12C7E0C7-A7DF-3BC3-0D4B-1A706BCE6981} = {879545ED-D429-49B1-96F1-2EC55FFED31D}
 		{B06770D5-2F3E-4271-9F6B-3AA9E716176F} = {879545ED-D429-49B1-96F1-2EC55FFED31D}
+		{4B850B93-46D6-4F25-9DB1-90D1E6E4AB70} = {831DDCA2-7D2C-4C31-80DB-6BDB3E1F7AE0}
+		{F5124057-1DA1-4799-9357-D9A635047678} = {4B850B93-46D6-4F25-9DB1-90D1E6E4AB70}
+		{A0538079-AB6F-4C7D-9138-A15258583F80} = {4B850B93-46D6-4F25-9DB1-90D1E6E4AB70}
 		{7C092DD9-9985-4D18-A817-15317D984149} = {5D4C0700-BBB5-418F-A7B2-F392B9A18263}
 		{31F6608A-FD36-F529-A5FC-C954A0B5E29E} = {7C092DD9-9985-4D18-A817-15317D984149}
 		{08D84994-794A-760F-95FD-4EFA8998A16D} = {7C092DD9-9985-4D18-A817-15317D984149}

dotnet/docs/EXPERIMENTS.md

@@ -25,6 +25,7 @@ You can use the following diagnostic IDs to ignore warnings or errors for a part
 | SKEXP0100 | Advanced Semantic Kernel features |
 | SKEXP0110 | Semantic Kernel Agents |
 | SKEXP0120 | Native-AOT |
+| SKEXP0130 | Conversation State |
 
 ## Experimental Features Tracking
 

dotnet/samples/Demos/ModelContextProtocolPlugin/ModelContextProtocolPlugin.csproj

@@ -16,6 +16,7 @@
     <PackageReference Include="Microsoft.Extensions.Logging" />
     <PackageReference Include="Microsoft.Extensions.Logging.Console" />
     <PackageReference Include="Microsoft.Extensions.Logging.Debug" />
+    <PackageReference Include="System.Linq.Async" />
   </ItemGroup>
 
   <ItemGroup>

dotnet/src/Agents/Abstractions/AgentThread.cs

@@ -1,6 +1,7 @@
 // Copyright (c) Microsoft. All rights reserved.
 
 using System;
+using System.Diagnostics.CodeAnalysis;
 using System.Threading;
 using System.Threading.Tasks;
 
@@ -26,6 +27,52 @@ public abstract class AgentThread
     /// </summary>
     public virtual bool IsDeleted { get; protected set; } = false;
 
+    /// <summary>
+    /// Gets or sets the container for conversation state part components that manages their lifecycle and interactions.
+    /// </summary>
+    [Experimental("SKEXP0110")]
+    public virtual ConversationStatePartsManager StateParts { get; init; } = new ConversationStatePartsManager();
+
+    /// <summary>
+    /// Called when the current conversion is temporarily suspended and any state should be saved.
+    /// </summary>
+    /// <param name="cancellationToken">The <see cref="CancellationToken"/> to monitor for cancellation requests. The default is <see cref="CancellationToken.None"/>.</param>
+    /// <returns>An async task.</returns>
+    /// <remarks>
+    /// In a service that hosts an agent, that is invoked via calls to the service, this might be at the end of each service call.
+    /// In a client application, this might be when the user closes the chat window or the application.
+    /// </remarks>
+    [Experimental("SKEXP0110")]
+    public virtual Task OnSuspendAsync(CancellationToken cancellationToken = default)
+    {
+        return this.StateParts.OnSuspendAsync(this.Id, cancellationToken);
+    }
+
+    /// <summary>
+    /// Called when the current conversion is resumed and any state should be restored.
+    /// </summary>
+    /// <param name="cancellationToken">The <see cref="CancellationToken"/> to monitor for cancellation requests. The default is <see cref="CancellationToken.None"/>.</param>
+    /// <returns>An async task.</returns>
+    /// <remarks>
+    /// In a service that hosts an agent, that is invoked via calls to the service, this might be at the start of each service call where a previous conversation is being continued.
+    /// In a client application, this might be when the user re-opens the chat window to resume a conversation after having previously closed it.
+    /// </remarks>
+    [Experimental("SKEXP0110")]
+    public virtual Task OnResumeAsync(CancellationToken cancellationToken = default)
+    {
+        if (this.IsDeleted)
+        {
+            throw new InvalidOperationException("This thread has been deleted and cannot be used anymore.");
+        }
+
+        if (this.Id is null)
+        {
+            throw new InvalidOperationException("This thread cannot be resumed, since it has not been created.");
+        }
+
+        return this.StateParts.OnResumeAsync(this.Id, cancellationToken);
+    }
+
     /// <summary>
     /// Creates the thread and returns the thread id.
     /// </summary>
@@ -45,6 +92,10 @@ protected internal virtual async Task CreateAsync(CancellationToken cancellation
         }
 
         this.Id = await this.CreateInternalAsync(cancellationToken: cancellationToken).ConfigureAwait(false);
+
+#pragma warning disable SKEXP0110 // Type is for evaluation purposes only and is subject to change or removal in future updates. Suppress this diagnostic to proceed.
+        await this.StateParts.OnThreadCreatedAsync(this.Id!, cancellationToken).ConfigureAwait(false);
+#pragma warning restore SKEXP0110 // Type is for evaluation purposes only and is subject to change or removal in future updates. Suppress this diagnostic to proceed.
     }
 
     /// <summary>
@@ -65,6 +116,10 @@ public virtual async Task DeleteAsync(CancellationToken cancellationToken = defa
             throw new InvalidOperationException("This thread cannot be deleted, since it has not been created.");
         }
 
+#pragma warning disable SKEXP0110 // Type is for evaluation purposes only and is subject to change or removal in future updates. Suppress this diagnostic to proceed.
+        await this.StateParts.OnThreadDeleteAsync(this.Id!, cancellationToken).ConfigureAwait(false);
+#pragma warning restore SKEXP0110 // Type is for evaluation purposes only and is subject to change or removal in future updates. Suppress this diagnostic to proceed.
+
         await this.DeleteInternalAsync(cancellationToken).ConfigureAwait(false);
 
         this.IsDeleted = true;
@@ -92,6 +147,10 @@ internal virtual async Task OnNewMessageAsync(ChatMessageContent newMessage, Can
             await this.CreateAsync(cancellationToken).ConfigureAwait(false);
         }
 
+#pragma warning disable SKEXP0110 // Type is for evaluation purposes only and is subject to change or removal in future updates. Suppress this diagnostic to proceed.
+        await this.StateParts.OnNewMessageAsync(this.Id, newMessage, cancellationToken).ConfigureAwait(false);
+#pragma warning restore SKEXP0110 // Type is for evaluation purposes only and is subject to change or removal in future updates. Suppress this diagnostic to proceed.
+
         await this.OnNewMessageInternalAsync(newMessage, cancellationToken).ConfigureAwait(false);
     }
 

dotnet/src/Agents/AzureAI/Agents.AzureAI.csproj

@@ -34,6 +34,7 @@
 
   <ItemGroup>
     <ProjectReference Include="..\Abstractions\Agents.Abstractions.csproj" />
+    <ProjectReference Include="..\..\SemanticKernel.Core\SemanticKernel.Core.csproj" />
   </ItemGroup>
 
   <ItemGroup>