.NET: [BUG] Flaky Integration Test: CreateAIAgentAsync_WithHostedFileSearchTool_SearchesFilesAsync - Vector Store Indexing Race Condition

[joslat]

Summary

The integration test CreateAIAgentAsync_WithHostedFileSearchTool_SearchesFilesAsync in OpenAIAssistantClientExtensionsTests is flaky and intermittently fails due to a race condition with OpenAI's vector store indexing process.

Test Failure

Assert.Contains() Failure: Sub-string not found
String:    "I couldn't find any documented code for '"···
Not found: "673457"

Root Cause
The test creates a vector store with file content and immediately queries the AI agent:

// Line 146-152: Create vector store and add file
var vectorStoreCreate = await vectorStoreClient.CreateVectorStoreAsync(options: new VectorStoreCreationOptions()
{
    Name = "WordCodeLookup_VectorStore",
    FileIds = { uploadedFileId }
});
string vectorStoreId = vectorStoreCreate.Value.Id;

// Lines 177-179: IMMEDIATELY query (no wait for indexing!)
var response = await agent.RunAsync("Can you give me the documented code for 'banana'?");
var text = response.ToString();
Assert.Contains("673457", text);  // ❌ FAILS - vector store not ready yet!

The Problem: Vector stores require time to index files before they're searchable. The test doesn't wait for indexing completion, creating a race condition.
Evidence of Race Condition
The agent responds with:
"I couldn't find any documented code for '"

This indicates the agent attempted file search but the vector store wasn't fully indexed yet, so no results were found.
Similar Pattern Found
The same pattern exists in AzureAIAgentsPersistentCreateTests.CreateAgent_CreatesAgentWithVectorStoresAsync (line 68-127), which also creates a vector store and immediately queries without waiting for indexing:

// tests/AzureAIAgentsPersistent.IntegrationTests/AzureAIAgentsPersistentCreateTests.cs
var vectorStoreMetadata = await this._persistentAgentsClient.VectorStores.CreateVectorStoreAsync([uploadedAgentFile.Id], name: "WordCodeLookup_VectorStore");

var agent = /* ... create agent with vector store ... */;

// IMMEDIATE query without waiting
var result = await agent.RunAsync("Can you give me the documented code for 'banana'?");
Assert.Contains("673457", result.ToString());  // Potentially flaky!

Impact

• ❌ CI/CD pipelines fail intermittently which makes the PR pipeline a bottleneck...
• ❌ False test failures waste developer time...
• ❌ Difficult to reproduce locally (timing-dependent)
• ❌ Reduces confidence in test suite
• ❌ Uses valuable compute and Developers time

---

Proposed Solutions
Solution 1: Wait for Vector Store Indexing Status (Recommended)
Add polling logic to wait until the vector store status is Completed:

// After creating the vector store
var vectorStoreCreate = await vectorStoreClient.CreateVectorStoreAsync(options: new VectorStoreCreationOptions()
{
    Name = "WordCodeLookup_VectorStore",
    FileIds = { uploadedFileId }
});
string vectorStoreId = vectorStoreCreate.Value.Id;

// ✅ NEW: Wait for indexing to complete
await WaitForVectorStoreReadyAsync(vectorStoreClient, vectorStoreId);

// Helper method
private static async Task WaitForVectorStoreReadyAsync(
    VectorStoreClient client, 
    string vectorStoreId,
    int maxWaitSeconds = 30)
{
    var sw = Stopwatch.StartNew();
    while (sw.Elapsed.TotalSeconds < maxWaitSeconds)
    {
        var vectorStore = await client.GetVectorStoreAsync(vectorStoreId);
        if (vectorStore.Value.Status == VectorStoreStatus.Completed)
        {
            return;  // ✅ Ready!
        }
        
        if (vectorStore.Value.Status == VectorStoreStatus.Failed)
        {
            throw new InvalidOperationException($"Vector store indexing failed");
        }
        
        await Task.Delay(1000);  // Poll every second
    }
    
    throw new TimeoutException($"Vector store did not complete indexing within {maxWaitSeconds}s");
}

---

Affected Tests

1. ✅ Confirmed Flaky:
   • OpenAIAssistantClientExtensionsTests.CreateAIAgentAsync_WithHostedFileSearchTool_SearchesFilesAsync
   • Location: tests/OpenAIAssistant.IntegrationTests/OpenAIAssistantClientExtensionsTests.cs:129-189
2. 🔍 Potentially Flaky (Same Pattern):
   • AzureAIAgentsPersistentCreateTests.CreateAgent_CreatesAgentWithVectorStoresAsync
   • Location: tests/AzureAIAgentsPersistent.IntegrationTests/AzureAIAgentsPersistentCreateTests.cs:68-127

---

Recommendation

Implement Solution 1 (wait for vector store status) for both tests:

1. ✅ Most reliable - no guessing on timing
2. ✅ Self-documenting - makes async nature explicit
3. ✅ Fails fast if indexing actually fails
4. ✅ Works across different environments (local, CI, slow networks)

---

Related Information
• Test Run Log: GitHub Actions run showing failure
• OpenAI API Docs: Vector store indexing is asynchronous
• Similar Issues: Azure AI services also require waiting for indexing completion

---

Tasks
• [ ] Fix OpenAIAssistantClientExtensionsTests.CreateAIAgentAsync_WithHostedFileSearchTool_SearchesFilesAsync
• [ ] Verify AzureAIAgentsPersistentCreateTests.CreateAgent_CreatesAgentWithVectorStoresAsync and fix if needed
• [ ] Add helper method WaitForVectorStoreReadyAsync to shared test utilities
• [ ] Update test documentation to note async indexing behavior
• [ ] Consider adding similar checks for other async operations (file uploads, code interpreter)

[jakesdupreez]

Good morning everyone. I'm preparing my environment to compile and become part of the collaboration but I'm currently struggling with getting all of the tests especially the integration tests to run as it seems I need a lot of different environment variable for some test. Is there some documentation or guidance on what I need to setup or have in place to run them all successfully. .

I had these initially o my first failed test but seems others need other variables. Do we have this centralized or some pattern to follow. This obviously does not go into GIT so just want to follow the teams standard please

dotnet user-secrets set "OpenAI:ChatModelId" ""
dotnet user-secrets set "OpenAI:ChatReasoningModelId" ""
dotnet user-secrets set "OpenAI:ApiKey" ""

dotnet user-secrets set "AzureAI:Endpoint" ""
dotnet user-secrets set "OpenAI:DeploymentName" ""
dotnet user-secrets set "OpenAI:BingConnectionId" ""

dotnet user-secrets set "Mem0:ServiceUri" ""
dotnet user-secrets set "Mem0:ApiKey" ""

Kind regards to all. Have a awesome bugless day. Fingers crossed

[rogerbarreto]

@jakesdupreez, Thanks for the interest, currently we don't have a central document for all the variables, but you may whoever check our workflow pipeline on which ones are used for the integration settings.

• https://github.com/microsoft/agent-framework/blob/main/.github/workflows/dotnet-build-and-test.yml#L178-L193

Normally when we implement features we run only the integration tests that are related to the projects we modified or the new Integration that were added as part of the changes.

The full integration run normally we let the pipeline ensuring all pass after build and UT are done and pushed as part of the feature branch.

[jakesduagile]

Thanks you @rogerbarreto