-
Notifications
You must be signed in to change notification settings - Fork 275
Fix tool eval scripts and update persisted tool hierarchy json data #1035
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Fix tool eval scripts and update persisted tool hierarchy json data #1035
Conversation
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Pull Request Overview
This PR refactors prompt handling in the tool description evaluator to simplify naming conventions and improve consistency. The changes remove the azmcp_ prefix requirement from prompt keys and consolidate tooling extensions under a unified namespace.
Key Changes
- Removed the
azmcp_prefix filtering requirement in prompt parsing logic - Consolidated multiple CLI extension tools (
extension_cli_generate,extension_cli_install,extension_azqr) under a singleextensionnamespace - Updated Azure AI Foundry prompts to consistently reference "Azure AI Foundry resource" instead of "AI foundry project"
- Removed angle brackets from a test prompt in the e2e test documentation
Reviewed Changes
Copilot reviewed 7 out of 7 changed files in this pull request and generated 3 comments.
Show a summary per file
| File | Description |
|---|---|
| servers/Azure.Mcp.Server/docs/e2eTestPrompts.md | Removed angle brackets from extension_cli_install test prompt |
| eng/tools/ToolDescriptionEvaluator/prompts.json | Alphabetically reorganized tool prompts and updated Azure AI Foundry terminology |
| eng/tools/ToolDescriptionEvaluator/namespace-tools.json | Consolidated CLI extensions into unified "extension" namespace |
| eng/tools/ToolDescriptionEvaluator/namespace-prompts.json | Updated prompts to reflect consolidated extension namespace and consistent Azure AI Foundry terminology |
| eng/tools/ToolDescriptionEvaluator/consolidated-prompts.json | Updated consolidated prompts with consistent terminology and unified extension handling |
| eng/tools/ToolDescriptionEvaluator/Generate-PromptsJsonFromMarkdown.ps1 | Removed azmcp_ prefix filtering logic from prompt parsing |
| eng/tools/ToolDescriptionEvaluator/Generate-GroupedPromptsJson.ps1 | Simplified namespace prompt key generation logic and removed special case for azqr |
|
@vcolin7 @fanyang-mono Please help review the changes. |
|
Two high level comments:
Should we start checking them in CI and make sure tool authors keep them up-to-date?
|
|
It is nice to rerun the tests and check in the new test results, since you've updated the testing artifacts. |
|
This tool should support all servers in the repo. Is there an existing issue to remove references to Azure.Mcp.Server? |
What does this PR do?
This PR fixes the tool eval related scripts to account for the following changes:
GitHub issue number?
N/A
Pre-merge Checklist
servers/Azure.Mcp.Server/CHANGELOG.mdand/orservers/Fabric.Mcp.Server/CHANGELOG.mdfor product changes (features, bug fixes, UI/UX, updated dependencies)servers/Azure.Mcp.Server/README.mdand/orservers/Fabric.Mcp.Server/README.mddocumentationeng/scripts/Process-PackageReadMe.ps1. See Package README/servers/Azure.Mcp.Server/docs/azmcp-commands.mdand/or/docs/fabric-commands.md.\eng\scripts\Update-AzCommandsMetadata.ps1to update tool metadata in azmcp-commands.md (required for CI)ToolDescriptionEvaluatorand obtained a score of0.4or more and a top 3 ranking for all related test prompts/servers/Azure.Mcp.Server/docs/e2eTestPrompts.mdcrypto mining, spam, data exfiltration, etc.)/azp run mcp - pullrequest - liveto run Live Test Pipeline