Skip to content

Conversation

@JasonYeMSFT
Copy link
Member

What does this PR do?

This PR fixes the tool eval related scripts to account for the following changes:

  • commands no longer have 'azmcp_' prefix
  • Some tools have been renamed

GitHub issue number?

N/A

Pre-merge Checklist

  • Required for All PRs
    • Read contribution guidelines
    • PR title clearly describes the change
    • Commit history is clean with descriptive messages (cleanup guide)
    • Added comprehensive tests for new/modified functionality
    • Updated servers/Azure.Mcp.Server/CHANGELOG.md and/or servers/Fabric.Mcp.Server/CHANGELOG.md for product changes (features, bug fixes, UI/UX, updated dependencies)
  • For MCP tool changes:
    • One tool per PR: This PR adds or modifies only one MCP tool for faster review cycles
    • Updated servers/Azure.Mcp.Server/README.md and/or servers/Fabric.Mcp.Server/README.md documentation
    • Validate README.md changes using script at eng/scripts/Process-PackageReadMe.ps1. See Package README
    • Updated command list in /servers/Azure.Mcp.Server/docs/azmcp-commands.md and/or /docs/fabric-commands.md
    • Run .\eng\scripts\Update-AzCommandsMetadata.ps1 to update tool metadata in azmcp-commands.md (required for CI)
    • For new or modified tool descriptions, ran ToolDescriptionEvaluator and obtained a score of 0.4 or more and a top 3 ranking for all related test prompts
    • For new tools associated with Azure services or publicly available tools/APIs/products, add URL to documentation in the PR description
  • Extra steps for Azure MCP Server tool changes:
    • Updated test prompts in /servers/Azure.Mcp.Server/docs/e2eTestPrompts.md
    • 👉 For Community (non-Microsoft team member) PRs:
      • Security review: Reviewed code for security vulnerabilities, malicious code, or suspicious activities before running tests (crypto mining, spam, data exfiltration, etc.)
      • Manual tests run: added comment /azp run mcp - pullrequest - live to run Live Test Pipeline

Copilot AI review requested due to automatic review settings November 3, 2025 19:40
@JasonYeMSFT JasonYeMSFT requested review from a team as code owners November 3, 2025 19:40
@github-project-automation github-project-automation bot moved this to Untriaged in Azure MCP Server Nov 3, 2025
Copy link
Contributor

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull Request Overview

This PR refactors prompt handling in the tool description evaluator to simplify naming conventions and improve consistency. The changes remove the azmcp_ prefix requirement from prompt keys and consolidate tooling extensions under a unified namespace.

Key Changes

  • Removed the azmcp_ prefix filtering requirement in prompt parsing logic
  • Consolidated multiple CLI extension tools (extension_cli_generate, extension_cli_install, extension_azqr) under a single extension namespace
  • Updated Azure AI Foundry prompts to consistently reference "Azure AI Foundry resource" instead of "AI foundry project"
  • Removed angle brackets from a test prompt in the e2e test documentation

Reviewed Changes

Copilot reviewed 7 out of 7 changed files in this pull request and generated 3 comments.

Show a summary per file
File Description
servers/Azure.Mcp.Server/docs/e2eTestPrompts.md Removed angle brackets from extension_cli_install test prompt
eng/tools/ToolDescriptionEvaluator/prompts.json Alphabetically reorganized tool prompts and updated Azure AI Foundry terminology
eng/tools/ToolDescriptionEvaluator/namespace-tools.json Consolidated CLI extensions into unified "extension" namespace
eng/tools/ToolDescriptionEvaluator/namespace-prompts.json Updated prompts to reflect consolidated extension namespace and consistent Azure AI Foundry terminology
eng/tools/ToolDescriptionEvaluator/consolidated-prompts.json Updated consolidated prompts with consistent terminology and unified extension handling
eng/tools/ToolDescriptionEvaluator/Generate-PromptsJsonFromMarkdown.ps1 Removed azmcp_ prefix filtering logic from prompt parsing
eng/tools/ToolDescriptionEvaluator/Generate-GroupedPromptsJson.ps1 Simplified namespace prompt key generation logic and removed special case for azqr

@JasonYeMSFT
Copy link
Member Author

@vcolin7 @fanyang-mono Please help review the changes.

@JasonYeMSFT
Copy link
Member Author

Two high level comments:

  1. The tool name mismatches due to changing tool names could have been fixed by re-running
  • Generate-PromptsJsonFromMarkdown
  • Generate-GroupedPromptsJson

Should we start checking them in CI and make sure tool authors keep them up-to-date?

  1. namespace-tools.json seems to be manually authored. I didn't find a script to create it. How should it be generated?

@fanyang-mono
Copy link
Contributor

It is nice to rerun the tests and check in the new test results, since you've updated the testing artifacts.

@joshfree joshfree moved this from Untriaged to In Progress in Azure MCP Server Nov 4, 2025
@joshfree joshfree added this to the 2025-11 milestone Nov 4, 2025
@hallipr
Copy link
Member

hallipr commented Nov 5, 2025

This tool should support all servers in the repo. Is there an existing issue to remove references to Azure.Mcp.Server?

@JasonYeMSFT JasonYeMSFT merged commit 5de8f21 into microsoft:main Nov 5, 2025
24 checks passed
@github-project-automation github-project-automation bot moved this from In Progress to Done in Azure MCP Server Nov 5, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

Status: Done

Development

Successfully merging this pull request may close these issues.

5 participants