Conversation
There was a problem hiding this comment.
Pull request overview
This PR updates the nightly MCP server stress test configuration by replacing 5 servers with 6 new ones and updating a container reference. The changes affect both the workflow definition file and its compiled lock file.
Changes:
- Updated workflow description to be more concise
- Removed 5 MCP servers: postgres, aws-kb-retrieval, linear, raygun, and axiom
- Added 6 new MCP servers: playwright, gitlab, notion, youtube-transcript, atlassian, and kubernetes
- Updated sequential-thinking container name from "mcp/sequential-thinking" to "mcp/sequentialthinking"
Reviewed changes
Copilot reviewed 2 out of 2 changed files in this pull request and generated 10 comments.
| File | Description |
|---|---|
| .github/workflows/nightly-mcp-stress-test.md | Updates MCP server list, removes 5 servers and adds 6 new ones, modifying the workflow description and container references |
| .github/workflows/nightly-mcp-stress-test.lock.yml | Reflects the changes from the .md file in the compiled lock file, including updated Docker image downloads, server configurations, and workflow metadata |
Comments suppressed due to low confidence (1)
.github/workflows/nightly-mcp-stress-test.lock.yml:1042
- The description states "Load 20 MCP servers" but the configuration actually includes 21 servers. The description should be updated to reflect the actual count of 21 servers, matching the configuration.
WORKFLOW_DESCRIPTION: "Load 20 MCP servers, discover and summarize the tools exported by each server, test tool invocations, and post a comprehensive report as a GitHub issue"
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
| }, | ||
| "gitlab": { | ||
| "type": "stdio", | ||
| "container": "mcp/wikipedia-mcp", |
There was a problem hiding this comment.
The server name "gitlab" is mapped to container "mcp/wikipedia-mcp". This appears to be a mismatch - the server name suggests GitLab functionality but the container is for Wikipedia. Either the server name should be changed to "wikipedia" or the container should be changed to "mcp/gitlab".
| "container": "mcp/wikipedia-mcp", | |
| "container": "mcp/gitlab", |
| "atlassian": { | ||
| "type": "stdio", | ||
| "container": "mcp/aws-kb-retrieval", | ||
| "tools": [ | ||
| "*" | ||
| ] | ||
| }, | ||
| "axiom": { | ||
| "type": "stdio", | ||
| "container": "mcp/axiom", | ||
| "container": "mcp/hackernews-mcp", | ||
| "tools": [ | ||
| "*" | ||
| ] |
There was a problem hiding this comment.
The server name "atlassian" is mapped to container "mcp/hackernews-mcp". This appears to be a mismatch - the server name suggests Atlassian functionality but the container is for Hacker News. Either the server name should be changed to "hackernews" or the container should be changed to "mcp/atlassian".
| axiom: | ||
| playwright: | ||
| type: stdio | ||
| container: "mcp/playwright" |
There was a problem hiding this comment.
The container reference in the .md file is "mcp/playwright" but in the .lock.yml file it's "mcr.microsoft.com/playwright/mcp" with additional configuration (args, entrypointArgs, mounts). This inconsistency between the source file and the compiled lock file suggests that either the .md file should specify the full Microsoft container reference or the .lock.yml file should use the simpler "mcp/playwright" reference. The discrepancy will cause confusion and may indicate the files are out of sync.
| container: "mcp/playwright" | |
| container: "mcr.microsoft.com/playwright/mcp" |
| await determineAutomaticLockdown(github, context, core); | ||
| - name: Download container images | ||
| run: bash /opt/gh-aw/actions/download_docker_images.sh ghcr.io/github/gh-aw-mcpg:v0.0.98 ghcr.io/github/github-mcp-server:v0.30.2 ghcr.io/github/github-mcp-server:v0.30.3 mcp/aws-kb-retrieval mcp/axiom mcp/brave-search mcp/everart mcp/fetch mcp/filesystem mcp/gdrive mcp/git mcp/google-maps mcp/linear mcp/memory mcp/postgres mcp/puppeteer mcp/raygun mcp/sentry mcp/sequential-thinking mcp/slack mcp/sqlite mcp/time node:lts-alpine | ||
| run: bash /opt/gh-aw/actions/download_docker_images.sh ghcr.io/github/gh-aw-mcpg:v0.0.98 ghcr.io/github/github-mcp-server:v0.30.2 ghcr.io/github/github-mcp-server:v0.30.3 mcp/brave-search mcp/duckduckgo mcp/everart mcp/fetch mcp/filesystem mcp/gdrive mcp/git mcp/google-maps mcp/hackernews-mcp mcp/kubernetes mcp/memory mcp/playwright mcp/puppeteer mcp/sentry mcp/sequentialthinking mcp/slack mcp/sqlite mcp/time mcp/wikipedia-mcp mcp/youtube-transcript mcr.microsoft.com/playwright/mcp node:lts-alpine |
There was a problem hiding this comment.
The Docker images download command includes both "mcp/playwright" and "mcr.microsoft.com/playwright/mcp". Since the playwright server configuration uses "mcr.microsoft.com/playwright/mcp", the "mcp/playwright" image appears to be unnecessary and should be removed from the download list to avoid downloading unused images.
| run: bash /opt/gh-aw/actions/download_docker_images.sh ghcr.io/github/gh-aw-mcpg:v0.0.98 ghcr.io/github/github-mcp-server:v0.30.2 ghcr.io/github/github-mcp-server:v0.30.3 mcp/brave-search mcp/duckduckgo mcp/everart mcp/fetch mcp/filesystem mcp/gdrive mcp/git mcp/google-maps mcp/hackernews-mcp mcp/kubernetes mcp/memory mcp/playwright mcp/puppeteer mcp/sentry mcp/sequentialthinking mcp/slack mcp/sqlite mcp/time mcp/wikipedia-mcp mcp/youtube-transcript mcr.microsoft.com/playwright/mcp node:lts-alpine | |
| run: bash /opt/gh-aw/actions/download_docker_images.sh ghcr.io/github/gh-aw-mcpg:v0.0.98 ghcr.io/github/github-mcp-server:v0.30.2 ghcr.io/github/github-mcp-server:v0.30.3 mcp/brave-search mcp/duckduckgo mcp/everart mcp/fetch mcp/filesystem mcp/gdrive mcp/git mcp/google-maps mcp/hackernews-mcp mcp/kubernetes mcp/memory mcp/puppeteer mcp/sentry mcp/sequentialthinking mcp/slack mcp/sqlite mcp/time mcp/wikipedia-mcp mcp/youtube-transcript mcr.microsoft.com/playwright/mcp node:lts-alpine |
| --- | ||
| name: Nightly MCP Server Stress Test | ||
| description: Comprehensive stress test that loads 20 well-known MCP servers, tests their tools, and reports results with automated issue creation for failures | ||
| description: Load 20 MCP servers, discover and summarize the tools exported by each server, test tool invocations, and post a comprehensive report as a GitHub issue |
There was a problem hiding this comment.
The description states "Load 20 MCP servers" but the configuration actually includes 21 servers (github, sqlite, brave-search, fetch, filesystem, memory, slack, puppeteer, gdrive, google-maps, everart, sequential-thinking, sentry, git, time, playwright, gitlab, notion, youtube-transcript, atlassian, kubernetes). The description should be updated to reflect the actual count of 21 servers.
| "notion": { | ||
| "type": "stdio", | ||
| "container": "mcp/postgres", | ||
| "container": "mcp/duckduckgo", | ||
| "tools": [ | ||
| "*" | ||
| ] | ||
| }, |
There was a problem hiding this comment.
The server name "notion" is mapped to container "mcp/duckduckgo". This appears to be a mismatch - the server name suggests Notion functionality but the container is for DuckDuckGo. Either the server name should be changed to "duckduckgo" or the container should be changed to "mcp/notion".
| # For more information: https://github.com/github/gh-aw/blob/main/.github/aw/github-agentic-workflows.md | ||
| # | ||
| # Comprehensive stress test that loads 20 well-known MCP servers, tests their tools, and reports results with automated issue creation for failures | ||
| # Load 20 MCP servers, discover and summarize the tools exported by each server, test tool invocations, and post a comprehensive report as a GitHub issue |
There was a problem hiding this comment.
The description states "Load 20 MCP servers" but the configuration actually includes 21 servers. The description should be updated to reflect the actual count of 21 servers, matching the configuration.
| container: "mcp/playwright" | ||
| gitlab: | ||
| type: stdio | ||
| container: "mcp/wikipedia-mcp" |
There was a problem hiding this comment.
The server name "gitlab" is mapped to container "mcp/wikipedia-mcp". This appears to be a mismatch - the server name suggests GitLab functionality but the container is for Wikipedia. Either the server name should be changed to "wikipedia" or the container should be changed to "mcp/gitlab".
| container: "mcp/wikipedia-mcp" | |
| container: "mcp/gitlab" |
| container: "mcp/wikipedia-mcp" | ||
| notion: | ||
| type: stdio | ||
| container: "mcp/duckduckgo" |
There was a problem hiding this comment.
The server name "notion" is mapped to container "mcp/duckduckgo". This appears to be a mismatch - the server name suggests Notion functionality but the container is for DuckDuckGo. Either the server name should be changed to "duckduckgo" or the container should be changed to "mcp/notion".
| container: "mcp/duckduckgo" | |
| container: "mcp/notion" |
| container: "mcp/youtube-transcript" | ||
| atlassian: | ||
| type: stdio | ||
| container: "mcp/hackernews-mcp" |
There was a problem hiding this comment.
The server name "atlassian" is mapped to container "mcp/hackernews-mcp". This appears to be a mismatch - the server name suggests Atlassian functionality but the container is for Hacker News. Either the server name should be changed to "hackernews" or the container should be changed to "mcp/atlassian".
| container: "mcp/hackernews-mcp" | |
| container: "mcp/atlassian" |
1 participant
No description provided.