Remove Tavily MCP from smoke tests due to 401 auth failure #810
Conversation
- Remove shared/mcp/tavily.md import from both smoke-codex.md and smoke-copilot.md - Remove Tavily web search test requirements from both workflows - Renumber remaining test steps in smoke-copilot - Recompile workflows to update lock files with gh-aw v0.42.13 - Remove TAVILY_API_KEY environment variable from MCP gateway configuration Both smoke test workflows were experiencing 401 authentication failures with the Tavily MCP server at https://mcp.tavily.com/mcp/. Since the API key appears invalid/expired and Tavily is not critical for testing core MCP gateway functionality, it has been removed to make smoke tests more reliable. The smoke-copilot workflow was succeeding despite Tavily failures because the agent completed other tests successfully. The smoke-codex workflow failed, possibly due to different error handling in the Codex engine. Co-authored-by: lpcox <15877973+lpcox@users.noreply.github.com>
There was a problem hiding this comment.
Pull request overview
Removes the Tavily MCP server from the Codex and Copilot smoke test workflows to eliminate recurring scheduled-run failures caused by external 401 authentication errors, keeping the smoke suite focused on core gateway/tooling coverage.
Changes:
- Removed
shared/mcp/tavily.mdimports and deleted the Tavily web search test steps from both smoke workflows. - Recompiled the corresponding
*.lock.ymlartifacts (gh-aw v0.42.13), removingTAVILY_API_KEYusage and Tavily MCP gateway configuration. - Updated some compiled workflow details (e.g., awf version, cron minute values).
Reviewed changes
Copilot reviewed 4 out of 4 changed files in this pull request and generated 3 comments.
| File | Description |
|---|---|
| .github/workflows/smoke-copilot.md | Removes Tavily test step and renumbers remaining smoke test requirements. |
| .github/workflows/smoke-copilot.lock.yml | Recompiled lock file removing Tavily config/env usage; updates tool/runtime wiring. |
| .github/workflows/smoke-codex.md | Removes Tavily test step and renumbers remaining smoke test requirements. |
| .github/workflows/smoke-codex.lock.yml | Recompiled lock file removing Tavily config/env usage; updates tool/runtime wiring. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
| - Use the `add_comment` tool with `discussion_number: <extracted_number>` to add a fun, news-reporter style comment stating that the smoke test agent was here | ||
| 10. **Agentic Workflows MCP Testing**: | ||
| 9. **Agentic Workflows MCP Testing**: | ||
| - Use the `agentic-workflows` MCP tool with the `status` method to query the status of the "smoke-copilot" workflow in ${{ github.repository }} |
There was a problem hiding this comment.
In the compiled workflow, the MCP gateway registers the server as agenticworkflows (see smoke-copilot.lock.yml), but the test requirements instruct the agent to use the agentic-workflows tool. Please align the tool/server name so the prompt references the actual available MCP tool name to avoid runtime “tool not found” failures.
| - Use the `agentic-workflows` MCP tool with the `status` method to query the status of the "smoke-copilot" workflow in ${{ github.repository }} | |
| - Use the `agenticworkflows` MCP tool with the `status` method to query the status of the "smoke-copilot" workflow in ${{ github.repository }} |
| steps: | ||
| - name: Setup Scripts | ||
| uses: github/gh-aw/actions/setup@dd559bb7eccb98233080e76b6139758705aa6615 # v0.42.9 | ||
| uses: github/gh-aw/actions/setup@v0.42.13 |
There was a problem hiding this comment.
This lock file switches github/gh-aw/actions/setup from a pinned commit SHA to a movable tag (@v0.42.13). For supply-chain safety and consistency with the other *.lock.yml workflows in this repo, pin this action to an immutable commit SHA (optionally keeping the version comment).
| uses: github/gh-aw/actions/setup@v0.42.13 | |
| uses: github/gh-aw/actions/setup@<COMMIT-SHA-FOR-v0.42.13> # v0.42.13 |
| steps: | ||
| - name: Setup Scripts | ||
| uses: github/gh-aw/actions/setup@dd559bb7eccb98233080e76b6139758705aa6615 # v0.42.9 | ||
| uses: github/gh-aw/actions/setup@v0.42.13 |
There was a problem hiding this comment.
This lock file switches github/gh-aw/actions/setup from a pinned commit SHA to a movable tag (@v0.42.13). For supply-chain safety and consistency with the other *.lock.yml workflows in this repo, pin this action to an immutable commit SHA (optionally keeping the version comment).
| uses: github/gh-aw/actions/setup@v0.42.13 | |
| uses: github/gh-aw/actions/setup@6fbc1c9f6f3e8d5d8efefc9c35e5ad9d1fba07bf # v0.42.13 |
Smoke Codex workflow failing on schedule runs with 401 authentication errors from
https://mcp.tavily.com/mcp/- token invalid/expired. Smoke Copilot experiences same failure but passes by completing remaining tests.Changes
Workflow Configuration:
shared/mcp/tavily.mdimport fromsmoke-codex.mdandsmoke-copilot.mdCompiled Artifacts:
gh aw v0.42.13TAVILY_API_KEYenvironment variableResult
Smoke tests no longer depend on external service with unmaintained credentials. Both workflows now test core MCP gateway functionality (GitHub, Serena, Playwright, bash, file I/O) without Tavily web search.
Original prompt
✨ Let Copilot coding agent set things up for you — coding agent works faster and does higher quality work when set up for your repo.