Optimize workflows based on live log analysis: MCP pagination and JSON parsing

[Copilot]

Live analysis of 11 workflow runs identified two actionable failure patterns: MCP response size limit exceeded (75k+ tokens from list_pull_requests) and JSON parsing errors from Unicode symbols in npm output.

Changes

Created shared pagination guidance

• .github/workflows/shared/mcp-pagination.md: Documents 25k token limit, pagination patterns for GitHub API tools
• Provides perPage examples for PRs, issues, search operations
• Lists error signatures to watch for

Updated smoke-claude workflow

• Import pagination guidance: imports: [shared/mcp-pagination.md]
• Explicit toolsets: toolsets: [repos, pull_requests] (was wildcard)
• Bounded iterations: max-turns: 15

Enhanced cli-version-checker

• Added "Common JSON Parsing Issues" section
• Stderr filtering pattern: npm view package --json 2>/dev/null | jq -r '.version'
• Documents Unicode symbol breakage (ℹ, ⚠, ✓) with grep/jq workarounds

Expected impact

• 80%+ reduction in MCP response size errors through proactive pagination
• Eliminate JSON parsing failures from CLI tools
• ~100k-200k tokens/month saved on retry turns

Original prompt

---

This section details on the original issue you should resolve

<issue_title>[q] Workflow optimizations based on live log analysis</issue_title>
<issue_description># Q Workflow Optimization Report

Issues Found (from live data)

Analysis Overview

• Analyzed: 11 workflow runs from the last 7 days
• Total Duration: 49.1 minutes
• Total Tokens: 884,087
• Total Cost: $0.51
• Total Errors: 57
• Total Warnings: 43
• Missing Tools: 0 (✓ all tools properly configured)

Issue #1: MCP Response Size Limits (Claude Engine)

Severity: Medium
Affected Workflows: Smoke Claude
Run IDs Analyzed: 19103049716, 19114358281

Log Evidence:

• MCP tool "list_pull_requests" response (75897 tokens) exceeds maximum allowed tokens (25000)
• MCP tool "pull_request_read" response (31675 tokens) exceeds maximum allowed tokens (25000)

Root Cause: Large result sets from GitHub API calls exceed the 25k token limit for MCP tool responses, forcing workflows to use pagination reactively rather than proactively.

Impact: Workflows must retry with pagination parameters, adding extra turns and API calls.

Issue #2: JSON Parsing Errors (Copilot Engine)

Severity: Low-Medium
Affected Workflows: CLI Version Checker, Q
Run IDs Analyzed: 19106334374, 19102935260

Log Evidence:

• Unexpected token 'W', "When fetch"... is not valid JSON
• Unexpected token 'ℹ', "ℹ Timeout "... is not valid JSON
• Unexpected token '⚠', "⚠ pip pack"... is not valid JSON

Root Cause: Tool outputs (npm, bash commands) include informational messages with Unicode symbols (ℹ, ⚠, ✓) before or after JSON content.

Impact: Parsing failures require additional error handling and retry logic.

Issue #3: Network Unreachable (IPv6)

Severity: Low (Infrastructure)
Affected Workflows: Changeset Generator, Smoke Copilot
Run IDs: 19114358285, 19114358286

Log Evidence:

• sending to ICMPv6 packet to [2606:4700::6810:1d22]: (101) Network is unreachable

Note: Infrastructure-related, not addressable via workflow configuration.

Changes Made

1. Created Shared MCP Pagination Guidance (.github/workflows/shared/mcp-pagination.md)

Purpose: Provide reusable best practices for avoiding MCP response size errors

Content:

• Documents 25k token limit for MCP tool responses
• Provides pagination examples for common GitHub API calls
• Explains how to use perPage parameter effectively
• Includes error message patterns to watch for
• Shows how to fetch large PRs incrementally

Benefits:

• Proactive pagination reduces wasted turns
• Consistent guidance across all workflows
• Reusable import for any workflow using GitHub API

2. Updated Smoke Claude (.github/workflows/smoke-claude.md)

Changes:

• Added imports: [shared/mcp-pagination.md] for pagination guidance
• Configured explicit toolsets: [repos, pull_requests] instead of wildcard
• Set max-turns: 15 to prevent excessive iterations
• Simplified GitHub tools configuration

Expected Improvements:

• Reduce MCP response size errors by 80%+ (based on proactive pagination)
• Clearer tool access scope with explicit toolsets
• Bounded iteration count prevents runaway workflows

3. Enhanced CLI Version Checker (.github/workflows/cli-version-checker.md)

Changes:

• Added "Common JSON Parsing Issues" section with:
  • Guidance on handling Unicode symbols in tool output
  • Examples using grep -v to filter informational messages
  • Alternative approaches with jq for reliable JSON extraction
  • Recommendation to filter stderr: npm view ... 2>/dev/null

Expected Improvements:

• Eliminate JSON parsing errors when using npm commands
• More robust error handling for tool outputs
• Clear documentation for future workflow authors

Validation

All modified workflows were validated:

• ✅ smoke-claude.md - Modified with pagination guidance and toolset configuration
• ✅ cli-version-checker.md - Enhanced with JSON parsing guidance
• ✅ mcp-pagination.md - Created as shared resource

Note: .lock.yml files will be generated automatically after merge via standard compilation process.

Expected Impact

Performance Improvements

• Reduced turns: Proactive pagination eliminates ~2-3 retry turns per Claude workflow
• Token savings: ~100k-200k tokens saved per month on smoke tests (fewer retries)
• Cost reduction: ~$0.10-$0.20 per month (small but measurable)

Reliability Improvements

• Fewer errors: JSON parsing guidance reduces CLI tool errors
• Better DX: Clear documentation helps future workflow authors
• Reusable patterns: Shared mcp-pagination.md can be imported by any workflow

References

Log Analysis Files

• Run summaries: /tmp/gh-aw/aw-mcp/logs/run-*/run_summary.json
• Error detection: /tmp/gh-aw/aw-mcp/logs/run-*/detection.log

Specific Runs Investigated

1. 19114358281 ...

• Fixes [q] Workflow optimizations based on live log analysis #3292

---

💡 You can make Copilot smarter by setting up custom instructions, customizing its development environment and configuring Model Context Protocol (MCP) servers. Learn more Copilot coding agent tips in the docs.

[Copilot]

Pull Request Overview

This PR enhances the smoke-claude workflow with pagination guidance and improved configuration. The changes add comprehensive documentation about MCP response size limits and best practices for handling large GitHub API responses, along with workflow configuration improvements.

Key Changes

• Added MCP pagination guidance documentation to help workflows handle the 25,000 token response limit
• Configured the Claude engine with explicit max-turns: 15 limit
• Updated GitHub toolsets from default to specific repos, pull_requests toolsets
• Added JSON parsing troubleshooting documentation for CLI tools

Reviewed Changes

Copilot reviewed 5 out of 5 changed files in this pull request and generated no comments.

Show a summary per file

File	Description
.github/workflows/smoke-claude.md	Updated workflow to use engine object notation with max-turns, added pagination import, and specified GitHub toolsets
.github/workflows/smoke-claude.lock.yml	Compiled workflow output reflecting the pagination guidance injection, max-turns configuration, and toolset changes
.github/workflows/shared/mcp-pagination.md	New shared documentation providing comprehensive pagination best practices for MCP tool responses
.github/workflows/cli-version-checker.md	Added troubleshooting section for JSON parsing issues with npm and other CLI tools
.github/workflows/cli-version-checker.lock.yml	Compiled workflow output including the new JSON parsing troubleshooting guidance

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.