🔍 Agentic Workflow Audit Report - 2026-01-16

[github-actions[bot]]

Audit Summary

Period: Last 24 hours (2026-01-15 to 2026-01-16)

Metric	Value
Runs Analyzed	10
Workflows Active	8
Success Rate	80.0% (8/10)
Issues Found	4 (2 errors + 2 missing tools)
Total Tokens	4,816,351
Total Cost	$0.89
Average Turns	51 turns (across 1 Claude run)

---

⚠️ Critical Findings

1. MCP GitHub Server Schema Error (HIGH SEVERITY)

Workflow Affected: AI Moderator (run §21053727950)

Error Pattern:

Model call failed: Invalid schema for function 'github-get_commit': 
In context=(), object schema missing properties.

Impact:

• Workflow failed after 5 retry attempts
• Execution time: 155 seconds (2m 35s)
• Agent could not complete moderation task

Root Cause: The MCP GitHub server is returning an invalid schema for the github-get_commit tool, causing the AI model to reject the function definition.

Recommendation:

1. Review MCP GitHub server schema definitions
2. Ensure all required properties are defined in the github-get_commit function schema
3. Add schema validation tests to prevent future regressions

---

2. Changeset Generator Failure

Workflow Affected: Changeset Generator (run §21053692938)

Details:

• Event: pull_request on branch copilot/mount-fs-in-agent-container
• Duration: 44 seconds before failure
• Agent: codex
• No detailed error logs available in agent-stdio.log

Impact: PR changeset generation failed, no changeset summary produced

Recommendation:

1. Investigate why agent-stdio.log was not generated
2. Add better error logging for codex agent failures
3. Review agent container configuration for the PR event trigger

---

🔧 Missing Tools

1. GitHub Code Scanning Alerts Tool

Tool: github-list_code_scanning_alerts
Count: 1 occurrence
Workflow: Code Scanning Fixer (run §21054612970)

Issue: Parameter validation error - "missing required parameter: owner"

Context: The workflow successfully completed despite this error by using the missing_tool safe output. However, the tool should be available for future use.

Recommendation:

1. Add github-list_code_scanning_alerts to the MCP GitHub server
2. Ensure proper parameter mapping (owner, repo) from workflow context
3. Update workflow to handle this tool when available

---

2. Safe-Input GitHub Query Tools

Tool: Safe-input tools for GitHub queries (github-pr-query/issue/discussion)
Count: 1 occurrence
Workflow: Daily Project Performance Summary Generator (run §21054147873)

Issue: GitHub query tools not available in safe-input mode

Context: The workflow needs to gather PR/issue/discussion data but safe-input mode doesn't expose these query tools.

Recommendation:

1. Consider adding read-only GitHub query tools to safe-inputs
2. Alternative: Use pre-fetched data from cache-memory
3. Document this limitation in workflow README

---

📊 Performance Metrics

Token Usage by Workflow

Workflow	Runs	Tokens	Cost	Avg Tokens/Run
Step Name Alignment	1	1,510,193	$0.89	1,510,193
Code Scanning Fixer	3	1,270,999	$0.00	423,666
Workflow Health Manager	1	897,253	$0.00	897,253
Daily Malicious Code Scan	1	668,768	$0.00	668,768
Daily Project Performance	1	280,764	$0.00	280,764
Issue Monster	1	188,374	$0.00	188,374

Highest Cost Workflow: Step Name Alignment ($0.89) - This workflow used the Claude agent and performed 50 turns, explaining the higher cost.

Agent Distribution

Agent	Runs	Tokens	% of Total
copilot	7	3,025,394	62.8%
claude	1	1,510,193	31.4%
codex	2	280,764	5.8%

---

🌐 Firewall Analysis

Total Network Requests: 167
Allowed Requests: 167
Denied Requests: 0

All network requests were allowed. The firewall is operating in monitoring mode without blocking any domains.

---

✅ Successful Workflows

The following workflows completed successfully:

1. Code Scanning Fixer (3 runs) - 100% success rate
2. Daily Malicious Code Scan Agent (1 run)
3. Issue Monster (1 run)
4. Step Name Alignment (1 run)
5. Daily Project Performance Summary Generator (1 run)
6. Workflow Health Manager - Meta-Orchestrator (1 run)

---

📈 Historical Context

This is the baseline audit for the new audit system. Future audits will include:

• Trend analysis comparing with previous periods
• Pattern recognition for recurring issues
• Success rate trends over time

Audit data stored in: /tmp/gh-aw/repo-memory/default/audits/2026-01-16.json

---

🎯 Recommendations

Immediate Actions (High Priority)

1. Fix MCP GitHub Server Schema - Address the github-get_commit schema error to prevent AI Moderator failures
2. Investigate Changeset Generator - Determine why the workflow failed without detailed logs
3. Add Missing Tool - Implement github-list_code_scanning_alerts in MCP GitHub server

Short-term Improvements (Medium Priority)

4. Enhance Error Logging - Ensure all agent failures produce detailed logs
5. Safe-Input Tools - Evaluate adding read-only GitHub query tools to safe-inputs
6. Schema Validation - Add automated tests for MCP server schema validity

Long-term Enhancements (Low Priority)

7. Monitoring Dashboard - Create visualization for audit metrics and trends
8. Automated Alerts - Set up notifications for critical failures
9. Cost Optimization - Review high-token workflows for optimization opportunities

---

📋 Next Steps

• Create issue for MCP GitHub server schema error
• Review Changeset Generator logs in detail
• Document missing tools in workflow READMEs
• Schedule follow-up audit in 24 hours
• Track success rate trends over next week

---

📁 Audit Artifacts

• Audit Data: /tmp/gh-aw/repo-memory/default/audits/2026-01-16.json
• Error Patterns: /tmp/gh-aw/repo-memory/default/patterns/errors.json
• Missing Tools: /tmp/gh-aw/repo-memory/default/patterns/missing-tools.json
• Logs Location: /tmp/gh-aw/aw-mcp/logs/

---

Audit completed: 2026-01-16T04:02:00Z
Next audit scheduled: 2026-01-17T04:00:00Z (daily at 04:00 UTC)

AI generated by Agentic Workflow Audit Agent