Create Shared Metrics Collection Infrastructure for Agent Performance

[github-actions]

Problem

The agent ecosystem currently lacks centralized performance metrics collection and storage. This prevents:

• Historical trend analysis
• Performance benchmarking
• Early detection of degrading agents
• Data-driven optimization decisions
• Objective performance comparisons

Meta-orchestrators (Agent Performance Analyzer, Campaign Manager, Workflow Health Manager) need shared metrics to coordinate effectively and make strategic decisions.

Solution

Create a shared metrics collection infrastructure using repo-memory for persistence.

Architecture

1. Metrics Schema (JSON format in repo-memory)

{
  "timestamp": "2024-12-24T12:00:00Z",
  "period": "daily",
  "workflows": {
    "workflow-name": {
      "safe_outputs": {
        "issues_created": 5,
        "prs_created": 2,
        "comments_added": 10,
        "discussions_created": 1
      },
      "workflow_runs": {
        "total": 7,
        "successful": 6,
        "failed": 1,
        "success_rate": 0.857
      },
      "engagement": {
        "issue_reactions": 12,
        "pr_comments": 8,
        "discussion_replies": 3
      },
      "quality_indicators": {
        "pr_merge_rate": 0.75,
        "avg_issue_close_time_hours": 48,
        "avg_pr_merge_time_hours": 72
      }
    }
  },
  "ecosystem": {
    "total_workflows": 120,
    "active_workflows": 85,
    "total_safe_outputs": 45,
    "overall_success_rate": 0.892
  }
}

2. Storage Location

/tmp/gh-aw/repo-memory-default/memory/meta-orchestrators/
├── metrics/
│   ├── daily/
│   │   ├── 2024-12-24.json
│   │   ├── 2024-12-25.json
│   │   └── ...
│   ├── weekly/
│   │   ├── 2024-W51.json
│   │   └── ...
│   └── latest.json (symlink or copy of most recent)
└── trends/
    ├── workflow-success-rates.json
    └── safe-output-volume.json

3. Collection Workflow

Create new workflow: .github/workflows/metrics-collector.md

---
description: Collects daily performance metrics for agent ecosystem
on:
  schedule:
    - cron: "0 0 * * *"  # Daily at midnight UTC
  workflow_dispatch:
permissions:
  contents: read
  issues: read
  pull-requests: read
  discussions: read
  actions: read
engine: copilot
tools:
  github:
    mode: remote
    toolsets: [default, actions]
  repo-memory:
    branch-name: memory/meta-orchestrators
    file-glob: "metrics/**/*"
timeout-minutes: 15
---

Responsibilities:

• Query GitHub API for last 24 hours of activity
• Calculate metrics per workflow
• Store in daily JSON file
• Update rolling aggregates
• Clean up old daily files (keep 30 days)

4. Consumer Integration

Update meta-orchestrators to read metrics:

• Agent Performance Analyzer: Read metrics for performance analysis
• Campaign Manager: Use metrics for campaign health assessment
• Workflow Health Manager: Incorporate metrics into health monitoring

Implementation Plan

Phase 1: Basic Collection (Week 1)

1. Create metrics-collector workflow
2. Implement basic safe output counting
3. Store daily JSON files
4. Test storage and retrieval

Phase 2: Enrichment (Week 2)

1. Add workflow run statistics
2. Calculate success rates
3. Add engagement metrics (reactions, comments)
4. Implement quality indicators

Phase 3: Integration (Week 3)

1. Update Agent Performance Analyzer to consume metrics
2. Update Campaign Manager to use metrics
3. Update Workflow Health Manager integration
4. Create shared query utilities

Phase 4: Visualization (Week 4)

1. Add trend calculation
2. Create summary dashboards
3. Implement alerting for anomalies
4. Document metrics usage

Expected Benefits

✅ Enables:

• Historical trend analysis (week-over-week, month-over-month)
• Performance benchmarking (compare agents to ecosystem averages)
• Anomaly detection (sudden drops in success rate)
• Evidence-based prioritization
• Objective performance rankings

✅ Improves:

• Meta-orchestrator coordination (shared data foundation)
• Report accuracy and depth
• Early problem detection
• Strategic decision-making quality

Acceptance Criteria

• Metrics collector workflow created and running daily
• Metrics stored in repo-memory with defined schema
• 7 days of historical data collected
• Agent Performance Analyzer successfully reads metrics
• Documentation for adding new metrics
• No performance impact on existing workflows

Priority

High - Foundation for data-driven agent ecosystem management

Effort Estimate

Total: 12-16 hours across 4 weeks

• Phase 1: 4-5 hours
• Phase 2: 3-4 hours
• Phase 3: 3-4 hours
• Phase 4: 2-3 hours

Dependencies

• Requires: GitHub API access for Agent Performance Analyzer (separate issue)
• Blocks: Performance benchmarking, trend analysis, quality scoring

Related

• Agent Performance Analyzer meta-orchestrator
• Campaign Manager meta-orchestrator
• Workflow Health Manager meta-orchestrator

AI generated by Agent Performance Analyzer - Meta-Orchestrator