EvalOps GitHub Action

Official GitHub Action for EvalOps CLI - Run LLM code evaluations and enforce quality budgets directly in your CI/CD pipeline.

Features

• 🎯 Quality Gates: Enforce minimum quality scores for code evaluations
• 💰 Cost Budgeting: Set and monitor evaluation cost limits
• 📊 Performance Monitoring: Track latency and execution time metrics
• 💬 PR Comments: Automatic evaluation results posted to pull requests
• 🚀 CI/CD Integration: Seamless integration with GitHub workflows
• 🔧 Flexible Configuration: Support for multiple environments and custom thresholds

Usage

Basic Example

name: 'EvalOps CI Pipeline'

on:
  pull_request:
    branches: [main]
  push:
    branches: [main]

jobs:
  evaluate:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      
      - name: Run EvalOps Evaluation
        uses: evalops/evalops-action@v1
        with:
          api-key: ${{ secrets.EVALOPS_API_KEY }}
          check-budget: true
          comment-pr: true

Advanced Configuration

name: 'Advanced EvalOps Pipeline'

on:
  pull_request:
    branches: [main]
    paths:
      - '**/*.eval.ts'
      - '**/*.eval.js'
      - 'evalops.yaml'
      - 'budget.yaml'

jobs:
  cost-gate:
    runs-on: ubuntu-latest
    outputs:
      cost-estimate: ${{ steps.evalops.outputs.cost-estimate }}
      
    steps:
      - uses: actions/checkout@v4
      
      - name: EvalOps Cost Check
        id: evalops
        uses: evalops/evalops-action@v1
        with:
          api-key: ${{ secrets.EVALOPS_API_KEY }}
          config-file: './evalops.yaml'
          budget-file: './budget.yaml'
          environment: 'staging'
          check-budget: true
          fail-on-violation: true
          cost-threshold: '10.00'
          quality-threshold: '0.8'
          comment-pr: true

  deploy:
    needs: cost-gate
    if: needs.cost-gate.outputs.budget-passed == 'true'
    runs-on: ubuntu-latest
    steps:
      - name: Deploy to Production
        run: echo "Deploying with cost estimate ${{ needs.cost-gate.outputs.cost-estimate }}"

Inputs

Input	Description	Required	Default
api-key	EvalOps API key	✅	-
config-file	Path to evalops.yaml configuration	❌	./evalops.yaml
budget-file	Path to budget.yaml configuration	❌	./budget.yaml
check-budget	Enforce budget constraints	❌	true
environment	Environment for budget config	❌	production
comment-pr	Post results as PR comment	❌	true
fail-on-violation	Fail action on budget violations	❌	true
cost-threshold	Override cost threshold (USD)	❌	-
quality-threshold	Override quality threshold	❌	-

Outputs

Output	Description
quality-score	Overall quality score from evaluation
cost-estimate	Estimated cost in USD
budget-passed	Whether budget constraints were met
violations-count	Number of budget violations
warnings-count	Number of budget warnings

Configuration Files

evalops.yaml

Your main evaluation configuration file. Create one with:

evalops init

budget.yaml

Your budget and quality gates configuration. Create one with:

evalops budget --init

Example budget.yaml:

version: '1.0'
description: 'Budget configuration for EvalOps evaluations'

quality_score:
  min: 0.6
  warning: 0.7

cost:
  max_usd: 10.0
  warning_usd: 5.0
  max_tokens: 100000

performance:
  max_latency_ms: 5000
  warning_latency_ms: 3000

environments:
  development:
    cost:
      max_usd: 1.0
  production:
    quality_score:
      min: 0.8
    cost:
      max_usd: 50.0

Environment Variables

• EVALOPS_API_KEY: Your EvalOps API key (can be set instead of the api-key input)
• GITHUB_TOKEN: GitHub token for PR comments (automatically provided by GitHub Actions)

Examples

Quality Gate Pipeline

- name: Quality Gate
  uses: evalops/evalops-action@v1
  with:
    api-key: ${{ secrets.EVALOPS_API_KEY }}
    quality-threshold: '0.8'
    fail-on-violation: true

Cost Control Pipeline

- name: Cost Control
  uses: evalops/evalops-action@v1
  with:
    api-key: ${{ secrets.EVALOPS_API_KEY }}
    cost-threshold: '5.00'
    environment: 'staging'

Multi-Environment Pipeline

strategy:
  matrix:
    environment: [development, staging, production]
    
steps:
- name: Evaluate ${{ matrix.environment }}
  uses: evalops/evalops-action@v1
  with:
    api-key: ${{ secrets.EVALOPS_API_KEY }}
    environment: ${{ matrix.environment }}

Contributing

1. Fork the repository
2. Make your changes
3. Run tests: npm test
4. Build: npm run build
5. Submit a pull request

License

MIT License - see LICENSE file for details.

Support

• 📚 Documentation
• 🐛 Issues
• 💬 Discussions