Skip to content

Conversation

@connor-tyndall
Copy link
Contributor

Summary

Fixes #20 - Random testing results in duplicate testing (wasting tokens / time)

  • Adds regression_count column to track how many times each feature has been regression tested
  • Replaces random selection with least-tested-first ordering
  • Ensures even distribution of regression testing across all features

Problem

The current feature_get_for_regression uses func.random() to select features:

.order_by(func.random())

This causes:

  • Same features tested repeatedly while others are never tested
  • Wasted tokens on duplicate testing
  • Incomplete regression coverage

Solution

Track regression test frequency and prioritize least-tested features:

  1. New regression_count column - Tracks how many times each feature has been regression tested
  2. Least-tested-first ordering - order_by(regression_count.asc(), id.asc())
  3. Automatic increment - Counter increases each time a feature is selected

Before

Session 1: Features [5, 12, 8]  (random)
Session 2: Features [5, 3, 12]  (random - duplicates!)
Session 3: Features [5, 8, 5]   (random - more duplicates!)

After

Session 1: Features [1, 2, 3]   (count: 0 → 1)
Session 2: Features [4, 5, 6]   (count: 0 → 1)
Session 3: Features [7, 8, 9]   (count: 0 → 1)
...
Session N: Features [1, 2, 3]   (count: 1 → 2, round-robin)

Test plan

  • Verify new databases get regression_count column
  • Verify existing databases are migrated correctly
  • Call feature_get_for_regression multiple times - verify different features returned each time
  • Verify regression_count increments after each call

🤖 Generated with Claude Code

Replaces random selection in feature_get_for_regression with a
least-tested-first approach to ensure even distribution of regression
testing across all features.

Changes:
- Add regression_count column to Feature model to track test frequency
- Add database migration for existing databases
- Update feature_get_for_regression to order by regression_count (ascending)
- Increment regression_count when features are selected for testing

This prevents the same features from being tested repeatedly while others
are never tested, reducing wasted tokens and ensuring comprehensive
regression coverage.

Closes leonvanzyl#20

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
@meirm
Copy link

meirm commented Jan 19, 2026

I am using this PR on my server and it works.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Random testing results in duplicate testing (wasting tokens / time)

2 participants