feat: Add CLI-based Search Quality Testing Framework #144
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Completes Phase 3 of #139 with comprehensive dependency management, documentation generation, and UV evaluation capabilities. ## New Features Added ✅ ### Dependency Management - Add `check-deps` target - identifies outdated dependencies - Add `check-deps-tree` target - visualizes dependency hierarchy - Add `export-requirements` target - exports to requirements.txt files - Found 83 outdated packages in testing including security-critical updates ### Documentation Generation - Add `docs-generate` target - creates API documentation with pydoc - Add `docs-serve` target - serves docs locally on port 8080 - Provides quick access to module documentation ### UV Integration (Experimental) - Add `uv-install` target - installs UV package manager - Add `uv-sync` target - syncs dependencies with UV - Add `uv-export` target - exports requirements with UV - Provides migration path for future UV adoption ## Developer Workflow New commands available: - `make check-deps` - Check for outdated dependencies - `make check-deps-tree` - View dependency tree - `make export-requirements` - Export requirements files - `make docs-generate` - Generate API documentation - `make docs-serve` - Serve docs locally - `make uv-install` - Install UV (experimental) ## UV vs Poetry Analysis Evaluated UV as Poetry alternative: - **Pros**: 10-100x faster, unified tooling, simpler config - **Cons**: Migration effort, pyproject.toml restructuring needed - **Recommendation**: Keep Poetry for stability, UV available for experimentation ## Phase 3 Complete - Issue Can Be Closed **✅ All Phase 3 Requirements Met:** - Dependency checking implemented - Requirements export functional - Documentation generation working - UV support added as experimental option **📊 Real Impact:** - Identified 83 outdated dependencies - Critical security updates available (urllib3, cryptography, etc.) - Documentation generation successful - UV integration path established Addresses and completes Phase 3 of #139.
Implement comprehensive CLI tool for testing and diagnosing RAG search quality with: 🔧 CLI Commands: - search test: Test single queries with detailed metrics - search batch-test: Run quality tests on multiple queries - search test-components: Test individual RAG pipeline components 📊 Quality Metrics: - Answer completeness and keyword coverage - Retrieval precision and similarity scores - Component-level performance timing - Batch quality scoring and reporting 🎯 Features: - Rich console output with tables and progress indicators - JSON export for further analysis - Component isolation for debugging - Configurable query rewriting strategies - Makefile integration for easy execution 📁 Files Added: - backend/cli/: Core CLI implementation modules - backend/search_cli.py: Entry point script - backend/test_data/search_queries.json: Sample test queries - backend/tests/cli/: Unit tests for CLI functionality - Updated Makefile with search-test, search-batch, search-components targets ✅ Addresses issue #131 requirements for systematic RAG search quality testing
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Summary
Implements a comprehensive CLI tool for testing and diagnosing RAG search quality as specified in #131.
Changes
Features
🔧 CLI Commands
search test- Test single queries with detailed metrics and verbose outputsearch batch-test- Run quality tests on multiple queries with aggregated reportingsearch test-components- Test individual RAG pipeline components for debugging📊 Quality Metrics
🎯 Additional Features
Files Added/Modified
backend/cli/- Core CLI implementation modulesbackend/search_cli.py- Entry point scriptbackend/test_data/search_queries.json- Sample test queriesbackend/tests/cli/- Unit tests for CLI functionalityMakefile- Added search-test, search-batch, search-components targetsTesting
✅ CLI help commands validated
✅ Utility functions tested (metrics calculation, quality evaluation)
✅ Makefile integration verified
✅ Lazy loading prevents configuration errors for help commands
Usage Examples
Documentation
Comprehensive documentation added in
backend/cli/README.mdwith:Closes #131