add ColQwen multimodal PDF retrieval integration #162
Conversation
- Add ColQwenRAG class with easy-to-use CLI for multimodal PDF retrieval - Support for both ColQwen2 and ColPali models with automatic device selection - MPS optimization for Apple Silicon with memory-efficient loading - Complete pipeline: PDF→images→embeddings→HNSW index→search - Multi-vector indexing for fine-grained document matching - Comprehensive user guide and reproduction test script - Resolves #119: ColQwen Doc and Support Management Features: - python -m apps.colqwen_rag build --pdfs ./pdfs/ --index my_index - python -m apps.colqwen_rag search my_index "query text" - python -m apps.colqwen_rag ask my_index --interactive - Automatic CPU fallback for memory constraints - Robust error handling and progress tracking
- Add noqa comments for E402 errors (imports after sys.path modifications) - Remove unused variable assignment in colqwen_rag.py - Use importlib.util.find_spec for dependency checks instead of unused imports - Fix import ordering in test_colqwen_reproduction.py
|
The faiss submodule still seems to have some problem, we need to remember to submodule update |
|
hmm @yichuan-w okay sounds good will look into it |
- Add apps/image_rag.py for indexing and searching images using CLIP embeddings - Supports text-based image search queries - Uses CLIP ViT-L/14 model via sentence-transformers - Follows the same pattern as other RAG apps in the apps directory - Addresses feature request for CLIP support in apps (issue #94)
|
|
||
| # Load CLIP model | ||
| print("🔍 Loading CLIP model...") | ||
| model = SentenceTransformer(self.embedding_model_default) |
There was a problem hiding this comment.
are U sure we can run this? you should test yourself on some dataset, the default model should be clip stuff.
|
@ASuresh0524 Thanks for the PR make sure the faiss submodule is correct, and I think we create an unnecessary faiss submodule update |
Sounds good, will fix this tomorrow |
Reset faiss submodule to match main branch to avoid unnecessary changes
|
@yichuan-w think i fixed it let me know if it looks good |
|
@ASuresh0524 can you add a brief introduction to Colqwen and how to use in the Readme? |
|
sounds good! will work on this for the next day to make it in-depth and squash merge it as well |
Add brief introduction and usage guide for ColQwen integration, similar to other RAG application sections in the README. - Quick start examples for building, searching, and interactive Q&A - Setup instructions with prerequisites - Model options (ColQwen2 vs ColPali) - Link to detailed ColQwen guide
|
plz fix CI @ASuresh0524 |
Features:
Checklist
uv run pytest)ruff formatandruff check)pre-commit run --all-files)