-
Notifications
You must be signed in to change notification settings - Fork 370
fix: Prevent mock data implementations with infrastructure features #95
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Changes from all commits
8e23fee
dae16c3
e756486
95b0dfa
cd9f5b7
d1233ad
03504b3
d652b18
11cefec
4cec4e6
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
| Original file line number | Diff line number | Diff line change | ||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
|
|
@@ -95,6 +95,27 @@ Ask the user about their involvement preference: | |||||||||||||
|
|
||||||||||||||
| **For Detailed Mode users**, ask specific tech questions about frontend, backend, database, etc. | ||||||||||||||
|
|
||||||||||||||
| ### Phase 3b: Database Requirements (MANDATORY) | ||||||||||||||
|
|
||||||||||||||
| **Always ask this question regardless of mode:** | ||||||||||||||
|
|
||||||||||||||
| > "One foundational question about data storage: | ||||||||||||||
| > | ||||||||||||||
| > **Does this application need to store user data persistently?** | ||||||||||||||
| > | ||||||||||||||
| > 1. **Yes, needs a database** - Users create, save, and retrieve data (most apps) | ||||||||||||||
| > 2. **No, stateless** - Pure frontend, no data storage needed (calculators, static sites) | ||||||||||||||
| > 3. **Not sure** - Let me describe what I need and you decide" | ||||||||||||||
|
|
||||||||||||||
| **Branching logic:** | ||||||||||||||
|
|
||||||||||||||
| - **If "Yes" or "Not sure"**: Continue normally. The spec will include database in tech stack and the initializer will create 5 mandatory Infrastructure features (indices 0-4) to verify database connectivity and persistence. | ||||||||||||||
|
|
||||||||||||||
| - **If "No, stateless"**: Note this in the spec. Skip database from tech stack. Infrastructure features will be simplified (no database persistence tests). Mark this clearly: | ||||||||||||||
| ```xml | ||||||||||||||
| <database>none - stateless application</database> | ||||||||||||||
| ``` | ||||||||||||||
|
|
||||||||||||||
| ## Phase 4: Features (THE MAIN PHASE) | ||||||||||||||
|
|
||||||||||||||
| This is where you spend most of your time. Ask questions in plain language that anyone can answer. | ||||||||||||||
|
|
@@ -207,12 +228,23 @@ After gathering all features, **you** (the agent) should tally up the testable f | |||||||||||||
|
|
||||||||||||||
| **Typical ranges for reference:** | ||||||||||||||
|
|
||||||||||||||
| - **Simple apps** (todo list, calculator, notes): ~20-50 features | ||||||||||||||
| - **Medium apps** (blog, task manager with auth): ~100 features | ||||||||||||||
| - **Advanced apps** (e-commerce, CRM, full SaaS): ~150-200 features | ||||||||||||||
| - **Simple apps** (todo list, calculator, notes): ~25-55 features (includes 5 infrastructure) | ||||||||||||||
| - **Medium apps** (blog, task manager with auth): ~105 features (includes 5 infrastructure) | ||||||||||||||
| - **Advanced apps** (e-commerce, CRM, full SaaS): ~155-205 features (includes 5 infrastructure) | ||||||||||||||
|
Comment on lines
+231
to
+233
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. 🧩 Analysis chain🏁 Script executed: #!/bin/bash
# Extract feature count references from both files
echo "=== create-spec.md feature counts ==="
rg "Simple apps|Medium apps|Advanced apps|Complex apps" .claude/commands/create-spec.md -B 1 -A 1
echo ""
echo "=== initializer_prompt.template.md feature counts ==="
rg "Simple|Medium|Complex|Advanced" .claude/templates/initializer_prompt.template.md | grep -E "apps|features|tests|Simple|Medium|Complex|Advanced" | head -30Repository: leonvanzyl/autocoder Length of output: 735 Feature count inconsistency between create-spec.md and initializer_prompt.template.md. The feature/test counts for corresponding app tiers don't align:
Additionally, the tiers use different naming ("Advanced" vs "Complex"), making it unclear whether they represent the same tier. The magnitude differences suggest these may be counting different things or based on different scope definitions. Clarify which counts are authoritative and align both templates accordingly. 🤖 Prompt for AI Agents |
||||||||||||||
|
|
||||||||||||||
| These are just reference points - your actual count should come from the requirements discussed. | ||||||||||||||
|
|
||||||||||||||
| **MANDATORY: Infrastructure Features** | ||||||||||||||
|
|
||||||||||||||
| If the app requires a database (Phase 3b answer was "Yes" or "Not sure"), you MUST include 5 Infrastructure features (indices 0-4): | ||||||||||||||
| 1. Database connection established | ||||||||||||||
| 2. Database schema applied correctly | ||||||||||||||
| 3. Data persists across server restart | ||||||||||||||
| 4. No mock data patterns in codebase | ||||||||||||||
| 5. Backend API queries real database | ||||||||||||||
|
|
||||||||||||||
| These features ensure the coding agent implements a real database, not mock data or in-memory storage. | ||||||||||||||
|
|
||||||||||||||
| **How to count features:** | ||||||||||||||
| For each feature area discussed, estimate the number of discrete, testable behaviors: | ||||||||||||||
|
|
||||||||||||||
|
|
@@ -225,17 +257,20 @@ For each feature area discussed, estimate the number of discrete, testable behav | |||||||||||||
|
|
||||||||||||||
| > "Based on what we discussed, here's my feature breakdown: | ||||||||||||||
| > | ||||||||||||||
| > - **Infrastructure (required)**: 5 features (database setup, persistence verification) | ||||||||||||||
| > - [Category 1]: ~X features | ||||||||||||||
| > - [Category 2]: ~Y features | ||||||||||||||
| > - [Category 3]: ~Z features | ||||||||||||||
| > - ... | ||||||||||||||
| > | ||||||||||||||
| > **Total: ~N features** | ||||||||||||||
| > **Total: ~N features** (including 5 infrastructure) | ||||||||||||||
| > | ||||||||||||||
| > Does this seem right, or should I adjust?" | ||||||||||||||
|
|
||||||||||||||
| Let the user confirm or adjust. This becomes your `feature_count` for the spec. | ||||||||||||||
|
|
||||||||||||||
| **Important:** The first 5 features (indices 0-4) created by the initializer MUST be the Infrastructure category with no dependencies. All other features depend on these. | ||||||||||||||
|
|
||||||||||||||
| ## Phase 5: Technical Details (DERIVED OR DISCUSSED) | ||||||||||||||
|
|
||||||||||||||
| **For Quick Mode users:** | ||||||||||||||
|
|
||||||||||||||
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -156,6 +156,9 @@ Use browser automation tools: | |
| - [ ] Deleted the test data - verified it's gone everywhere | ||
| - [ ] NO unexplained data appeared (would indicate mock data) | ||
| - [ ] Dashboard/counts reflect real numbers after my changes | ||
| - [ ] **Ran extended mock data grep (STEP 5.6) - no hits in src/ (excluding tests)** | ||
| - [ ] **Verified no globalThis, devStore, or dev-store patterns** | ||
| - [ ] **Server restart test passed (STEP 5.7) - data persists across restart** | ||
|
|
||
| #### Navigation Verification | ||
|
|
||
|
|
@@ -174,10 +177,92 @@ Use browser automation tools: | |
|
|
||
| ### STEP 5.6: MOCK DATA DETECTION (Before marking passing) | ||
|
|
||
| 1. **Search code:** `grep -r "mockData\|fakeData\|TODO\|STUB" --include="*.ts" --include="*.tsx"` | ||
| 2. **Runtime test:** Create unique data (e.g., "TEST_12345") → verify in UI → delete → verify gone | ||
| 3. **Check database:** All displayed data must come from real DB queries | ||
| 4. If unexplained data appears, it's mock data - fix before marking passing. | ||
| **Run ALL these grep checks. Any hits in src/ (excluding test files) require investigation:** | ||
coderabbitai[bot] marked this conversation as resolved.
Show resolved
Hide resolved
|
||
|
|
||
| ```bash | ||
| # Common exclusions for test files | ||
| EXCLUDE="--exclude=*.test.* --exclude=*.spec.* --exclude=*__test__* --exclude=*__mocks__*" | ||
|
|
||
| # 1. In-memory storage patterns (CRITICAL - catches dev-store) | ||
| grep -r "globalThis\." --include="*.ts" --include="*.tsx" --include="*.js" $EXCLUDE src/ | ||
| grep -r "dev-store\|devStore\|DevStore\|mock-db\|mockDb" --include="*.ts" --include="*.tsx" --include="*.js" $EXCLUDE src/ | ||
|
|
||
| # 2. Mock data variables | ||
| grep -r "mockData\|fakeData\|sampleData\|dummyData\|testData" --include="*.ts" --include="*.tsx" --include="*.js" $EXCLUDE src/ | ||
|
|
||
| # 3. TODO/incomplete markers | ||
| grep -r "TODO.*real\|TODO.*database\|TODO.*API\|STUB\|MOCK" --include="*.ts" --include="*.tsx" --include="*.js" $EXCLUDE src/ | ||
|
|
||
| # 4. Development-only conditionals | ||
| grep -r "isDevelopment\|isDev\|process\.env\.NODE_ENV.*development" --include="*.ts" --include="*.tsx" --include="*.js" $EXCLUDE src/ | ||
|
|
||
| # 5. In-memory collections as data stores | ||
| grep -r "new Map\(\)\|new Set\(\)" --include="*.ts" --include="*.tsx" --include="*.js" $EXCLUDE src/ 2>/dev/null | ||
|
Comment on lines
+183
to
+200
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Quote/array-ize ♻️ Suggested fix-# Common exclusions for test files
-EXCLUDE="--exclude=*.test.* --exclude=*.spec.* --exclude=*__test__* --exclude=*__mocks__*"
+# Common exclusions for test files
+EXCLUDE=(--exclude='*.test.*' --exclude='*.spec.*' --exclude='*__test__*' --exclude='*__mocks__*')
# 1. In-memory storage patterns (CRITICAL - catches dev-store)
-grep -r "globalThis\." --include="*.ts" --include="*.tsx" --include="*.js" $EXCLUDE src/
-grep -r "dev-store\|devStore\|DevStore\|mock-db\|mockDb" --include="*.ts" --include="*.tsx" --include="*.js" $EXCLUDE src/
+grep -r "globalThis\." --include="*.ts" --include="*.tsx" --include="*.js" "${EXCLUDE[@]}" src/
+grep -r "dev-store\|devStore\|DevStore\|mock-db\|mockDb" --include="*.ts" --include="*.tsx" --include="*.js" "${EXCLUDE[@]}" src/
# 2. Mock data variables
-grep -r "mockData\|fakeData\|sampleData\|dummyData\|testData" --include="*.ts" --include="*.tsx" --include="*.js" $EXCLUDE src/
+grep -r "mockData\|fakeData\|sampleData\|dummyData\|testData" --include="*.ts" --include="*.tsx" --include="*.js" "${EXCLUDE[@]}" src/
# 3. TODO/incomplete markers
-grep -r "TODO.*real\|TODO.*database\|TODO.*API\|STUB\|MOCK" --include="*.ts" --include="*.tsx" --include="*.js" $EXCLUDE src/
+grep -r "TODO.*real\|TODO.*database\|TODO.*API\|STUB\|MOCK" --include="*.ts" --include="*.tsx" --include="*.js" "${EXCLUDE[@]}" src/
# 4. Development-only conditionals
-grep -r "isDevelopment\|isDev\|process\.env\.NODE_ENV.*development" --include="*.ts" --include="*.tsx" --include="*.js" $EXCLUDE src/
+grep -r "isDevelopment\|isDev\|process\.env\.NODE_ENV.*development" --include="*.ts" --include="*.tsx" --include="*.js" "${EXCLUDE[@]}" src/
# 5. In-memory collections as data stores
-grep -r "new Map\(\)\|new Set\(\)" --include="*.ts" --include="*.tsx" --include="*.js" $EXCLUDE src/ 2>/dev/null
+grep -r "new Map\(\)\|new Set\(\)" --include="*.ts" --include="*.tsx" --include="*.js" "${EXCLUDE[@]}" src/ 2>/dev/null🤖 Prompt for AI Agents |
||
| ``` | ||
|
|
||
| **Rule:** If ANY grep returns results in production code → investigate → FIX before marking passing. | ||
|
|
||
| **Runtime verification:** | ||
| 1. Create unique data (e.g., "TEST_12345") → verify in UI → delete → verify gone | ||
| 2. Check database directly - all displayed data must come from real DB queries | ||
| 3. If unexplained data appears, it's mock data - fix before marking passing. | ||
|
|
||
| ### STEP 5.7: SERVER RESTART PERSISTENCE TEST (MANDATORY for data features) | ||
|
|
||
| **When required:** Any feature involving CRUD operations or data persistence. | ||
|
|
||
| **This test is NON-NEGOTIABLE. It catches in-memory storage implementations that pass all other tests.** | ||
|
|
||
| **Steps:** | ||
|
|
||
| 1. Create unique test data via UI or API (e.g., item named "RESTART_TEST_12345") | ||
| 2. Verify data appears in UI and API response | ||
|
|
||
| 3. **STOP the server completely:** | ||
| ```bash | ||
| # Kill by port (safer - only kills the dev server, not VS Code/Claude Code/etc.) | ||
| # Unix/macOS: | ||
| lsof -ti :${PORT:-3000} | xargs kill -TERM 2>/dev/null || true | ||
| sleep 3 | ||
| lsof -ti :${PORT:-3000} | xargs kill -9 2>/dev/null || true | ||
| sleep 2 | ||
|
|
||
| # Windows alternative (use if lsof not available): | ||
| # netstat -ano | findstr :${PORT:-3000} | findstr LISTENING | ||
| # taskkill /F /PID <pid_from_above> 2>nul | ||
|
|
||
| # Verify server is stopped | ||
| if lsof -ti :${PORT:-3000} > /dev/null 2>&1; then | ||
| echo "ERROR: Server still running on port ${PORT:-3000}!" | ||
| exit 1 | ||
| fi | ||
| ``` | ||
|
|
||
| 4. **RESTART the server:** | ||
| ```bash | ||
| ./init.sh & | ||
| sleep 15 # Allow server to fully start | ||
| # Verify server is responding | ||
| if ! curl -f http://localhost:${PORT:-3000}/api/health && ! curl -f http://localhost:${PORT:-3000}; then | ||
| echo "ERROR: Server failed to start after restart" | ||
| exit 1 | ||
| fi | ||
| ``` | ||
|
|
||
| 5. **Query for test data - it MUST still exist** | ||
| - Via UI: Navigate to data location, verify data appears | ||
| - Via API: `curl http://localhost:${PORT:-3000}/api/items` - verify data in response | ||
|
|
||
| 6. **If data is GONE:** Implementation uses in-memory storage → CRITICAL FAIL | ||
| - Run all grep commands from STEP 5.6 to identify the mock pattern | ||
| - You MUST fix the in-memory storage implementation before proceeding | ||
| - Replace in-memory storage with real database queries | ||
|
|
||
| 7. **Clean up test data** after successful verification | ||
|
|
||
| **Why this test exists:** In-memory stores like `globalThis.devStore` pass all other tests because data persists during a single server run. Only a full server restart reveals this bug. Skipping this step WILL allow dev-store implementations to slip through. | ||
|
|
||
| **YOLO Mode Note:** Even in YOLO mode, this verification is MANDATORY for data features. Use curl instead of browser automation. | ||
|
|
||
| ### STEP 6: UPDATE FEATURE STATUS (CAREFULLY!) | ||
|
|
||
|
|
||
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
🧩 Analysis chain
🏁 Script executed:
Repository: leonvanzyl/autocoder
Length of output: 607
🏁 Script executed:
Repository: leonvanzyl/autocoder
Length of output: 240
🏁 Script executed:
Repository: leonvanzyl/autocoder
Length of output: 1151
🏁 Script executed:
Repository: leonvanzyl/autocoder
Length of output: 46
🏁 Script executed:
Repository: leonvanzyl/autocoder
Length of output: 116
🏁 Script executed:
Repository: leonvanzyl/autocoder
Length of output: 19205
🏁 Script executed:
Repository: leonvanzyl/autocoder
Length of output: 8294
🏁 Script executed:
Repository: leonvanzyl/autocoder
Length of output: 1267
Reconcile stateless app handling between spec creation and initializer implementation.
The spec creation process mentions "simplified infrastructure" for stateless apps, but the initializer template doesn't implement this scenario:
Undefined simplification (line 114): States infrastructure will be "simplified (no database persistence tests)" for stateless apps, but doesn't specify:
Unclear XML tag placement (lines 115-117): The XML tag
<database>none - stateless application</database>is shown but its location in the spec structure is not defined.Missing initializer implementation: The initializer template (lines 39-41, 119-131, 225) has no conditional logic for stateless apps. It always mandates exactly 5 infrastructure features (indices 0-4) for all complexity tiers, regardless of whether the app needs a database. This creates an inconsistency: the spec process promises simplified infrastructure for stateless apps, but the initializer can't deliver it.
Resolve by either:
🤖 Prompt for AI Agents