Make .blarignore additive to .gitignore for automatic exclusion of git-ignored files

[rysweet]

Problem Statement

Currently, Blarify's .blarignore file works independently of .gitignore, leading to:

• Duplication of ignore patterns between the two files
• Git-ignored files (like node_modules, .env, build artifacts) being processed by Blarify
• Manual maintenance burden to keep both files in sync
• Wasted computational resources analyzing files that shouldn't be included

Proposed Solution

Make .blarignore additive to .gitignore so that:

1. All patterns in .gitignore are automatically applied
2. .blarignore adds additional exclusions on top of .gitignore
3. Full gitignore pattern syntax is supported (not just basename matching)
4. Backward compatibility is maintained

Implementation Plan

1. Add gitignore parsing library

• Add pathspec or gitignore-parser to dependencies
• These libraries properly handle gitignore pattern syntax including:
  • Glob patterns (*, **, ?)
  • Directory patterns (/)
  • Negation patterns (!)
  • Comments and blank lines

2. Create GitignoreManager class

class GitignoreManager:
    def __init__(self, root_path: str):
        # Find and parse all .gitignore files
        # Cache compiled patterns for performance
        
    def should_ignore(self, path: str) -> bool:
        # Check if path matches any gitignore pattern
        
    def get_gitignore_files(self) -> List[str]:
        # Find all .gitignore files in the project

3. Modify ProjectFilesIterator

• Add use_gitignore parameter (default True for backward compatibility)
• Integrate GitignoreManager for pattern matching
• Replace basename-only matching with full path pattern matching
• Support both .gitignore and .blarignore patterns

4. Update GraphBuilder API

• Add configuration option to disable gitignore integration if needed
• Ensure .blarignore is loaded from project root by default

5. Comprehensive test suite

• Unit tests for gitignore pattern parsing
• Integration tests for .gitignore + .blarignore interaction
• Edge case tests:
  • Nested .gitignore files
  • Complex patterns
  • Performance with large ignore lists

6. Documentation updates

• Update README with new behavior
• Add configuration examples
• Document pattern syntax support

Benefits

• Automatic exclusion of git-ignored files
• Reduced maintenance burden
• Better performance by skipping unnecessary files
• Consistent behavior with git

Success Criteria

• Files in .gitignore are automatically excluded
• .blarignore patterns work additively
• All gitignore syntax is supported
• Performance is not significantly impacted
• Backward compatibility is maintained
• Tests pass and documentation is updated