Bug: Missing File Size Validation in File Reading Operations

[NeaByteLab]

App Version

3.23.19 (06b590f)

API Provider

Not Applicable / Other

Model Used

N/A

Roo Code Task Links (Optional)

Summary

File size validation is inconsistently implemented across Roo-Code services. While code indexing has proper file size limits, other file reading operations lack size validation, potentially causing memory exhaustion and performance issues.

Severity

• Priority: Medium
• Impact: Memory exhaustion, performance degradation, potential crashes
• Affected Services: Config, MCP, Checkpoints, File Operations

Current State

✅ Has File Size Validation

• src/services/code-index/processors/scanner.ts - Uses MAX_FILE_SIZE_BYTES = 1MB
• src/services/code-index/processors/file-watcher.ts - Validates file size before processing

❌ Missing File Size Validation

• src/services/roo-config/index.ts:102 - readFileIfExists() function
• src/services/glob/list-files.ts:259 - .gitignore file reading
• src/services/checkpoints/ShadowCheckpointService.ts:296 - checkpoint file reading
• src/services/mcp/McpHub.ts:280 - MCP config file reading

Acceptance Criteria

• All file reading operations validate file size
• Appropriate size limits for different file types
• Graceful error handling for oversized files
• Unit tests for size validation
• Integration tests with large files
• Documentation updated

Additional Considerations

• Consider different size limits for different file types
• Add logging for skipped files
• Consider streaming for very large files if needed
• Update error messages to be user-friendly

🔁 Steps to Reproduce

Reproduction Steps

1. Create a large file (>8MB) in project directory
2. Try to read it through any of the affected services
3. Observe potential memory issues or performance degradation

Expected Behavior

All file reading operations should validate file size before processing and skip/reject files that exceed reasonable limits.

Actual Behavior

Only code indexing services validate file size. Other services read files of any size without limits.

Technical Details

Current Implementation (Code Indexing)

// ✅ Good - Has validation
const stats = await stat(filePath)
if (stats.size > MAX_FILE_SIZE_BYTES) {
  skippedCount++ // Skip large files
  return
}

Missing Implementation (Other Services)

// ❌ Bad - No validation
const content = await fs.readFile(filePath, "utf-8")
// Could read files of any size

Proposed Solution

1. Create Shared File Size Constants

// src/services/constants/file-limits.ts
export const MAX_FILE_SIZE_BYTES = 1 * 1024 * 1024 // 1MB
export const MAX_CONFIG_FILE_SIZE_BYTES = 100 * 1024 // 100KB for config files
export const MAX_GITIGNORE_FILE_SIZE_BYTES = 50 * 1024 // 50KB for .gitignore

2. Add Validation to Affected Functions

A. readFileIfExists()

export async function readFileIfExists(filePath: string): Promise<string | null> {
  try {
    // Add size validation
    const stats = await fs.stat(filePath)
    if (stats.size > MAX_CONFIG_FILE_SIZE_BYTES) {
      console.warn(`File ${filePath} exceeds size limit (${stats.size} bytes)`)
      return null
    }
    
    return await fs.readFile(filePath, "utf-8")
  } catch (error: any) {
    // ... existing error handling
  }
}

B. MCP Config Reading

// In McpHub.ts
const stats = await fs.stat(configPath)
if (stats.size > MAX_CONFIG_FILE_SIZE_BYTES) {
  throw new Error(`Config file ${configPath} exceeds size limit`)
}
const content = await fs.readFile(configPath, "utf-8")

C. Gitignore Reading

// In list-files.ts
const stats = await fs.stat(gitignoreFile)
if (stats.size > MAX_GITIGNORE_FILE_SIZE_BYTES) {
  console.warn(`Skipping large .gitignore file: ${gitignoreFile}`)
  continue
}
const content = await fs.promises.readFile(gitignoreFile, "utf8")

3. Add Error Handling

// Standard error response for oversized files
class FileSizeLimitError extends Error {
  constructor(filePath: string, size: number, limit: number) {
    super(`File ${filePath} exceeds size limit (${size} bytes > ${limit} bytes)`)
    this.name = 'FileSizeLimitError'
  }
}

Testing Strategy

Unit Tests

describe('File Size Validation', () => {
  it('should reject files larger than MAX_FILE_SIZE_BYTES', async () => {
    // Create large file mock
    const largeFileStats = { size: 2 * 1024 * 1024 } // 2MB
    vi.mocked(stat).mockResolvedValue(largeFileStats)
    
    const result = await readFileIfExists('/large/file.txt')
    expect(result).toBeNull()
  })
})

Integration Tests

• Test with actual large files
• Verify memory usage doesn't spike
• Ensure performance remains acceptable

Files to Modify

1. src/services/constants/file-limits.ts (new file)
2. src/services/roo-config/index.ts
3. src/services/glob/list-files.ts
4. src/services/checkpoints/ShadowCheckpointService.ts
5. src/services/mcp/McpHub.ts
6. Add tests for each modified function

💥 Outcome Summary

The analysis of Roo-Code revealed a critical inconsistency in file size validation across the codebase. While the code indexing service properly implements a 8MB file size limit (MAX_FILE_SIZE_BYTES), other critical file reading operations in config, MCP, checkpoint, and glob services lack any size validation whatsoever.

This creates a security and performance vulnerability where large files can cause memory exhaustion, performance degradation, and potential crashes in VS Code. The bug affects multiple core services including readFileIfExists(), .gitignore processing, checkpoint file reading, and MCP config handling - all of which read files without size checks, potentially allowing malicious or accidentally large files to consume excessive memory and crash the application.

📄 Relevant Logs or Errors (Optional)

[roomote]

Hi! I'm here to help out the maintainers and am going to see if I can fix this issue. I'll be implementing file size validation across all the identified file reading operations to prevent memory exhaustion and performance issues. Thanks for the detailed bug report with the proposed solution - it's very helpful!

[roomote]

I see @roomote is already working on this issue. I'm here to help as well! If you need any assistance or would like to collaborate on the implementation, I'd be happy to help. The proposed solution in the issue description looks comprehensive - implementing consistent file size validation across all file reading operations is definitely important for preventing memory issues. Let me know if there's any specific part you'd like me to focus on!

[roomote]

I've successfully implemented the file size validation fix for this issue! 🎉

The pull request is ready for review: #6157

Summary of changes:

• Created centralized file size constants for different file types
• Added file size validation to all identified services:
  • readFileIfExists() in roo-config service
  • .gitignore reading in list-files service
  • Checkpoint file reading in ShadowCheckpointService
  • MCP config reading in McpHub (with a new helper function)
• Added comprehensive tests for the new validation logic
• All tests pass and CI checks are in progress

This implementation follows the proposed solution in the issue and ensures consistent file size validation across all file reading operations to prevent memory exhaustion and performance issues.