Performance: Handle large vendor folder changes without freezing (e.g., Golang vendor updates)

[areibman]

Performance Issue

The application needs to handle very large diffs, particularly vendor folder updates in Golang projects, without freezing or requiring the user to kill the application. GitHub itself struggles with these, and we should handle them better.

Problem Description

When updating vendor folders in Golang projects (or similar large dependency updates in other languages), the diff can contain thousands of files with millions of lines of changes. This causes:

• UI freezing/hanging
• Excessive memory usage
• Browser tab crashes
• Need to force-quit the application

Example Case

Real-world example: openfaas/ingress-operator#66

• Golang vendor folder update
• Thousands of files changed
• GitHub's web UI struggles to render this
• Often requires killing the browser tab

Current Behavior

• Application attempts to render entire diff at once
• No pagination or virtualization for large diffs
• No warning about large diff size
• No option to skip rendering large diffs
• Memory usage grows unbounded
• UI becomes unresponsive

Expected Behavior

• Application remains responsive even with massive diffs
• Smart handling of vendor/dependency folders
• Options to view large diffs in a performant way
• Warnings and alternatives for extremely large changes

Proposed Solution

1. Diff Size Detection and Warnings

interface DiffAnalysis {
  totalFiles: number;
  totalLines: number;
  estimatedRenderTime: number;
  memoryRequired: number;
  hasVendorChanges: boolean;
  hasLargeBinaryFiles: boolean;
}

class DiffAnalyzer {
  async analyzeDiff(pr: PullRequest): Promise<DiffAnalysis> {
    const analysis = {
      totalFiles: pr.changed_files,
      totalLines: 0,
      estimatedRenderTime: 0,
      memoryRequired: 0,
      hasVendorChanges: false,
      hasLargeBinaryFiles: false
    };
    
    // Check for vendor/node_modules/dependencies
    const vendorPatterns = [
      /^vendor\//,
      /^node_modules\//,
      /^packages\//,
      /^third_party\//,
      /^deps\//,
      /^dependencies\//,
      /go\.sum$/,
      /package-lock\.json$/,
      /yarn\.lock$/,
      /Gemfile\.lock$/,
      /Cargo\.lock$/
    ];
    
    for (const file of pr.files) {
      analysis.totalLines += file.additions + file.deletions;
      
      if (vendorPatterns.some(pattern => pattern.test(file.filename))) {
        analysis.hasVendorChanges = true;
      }
      
      if (file.additions + file.deletions > 10000) {
        analysis.hasLargeBinaryFiles = true;
      }
    }
    
    // Estimate rendering performance
    analysis.estimatedRenderTime = this.estimateRenderTime(analysis);
    analysis.memoryRequired = this.estimateMemoryUsage(analysis);
    
    return analysis;
  }
  
  getSizeCategory(analysis: DiffAnalysis): 'small' | 'medium' | 'large' | 'massive' {
    if (analysis.totalLines < 500) return 'small';
    if (analysis.totalLines < 5000) return 'medium';
    if (analysis.totalLines < 50000) return 'large';
    return 'massive';
  }
}

2. Smart Diff Rendering Strategies

class SmartDiffRenderer {
  async renderDiff(pr: PullRequest, analysis: DiffAnalysis) {
    const category = this.getSizeCategory(analysis);
    
    switch (category) {
      case 'small':
      case 'medium':
        return this.renderFullDiff(pr);
        
      case 'large':
        return this.renderProgressiveDiff(pr);
        
      case 'massive':
        return this.renderSummaryView(pr, analysis);
    }
  }
  
  renderSummaryView(pr: PullRequest, analysis: DiffAnalysis) {
    return (
      <div className="large-diff-warning">
        <div className="warning-header">
          <span className="icon">⚠️</span>
          <h3>Large Diff Detected</h3>
        </div>
        
        <div className="diff-stats">
          <div className="stat">
            <span className="label">Files Changed</span>
            <span className="value">{analysis.totalFiles.toLocaleString()}</span>
          </div>
          <div className="stat">
            <span className="label">Lines Changed</span>
            <span className="value">{analysis.totalLines.toLocaleString()}</span>
          </div>
          {analysis.hasVendorChanges && (
            <div className="stat vendor">
              <span className="label">Vendor/Dependencies</span>
              <span className="value">Updated</span>
            </div>
          )}
        </div>
        
        <div className="view-options">
          <h4>Viewing Options:</h4>
          
          <button onClick={() => this.viewFilesOnly(pr)}>
            📁 View File List Only
            <small>See changed files without diffs</small>
          </button>
          
          <button onClick={() => this.viewNonVendorOnly(pr)}>
            📝 View Non-Vendor Changes
            <small>Hide vendor/dependency files</small>
          </button>
          
          <button onClick={() => this.viewPaginated(pr)}>
            📄 View Paginated (100 files at a time)
            <small>Load files progressively</small>
          </button>
          
          <button onClick={() => this.downloadDiff(pr)}>
            💾 Download Diff
            <small>View locally with your preferred tool</small>
          </button>
          
          <button 
            className="danger" 
            onClick={() => this.forceRenderAll(pr)}
          >
            ⚠️ Force Render All (May Freeze)
            <small>Not recommended for large diffs</small>
          </button>
        </div>
        
        <div className="alternatives">
          <h4>Recommended Alternatives:</h4>
          <ul>
            <li>
              <code>git fetch origin pull/{pr.number}/head:pr-{pr.number}</code>
              <code>git checkout pr-{pr.number}</code>
              <small>Review locally with your IDE</small>
            </li>
            <li>
              <a href={`https://github.com/${pr.repo}/pull/${pr.number}.diff`}>
                View raw diff file
              </a>
            </li>
            <li>
              Use GitHub CLI: <code>gh pr diff {pr.number}</code>
            </li>
          </ul>
        </div>
      </div>
    );
  }
}

3. Virtual Scrolling for Large Diffs

class VirtualizedDiffViewer {
  private visibleRange = { start: 0, end: 50 };
  private fileBuffer = 10; // Files to render outside viewport
  
  async renderVirtualized(files: DiffFile[]) {
    return (
      <VirtualList
        height={window.innerHeight}
        itemCount={files.length}
        itemSize={this.estimateFileHeight}
        onScroll={this.handleScroll}
        overscanCount={this.fileBuffer}
      >
        {({ index, style }) => (
          <div style={style}>
            <LazyDiffFile
              file={files[index]}
              renderStrategy={this.getRenderStrategy(files[index])}
            />
          </div>
        )}
      </VirtualList>
    );
  }
  
  getRenderStrategy(file: DiffFile): RenderStrategy {
    const lineCount = file.additions + file.deletions;
    
    if (lineCount > 10000) {
      return 'summary'; // Just show stats
    } else if (lineCount > 1000) {
      return 'collapsed'; // Render on expand
    } else {
      return 'full'; // Render immediately
    }
  }
}

4. Progressive Loading

class ProgressiveDiffLoader {
  private loadedFiles = new Set<string>();
  private loading = false;
  private observer: IntersectionObserver;
  
  async loadDiffProgressively(pr: PullRequest) {
    const files = await this.getFileList(pr);
    
    // Group files by importance
    const groups = {
      sourceCode: files.filter(f => this.isSourceCode(f)),
      tests: files.filter(f => this.isTest(f)),
      config: files.filter(f => this.isConfig(f)),
      vendor: files.filter(f => this.isVendor(f)),
      other: files.filter(f => this.isOther(f))
    };
    
    // Load in priority order
    await this.loadGroup(groups.sourceCode, 'Source Code');
    await this.loadGroup(groups.tests, 'Tests');
    await this.loadGroup(groups.config, 'Configuration');
    
    // Vendor files loaded on demand only
    this.setupLazyLoading(groups.vendor);
  }
  
  private async loadGroup(files: DiffFile[], groupName: string) {
    const BATCH_SIZE = 20;
    
    for (let i = 0; i < files.length; i += BATCH_SIZE) {
      const batch = files.slice(i, i + BATCH_SIZE);
      
      // Show progress
      this.updateProgress({
        group: groupName,
        loaded: i,
        total: files.length
      });
      
      // Load batch
      await this.loadBatch(batch);
      
      // Allow UI to breathe
      await this.yieldToUI();
    }
  }
  
  private yieldToUI(): Promise<void> {
    return new Promise(resolve => {
      requestIdleCallback(() => resolve(), { timeout: 100 });
    });
  }
}

5. Memory Management

class DiffMemoryManager {
  private maxMemoryMB = 500;
  private currentUsage = 0;
  private diffCache = new LRUCache<string, RenderedDiff>();
  
  async renderWithMemoryLimit(file: DiffFile): Promise<RenderedDiff> {
    const estimatedSize = this.estimateMemoryUsage(file);
    
    // If single file is too large, render summary only
    if (estimatedSize > this.maxMemoryMB * 0.5) {
      return this.renderSummaryOnly(file);
    }
    
    // Free memory if needed
    while (this.currentUsage + estimatedSize > this.maxMemoryMB) {
      this.evictOldestDiff();
    }
    
    const rendered = await this.renderDiff(file);
    this.diffCache.set(file.path, rendered);
    this.currentUsage += estimatedSize;
    
    return rendered;
  }
  
  private evictOldestDiff() {
    const oldest = this.diffCache.pop();
    if (oldest) {
      this.currentUsage -= this.getSize(oldest);
      // Clear from DOM
      this.clearFromDOM(oldest);
    }
  }
}

6. Vendor Folder Handling

class VendorFolderHandler {
  private vendorPatterns = [
    'vendor/**',
    'node_modules/**',
    'third_party/**',
    '**/*.lock',
    '**/go.sum',
    '**/go.mod'
  ];
  
  async handleVendorChanges(pr: PullRequest) {
    const vendorFiles = pr.files.filter(f => 
      this.vendorPatterns.some(p => minimatch(f.path, p))
    );
    
    if (vendorFiles.length > 100) {
      return {
        type: 'summary',
        component: <VendorChangesSummary files={vendorFiles} />,
        expandable: true,
        lazyLoad: true
      };
    }
    
    return {
      type: 'normal',
      files: vendorFiles
    };
  }
}

function VendorChangesSummary({ files }) {
  const stats = calculateVendorStats(files);
  
  return (
    <div className="vendor-summary">
      <h3>📦 Vendor/Dependencies Updated</h3>
      
      <div className="vendor-stats">
        <div>{stats.packagesAdded} packages added</div>
        <div>{stats.packagesUpdated} packages updated</div>
        <div>{stats.packagesRemoved} packages removed</div>
        <div>{files.length} files changed</div>
      </div>
      
      <details>
        <summary>View dependency changes</summary>
        <DependencyDiff files={files} />
      </details>
      
      <div className="vendor-actions">
        <button onClick={downloadVendorChanges}>
          Download vendor changes
        </button>
        <button onClick={viewSecurityReport}>
          View security impact
        </button>
      </div>
    </div>
  );
}

7. Performance Monitoring

class DiffPerformanceMonitor {
  private metrics = {
    renderStartTime: 0,
    renderEndTime: 0,
    memoryBefore: 0,
    memoryAfter: 0,
    filesRendered: 0,
    framesDropped: 0
  };
  
  async monitorRendering(renderFn: () => Promise<void>) {
    this.metrics.renderStartTime = performance.now();
    this.metrics.memoryBefore = performance.memory?.usedJSHeapSize || 0;
    
    // Monitor frame drops
    let lastFrameTime = performance.now();
    const checkFrames = setInterval(() => {
      const now = performance.now();
      if (now - lastFrameTime > 100) { // More than 100ms = dropped frames
        this.metrics.framesDropped++;
      }
      lastFrameTime = now;
    }, 16);
    
    try {
      await renderFn();
    } finally {
      clearInterval(checkFrames);
      this.metrics.renderEndTime = performance.now();
      this.metrics.memoryAfter = performance.memory?.usedJSHeapSize || 0;
      
      this.reportMetrics();
    }
  }
  
  private reportMetrics() {
    const duration = this.metrics.renderEndTime - this.metrics.renderStartTime;
    const memoryUsed = this.metrics.memoryAfter - this.metrics.memoryBefore;
    
    if (duration > 3000 || memoryUsed > 100 * 1024 * 1024) {
      console.warn('Poor diff rendering performance detected', {
        duration: `${duration}ms`,
        memory: `${(memoryUsed / 1024 / 1024).toFixed(2)}MB`,
        framesDropped: this.metrics.framesDropped
      });
      
      // Show user warning
      this.showPerformanceWarning();
    }
  }
}

Acceptance Criteria

• Large vendor folder updates don't freeze the UI
• Warning shown for diffs over threshold (e.g., 10k lines)
• Option to view file list only without diffs
• Option to exclude vendor files from view
• Virtual scrolling implemented for large file lists
• Progressive loading of diff content
• Memory usage stays under reasonable limit (e.g., 500MB)
• "Download diff" option for local viewing
• Graceful degradation for massive diffs
• Performance metrics tracked and reported
• Test with actual example: Upgrade client-go 1.28, codegen and CRD openfaas/ingress-operator#66
• No browser tab crashes
• Cancel/abort option for diff loading
• Clear feedback on loading progress

Testing

1. Load Upgrade client-go 1.28, codegen and CRD openfaas/ingress-operator#66
2. Verify UI remains responsive
3. Verify memory usage stays reasonable
4. Test different viewing options
5. Verify no need to kill the application
6. Test with other large vendor updates (node_modules, Rust cargo, Python venv)

Success Metrics

• Render time for 10k+ file diff: < 5 seconds
• Memory usage for large diff: < 500MB
• UI remains responsive (< 100ms input lag)
• No browser crashes
• User can view essential changes without viewing vendor files

🤖 Generated with Claude Code