Skip to content

v1.2.3: Optimize Batch Processing

Compare
Choose a tag to compare
@vrknetha vrknetha released this 10 Feb 16:53
· 26 commits to main since this release

What's Changed

Improvements

  • Removed redundant batch configuration to rely on FireCrawl library's built-in functionality
  • Simplified batch processing logic by leveraging library's native implementation
  • Optimized parallel processing and rate limiting handling
  • Reduced code complexity and potential configuration conflicts

Technical Details

  • Removed custom CONFIG.batch settings:
    • Removed maxParallelOperations configuration
    • Removed delayBetweenRequests configuration
  • Simplified batch operation processing to use library's built-in batch handling
  • Updated server startup logging to remove batch configuration references
  • Maintained credit usage tracking and error handling functionality

Benefits

  • More reliable batch processing using library's battle-tested implementation
  • Better handling of rate limits and concurrent operations
  • Reduced maintenance overhead
  • Improved code maintainability