v0.9.8

Added:

chunk function: it can read and chunk, any file, directory or url with swappable reader function

---

Release Notes - Version 0.9.8

New Features

• Enhanced Chunker Functionality
  • Introduced an as_node option in the chunk_content function, which returns Node objects complete with metadata such as:
    • chunk_id
    • total_chunks
    • chunk_size
  • This enhancement facilitates smoother integration with graph-based processing pipelines.

Enhancements

• Improved File Processing
  • Updated the file_to_chunks and chunk functions in lionagi/libs/file/process.py:
    • Better parameter handling and consistency.
    • Enhanced support for outputting chunks in various formats (CSV, JSON, etc.) using Pile adapters.
• Module Exports
  • Updated lionagi/tools/types.py to export the chunk function, making it easier to access and use across the project.

Bug Fixes

• Corrected minor issues in parameter passing within the chunking functions.
• Resolved inconsistencies in metadata generation for chunks.

Version Update

• Version Bump: Updated from 0.9.7 to 0.9.8.
  • This update reflects the new features and improvements implemented in this release.

Summary

This release focuses on providing a more robust and flexible file chunking system, enabling better integration with downstream graph-based workflows and improved metadata handling. Users can now benefit from more granular control over file processing and enhanced output options.

Enjoy the new improvements!

What's Changed

• Add chunker by @ohdearquant in #560

Full Changelog: v0.9.7...v0.9.8