Refactor Prompt Builder for Efficient DataChunk Processing and Memory Optimization

[dorbanianas]

The current prompt_builder.cpp implementation in src/core/functions requires several optimizations and refactoring steps to improve efficiency and memory usage when processing DataChunks. Below are the main issues identified:

1. Function Naming and Parameter Handling:

   • Rename GetMaxLengthValues to GetMaxTokenLengthPerAttribute for better clarity.
   • Optimization: Update this function to find max token length directly over the DataChunk vectors, bypassing JSON-related overhead.

2. Concatenation and String Allocation:

   • In CombineValues, while += concatenation is currently manageable, the approach could be improved by allocating the full string size upfront to prevent reallocation overhead. This is especially relevant as the code is likely to be reused in scenarios with aggregate data handling.
   • Loop Optimization: Remove pop_back usage for trimming the last space after concatenation. Instead, concatenate up to N-1 times and handle the last item separately, reducing unnecessary operations.

3. DataChunk to JSON Transformation:

   • Avoid mapping entire DataChunks to JSON before chunking. Although JSON is currently used as a temporary solution, the goal is to remove this dependency.
   • For efficient memory handling, calculate and set the maximum number of tuples per LLM request based on token size limits, allocating memory in advance on the stack.

4. Prompt Rendering and Inja Removal:

   • In the long term, replace Inja for prompt rendering. Implement a custom rendering solution directly with DataChunks to avoid intermediary JSON representations and reduce memory usage.

• Source: remove redundant code #41