You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
When building an ETL pipeline, transformers may need to perform multiple actions, which can result in layers of function calls that are hard to maintain. Is it possible to design them like advisor ?
The following actions include chunking, keyword, and summarization. If more transformations are required in the future, additional layers will need to be added.
There is a previous discussion #1253 regarding the design of the ETL pipeline.
This discussion covers not only handling multiple transformers in a chain format based on Reactive Streams but also includes improvements for efficiently processing large volumes of data and enhancing performance.
Please feel free to join the discussion if you’re interested. 🙇🏻♂️
When building an ETL pipeline, transformers may need to perform multiple actions, which can result in layers of function calls that are hard to maintain. Is it possible to design them like advisor ?
The following actions include chunking, keyword, and summarization. If more transformations are required in the future, additional layers will need to be added.
vectorStore.write(splitter.split(loadTextAsDocuments( summaryDocuments(keywordDocuments(docs))));
The text was updated successfully, but these errors were encountered: