Enhancing the architecture of the ETL pipeline's transformers

[kevintsai1202]

When building an ETL pipeline, transformers may need to perform multiple actions, which can result in layers of function calls that are hard to maintain. Is it possible to design them like advisor ?

The following actions include chunking, keyword, and summarization. If more transformations are required in the future, additional layers will need to be added.

vectorStore.write(splitter.split(loadTextAsDocuments( summaryDocuments(keywordDocuments(docs))));

[youngmoneee]

There is a previous discussion #1253 regarding the design of the ETL pipeline.

This discussion covers not only handling multiple transformers in a chain format based on Reactive Streams but also includes improvements for efficiently processing large volumes of data and enhancing performance.

Please feel free to join the discussion if you’re interested. 🙇🏻‍♂️